PUBG LABS Report-1st Skill Based Rating (SBR) Test

With your help, PUBG Labs has just finished its first trial of the Skill Based Rating (SBR) system. A big thank you to all the players who checked out PUBG Labs and took part in this test! We’re pouring through all your feedback and other data from the test so that we can improve the system for the next iteration of SBR coming to Labs.

Some people were curious why we used PUBG Labs to conduct the skill based rating test.

Here are a few of the reasons:

We required large-scale live testing data.

A rating system cannot be meaningfully built in isolation. Similarly to when the Mastery systems were developed, we performed a ‘dark launch’ of Skill Based Rating in order to gather an initial set of data that we used to tune the rating algorithm and create a solid starting point to work off of.

This initial data was not enough however as, among other problems, observing players secretly can’t truly measure how players will perform when they are aware that they are being evaluated. We required real PUBG style games played by real players in real situations over a long-ish period of time to obtain the data required to take the next steps in refining these calculations.


We needed feedback on how the system feels.

Numbers and design philosophies are all well and good, but at the end of the day the system needs to feel good to players. For skill based rating, we wanted to be more open to feedback and work closer with the community than has previously been possible. To that end we included a few different places in Labs for players to directly indicate their feelings on the way the system worked.

Labs gave us the opportunity to share the development of skill based rating early and get a head start on pushing the system in the direction that players want to see.


We wanted to engage the community in the conversation.

We know the community takes skill based ratings very seriously. We felt that it wouldn’t be right to build this in a silo without a solid and up to date idea of ​​what the community wants such a system to look like for PUBG.

Launching skill based rating in Labs jump started the community into a focused discussion around the topic that has proven to be very beneficial to development efforts. So many players shared so many exciting ideas over the course of the test. A few enterprising players even posted up concepts for entire rating systems of their own design! We want all of you to know that we’ve been carefully reading all of your comments and ideas, and will continue to do the best we can to try and deliver a rating system that everyone can be happy with.


Test Cycle Overview

Skill Based Rating went live in PUBG Labs at the end of November and ran for 6 weeks. During this testing period we were able to apply a few adjustments to the algorithm based on what we observed.
https://twitter.com/PUBG_Support/status/1202768163470839808?s=20

Due to the way the infrastructure for Skill Based Rating was built, we were able to make these changes to live servers without needing to bring them down for maintenance, which greatly increased the rate at which we were able to take action to improve the experience.
https://clips.twitch.tv/SpineyHonorablePorcupineOSsloth

After gathering survey results, analyzing the live data, and collecting feedback from various social channels we made a decision to run a second test in February that addresses some of the issues we noticed with the configuration of the algorithm in the first test of Skill Based Rating .

Observations

Next we would like to share some fun and interesting data and feedback we gathered during the skill based rating labs and talk a little about how we plan to move forward into the second test.

18.8% of PC players opted-in to play competitively
22.3% of console players joined competitive ranks of PUBG players (That’s pretty cool!)

Opt-in Percentage by Region

Region PC Console
Asia 19.7% 23.4%
EMEA 15.2% 23.5%
JP 20.9% 28.6%
KR 18.5% 32.8%
NA 15.3% 21.5%

The Skill Based Rating Labs test was made to measure the performance of players in existing queues. For this reason among others, the decision was made that players would have the option whether to participate in this test or not by choosing to ‘opt-in’ from the Labs page or leaving it alone. The table above shows the percentage of active users per region and platform type who chose to participate in the Skill Based Rating Labs.

For some context here, we would not have been surprised to see 10% or less of players opt-in based on research and experience with other games-so these numbers were very surprising and exciting! It really drives home how passionate PUBG players are about the topic of measuring skill that in some cases as much as a third of all active players chose to get involved in the test.


Rating distribution

PC (Squad TPP)
Tier Distribution
Bronze 2.85%
Silver 37.47%
Gold 30.16%
Platinum 24.15%
Diamond 5.24%
Master 1.05%

The table above represent a snapshot of the current state of a key metric we focus on when tuning the Skill Based Rating algorithm-rating distribution. The rating distribution is a breakdown of how the player pool is divided in terms of RP and Tiers. There are many ways to look at this kind of data, but to keep it simple we will be using the above table from squad TPP players on PC who have played at least 50 rated games in the squad queue.

Looking at these results, it’s clear that there are things that need to be improved here. Some standout issues include the small number of Bronze players and the Master tier being a bit larger than is ideal. The diamond tier is on the small side as well. We will be making adjustments to smooth out the difficulty of the tiers while also shifting every tier to be a bit harder to earn in general.


Feedback

Next we would like to recognize and address a couple of the most common and popular pieces of feedback that players passed along during the test.

“Damage should be a part of the calculation. It feels bad to play team modes and have to compete with teammates for RP ”

This was one of the first pieces of feedback we received when the test went live, and it was one of the first adjustments we knew we needed to make to improve the experience. We did seriously consider the possibility of adding damage directly into the calculation as it seemed like the simplest solution on the surface, but after a lot of thought on the matter it turns out that adding damage hurts the experience more than it helps. There were just too many edge cases and ways to abuse having damage in the calculation.

The problem of team play feeling bad still needed to be solved though, even if damage wasn’t the answer. We eventually settled on adding the concept of ‘assists’. With assists, players who do meaningful damage to an enemy within a reasonable amount of time of that enemy dying will receive an assist-and the algorithm values ​​assists with the same weight as kills so that there’s no need for a team to be conflicted over which member scores the final blow.

“Gaining RP feels too easy compared to losing it, like a progression system. Master tier players can climb forever ”

Uh oh. The very last thing that we want is for Skill Based Rating, a system whose purpose is to accurately measure player skill and performance, to be confused for a progression system or to feel like you can get a high rank just by playing a lot of games . Please rest assured that our number one objective throughout all of this is to provide the best experience possible for our players, and that we are firm in believing that the way to accomplish that goal is to create the fairest and most accurate rating system possible.

We’ve heard your feedback that the system is not yet where it needs to be, and our data agrees . While it is heartening to see that players are in fact being consistently sorted into tiers based on performance and not time played, it is clear from metrics such as the rating distribution table we talked about previously, that some changes need to be made.

For the second test, you can expect all rating tiers to be stricter with their performance requirements-resulting in an increase in difficulty across the board. Higher tier players will also find that the amount of RP they can earn from a single game will decrease compared to lower tier players. Master tier players will find that they can no longer gain rating forever as there will be both a soft cap that raises the difficulty for every 100 RP earned above 3500 as well as a hard cap of 5000 RP.


Closing Thoughts

Skill Based Rating is a work in progress, and we will continue to work together with the community to make sure we are always pushing the ball forward in the right direction.

This is the type of living system that requires constant attention from both developers and players to keep quality high and the experience enjoyable.

As we move onward to the second Labs test of SBR and beyond we will do our best to ensure that the direction of development stays aligned with the needs and wants of our community, and keep our fingers on the pulse of the community to make sure we stay on the right track.

Thank you all again for your participation in SBR test one. Good luck out there on the battlegrounds! We will be back again with another update on the state of SBR on the other side of test 2.

-PUBG Skill Based Rating Team