PUBG: PC Dev Letter

Hello players,

This is Taeseok Jang, Executive Producer of PUBG PC.

With the entire world facing the threat of COVID-19, let me start by wishing you and your loved ones good health. Like everyone, PUBG’s teams around the world have had to change our work practices a bit, but we’re doing all we can to safely provide new content and uninterrupted live service to you all.

2019 was a busy year for us, with lots of new content, updates to some of our maps, and a number of quality of life fixes. While much of this content was very well received, we know some of our other efforts didn’t land as well with you. We also had trouble nailing down the causes of a few persistent bugs and struggled against a relentless onslaught of cheat programs. These are some of the things I’ll be talking with you about today.

Last year was also a year of growth and change for our teams. To say that we suffered from growing pains would be an understatement, and most unfortunate of all is that during the massive expansions of our teams, we failed to communicate properly with you all, giving the unfortunate illusions that we weren’t listening or did not care. It’s this that I’m most sorry for, as one of the reasons PUBG was able to grow as much as it has was the love and passion of our community. That dedication has carried us through three years of service and I’d like to reaffirm to you all today that we will focus on the issues that are most important to you and continue to improve the game for years to come.

There is quite a bit to go over in each category, so I’ll be releasing additional letters in the future to talk about other pressing topics. In this letter, I’ll be giving you an update on some of those major areas of focus for us, including anti-cheat, performance, and stability. We may be throwing around a lot of technical terms, but we’ve tried our best to make everything as easy to follow as possible. Also keep in mind that, for anti-cheat especially, we’ve tried to share as much of our plan and progress as we can without giving too much information out to the people we’re trying to stop.

Let’s get started.

  • Countermeasures for Hacking and Cheat Programs
    • Improve preventive measures
    • Strengthen monitoring/banning reliability
  • Performance & Stability Improvements
    • Performance improvements
    • Stability improvements
    • Network improvements

Countermeasures for Hacking and Cheat Programs

The fight against cheat programs is never-ending and something we’ve been trying very hard to find a solution for. Last year, we spent time working on various measures to block cheat programs. Most of these actions focused on blocking cheat program developers to make it more difficult for them to create these highly lucrative cheats. As these cheat developers excel at adapting to our measures, we chose to keep these efforts secret to increase the time it takes for them to react, as much as possible. The unfortunate side effect of this is that it makes us look like we’re doing nothing against a high-priority issue. This year, we will be making improvements to ensure players can feel the impact of our efforts.

Enhanced Preemptive Measures

Preemptive anti-cheat actions mean making cheat programs unable to function before a cheater ever runs the program. This is mainly accomplished by targeting the development of cheat programs to make the effort outweigh the gains. We’re attacking this in a couple ways.

Two-factor Authentication System
We receive countless support tickets from players whose accounts have been permanently banned after their Steam account was hacked and used with cheat programs. The sellers of cheat programs often trade these accounts due to the high account turnover their customers experience. The vast majority of hacked accounts do not have Steam Guard activated and many have their login information compromised through phishing sites that lure players in with the fake promise of free in-game items.

To help prevent account hacking and improve security overall, we will be adding SMS authentication and a secondary authentication system using a one-time password. Your account will need to have two-factor authentication enabled in order to participate in ranked games to help ensure the fairest environment possible.

Improve hack vulnerabilities in code
Cheat programs target specific factors to manipulate our code. Here’s what we’re doing to prevent manipulation to weapons, vehicles, and character movement.

  • Weapons
    • Reinforce server authentication related to the gun’s firing angle, recoil, ballistic trajectory, etc.
    • Analyze and overcome the vulnerabilities of client logic related to aiming.
  • Vehicles
    • Reinforce server authentication related to the vehicle’s speed, moving angle, the time period it can remain in the air, etc.
    • This will not only ban accounts that are detected, but will also deter cheating tools in real-time, kick cheaters out of the match, and issue instant bans.
  • Character Movement
    • Reinforce server authentication related to a character’s move speed, the time period they remain in the air, etc.

Enhanced Cheat Detection and Bans

To date, we’ve banned millions of accounts for using cheat programs. However, even with bans happening on a massive scale, cheaters still make it through and in some cases remain for much longer than they should. This year, we’re focusing on enhanced detection for our automated ban system as well as improving our hardware ban systems.

Expand Detection Range
Our teams will further expand the detection range to better prevent cheat programs. While our initial efforts yielded significant results, the developers of these cheat programs soon found ways to bypass our detection methods. We’ve been working to mitigate this issue by pushing updates on a regular basis. Cheat programs have also been targeting kernel domains to help their programs greatly avoid detection. To help close this loophole, we’ll soon be expanding our focus here so that our anti-cheat solutions have higher authority to detect and stop these persistent cheats.

Automatic Analysis
The disadvantage of the manual analysis method is that it requires repeated countermeasure updates due to the cheat programs being constantly updated to stay a step ahead. One example is the automatic shutdown of a PC when our programs initialize the authentication process of a suspected account.

The goal is to automatically find clients using cheat programs by looking for exact differences in values between environments that are legitimate and those using cheats. We can then automatically block the account if the environment is not pure. Our expected timeline for applying this feature is currently scheduled for July 2020.

Toughen HWID Bans
When a Hardware ID ban is issued, it records information on the specific components inside a PC so that even when a new account is purchased or stolen, our servers know that PC is blacklisted. The more information collected, the more accurate the ban is but by seeking this additional information, our methods become easier to get around as sometimes replacing a single component is enough to bypass the ban. In addition to this, the developers of cheat programs have found ways to falsify Hardware ID data, which renders these efforts much less effective as well.

We have recognized the limitations of the existing HWID ban system and are developing improvements. These improvements will seek hardware information at the user and kernel level using specific algorithms so that even if a cheat program tries to bypass, we will be able to quickly block the program again. This should help apply bans to PCs as well as accounts that use cheat programs.

Strengthen the Level of Bans
One of the ways we detect cheaters is through the reports you send in during your games. Among those reported users, we instantly ban those who have direct evidence of cheating and impose a temporary 24-hour ban on others that need further investigation.

The majority of temp banned users convert to a permanent ban, but a small portion are cleared due to either lack of evidence or us finding that the account was falsely flagged.
Our goal is to avoid false bans as often as possible, so we’re cautious when taking action against an account; however, we’ve come to realize that being too cautious here can negatively impact far more people than the suspected account. Thus, we’re changing our investigation methods to be more strict against accounts that have been temporarily banned on suspicion of using cheats.

In addition, we will target suspected cheaters for additional monitoring. If an account has been temp banned at least once, it will be added to a management list to be investigated further, sometimes including live monitoring of how they play the game.

If an account is temp banned multiple times on suspicion of cheating, we will automatically add additional time to their ban to ensure we have time to properly investigate the account. We’re also looking to further automate the investigation process so that users who are found innocent can quickly return to the game by accumulating appropriate proof across multiple innocent accounts to check against.

In order to apply these measures, we’ll need to re-establish our existing policy standards. Our plan is to apply these in June of 2020.

Performance and Stability Improvement

Now that we’ve discussed our anti-cheat plans, let’s move onto some of your other pressing issues, performance and stability.

First of all, we know it hasn’t been the best year for this category so far, with persistent network, server, and game client issues. Although providing a high level of service is always one of our top priorities, specific issues have endured for far too long and overshadow any good improvements we make. Hitching and game crashes especially have increased and remained persistent in the recent months. Your feedback has been heard loud and clear and is completely justified.

Recently, we deployed a fix that greatly reduced instances of hitching for many users in the game. We also rolled back the blood changes we made in Update 5.3 after they were found to be causing crashes for some players. While it’s nice to have found solutions to these issues, we know we need to be faster at detecting these issues if they make it into the game and, more importantly, finding them before they get there. To this end, we’re working on making our pre-deployment tests more efficient.

In the case of our game servers experiencing issues, the biggest cause here has been an onslaught of DDoS attacks that had persisted for many months. Earlier this year, we rolled out a new defense we developed to mitigate these attacks, which has proved to be effective. Now that it’s in place, we’ll continue to strengthen it where possible to make sure these attacks do not disrupt your experience when they occur.

Performance Improvements

Optimization of Streaming/Hitching

  • Improve Seek Time to find content in the package through pak file optimization
    • Pak file: A “package” file that compresses all assets (weapons, vehicles, buildings, costumes, etc.) used in the game
  • Improve performance by reducing the number of Level Streaming and through optimization
    • Merge Landscape → Reduce the number of Level Streaming and GC (Garbage Collection) → Improve hitching issues
  • Improve hitching issues caused by graphic data loading/unloading
    • Change the method of Level Streaming from graphic data loading to one frame to loading separately per frame to reduce hitching issues
  • List of improvement for stuttering issues
  • When players open Inventory
    • When players turn on Character Preview
    • When players equip weapons
    • In situations where players make kills
    • When players throw items
    • When players open UI including Option/Mission, etc
    • Minimize the number of Font types used in the game for optimization
    • Optimize various skins shown in the game – Items (Weapon, character)
      • In particular, improve hitching issues by asynchronous loading of gun skins that take a long time to load

Optimization of GPU

  • Optimize and improve GPU performance by further utilizing HLOD
    • HLOD refers to a system that combines multiple static mesh actors into a single static mesh actor when these are viewed from a long distance. This can help improve performance by reducing the number of the actors that need to be rendered on a scene and thus reducing per-frame draw calls.
  • Optimize it by improving landscape structures

Optimization of Animations

  • Optimize the animations so the amount of calculations can be adjusted based on a gameplay-wise priority to address the overload issue due to dense character population or excessive actions played

Improvement for Low-spec PCs

  • We’ve seen cases where the game can’t be played due to performance issues on various PCs. To minimize such problems, we will make improvements to automatically adjust the degree of optimization based on CPU use so the game performance can be stable on low-spec PCs as well. Also, we are currently working to reduce the cost of animation processing on the machines with low CPU cores.

Stability Improvement

We still have instances where players experience intermittent crash issues in the lobby, in-game, on the loading screen, etc. Our dev team is working to improve these issues by enhancing internal verification processes and further utilizing crash reports.

Fix Client Crash Issues in a Timely Manner

  • Client crash issues occur mainly due to memory loss and we’re strengthening our check process when builds are created
    • Remove memory being unnecessarily used through continuous memory filing
    • Optimize art resources being used more than needed
    • Optimize content loading methods in general through code refactoring
    • Prevent memory leakage through live profiling
      • This should prevent issues that stem from extended play

Enhance Build Verification Process

  • Introduce an automatic verification tool while enhancing manual verification methods
  • Introduce various check-up system to monitor program’s health
    • Tool for checking memory leakage
    • Tool for checking memory usage
    • Tool for detecting memory damage which detects crashes caused by unintended access to memory
    • Hitch checks
    • CPU/GPU state checks
  • Create a testing environment with 100 users using AI
    • Performance tests are already run when there are 100 players in a match, but player count and results can vary.
    • By utilizing bots, we can run large scale tests similar to the actual gameplay environment as often as we want

Network Improvements

DDoS

Distributed Denial-of-Service (DDoS) attacks affect the majority of popular online games at some point. When the game server experiences DDoS attacks, all users on that server go through connection errors such as network delay, high ping, and packet loss. DDos attacks greatly increased since November 2019 and became a very serious problem in February 2020. To address this, our teams have introduced additional external solutions, while developing our own DDoS defenses. Thanks to this, most DDoS attacks are being blocked successfully. We will share our measures taken against DDoS in more detail in a separate post.

Packet Loss

After some of the external DDos solutions were applied, we began to receive a higher amount of packet loss reports from NA and EU players. We’ve mitigated this quite a bit, but some of these issues still remain. Since various errors can contribute to this, we’re working on cause analysis and route optimization to further reduce this packet loss and improve the overall network environment.

In Closing

We understand that when we do not communicate with you, your natural instincts are to assume that we don’t care about these issues. We’re working very hard to improve not only our communication, but also the issues we described in this letter.

We’ve got a lot of great content planned for this year, and we want you to be able to enjoy it without wondering when your next crash or issue is going to be. I’ve said quite a bit already for one letter, so we’re going to release further communication on other issues at a later date. Also, thank you for your patience while we got this letter out. We felt it was worth a little extra time to fully articulate what we’ve been working on and what our upcoming priorities are.

The PUBG Dev team is aware of the importance of providing a stable environment and thank you for your continued patience and feedback.

Thank you and stay safe.

– Taeseok Jang, PUBG Executive Producer