3.7.4 Deployment Downtime Incident Report

Today during the deployment of the 3.7.4 patch we encountered some problems that led to an extended amount of downtime for Path of Exile as well as progress loss for any gameplay time played between when we initially deployed 3.7.4 and when we were finally able to deploy the fixed version approximately 2 hours later. I would like to take some time to explain what happened and how we will be preventing it in the future.

Timeline

At 12:00 NZT today we took the realm down to deploy 3.7.4. The downtime took approximately 18 minutes and the realm was back up at 12:18 NZT. At 12:25 NZT we were informed by Kakao Games, our Korean publisher, that Korean users who were logging in were being presented with the Create New Account dialog instead of logging into the existing accounts.Without knowing what the problem was, my first call was to order the shutdown of the Korean login gateway in order to potentially reduce the amount of damage tha... Read more