Rocket League Dev Tracker

(2020) Update on PsyNet Stability Issues

This thread was added on February 04, 2020, with posts from Psyonix_Devin.

Original Post

Hi everyone, unfortunately we had to drop Rocket League into maintenance mode again today, and we wanted to share some info on what’s been happening behind the scenes since last week.

On Thursday last week, we performed scheduled maintenance to move our backend infrastructure (also known as PsyNet) to an updated version of MySQL. Once we hit a peak number of 315,000 concurrent players on Friday, PsyNet started experiencing critical errors, which continued throughout the weekend, and today. We had to repeat this process several times over the weekend, which is why we made the difficult decisions to delay the starts of RLCS, Rivals Play-Ins, and CRL.

We are still investigating the root cause of the problem within PsyNet. This is something we’ve been working on non-stop over the weekend, and we are committed to getting PsyNet back into a stable state as soon as we possibly can. Once we have PsyNet back in a stable state, we will share more on the specifics on what happened, and what we’re going to do to prevent stability issues like this in the future.

Finally, we are still moving forward with tomorrow’s content update at 10 a.m. PST / 6 p.m. UTC, and we’ll post patch notes here before the update goes live on all platforms.

We want to apologize for all of the issues over the last four days. Know that we are working as hard and quickly as possible to fix this, and get it right. Thank you.

EDIT: In case there is any confusion, servers are up right now. The maintenance mode referred to at the top of this post ended several hours ago.

lucki48

So servers are still down?

No, game is online as of this post.

Slothies

"We are struggling like crazy ...so we're also going to toss an update onto the bonfire that is our servers. Hopefully it goes ok."

Why? Just hold off on the update. One thing at a time!

Why? Just hold off on the update. One thing at a time!

Appreciate why it seems odd, but deploying the update tomorrow is handled by folks here who aren't working on the PsyNet issues. Downloading the update won't have an effect on PsyNet either, since you download directly from Steam/MSFT/Sony/Nintendo.

jcreek

I know it's super frustrating (I had to cancel a charity tournament on Saturday as part of a 12 hour livestream for a mental health charity) but obviously you guys are doing your best to get it all fixed and get us back to playing the game we love. Thanks for communicating about it and letting people know what's going on 😊

Can you DM me details about your stream?

Kallumon

So do we have any new dates or information based for when RLCS/RLRS will be starting?

Waiting on a few things internally before we make any additional changes to the schedule.

[deleted]

[deleted]

I was totally fine with this thread until reading this comment.

supra_snak3r

Use me as an "oof" button

Oof.

Marchey1

what did he say?

It was an Episode I joke, not sure why they deleted it.

velixo

u/Psyonix_Devin any update regarding the planned post regarding specifics of what went wrong, and how you're planning to prevent future stability issues? Is the post in the works? Would be lovely to see!

Hiya! Nothing specific to share yet but this is going to happen, for sure.