UPDATE: Server stability issue appears fixed. Be careful with your database page sizes, people.

Hey everyone,

It's been a long day but we wanted to put together a few thoughts while we have a moment waiting for our next server fix to build. This launch has been rough, to say the least. In this post, we plan to address both the ongoing technical realm stability issues and the conversation around streamers getting priority in the login queue. We are sorry that this is being addressed so late in the day - we have been giving the server issues absolute priority and haven't had time until now to write up this explanation.

Let's start with the technical issues.

Immediately upon launch of the league, we could see that the queue was running incredibly slowly. At the rate that it was emptying, it'd be at least two hours to get everyone into the game. The reason was that when players logged into their accounts, the server would migrate any previously un-migrated Ritual characters to Standar...

UPDATE/TL;DR: Queue currently fixed. There was an hour of it going super slowly. We will make sure this never happens again. See below updates for notes about current realm stability.

ORIGINAL POST: When the Ulstatimatum league started this morning, it was immediately apparent that the login queue was moving quite slowly. We are investigating this, and so far it appears that the reason is that this league's character migrations (which are a process that runs when a character logs in, to convert it to the new internal version) are much slower than normal.

Users are getting in, but it's going to take a while for the queue to clear and we're very sorry about that. We're acutely aware that a similar problem occurred last league launch and we thought we had resolved it.

Queue processing should speed up as more characters are converted, and we are trying to find other solutions that will help in the meantime.

Once again, we're very sorry about the delayed start to the league for most us...

UPDATE/TL;DR: Queue currently fixed. There was an hour of it going super slowly. We will make sure this never happens again. See below updates for notes about current realm stability.

ORIGINAL POST: When the Ulstatimatum league started this morning, it was immediately apparent that the login queue was moving quite slowly. We are investigating this, and so far it appears that the reason is that this league's character migrations (which are a process that runs when a character logs in, to convert it to the new internal version) are much slower than normal.

Users are getting in, but it's going to take a while for the queue to clear and we're very sorry about that. We're acutely aware that a similar problem occurred last league launch and we thought we had resolved it.

Queue processing should speed up as more characters are converted, and we are trying to find other solutions that will help in the meantime.

Once again, we're very sorry about the delayed start to the league for most us...

chris_wilson

I am concerned by the reports of this and the other problems mentioned and I will find out what is going on. I am aware that people are fixing stuff aggressively but I have been personally focusing on 3.14.0 stuff so am not totally in the loop about what's happening with other performance issues in 3.13.0. I'll find out.

Replying to myself with an update:

I spoke to the lead engine programmer and we're going to be making a series of changes that address the problems people are describing. These changes will all have trade-offs in terms of other factors like memory usage, but we'll find a sweet spot where the game looks perfect and performs well. The end goal is that slower computers will have longer load times and slower (stable!) frame rates, but everyone's experience looks good and plays smoothly. In terms of timeframes, I'm hoping it's measured in days, but we'll update you once we have a better idea of timing.

chris_wilson

I am concerned by the reports of this and the other problems mentioned and I will find out what is going on. I am aware that people are fixing stuff aggressively but I have been personally focusing on 3.14.0 stuff so am not totally in the loop about what's happening with other performance issues in 3.13.0. I'll find out.

Replying to myself with an update:

I spoke to the lead engine programmer and we're going to be making a series of changes that address the problems people are describing. These changes will all have trade-offs in terms of other factors like memory usage, but we'll find a sweet spot where the game looks perfect and performs well. The end goal is that slower computers will have longer load times and slower (stable!) frame rates, but everyone's experience looks good and plays smoothly. In terms of timeframes, I'm hoping it's measured in days, but we'll update you once we have a better idea of timing.

staudd

Thanks for letting us know the state of things.

Just for the sake of mentioning it: A huge problem seems to be the fact that some ground tiles dont seem to load at all. Ive encountered this problem of transparent floor most noticably in Delve, but in maps too.

I am concerned by the reports of this and the other problems mentioned and I will find out what is going on. I am aware that people are fixing stuff aggressively but I have been personally focusing on 3.14.0 stuff so am not totally in the loop about what's happening with other performance issues in 3.13.0. I'll find out.

staudd

Thanks for letting us know the state of things.

Just for the sake of mentioning it: A huge problem seems to be the fact that some ground tiles dont seem to load at all. Ive encountered this problem of transparent floor most noticably in Delve, but in maps too.

I am concerned by the reports of this and the other problems mentioned and I will find out what is going on. I am aware that people are fixing stuff aggressively but I have been personally focusing on 3.14.0 stuff so am not totally in the loop about what's happening with other performance issues in 3.13.0. I'll find out.

I'm really glad that you are enjoying the league!

It was three months of development for most of our staff, with some QA and a few last-minute changes by whoever was in over the Christmas period. I believe the reason it worked out well was our new development processes: https://www.pathofexile.com/forum/view-thread/2977334 (basically extreme scope control)

So the plan going forward is to continue the three months of development plus a month of QA and tweaks, but that still means that we're doing releases every three months (with the content ideally ready a month before each respective release). For example, 3.14 development started in mid-December, will likely be finished in mid-March and will likely release in mid-April.

And of course, not all releases are going to have endgame expansions alongside the league, so do expect the sizes of the releases to vary this year.

I'm really glad that you are enjoying the league!

It was three months of development for most of our staff, with some QA and a few last-minute changes by whoever was in over the Christmas period. I believe the reason it worked out well was our new development processes: https://www.pathofexile.com/forum/view-thread/2977334 (basically extreme scope control)

So the plan going forward is to continue the three months of development plus a month of QA and tweaks, but that still means that we're doing releases every three months (with the content ideally ready a month before each respective release). For example, 3.14 development started in mid-December, will likely be finished in mid-March and will likely release in mid-April.

And of course, not all releases are going to have endgame expansions alongside the league, so do expect the sizes of the releases to vary this year.

For the last six or so hours, we've been having some quite bad server issues. This manifests in several ways: some players getting disconnected from time to time, various actions serverside timing out, and rarely, items being duplicated or destroyed. This news post explains what we know about the situation and our timelines for fixing it.

This release has seen 11% higher peak player numbers than any previous Path of Exile release, and that would likely be even higher without the server issues. While this would normally be something to celebrate, right now it's causing some big server problems.

Path of Exile's server architecture consists of a central set of database servers and an arbitrarily large pool of game servers. We can add more game servers easily (and have to vary that number constantly to take into account player numbers), but the database servers are much harder to change.

For the first time since 2013, our database servers are overloaded. This is manifesting as various ...

For the last six or so hours, we've been having some quite bad server issues. This manifests in several ways: some players getting disconnected from time to time, various actions serverside timing out, and rarely, items being duplicated or destroyed. This news post explains what we know about the situation and our timelines for fixing it.

This release has seen 11% higher peak player numbers than any previous Path of Exile release, and that would likely be even higher without the server issues. While this would normally be something to celebrate, right now it's causing some big server problems.

Path of Exile's server architecture consists of a central set of database servers and an arbitrarily large pool of game servers. We can add more game servers easily (and have to vary that number constantly to take into account player numbers), but the database servers are much harder to change.

For the first time since 2013, our database servers are overloaded. This is manifesting as various ...

The number of zones it takes for a deferred item to reappear is random and based on how rare the item is. For some items, it's low like 3 zones, and for others it's higher.

The number of zones it takes for a deferred item to reappear is random and based on how rare the item is. For some items, it's low like 3 zones, and for others it's higher.

Hey everyone,

As you know, tomorrow we will release the Echoes of the Atlas expansion. At the moment we're putting the finishing touches on our preparations for the launch.

As you know, 2020 was a difficult year for our team and for the world at large. Echoes of the Atlas is the first expansion we've developed under our new development principles. Our team has worked really hard in an effort to make this expansion one of our best yet.

It has been very helpful to have the extra month for polish, as it has given us time to find problems that normally might slip out in the released version. Just yesterday we added a bunch of new rewards for the pinnacle fights based on our internal feedback that they needed to be more rewarding, for example.

I can't wait until tomorrow when we get to see how players react to the new content. Watching developers playtest the multi-boss fights has been a lot of fun, so I'm really looking forward to having some streams going tomorrow (as the team handle...

Hey everyone,

As you know, tomorrow we will release the Echoes of the Atlas expansion. At the moment we're putting the finishing touches on our preparations for the launch.

As you know, 2020 was a difficult year for our team and for the world at large. Echoes of the Atlas is the first expansion we've developed under our new development principles. Our team has worked really hard in an effort to make this expansion one of our best yet.

It has been very helpful to have the extra month for polish, as it has given us time to find problems that normally might slip out in the released version. Just yesterday we added a bunch of new rewards for the pinnacle fights based on our internal feedback that they needed to be more rewarding, for example.

I can't wait until tomorrow when we get to see how players react to the new content. Watching developers playtest the multi-boss fights has been a lot of fun, so I'm really looking forward to having some streams going tomorrow (as the team handle...

I'm keen for feedback on how we presented the Echoes of the Atlas announcement stream that we did today. It was a massive amount of work for us to put together compared to the way we usually do announcements, but it appears to have gone really well. What did you think about how we presented it? What was good, what was bad? What should we do more of, or try to do differently? We're not so much looking for feedback on the actual expansion/content, but on the livestream itself.

We have to decide whether to do similar livestreams for future announcements. We're worried that smaller releases that are just a challenge league (see Delirium for example) might look underwhelming in comparison and could be disappointing if people turned up expecting an amount of content equivalent to today's announcement. But if it's the right format for revealing information then we'll certainly consider whether the large amount of extra work is worth it.

Thanks again for checking out the announcement and al...

I'm keen for feedback on how we presented the Echoes of the Atlas announcement stream that we did today. It was a massive amount of work for us to put together compared to the way we usually do announcements, but it appears to have gone really well. What did you think about how we presented it? What was good, what was bad? What should we do more of, or try to do differently? We're not so much looking for feedback on the actual expansion/content, but on the livestream itself.

We have to decide whether to do similar livestreams for future announcements. We're worried that smaller releases that are just a challenge league (see Delirium for example) might look underwhelming in comparison and could be disappointing if people turned up expecting an amount of content equivalent to today's announcement. But if it's the right format for revealing information then we'll certainly consider whether the large amount of extra work is worth it.

Thanks again for checking out the announcement and al...