We were not intending to reboot unless the cluster became inoperable. This will have corrected itself at DT this morning.

I was looking at the relevant code yesterday, what is your source?

We did and they were caught.

Why is that odd? We know of some issues, but we are not sure which of them might be most player-noticeable and have impact on the experience. During the experiment we have also discovered new issues, which may be more important to fix than some of the issues we know of.

We are aware of this issue; across the cluster right now then different nodes can disagree on the correct time by up to 3.3 seconds. If the Location Node hosting the solar system you are in and the Proxy Node your Client is connecting through are at opposite ends of that spectrum then your Client will simulate the actions differently than the Location Node is executing them, as in at a different time.

These are the chat presence issues I have referenced above.

We are following up on reports on issues with chat presence.

We believe we have found the relevant code and it is linked to this #nodowntime experiment.

We are following up on chat presence issues and believe we understand why this is happening. Will resolve itself during the next downtime.

Yep, noted, thanks; we’ve found the code for this…

We are following up on reports on this.

Indeed, a known issue that asteroids only respawn during downtime.

Very probably different issues; but, yes, we are following up on chat presence issues that may be related to the #nodowntime experiment.

Yep, thanks for the report; we are indeed following up on some chat presence issues that are possibly related to the #nodowntime experiment.

The experiment only really started at 11 o’clock UTC today, before that TQ was just on its normal 24 hours run. So nothing you experienced prior is related to this experiment.

Nothing you experienced before 11:00 UTC today would be related to this experiment.

Nothing you experienced before 11:00 UTC today would be related to this experiment.

So, this is a bit tricky since some things may break or not behave as anticipated and we needed to be upfront about that aspect of this experiment. So we had to tell players.

I already replied to this question; that was not related but demonstrated that we have further improved our ability to fix things without a reboot.

If we were to switch from MS SQL, then Postgres would be at the top of the list.

Not related; interesting nevertheless since this is an issue that has happened once before and the first time it happened we had to reboot both the chat cluster and TQ. This time around we managed to resolve the issue while the TQ was live; so progress on fixing things without needing downtime/reboot.