Posts by wotc_beastcode

voltaiclv

Any thoughts to pre-allocating larger blocks of memory, and using those, instead of hitting the wall with memory fragmentation?

Something more akin to the object pool pattern would probably be more helpful and squeeze performance out of your engine before even further tweaks would be needed.

Automatic garbage collecting is (usually) something to be avoided in the world of optimizing game engines, as I'm sure you're learning. I'm glad your team is making strides!

For sure. We use a fair amount of object pooling, both with GameObjects (like cards) to avoid instantiation costs and with Unity-agnostic data types (like Lists) to avoid allocations.

Alikont

There is a thing that finds most of the "leaks", it's called Garbage Collector, and .net already have it.

The problem is that it's very conservative in what it considers a leak, because if program can theoretically reach the object, even if it will never use it again, it's still a valid object.

There are few ways to make memory leak in C#, one of the common leaks is event handlers.

For example, when you create a new game, it creates an object, and game object needs to handle mouse input. So it's subscribing to mouse input evens. What it means is that input manager now have a valid reference on Game object, and notifies it about every mouse movement.

If after game is finished you throw away Game object, the reference on it still exists in Input Manager, and because it's theoretically reachable it will be invalid to remove Game from memory, because it will crash Input Manager.

This person knows what's up! We've had a few of the event handler one. :D

oreguayan

Can you nerd out for a bit on this and elaborate? I’m curious what that entails

DID SOMEONE SAY NERD OUT?!

I can do that. :D

SoneEv is pretty much correct with their explanation.

As a programming team gets more familiar with the problem they are solving, they learn better and better ways to approach it. It's always a careful balance between adding new features and "cleaning up" what you've done in the past. Too much cleanup and you stop making progress. Too little and you start moving too slowly make any progress.

Another manifestation of this is when you start hitting scaling limits. For example, Arena has a lot of rich visual effects and situational graphical elements. We have an elaborate set of rules for which one to play in a given situation. Hand-coding is inefficient because it's hard to maintain and limits the work to only engineers. So we have a flexible data system to specify all the rules. Our first system was fine when we had a few hundred visual assets and a dozen variables that affected them. But now we have thousands of elements and hundreds of...

Read more

Awww. Thanks for the kind words and understanding where we are coming from. I will say the "and the rest of the team" part is really where the work came from. Even a small release takes a lot of mobilization. I just looked and numbers and pointed at people and code. Also, a special shout out to Megan O'Malley, who realized she had an HD3000 at home and carted the thing in at significant personal inconvenience so that we could get working on it immediately.

GoboWarchief

This is great news. Any time frame by chance?

Try now :D

Crocodales

Thank you for making this thread sticky again. In the last one, one of the WotC devs mentioned they were giving the Intel HD 3000 the side eye and had one coming into the office to bang on. Have there been any more developments on that? I’m one of those running the Intel HD 3000 and have been locked out since the 09/04 update, any new information is much appreciated.

We've identify and implemented a mitigation for that chip and others like it. It's just a matter of getting it to you. Stay tuned.

marumari

Can I ask why Arena is a 32-bit program and not a 64-bit program?

Lots of Magic players are still on 32-bit windows and we want to be as inclusive as possible. We'd love to provide both, but that's further down the road.

A quick update on our progress on our findings. We have identified a couple of issues that, in conjunction, lead to frequent crashes for some users. We are working on getting a fix to you. As always, I cannot promise a specific timeline, but it's in the works.

[Techy details follow for those who are curious]

Since integrated GPUs don't have any VRAM, they borrow from main memory. So, for example, if we are using 2GB of graphics memory and 2GB of system memory, an integrated GPU will use 4GB and crash a 32-bit program. Kind of a simplification, but it's pretty close.

The HD3000 (and probably a few others from that era) also seem to exhibit weird behavior where it uses up way more than we'd expect, even from an integrated GPU. For example, am HD5500 winds up using less than 1.5GB at game start, whereas an HD3000 winds up using more than 2.5GB. We are looking into the source of this discrepancy.

But why did it get so much worse recently? In a recent release, memory usage increased i...

Read more
MoodyMcSorley

Thank you so much for the effort you put in this post, T.J. I felt bad for the players having their troubles and was pretty upset when the crashes started happening to me as well on the ELD update. It means a lot as a fan to see a response that shows these problems are 1. On your radar 2. Being taken seriously.

A trend that some other players noticed is how older laptops with the Intel HD 3000 Graphics card seem to be a common variable (I'm in this camp as well).

Good luck to you and the team, and thanks again.

Yah, we've definitely been giving the HD3000 the side eye. You aren't the only group with the issue, but the 3000s have gotten a pretty big uptick since 9/4. We have one coming into the office tomorrow so we can bang on it. Thanks for sticking with us.

Thank you very much for putting together this post!

Rest assured, we are aware of these issues and we are working hard to resolve them. Definitely keep the bug reports coming. We do read them and we do take them seriously.

I'm the tech lead for the client, so I wanted to chime in to give you guys some context on the issues we are investigating and give some ideas on how to help us solve them.

[Warning: You are about to get a whole bunch of engineer speak.]

1. Unity Crash

This is the one we are most concerned about right now. It definitely seemed to start hitting a lot more people around the 9/4 release. We are looking into what changed but also trying to understand what's causing the issue. It generally appears to be an issue with the game running out of memory. That's why you'll be fine for a bit, but then it will crash seemingly randomly.

What's weird to us is that we are seeing situations where we didn't actually use up that much memory with the game itself, but somehow the...

Read more
And3riel

One factor that probably confuses the issue is that a lot of stuff happens when we receive a message. Magic game states are complex and can cause a lot of changes. This often results in asset loads and a bunch of other hefty logic. So there is definitely a correlation between

receiving

a network message and frame rate hits. Naturally, we are looking into improving those.

Whoa hold a minute there. You dont have all assets loaded before the match starts? They only load after server issued an action?

Why would you decide to load assets on the fly like that? It seems like the amount of assets needed in one match is pretty much given by the decklists. Or are you telling me that the assets for 150 MTG cards do not fit into memory at once?

There are a number of reasons, but as an example, we can't pre-load your opponents deck (animations, card art, etc...) until you see a given card. If we loaded that data ahead of time, the client could be hacked to show your opponents deck. And naturally we don't want to load every possible asset.

You are not alone. A number of players are experiencing degraded performance and the tech team is working hard to sort the problem out. If you want to help, you can provide information according to this tweet: https://mobile.twitter.com/mtg_arena/status/1178124572878458882?s=21

A detailed description of your experience is very helpful. Does it get worse the more games you play? Specifically how does the lag manifest? Does the whole game freeze up for a moment? How frequently and how long are the freezes? Do they happen during particular actions? We have reporting for these details, but getting a user's perspective on this can help us make sense of the numbers.

Read more
FooberticusBazly

Providing more information about system specs wouldn't hurt anything. But from the evidence I've seen, most if not all of these performance issues are directly tied to network/netcode/server issues.

The game's netcode (client and server side) does not handle network interruptions well. Furthermore, I've seen evidence that the game's render loop is bound to the netcode, blocking until all requests/responses are resolved before rendering frames. I've black box tested this (and other possible causes of these issues) using the game's resource meter, Wireshark, Window's Resource Monitor and various cpu/gpu monitoring tools.

Using the game's resource meter, you can observe the game's framerate is directly tied to the network latency between the client and server. When the latency rises, the framerate drops in lock.

Using Wireshark, you can observe all of the network traffic going to/from the server gateway. Nearly all client-side actions, including those that do not change the game state such as picking up a card and dragging it around the screen without playing it, fire a constant stream of packets to the server. These network requests directly relate to observable spikes in latency, which in turn directly reduces the frame rate.

From these observations you can deduce that the game's main loop is waiting to render frames until network requests/responses are resolved.

Network/netcode/server issues would explain the variance in the reported issues. Two people have the same cpu, gpu and ram. Both are using the same version of Windows. One has problems, the other doesn't. Factor out the commons (hardware, software) and that leaves the network.

Or one person has a monster gaming rig and another is playing on an 8 year old laptop using the onboard gpu. The monster rig's gpu, cpu and ram aren't anywhere close to full load, the laptop is running at 90% cpu and gpu usage. The monster rig has performance issues, the laptop doesn't. Factor out the obvious disparities and that leaves the network.

When most people talk about "internet speed" they're thinking of bandwidth. Bandwidth is a measure of how much concurrent data you can transmit at one time, the width of your pipe, which isn't really a measure of speed at all. You can run one of those online "speed tests" that will tell you your bandwidth and your ping, or the latency between you and whatever internet speed test server you're testing against. This ping value is closer to a measure of internet speed, but all it tells you is the latency between you and that one specific server.

The only measure of latency that really matters in this case is the latency between the game client and server. All your other network connections could be running at 1ms latency, but if the latency between the game client and server is higher then the "internet speed" of all that other stuff doesn't matter.

To make things more complicated, this client/server latency could vary for many different reasons. Playing over wifi vs a hardwired connection will introduce variance. Busy network legs between your home and the server gateway will introduce variance. The servers themselves, if they're under heavy load and are responding slowly, will introduce variance.

The solution to this kind of problem would mean a refactor of the game's core code, including decoupling the render code from the network code so it's not waiting for requests/responses to resolve before rendering frames. The netcode needs to be more resilient to handle network variance without disconnecting.

There is also likely an issue with the server architecture. There may not be enough servers or resources to scale under peak load, or the messaging architecture needs to be rethought, or any number of issues with optimization. What I do know from reading WoTC's tech job postings, the server code is written in C#/.Net and the servers are running in a Microsoft cloud environment instead of using an industry proven, time tested stack and cloud environment.

I'm sure the developers at WoTC are well aware of all of this.

I'm an old software engineer (architect) with over 20 years of experience. These kinds of changes, especially when they touch the very core of both your client and server architecture, are not easy, fast or cheap. It's like keeping your Honda Civic from the 1990s vs buying a new Tesla. The civic is ugly and has 90 horsepower. But it's paid for, gets you where you need to go and costs less to maintain than buying a newer, more efficient car. This is exactly how the people with the checkbooks at Hasbro/WoTC look at the issue. They may eventually replace the Civic, but they're in no big hurry to do it because it still carries the groceries.

Thanks for the detailed and well-thought-out post. Thanks to the OP as well for urging users to send us detailed reports. Believe me that we get them and believe me that we find them useful.

As far as network latency is concerned, I can tell you fairly categorically that the game does not hold rendering or game loop processing to wait for network replies. Game actions like a land being played or a spell resolving are definitely paused to wait for the server, but rendering is still occurring. In fact, one of the dead giveaways that the server is under heavy load or that your connection is having trouble is when you play a land and it pauses mid-flight and hangs out in the full-card mode. We start the animation for playing the land while the network request is in flight, but don’t finish the animation and put the card in play until we get the response.

You can determine if the render loop is still running by looking at some piece of the game that is always in motion (flying creatures...

Read more
Ootter31019

Its hard to say its the program if it isn't happening to everyone. But it honestly seems like they have done something that makes it run worse on higher spec machines. My terrible Lenova Thinkpad that is like 5 years old and shit tier specs runs it more or less fine. I have a few laggy screens and sometimes it will pass at the start of a draw. Very playable though. I would check your drivers, and probably just make a report to WotC for more information they can work with.

I'm on vacation (so you definitely don't see me posting) and I plan on digging into this more when I'm back, but I've been just playing as a customer and noticed something a little ironic. There is something to your theory that performance can degrade more on higher spec machines. We still have a number of per-frame memory allocations we are working on killing. So if you are running at a very high frame rate (I was around 250), you are allocating a lot more memory and, therefor, incurring a lot more garbage collections. Capping my FPS to 60 smoothed me out a lot. It's only one small piece of the puzzle, bit I though I'd mention it.

Read more
andreliverod

You mean the MTGA only set that is called ANA? the Starter decks contain cards from sets like M20 etc.

As far as I can see you cannot exclude sets like writing "S!=ANA" does not work. Does not seem like it.

Not at a computer to double check my work, but you can add a minus sign to exclude a given term:

-e:ana

elHahn

Wow - that's really neat.

Do you know if there is a documentation of the advanced filters?

Our super awesome community folks put something together here:

https://forums.mtgarena.com/forums/threads/46567

ChooJeremy

While this works, note that this is dangerous, because hybrid cards aren't shown. For example, if I apply c<=rw on my sealed pool, which also contains [[Dovin, Hand of Control]] and [[Kaya, Bane of the Dead]], neither of them are shown even though they could easily fit into my deck.

SHUCKS! You are totally right. Hm... I wonder what the right filter for that would be. "Castable with"?

I've been playing a bunch of sealed and I kept wanting to visualize different color combos. When you are often building a 3-color deck, toggling on those three colors really starts to be "all those colors and all your gold cards". If you want to only see gold cards that are in your chosen colors, you can use the advanced filters in the Search box.

As an example, I am thinking that I'll go Mardu with this pool (Red/White/Black), so I put this in the filter:

c<=rwb

It looks a little goofy but that basically says "Show only cards that have these colors or fewer". This will also include colorless cards.

Read more