Part 2: How We Built It
(Step back to Part 1 for an overview of the network models we assessed)

Choosing a Network Model

It turns out that determinism is quite an endeavour - on PC you’re dealing with effectively infinite hardware profiles and nondeterministic behavior shows up in the darnedest places - virtual machines for certain languages, differing compilers, it might even come up in floating point numbers. To make things worse as an independent UDK developer, we don’t have access to change the underlying engine code in Unreal 3. All in all, we ran too high a risk of running into non-deterministic behavior that we simply could not overcome.

This left us with the traditional Tribes/Unreal model with high bandwidth requirements. In an RTS with thousands of units moving at once, the per-player bandwidth requirements would have been too restrictive - we wouldn’t have been able to make the game. Fortunately, we wanted The Maestros to be something a little smaller, a lot scrappier, and much, much faster. The update frequency of the Unreal networking architecture would give us excellent responsiveness - one of our core design pillars. With a small enough unit cap, the bandwidth requirements wouldn’t hinder players either, as long as they had modern internet connections.

We also wanted the game to be accessible for new players, and the hassle of configuring routers and/or firewalls wasn’t something we wanted players to deal with. We were also going to need servers with pretty large outbound bandwidth, which many home internet connections wouldn’t be able to support. For these reasons, we settled on a dedicated server model, putting the onus on us to host game servers.

Will It Work?

We didn’t take this decision lightly, though. We started with a few assumptions. One, our users would need a reliable 1 mbps internet connection to play The Maestros. This definitely doesn’t apply to every potential player, but after hearing about others coming to the same conclusions, we felt reassured that it was still worth building. How did we know this was good enough for such a bandwidth-intensive network model? First, we did some naive math. If the x, y, z of both position and velocity are stored as 4-byte floats, sending them 30 times a second gives you: 3 2 4 30 = 720 byte / second / unit. Ok, and we’ve got a megabit per second of data which is 1/8th of a megabyte = (1024 1024) / 8 = 131072 bytes. So our theoretical cap for moving units, for all players at a single point time should be 131072 / 720 = ~182 units. Now, this is little more than a gut-check number, given how much more complex things could be, but 150-200 units was just enough for us to make a compelling RTS.

As soon as we got the basics of our game up and running in-engine, we put our math to the test. Unreal performs many improvements on the naive model we described above, so we were able to get 200 units moving on the screen simultaneously with little effort, after aggressively pushing the maximum bandwidth per client upwards. We used the ‘net stat’ command in UDK to monitor our network usage. Here’s a snapshot of the stats with nearly 200 units moving at once.

In a 3v3 match, we put the cap for each player at ~20 units, leaving half again as many for neutral monsters that players would fight around the map.

Client to Server Communication

We expected moving units to be our biggest bandwidth hog, but there is a whole category of issues outside of server -> client position updates that we had to solve. Fortunately, we had a lot more freedom to come up with answers. First question: how do we tell the server to start moving units in the first place?

In Unreal, there are two primary ways to send data between client & server: replicated variables, and Remote Procedure Calls (RPC). In UDK, replicated variables are sent from the server to the clients every so often if they have changed, and they can trigger events when the client receives them (i.e. isFlashlightOn goes from false to true and causes an animation to play). RPCs are just functions that a client can tell the server to execute, or vice versa. A “server” function will always be called on the server, and a “client” function will always be executed on the relevant client.

The First Command Payload

So obviously, our commands need to be sent as server RPCs if we want a client to tell the server to do something - like move their units from one place to another. The next thing to determine was the payload. We wanted to come up with something generic that would encompass any kind of command - attack, move, use an ability, etc. Our first pass was sending a json string over the wire with a location and unit IDs. A move command for a full army might have looked something like this:

{“commandType”:”MOVE”,“unitIds”:[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21],”location”:{“x”:100.0,”y”:200.0,”z”:300.0}}

Jsons are certainly generic, and we could imagine an attack or ability command having unitIds and targetUnitId, etc. There are 131 characters in that string so theoretically ~131 bytes in each payload. When tested in-engine, with whatever function data and reliable-transfer data the engine adds, the full call turns out to be ~260 bytes, and we got up to about 8,000 B/s by clicking around as fast as we could. That’s pretty high, but well under a typical player’s outbound bandwidth. It doesn’t help that we’d also to have to send those commands to each player (in most cases), adding up to 48 KB/s to each player’s already-taxed download speed.

A Smaller Command Payload

If you’ve ever made a real-time networked multiplayer game before though, all this probably sounds pretty ridiculous to you. Keep in mind that this was a bunch of college students, trying their hand at multiplayer game programming ;) In reality, that payload is much larger than it needs to be, and the time to serialize and deserialize those strings could really hamper performance. When we put this into practice, more than 2 players in a game tanked server performance to unplayable levels.

So we went back to the drawing board. Json strings inherently have name information (e.g. “commandType”) that is unnecessary if a strict order for each piece of data was maintained. For example, commandType will always be the first four bytes, unitIds will always be an array after that, etc. Additionally, Unreal Engine wasn’t built to tightly pack strings for network transfer. Floats, vectors, and integers, however, are Unreal Engine’s bread and butter, and they’re much cheaper for it to serialize and deserialize. In came the idea of the “FastEvent” which was just a tiny little object packed with primitive data that could be efficiently serialized, transferred, and deserialized. A movement payload still contains 21 integers, a 3-float vector and a 4 character string (21 4 + 3 4 + 4 = 100 bytes), but when we give Unreal these raw types, our typical command goes from 260 bytes to 110. Clicking around as hard as we could would barely tip 2,000 B/s in upload speed. Packing and unpacking those primitives into little objects didn’t cause so much as a flutter in the CPU usage either, and our 3v3 games became buttery smooth.

We even had space to fit a bunch more floats, booleans, and vectors used by other commands, and our theoretical max payload still sat pretty at ~160 bytes. Of course, UnrealScript didn’t make this transition terribly simple. You can’t pass objects over RPC functions, and you can’t really pass arrays either. On top of that, there’s a maximum number of arguments, so you can’t very well send all those unit IDs one by one. This left us with strings, primitives, and structs of primitives. We settled on an object that looked like this image below.

Yep, we had to hand-build arrays as structs. Coding-terror aside, each command had it’s own object that implements a simple interface with one method for translating the specific command into a generic FastEvent. Once it was translated, we would manually unpack each variable into a function argument, so that they could be individually disregarded if they were 0 or null-type and then we would piece it back together on the other side, and translate it back into its specific command type to be processed by an appropriate handler. It looked something like the snippet below.

It’s quite the function call, but it’s performed beautifully, and upload bandwidth hasn’t been a problem since!

As always, technology places certain limitations on game developer, but in The Maestros case, working with small units caps has really allowed us to make a game that looks nothing like anything else out in the genre right now, which has been pretty cool.