Space MMO – Pioneer-alike

Posted by | Posted in Game Development, Pioneer | Posted on 29-06-2013

Space game MMOs… A brain dump:

I’ve been inspired to brain dump by a post on the Paragon forum.

To quote:

Will multiplayer be instanced or MMO style?
If MMO will players be able to transverse to other servers/galaxy’s with their ships,equipment and credits?

Now, leaving aside the question of what they might mean behind instancing, lets instead think about how a space game like Paragon, not Pioneer note, Paragon might choose to implement an MMO gameplay model.

NB: Paragon is a sort of daughter project to Pioneer. It’s forked from the Pioneer codebase and occasionally pulls in work we do. However Paragon has it’s own developers, development and game specific code, scripting, storyline and art. I’ll be discussing these things from being a Pioneer developers point of view.

Why no Pioneer MMO?

One of the most often asked questions is why don’t we have multiplayer in Pioneer or make it an MMO.

The answer is that in Pioneer everyone is running on their own time. This is because we use time acceleration to speed up the boring bits of space flight that results from using a fully Newtonian flight model. Without it you’d takes days, weeks, months and even years to travel around in Pioneer. That would make it a lot less fun ;)

So whilst you might briefly occupy the same space as another player they might be hundreds of years ahead of you in the future, or conversely behind you in the past. Because of this we can’t have players meeting up, almost ever.

Not much of an MMO when you can’t ever meet another player.

Why can Paragon be an MMO?

Paragon has ditched the fully Newtonian travel part, or rather, it’s still there, and in combat it’ll be interesting to see how players cope with it. However they’ve supplemented the game with faster-than-light travel within star systems. Because of this they have also discarded the time acceleration feature, or will have totally quite soon.

That gets around the “when” part of the problem Pioneer has, and it means that players could all stay on the same universal game time whilst flitting about the stars.

That is the crucial difference that makes it feasible. Not easy, feasible.

The Simplest Approach:

Let begin with how we could naively implement the MMO side of things by considering a single star system with two players in it.

I’ll try to cover some of the extraneous servers and processes you need for an MMO later but for now lets assume that you have things like a login server that issues a valid token and skip to the “game server” part.

Naive Game Server:

Here I am, I’m a game server.

A client tries to connect and hands me a token, I check that token is valid, and allow them too connect.

Order of business now is to:

  1. Find out who they are,
  2. Request their data from our database,
  3. Load any data they’ll need,
    1. ships, textures, sounds, equipment etc.
    2. This is a bit of a moot point since everything is preloaded in our engine.
  4. Add them to “the game”,
  5. Inform other connected clients about them,
  6. Send a snapshot of the current game state to them,
    1. Including information about other players,
  7. Add them to the various update loops to keep them informed about all future state changes.

This is a pretty simple client-server arrangement.

You have one server dealing with many clients and managing the entire game world. Each client connects and no matter where they are or what they’re doing everyone else finds out about everything they do and vice versa. The game, the players ships in this case, is all simulated on the server and positions and updates are sent to the clients who are all running a representation of the game. They can try and do something illegal but all that should do is request that the server does something, the server can validate that request and only needs to process it if it’s valid. Invalid requests can either be quietly discarded or the requester can be monitored for a while to see if they’re making lots of invalid requests and then kicked (disconnected) entirely.

This basic approach will work at first, you can do various things to hide the latency you’ll get from waiting for the server to simulate the game step and then communicate over the internet to update your client. Usually you’ll be simulating part of it on your own machine and just playing that, only making changes to it when the servers version is different.

This will work nicely for between 8 and 64 players usually depending on the amount of processing required each game step, the amount of data that needs to be transferred between each connected client and some other factors.

8 to 64 players however, is not a lot of people. It’s a fairly conservative figure for a game like Paragon and there’s some obvious optimisations like:

  • Paragon has many star systems, exploit this fact:
    • Players in one star system can’t see ones in another so don’t bother telling them about each other unless they’re in the same one.
  • Each star system can be vast:
    • Only tell players about each other at allwhen they’re within a certain distance of each other,
    • If one is on Mercury near the Sun and another is orbiting Pluto then simply don’t update each other position etc,
    • Scale the number of position and other updates depending on distance, for our Mercury guy:
      • Out by Pluto? No updates!
      • Near Mars orbit? Tell them they’re in the system, give basic ship data and rough position, update every minute.
      • Near Earth orbit? Update every 30 seconds.
      • Near Venus orbit? Update every 10 seconds, start to give any customisation info they need for rendering.
      • Nearing Mercury orbit? Update every second, add in orientation and acceleration info.
      • Less than 100km? Constant updates as often as your own ships data, all the info you want to give about another ship.
    • Take advantage of being docked in stations and stop updating people!

There’s many more but take note of the above because they apply to almost every later situation I’ll discuss.

By now if you’ve done all of that we could be looking at between 200 and 1000 ships on our single server, maybe more under ideal circumstances and with some old fashioned profile guided optimisations of the code itself.

Under Ideal Circumstances:

I swear “Ideal Circumstances” are the Unicorns of the MMO world.

If you’ve ever worked on an MMO what you find is that you’ve carefully balanced everything, started players off in far distant lands, given them quests to head of into the wilderness, alone or in small parties and then… then you find 99% of them stood around in a town somewhere complaining about lag because each and every one of them can see 5000 other players.

Lets tackle having a few more players before we cover their habit of congregating in specific places though.

The first problem is that you can now handle, lets be optimistic, 1000 players on your server.

This is fine at first on release day when you start of well below that number and are just happy to see people in your game, but quickly it climbs to it’s limit, and then beyond.

You want to avoid bad play experiences caused by crashes and other issues so you put in a limit on how many people can be in each “instance” of the game world and then you run many servers, each a separate “game” instance. You’re not an MMO in one sense, because you might have many players, but they’re all playing in little copies of a much greater idea and they can’t interact across the boundaries you’ve established.

That’s not an approach I’ve ever liked, it’s one used by many successful MMOs and indeed it might have it’s place at greater populations but right now there’s a lot more we can do before turning to that answer.

A Less Naive Approach:

Scaling Outwards:

What I’m going to propose now is that we exploit the first fact we observed about how Paragon (/Pioneer) work with regard to stars:

  • Paragon has many star systems, exploit this fact:
    • Players in one star system can’t see ones in another so don’t bother telling them about each other unless they’re in the same one.

Then change it to:

  • Paragon has many star systems, exploit this fact:
    • Players in one star system can’t see players in another, so move them onto a different server when they go to another star system.

This changes the equation a bit because now instead of having 1000 player per-server, you can have 1000 players per-star-system(-server).

Therefore if you had 1000 players spread across 20 different star systems before you, it means you could now have 20Star-System servers” running and handling up to 20,000 players. The “Game server” has become a glorified telephone exchange, it verifies the token from your “login server“, confirms some information with the database including what star system you should be in, then it tells you which “Star-System server” to connect to and off the player goes to talk to it and play their game.

When the player wants to enter hyperspace to reach another star system their Star-System server contacts the Game server and asks for the new address they want, it hands it back and away they go.

How you handle that switchover is interesting:

  • Is it all mediated by the Star-System servers as above?
  • Do you disconnect, have your data saved to the database, contact the Game server, get address for new Star-System and then reconnect?
  • What happens if you lose connection between?
  • What if there isn’t a Star-System server available?
  • What if it’s full? Scratch that, we’ll come to it shortly.

Wherever I’ve worked this process is called “server migration“, different places handle it in different ways and the best way is… to analyse your game and decide on the best approach that suits your gameplay because there sure as shit isn’t a universal right answer that fits them all!

My God, it’s Full of Servers:

So we want a server per Star-System, okay, that’s a disk, CPU, motherboard, RAM, case blah blah blah, lets go low-end and call it £500 per machine, we just need to know how many Star-Systems we’re going to have which is:

  • Infinity

Bugger. That’s going to hurt the wallet. Maybe that was a stupid approach.

We have virtualisation and all kinds of fancy networking shit these days, so lets run each server as a process on a multi-core machine. Each machine can now run many of these Star-System server processes and we can separate them using virtualisation or something more custom. When one machine gets too heavily loaded for it’s CPU or networking to cope, we pack up one of the server processes and then start it up on another machine with plenty of spare capacity. Thus we spread the hardware load so that a very busy Star-System doesn’t starve others on the same machine.

You might choose to move the heaviest process or the lightest (I’d personally go with the lightest unless you have a non-uniform choice of physical machines with varying performance) and you might use virtualisation and hardware load balancing or you could mirror your server using the Game server as an intermediary to manage the switchover. Your call according to your budget and understanding.

So you’ve got a server per-Star System, and they’re all lovingly load balanced by the best available practice, problem solved right? No, now you’ve got another problem.

This time it’s the anti-social bastards (ASB’s from here on). They want to play an MMO, with no-one else. I know, I know, I hate them too, almost as much as the overly sociable congregators.


The problem with the ASB’s, aside from them ignoring your amazing group gameplay sections thus hurting your feelings, is that they’ve all headed off into the wilderness for solitude and that means you need 1 Star-System server per-ASB. Now 1 whole server per player is an awful situation to be in. Sure you can run many server processes on a single machine but there’s overhead per-server and you’ve spent time and effort optimising each instance to handle up to 1000 players, not one.

I myself am mildly stumped by these ingrates but we must look beyond that and deal with them, after all they might be exploring, about to return to inform their fellow congregrating gits that they’ve found a lovely new planet to all hang around where they should all head right now, on mass. Super.

Thankfully we do have a possible way of dealing with the ASB’s, at least in low numbers, up to … ooh, about 1000 players per server? Yes you’ve guessed right we’re going to promote the Game server to being a Game server again.

Return of the Great White Dope:

Effectively we’ll tier the service, we’ve done most of the work to support this already above:

  • The Game server:
    • It handles up to 1000 players spread across many different Star Systems.
    • When a Star System gets full enough we spawn a Star-System server process somewhere on the network and migrate all of those players into it.
    • We keep track of all existing Star-Systems and when they get low enough in population they might try and shut down and migrate players back into the Game server.
      • The Game server must moderate this activity based on it’s own player load and prevent it shutting down if necessary.
      • The low-population Star System server can always ask again after a timeout period.
  • The Star-System server:
    • A migratory beast, roaming the network looking for enough processing time to shoulder it’s burden.
    • When no longer in use it can go idle for long periods in case things pick up, or it can after a certain period of time contact it’s parent Game server and inform it that it should take over it’s duties before shutting down.
    • When under low-load, but still with some players present it can try to pass these back to the parent Game server if it’s own load is light enough.
    • Manages a single Star-System and it’s players.

Those are the two main servers that I propose.

From an implementation view they’re both very lightly threaded beasts, good server neighbours to other processes.

They’re also extremely similar, really one is conceptually the same as the other just with some bits turned off because the Star-System server is a limited case of the Game server.

The Game server runs many Star-Systems, and manages external Star-Systems. So the Star-System server just needs to not do the management and run only a single Star system. Thus meaning you can probably avoid some complexity and maintenance by writing a single Game server that can function in a limited way as a Star-System server depending on its configuration.

There are other servers you’ll need; login, patching, chat/messaging, database(s). However these are side servers, you pass through them making temporary requests and you can have many of them in redundant configurations so that if, for example; a single database server goes down it’s ok because the data is replicated across several and you only need one to reply.

So it’s time to address those congregating gits.

Goldilocks and the 5000 Gits:

Our ideal player, allow me speak in dreamy tones for a moment;

  • Logs in,
  • Does a little management of his ships inventory,
  • Buys some stock,
  • Takes a mission,
  • Launches,
  • Flies to a jump point and leaves the Star System he was in,
  • Plays for a couple of hours causing no fuss,
  • Follows the content (generated or otherwise),
  • Groups with a few other players,
  • Logs out deliberately.

If you’re a commercial MMO let me add:

  • Pays in advance by direct debit,
  • Leaves favourable reviews on MetaCritic.

Let me start by saying that this would be a marvellous player and that sadly, they don’t exist.

They’re either ASB’s or they’re Congregating Gits.

We’ve tried to deal with the ASB’s above, and that approach will have limited success, eventually I suspect you’d be forced to make two choices.

First you can split the world into shards, effectively de-MMO’ing it slightly by putting up invisible walls between players on one shard and players on another with no way of sharing their ships across the divide. It’s a valid solution but I find it “icky”. So the second solution is that you’ll try to split the “Game” server like an amoeba. You already have the migration worked out for the Star-System service so you’ll divide the universe into little chunks, maybe even into Sectors (8x8x8 light ingame years, how we divide the galaxy) and then have each Game server talk to each other to migrate players going across them. It’s reasonable. It’s just another level in the tier of servers.

Eventually I think you’d have to admit defeat and do both.

This still leaves the extroverts, the ones who can’t bare to be alone, and indeed absolutely must hang around in clumps of thousands of ships at a time.

You might think it’s great that so many player want to be hanging around but we must deal with reality here and the reality is that many players in one place mean that many players can all see each other, and get updates about each other, all of the time. This takes processing power, this takes bandwidth, and both of those cost money – that’s even ignoring the fact that beyond even quite low numbers you can’t actually send that much data to that many players at once.

I propose a number of things but lets highlight the most contentious because there is no technical fix for this, lets shut the shop…

Closed for Business:

The only way to deal with these players is to make them go elsewhere and that means manipulating the gameplay.

Not changing the gameplay, no I mean manipulating it, exploiting it, taking advantage of your exalted position as developers and cheating every grim little drop out of the fact that you’re writing it.

  • Weight the destinations of every job on the mission list towards empty or low-population star systems in the opposite direction of populated ones.
  • Make jobs going away from high load servers more profitable than ones going towards them.
  • Decrease the value of things that a highly loaded Star-System would want and increase their cost.
    • Reverse that for low player-population systems.
  • Make hyperspace jumps fallible, with the chance of being dumped at the edge of your current system increasing based on the population of the one you’re trying to reach.
    • Alternatively just make it unable to calculate the jump so you don’t inconvenience the player.
    • Make the likelihood of a good jump visible and known in advance of selecting a mission/hyperspace-target.
  • Spread content like missions, hidden ships, secrets and trade route across a wide volume in all 3 dimensions.
  • Keep a few uninhabited (by NPCs) star systems around populated one so that there will be escape routes out of the heavily oversubscribed “thickets” of Star systems.

Do all of the above, together, and try not to be too obvious by using some randomness.

Encourage the player away from these zones, then provide a lot of them spread out over a large volume and perhaps you’ll reduce the clumping.

Eventually you’ll get to a point where it’s inevitable that everyone will try to go somewhere to protest something you do, celebrate their favourite games 1st birthday, or because someone on Twitter told them too. With any luck you’ll have prevented them from trapping people in there by sheer attempts by everyone to get to one particular system but there is no way of avoiding that eventually, the architecture and nature of the game is just so amenable too it.

What you can do is minimise it happening as much as possible.

Conclusions as for Wimps! Here’s Mine:

The above is quite high level, I could go into how Paragon/Pioneer currently handle Star Systems and hyperspace (we delete the world and create a new one each jump) or how login server etc work but lets leave those for future posts.

What we have above is the rough outline, what I’d start with working towards if making a “Paragon MMO” were my goal. What servers I’d think about writing and the order in which I’d tackle the high level problems.

There’s a lot more to finesse and wrangle with than the above so if you want to hear about that stuff then leave a comment and a question and I’ll do another post in more depth about specific things.

Andy (aka, FluffyFreak)


Comments posted (2))

  1. You could think of it as star clusters and have a group of stars that can support n players. In general a star system is going to have a handful of objects in it that are going to have very well regulated behaviours unless you start introducing weapons that can carve up planets.

    what would be absolutely epic though would be if you simulated an Alcubierre Drive in game, and show the spacetime distortions from the POV of the ship…. go, do it I say!

  2. Inter-system travel, with a warp-like drive, isn’t possible in the engine though. Not without further work, hence the “hyperspace” workaround.

    The trouble is that the granularity needs to really go down even lower than system or cluster level. Ideally you’d like to get right down to parts of a planet in some cases because your limits aren’t based on what you simulate, but your networking infrastruture and what you can communicate about the world to your players.