8 things to avoid when building an MMOG server

A few years ago, I wrote an article for Develop magazine – “10 MMOs you don’t want to do”.

Here’s 8 things you really shouldn’t do but that might seem like a good idea if you’ve never made an MMOG before.

All these are examples of things that have been done on real MMO projects, usually MMORPGs.

  1. use off-the-shelf middleware from the enterprise industry. It’s designed for completely different usage-patterns and cannot cope with MMO style usage. Equally, initially distrust anything from traditional Big Iron companies.
  2. think that Grid Computing will somehow magically solve the problems. It won’t (c.f. previous point).
  3. aim to code the server in a scripting language. You *can* run some scripts embedded in the server, but not the server itself – but even that can screw you when you’re trying to run thousands of scripts at once
  4. assume that front-end load-balancing will solve any problems. It won’t, all it does is increase the efficiency of your cluster by a small amount. And it usually won’t provide you with failover, because most game designs will end up kicking you from your server if it dies, so the failover never gets used at that level.
  5. ignore performance testing until mid-way through the project. If performance tests at 10% of the way through production say it’s slow, that means you’re in deep trouble – it does NOT mean that “we’ll come back and optimize it later”. Optimizing netcode and server code is NOT like traditional single-threaded local-only optimization: many of the things you’re dealing with (like LANs, and TCP/IP stacks) run orders of magnitude too slowly, so your optimization comes from imaginative system-architecture, not from optimizing small chunks of code at a time.
  6. ignore billing concerns in your core game design. Non-free MMOG’s are entirely about billing, which means that you have to design it in, and build it in to the tech design from an early stage. Retrospectively adding billing hooks and billing information to existing server codebases is often about as easy and effective as retrospectively making your code secure. Just don’t go there.
  7. hire an academic who specializes in networking, especially a PhD student (this gets done quite often). All this means is that they’ve obsessed with a very narrow slice of the many many problems, and generally they won’t know WTF to do about the rest of the problems. That’s no better than just promoting a general programmer to become “the new Server specialist”
  8. innovate on both technology AND game design at the same time. Either do a traditional MMO so you can re-use all the existing common wisdom for design, and get to market (or at least a stable GDD) fast, and use the slack that buys you to focus on better tech, or use the most boring tech you can think of (instance lots; do lowest-hanging-fruit in your tech design) and innovate on the gameplay

I reserve the right to come back and edit this to make it ten once I’ve had more sleep and can remember two more :)…

12 thoughts on “8 things to avoid when building an MMOG server

  1. Richard Tew

    #3: I am curious why you doubt the awesome implementational power of the scripting language. Perhaps your experience was not with the right one? :-)

    #8: This reminds me of a typical forum post you might have read. “You have 30 programmers working on your product, I don’t understand why they aren’t all working on “.

    Any chance you can flesh out these two things with some detail as to why you recommend avoiding them?

  2. adam Post author

    I don’t know – has anyone implemented a serious physics engine in python yet? :) One that can run hundreds if not thousands of players on a single server? Because that kind of thing – CPU-intensive calcs – is a serious issue when building MMO servers these days, both for movement in the world (physics) and for interesting NPC’s (AI). Another one that can cause big problems is RAM usage, which scripting languages are frequently not-so-great at. 8 Gb seems a lot of RAM to play with, until you divide it by the number of connected clients, and start holding multiple simultaneous copies of world state to help you reduce latency artifacts.

    The worst problem I’ve seen though is to think you can build the server-engine core in script – e.g. things like efficient and/or powerful scheduling and I/O management. That’s something that most scripting languages suck horrendously at. For instance, building a stackless system inside a scripting language and trying to make it decently performant usually isn’t going to be much fun.

  3. Richard Tew

    With regard to #8, that part was clipped due to confusion about whether markup is allowed in blog comments.

    If I took this thing to heart and I was working on my first MMO, I would probably end up churning out something mediocre and the development team would be restrained from straying outside the mold and from developing things which might have made the game. To me, well reasoned innovation in both design and technology, especially where one facilitates the other, is where opportunity lies to make better games.

    I recently tried the new crop of MMOs and while I may have loved the marketing media I saw before release, playing them felt like more of the same with different window dressing. There are too many well known popular games out there which offer the same basic gameplay this thing might have you end up with, I think new MMO developers shouldn’t be encouraged to restrain their efforts and compete with something that isn’t as good as it could be, just to get it out there when doing so will have them standing alongside and dwarfed by the “rock star” efforts (i.e. WOW, Tabula Rasa, etc).

    With regard to #3. Right, physics engine. I guess it comes down to my taking the underlying parts like this for granted. As a part of the server, the physics engine can be something which has bindings which allow the C++ engine to beaver away efficiently as directed by the scripting language. Chances are you are going to use a third party engine anyway with pre-existing bindings. Throw in some database bindings which need to be written once and then bugfixed occasionally, some network bindings which should be the same and so forth.

    Given this base, the bulk of your server can be done in a scripting language and in my experience, it makes all the difference and I consider it to be one of the biggest advantages you can have.

  4. adam Post author

    #3 – I’m not sure I understand how scripting can be “the bulk of your server” if you’re also saying that physics, database, networking are all implemented in C++? And even those things are mostly a layer higher than a bunch of core services (like scheduling, low-level parts of networking, denormalized databases, etc) that you need to implement – usually from scratch – if you’re making a modern MMO.

    It sounds to me as if you’re thinking of “the server” as being “the gameplay logic on the server”? I’d say that is only a small part of the whole server.

    Seriously, if you know of any licensable MMO engines that work properly and do all the low-level (scheduling) and mid-level (physics) stuff as you describe, so that all you need to do is scripting, I’d love to hear from you. I have money to spend :), but haven’t yet found anything that could meet my requirements :( …

  5. adam Post author

    #8 – Sorry about the HTML, I’m not entirely happy with wordpress’s behaviour there, but haven’t had time to improve it.

    Runescape: innovated on technology, did pretty much zero innovation on gameplay – the only innovation I can think of was the simplified LMB/RMB user interface. It was a simplified clone of Ultima Online.

    Games that only innovated on gameplay are harder to point out off the top of my head, but everything I can think of that comes close (I can think of lots of examples where they tried to re-invent the wheel, so technically they were innovating compared to their own ability, but not compared to the market) has done poorly commercially because of immense production costs that were NEVER matched by corresponding increases in the sales figures.

    I guess my main question to you is: how MUCH does double innovation increase your profits?

    Guild Wars innovated heavily on both – but then, it was founded by the guys who invented Battle.net and developed Starcraft and Diablo. These rules are there to be broken by people who know better (all rules are made to be broken) but for the vast majority of studios producing MMO’s, they don’t have that kind of exceptional previous experience in-house.

    It might be a good idea to ask them whether they would double-innovate again. Double innovation costs a LOT of money, and money is the biggest single problem for most decent MMOs – if you don’t quite make enough profit, you get stuck in AO Purgatory, never quite able to afford to improve the game enough to increase your player numbers.

    Care to comment on Eve? :)

  6. Richard Tew

    Adam: It sounds to me as if you’re thinking of “the server” as being “the gameplay logic on the server”? I’d say that is only a small part of the whole server.

    You are right, my perspective is skewed with regard to the term server.

    But not just the gameplay logic. I think there is a sweet spot where all the resources which cannot be reasonably implemented in the scripting language, can be exposed to the scripting language so that they can be used flexibly.

    Take for example the networking framework in EVE. Written in Python above an exposed layer of a socket wrapping extension. Any feature I might write for EVE (gameplay or not) was comprised of T-SQL and Python, and this implemented everything in the server, client and authoring/introspection mechanisms.

    This is where my perspective is skewed I guess. The C++ resources (DB, networking, physics, etc) were written early and only touched and modified for maintenance.

    Adam: Seriously, if you know of any licensable MMO engines that work properly and do all the low-level (scheduling) and mid-level (physics) stuff as you describe, so that all you need to do is scripting, I’d love to hear from you. I have money to spend :), but haven’t yet found anything that could meet my requirements :(

    I know of none. And I would expect if I found something even close to suitable, to potentially need to replace the graphics engine, physics engine, networking bindings and/or database bindings depending on the needs of the game I was making.

  7. Richard Tew

    Adam: I guess my main question to you is: how MUCH does double innovation increase your profits?

    Did EVE double innovate? Let us say that for arguments sake it did. I think without that double innovation it wouldn’t be around today. Should someone make an Everquest in space when they can choose to take advantage of some of the opportunities (rather than avoiding them because of some questionably quantified effect on the games future profits) and make an EVE? I don’t think either choice is bad, but I don’t think a safe choice is necessarily the right one.

    I think the design ideas chosen for EVE inspired certain technological solutions. And those technological solutions inspired other design ideas and possibilities. Advantage was taken of the opportunities afforded by each, contributing to something much better than it could have been had innovation only been taken in one direction.

    If double innovation is considered to be at fault for the failure of a game, I would be inclined to cast a wider net and look for other problems and causes and assume that any problems with double innovation were fallout from larger problems.

  8. Chris Ochs

    I’ve used ‘scripting’ languages at some scale for games, although very little with virtual worlds outside of personal projects.

    My general approach is to use multiple languages. So called scripting languages have come a long way. I’ve done some pretty amazing stuff with jruby for instance, because I can leverage the jvm and choose when to use pure java or do it in ruby. And I’ll sometimes drop out to C using extensions.

    For network servers, messaging systems, etc.., you are better off using java or C. Anything that has to handle thousands of operations a second, is going to benefit from a faster language. But the key here is that with distributed computing and scaling horizontally, there are fewer places in your code that have to handle that kind of volume, so it opens the door to doing a lot more with more productive languages. The big failure of the game industry has been to stick with vertical scaling. You don’t need to handle thousands of requests per second on a single server, although I’d say around a thousand is a good number for an intel box.

    I haven’t worked that much on physics engines in particular, but I’ve spent many years on client/server architectures, and I work with live games where we are using distributed systems and message passing to a good degree of success. I’ve seen first hand the source code for several commercial engines, and how can I say it nicely, they are antiquated. They can only scale vertically, they still have an rdbms as a central bottleneck. Very little use of tools like redis, mongodb, zeromq, etc.. If you are still scaling vertically and your only option is to squeeze as much as you can out of a single server, you are doing it wrong.

    Another major lesson I’ve learned is that the performance of the language does not equal the performance of the system. The main reason now days to choose languages like C or Java is the tools, that’s where dynamic languages are still pretty far behind. Things like thread/heap dumps, good memory management, profiling tools, that’s what you miss the most.

  9. Ted

    Eve Online MMO uses Stackless Python as server. It has come a long way brother, you should update your listing here..

  10. adam Post author

    From what I’ve heard and can see, it’s still a major pain point. They designed around it. That’s fine, but … it’s something to avoid, generally speaking.

  11. 'No Bugs' Hare

    I realize that I’m beating a dead horse now, but (having a bit of experience in the field myself) I would say that all of the points still stand, with a few minor clarifications:

    1. Hardware from Big Iron guys is generally good, you don’t need to distrust their hardware. As for the software – you’re perfectly right.

    4. You said: “assume that front-end load-balancing will solve any problems. It won’t, all it does is increase the efficiency of your cluster by a small amount.” I’d say that given a good client app of your own, even small increase in cluster efficiency isn’t going to happen.

    Also, to those interested: I’m writing a book on MMOG Development&Deployment right now, with “beta” chapters published as they’re written; I’m planning to cover these (and many other) items in detail.

Leave a Reply

Your email address will not be published. Required fields are marked *