Entity Systems are the Future of MMOs Part 4

Massively Multiplayer Entity Systems: Introduction

So, what’s the connection between ES and MMO, that I’ve so tantalisingly been dangling in the title of the last three posts? (start here if you haven’t read them yet).

The short answer is: it’s all about data. And data is a lot harder than most people think, whenever you have to deal with an entire system (like an MMO) instead of just one consumer of a system (like the game-client, which is the only part of an MMO that traditional games have).

Obviously, we need to look at it in a lot more detail than that. First, some background…

Massively Multiplayer Game Development 101

There’s a few key things you need to be aware of in MMO development. Many professional game developers know some or all of these already, but seeing as I’ve even met MMO developers who didn’t know them all, and much of what I’m going to say later on won’t make sense without it, I’m going to do a quick recap. Bear with me; I’ll have to do some gross generalization to keep this short (so it won’t be 100% true or accurate).

1. The vast majority of the cost of MMO development is content

Content is:
– quests (missions, storylines, scripted events)
– 3D areas (meshes, textures and logic for: zones, dungeons, towns, landscapes)
– loot (item graphics, drop rates, item stats, item abilities)

Content is NOT:
– Fancy 3D graphics
– Physics engines
– AI
– Core game rules

…which tend to make up the bulk of non-MMO games.

As Raph Koster (amongst others, but IIRC Raph was one of the first to come up with a clear concise description) pointed out close to 10 years ago, the rate at which players consume content vastly excedes the rate at which developers generate it – you have perhaps 50 people working purely on content generation on a modern MMO, and you have perhaps 500,000 people consuming it. The problem is, those 500,000 use Thottbot, so they *share* their discoveries, and consume at a rate proportional to the square of their number, whereas it’s generally produced at a rate more directly proportional to the number of developers.

There are many ways to work around this issue, but most of them end up producing or managing vast amounts of content (they just get more cunning in exactly how that content is generated).

2. The vast majority of the development of an MMO takes place AFTER launch

The ten-year-old MMO’s have been releasing expansions and content updates every 6 to 18 months; modern MMO’s release new content every 3 to 12 months. “New content” in this context generally means entirely new 3D areas with entirely new quests and entirely new items, not to mention entirely new special abilities and new plotlines. i.e. complete “miniature MMOs”.
The only bits you don’t need to keep rewriting are the technology, although nearly all MMOs have done minor updates throughout their lifetime, usually when a new expansion pack is released as a retail boxed product (if you buy and install the expansion, not only do you get the bonus content, but all the “old” content becomes higher resolution / more pretty / added sparkly bits). Although … I long wished there was more updating of tech, and three major MMOs are currently doing huge updates: Ultima Online (1997) recently massively renovated their 10-year old graphics as Kingdom Reborn, Eve Online (2001) just completed their “shininess and detailed spaceships with added humanoids” update known as Trinity, and Anarchy Online (2001) recently previewed massive sweeping changes to their very poor detail and draw distance client (they say “the current engine was designed before there really were GPUs to utilize” but that is blatantly false as the GeForce had been on sale for almost 2 years when they launched, and the TNT,Voodoo, et al for years before that. Yes, I was bitter that when it launched the most recent hardware it supported was 5 years old :)). All represent major changes to the look and feel of the games – it’s hard to overestimate the visual impact of these. Runescape did their mega update (“Runescape 2”) a couple of years ago.

However … all are bringing games that were NOT cutting edge at launch up to a standard that is still well short of what is cutting edge now. This is important to understand: graphical quality is (according to what the successful MMO’s deploy) definitely not the main selling point for these games, unlike almost all other successful games you can possibly buy.

So … what is that “development” that takes place after launch? It’s content, not engine. And even when it’s engine (better graphics, better physics) it’s always a minimal improvement by the standards of standalone PC and console titles of the time.

For reference, many MMO’s after launch end up with LARGER development teams than they had in the 3-5 years of development leading up to the launch. All that extra content requires a lot of people (not to mention keeping all the old content working, updating it where it conflicts with new content, etc).

On top of all that, though, there is the issue of customer support. When you launch a traditional game, that’s it. Done. Over the last 15 years there’s been a trend towards launching occasional “patches” and even minor content updates – but mostly bugfixes – which, incidentally, I suspect has been largely driven by publishers learning from the MMOs: releasing a content patch for an old game is a good way to get extra marketing / publicity and increase some sales. Bethesda’s TES4/Oblivion even went as far as to charge for the patches.

MMO’s require a lot more work than that: since MMO’s are all monetized off continuous play (i.e. the more each player plays, the more money the publisher makes), either through monthly subscriptions or through in-game commerce, it is imperative to keep all the players playing as long as possible. If a player plays your MMO for 12 months instead of 6, you’ll make twice as much profit; if they play your console title for 12 months instead of 6, you’ll make no more money and will probably cannibalize your sales of the sequel.

So. Total lifetime development cost is a lot more important, financially, to an MMO than the pre-launch development cost.

3. The way the core logic works now is not how it will work forever

Sooner or later, you nerf.

Usually because you(r players) discover a disproportionately powerful game tactic in PvP that you didn’t think of which is unbalancing and ruining the game, and so you “have” to fix it.

Sooner or later, you choose to break every absolute rule of your game logic, because a new expansion wants to slightly redefine and “freshen” the core game.

And that means you have to be able to break every rule you ever hardcoded into the game-logic. Worse, it means you have to be able to handle the fallout (all the bugs that the change introduces, plus all the invisible bugs that were originally there but had no noticeable effect and now suddenly have visible effect, the new security holes, etc).

Since the ongoing development of the game (post launch) is greater than the initial development in scope and cost, over the lifetime of the game it is more important to make it easy to change the core logic – and to react to side effects of that change – than to be able to make the core logic easily in the first place.

4. Even a moderately successful MMO generates gigabytes of data every 24 hours

The progress of every player of the MMO has to be uniquely independently tracked, so that the game can remember for each player which quests they’ve completed, which equipment they have, etc. With 100,000 players, that data racks up quickly. A lot of it can be thrown away every day (for instance, you don’t generally need the history of how many hitpoints you had at the end of every day), but a lot of it has to be kept forever (whether you completed a given quest, and for some quests how exactly you completed it).

However, the trend at the moment is towards keeping ALL this data, in the form of historical records of your personal game experience – e.g. the automatic “blog” that is created whilst you play Vanguard, which e.g. updates every time you level up, or kill a monster, or die, etc.

This latter kind of data alone is generated at the rate of around 30Gb every 24 hours.

And any of that data that gets used in-game (e.g. even if just referenced in a conversation with an NPC) has to be programmatically accessible.

Ultimately, MMOs have to store, retrieve, and update vast databases. There is a huge decades-old industry that specializes in providing software to manage databases of this size and larger, but MMO developers don’t like SQL and Relational programming. This is a real problem – relational databases can handle the load, but relational programming is fundamentally incompatible with Object-Oriented programming. The net effect is that programmers who have to write their data to disk find it boring, difficult, and irritating to do – and that leads to mistakes and greatly increased bugs, as well as reduced functionality (because no-one wants to write or add to the code to add the new features more than they absolutely have to).

MMO Development Priorities

The net effect of these aspects of MMO development is that the long term profitability of an MMO can be greatly affected by a bunch of simple issues:

  • How much effort is required to change the data used to describe an in-game item
  • How much effort is required to add (or remove) new (old) types of in-game item
  • How re-usable is the content
  • How easy it is in practice (i.e. how much coding is needed) to re-use re-usable content
  • How much specialist knowledge of the game code, overall game design, and details of the game systems is needed to modify the content and logic (can new team members be effective as the team who originally wrote the game pre-launch?)
  • How exportable is the data generated by playing the game (progress, achievements, player-history, etc)
  • How analysable is the game-data generated by playing (to make decisions about what to change, and check that game improvements have improed things)
  • How modifiable the core-content is
  • How easy it is to change individual rules and check the side-effects of the changes
  • How many of all the above changes can be done WITHOUT requiring a programmer (can designers change code themselves? can artists?)
  • How much can third-party specialist systems be utilised for server-side service provision

Leading to a few rules of thumb that are often adopted by commercial MMOs:

  1. Use commercial databases for all persistent data-storage
  2. Use Relational Databases and allow arbitrary general data requests
  3. Implement as much as possible of the game-logic as raw data rather than as compiled data (and preferably as data rather than source code)
  4. Use standard but simple to learn scripting languages where data isn’t expressive enough
  5. Do lots and lots of testing during development
  6. Do even more testing post-launch EVERY SINGLE TIME you consider changing ANYTHING on the live servers

Some of those aren’t very effective (too vague), others are horrifically expensive (it takes a lot of people to “test EVERYTHING” every time you make a change to a live game). None of them are great for performance in and of themselves (although they can be made highly performant, that requires extra work), and none of them are “programmer friendly” – in fact, they’re all deliberately programmer UNfriendly, being in there for the benefit of non-programmers.

NB: this is in no way a comprehensive or exhaustive list; I’m trying to give merely a flavour of the thing here for people who aren’t familiar with typical MMO dev processes. I don’t know any real MMO that faces only those challenges and uses only those rules of thumb, but they are indicative of what is used, and very very roughly explain why those things are used.

And so on to Entity Systems, and how they can affect the typical MMO development process by helping with problems or making the current solutions more programmer-friendly… (which is going to be Part 5. This is getting long enough already).

Did this post help you?

Support me on Patreon, writing about Entity Systems and sharing tech demos and code examples

Next post

This entry was posted in computer games, databases, entity systems, games design, massively multiplayer, programming, system architecture on by .

32 thoughts on “Entity Systems are the Future of MMOs Part 4

  1. Pingback: T=Machine » Entity Systems are the future of MMOG development - Part 1

  2. Juha Lindfors

    “1. Use *commercial* databases for all persistent data-storage”

    So what does, from your point of view, disqualify open source databases from backing an MMO? Lack of confidence in their high availability features?

    “3. Implement as much as possible of the game-logic as raw data rather than as compiled data”

    How about rule engines, do you see those being currently used to externalize the game logic and to improve the accessibility of game-logic and game rules to non-programmers?

  3. adam Post author

    @ Juha – Doh! By “commercial” I meant “non-proprietary”. So far, I’ve found the best open-source databases are easily up to the task, and even better than some of the commercial ones (e.g. Microsoft).

    Rule engines are still orders of magnitude too slow to run a typical MMOG game server. They’re great, but not suited to any real-time computation yet.

  4. Random reader

    Just wanted to say that I really enjoy this series of post, not that I work on a MMOG or anything, and I hope that you’ll find time soon to write the next installment. Cheers!

  5. Ken

    I’m going to second Random reader here–really looking forward to you wrapping this all together! Thanks!

  6. Mark Bernard

    Great set of articles. I am anxiously looking forward to the next installment. Hope you get it up soon.

    Thanks.

  7. John

    Perhaps I’m a little late coming to this series of articles, however I’ve just been engrossed in the first 4 parts, now I’m looking forward to the 5th part :o)

  8. aaulia

    Wow this series is helping me so much already, I always want to do entity system since I read Mick West’s article. But his article is to basic, and only touch the surface. Hence I’ve been scratching my head trying to fit his concept to OOP, which through this series I found out, that I’ve been doing it wrong >.<. Thx, waiting for the next installment patiently :)

  9. Logan

    Extremely interesting articles. Looking at the dates however… I guess we can’t expect to see part 5? As far as I can tell, it’s not up, but it’s been over a year now.

  10. adam Post author

    Sorry, guys … just very busy with my day-job right now :).

    In the meantime, I’d recommend having a look at Chris Lunford’s recent post which goes off in the same kind of direction I was going in next – http://chrislunsford.com/?p=7

    NB: the main reason I haven’t posted part 5 yet is that I don’t want to post something empty and vacuous, and I haven’t had time to write something more meaningful.

  11. ConcernedFan

    “NB: the main reason I haven’t posted part 5 yet is that I don’t want to post something empty and vacuous, and I haven’t had time to write something more meaningful.”

    Bah whatever you write will be good! Please sir we want some more!

  12. Arly

    I have a simple question to make my mind clear of this amazing approach which has just been referred by a friend. So does this mean we have to think the “Query” way? like “select var A, B, C from Component D, E, F”? correct me if I’m wrong.

  13. adam Post author

    @Arly

    You don’t absolutely “have to” think in queries – you could get most of the benefit without that.

    And I know that on *some* platforms there isn’t a cheap-enough query engine available – so I’m loathe to insist upon it.

    However … in general … if you use queries as the basis of two things (which are independent things!):
    1. “what components are current for entity X”? … or … “what entities have component Y?”
    2. “what are the data-values for component Y on entity X?”

    …then you get a lot simpler code in many places, and a lot simpler debugging. If there’s any way you can afford to use queries for either of those – go for it.

    You don’t need the expressive power of SQL (although … where you can afford it, that provides some truly awesome dynamic / runtime editability), you just need something that’s evaluated each time it’s read.

    A lot of people seem to use publish/subscribe, and/or an event-driven-method-calls approach, which *can be* a half-way house, depending upon what you do with it. I’ve had a lot of problems with event-driven coding for normal commercial projects, so I tend to avoid it.

  14. Pingback: T=Machine » Entity Systems are the future of MMOG development – part 3

  15. Masher

    This “entity” paradigm sounds suspiciously like the decorator design software design pattern. What exactly do you see this accomplishing that decoration can’t do in a more general, elegant manner?

  16. adam Post author

    I’m afraid I fail to see any similarity at all. What parts do you think are the same?

  17. Masher

    The similarities are you’re unfolding an object hierarchy by compositing pseudo-objects from collections of simpler objects (though you refer to them as entities). This is probably closer to the compositing design pattern than decoration in actual practice, but unless I’m missing something (and I very well may be) doesn’t otherwise appear to be anything new.

  18. Masher

    After reading my reply, I realize it sounds rather pompous and tendentious. My apologies … I don’t mean to be overly critical. Perhaps I’m wrong, but I fail to see how these “entity systems” differ in essentials from a composite or a wrapped object.

  19. adam Post author

    @Masher

    With decorator etc:

    Where’s your definition of which components exist in an entity?

    How much runtime is required to fetch an entity into memory?

    Where do you lookup (and perform type-checking) the components within a particular uniquely referenced entity?

  20. Masher

    Adam,

    The OOP Composite Design pattern is typically implemented by composing entities as a collection class of pointers to lower-level instantiated objects. This allows you the benefits of multiple inheritance without a deep class structure. Object instances can be composed on the fly by assembling the functionality of smaller units.

    Now, from both a paradigmatic and an implementational standpoint, doesn’t that sound like this entitiy system?

  21. adam Post author

    (NB: I recently came out of hospital, I’m still suffering after-effects of morphine and anaesthetic, so my responses might not be entirely rational)

    *gut feeling*: this sounds very similar to the approach I used on my very first ES, back when I didn’t really understand them, and I was implementing everything in a strong OOP language (IIRC early java). It was “close, but not quite there”, and it did just well enough to show there was real potential in the ideas, but it also made life so difficult that it proved not worth using in practice.

    …there were lots of small problems in the implementation that became huge problems once you built up a large system. They were mostly architectural and/or philosophical, so that they were extremely difficult to change without throwing away vast amounts of code.

    So. When you get into implementation details, what you describe starts to sound more similar, but the devil is still going to be in the details. If you skip over the details (e.g. the questions I asked in my last comment) then it’s easy to convince yourself that you’re implementing an ES when in truth you’re just doing traditional OOP in a very slightly modified way.

    For whatever reason one chooses to use an ES, one of the fundamental aims is always to change the paradigm that the programmers are using to design and write their application; I don’t know how similarly those patterns affect the thought processes of the programmers using them.

    NB: when I did that first ES, I knew some of the design patterns, but only about 1/2 of the core ones – sadly, my unviersity still thought they were “too modern” to teach yet. It’s quite possible that I used these patterns at the time, following the formal descriptions, without memorising them by name.

  22. adam Post author

    PS: when I did my first ES, I didn’t yet understand ES’s in any detail … so it’s *also* quite possible that if I ran into the same problems today, I might in fact be able to spot simpler fixes that would make the whole thing work.

    Back then, I was fumbling in the dark, working it out as I went. These days, I know the shape and form of what I’m aiming for, and it’s much easier to convert something to fit that mould.

  23. Ripley

    Is part 5 still in the works? I’m very interested in the conclusion of this set of articles – Very well written and certainly eye-opening to me, a database guy who is interested in game development.

  24. Alexander

    You got me going with the first four parts, I would really like to see the next parts O:

    Pretty please?

  25. Pingback: Drafting up an Engine Framework « Programmatical Musings

  26. Jacob

    Hey, I was wondering…

    have you ever tried mixins or other oop features that are available in other languages? For example, I find that CLOS(common lisp object system) makes entity systems a natural development and it is still oop. In fact, I often wonder if these patterns that are offered are just work arounds to weak abstractions. OOP in Java is not the same as OOP in newspeak/smalltalk after all.

  27. adam Post author

    I dabbled with them a long time ago, but found them very hard to develop with in practice – trivial to write code, but impossible to understand what you’d written. It felt like to make them seamless and user-friendly for coders you needed special editors that are fundamentally different from the text-editors we use for writing code today.

    Today’s IDEs are more than capable of providing that service, but … for whatever reason, most don’t (or: didn’t). We are still a bit stuck in the mindset of “a program is a collection of simple text files, where each file tells you waht it does”.

    In my limited experience, IIRC: I found mixins worked great for the simple examples pulled out over and over again (e.g. logging, self-testing – meta activities around a class), but were poor for humans when used on congruent code that was part of the main application. Simply too hard to work with on significant code bases.

    Re: CLOS – Lisp is already functional; IMHO that removes most of the reasons we need/want an entity system in the first place.

Leave a Reply

Your email address will not be published. Required fields are marked *