But they should.

So … here are the evaluations (from the audience) for our panel session at AGDC 09.

Judge for yourself whether you want to attend any future sessions featuring us again (Adam Martin, Bill Dalton, Rick Lambright, Joe Ludwig, Marty Poulin).

Head Count: 74; Evaluations: 32 (43% response rate)

• Overall rating of the presentation – 88% (AVG: 86%)
• How relevant was the topic to you? – 86% (AVG: 84%)
• How well did this class meet your expectations? – 94% (AVG: 84%)
• Would you recommend this session to a colleague? – 90% (AVG: 84%)
• Evaluate the speakers’ ability to communicate – 94% (AVG: 86%)
• If there were visual aids (slides) how were they? – 74% (AVG: 60%)

All of those are above average, and I’m glad that a particularly high number would recommend the session to their colleagues.

It seems that we did particularly well on fulfilling the remit (very high number for “met expectations”), and that our speakers had an awesome ability to communicate (almost 10% higher than average for the other speakers at the conference).

Audience Comments

1. The most entertaining session I attended, but didn’t sacrifice information value.
2. Interesting format, I wouldn’t mind seeing more of this, but it is time consuming
3. Good stuff
4. Slow, confused start lost valuable time for Q&A
5. Should have done middleware
6. Only 3 topics covered. Expected others

Comment 4 – yeah, something I’m unhappy about too, (it wasn’t our fault, it was the people running the conference), but there was nothing for it but to grin and carry on. Someone screwed-up the radio microphones, and we lost a lot of time at the start waiting for them to fix it. There was nothing we could do – they had connected the mics from a different room to *our* speakers. We didn’t find out until the person in the other room started talking, and it all came out through our speakers :(.

Comment 6 – we covered 4 topics (oops, audience can’t count :P). We all wanted to do more, but at GDC conferences, the organizers only give us 1-hour slots. With 4 speakers + moderator, I think that was pretty good, especially considering the time we lost at the start.

Perhaps someone will clone this format for a future conference (seems a good idea), and try to get a 2-hour slot for it?

Or, in other words, there’s now a page where I can effortlessly post all my various bookmarked links to do with MMO development – and add my own commentary to each link – which you can’t ordinarily do. Which is why it’s taken me some time to get around to it (previous efforts to do this without customizing WordPress, or using plugins only, failed).

The (practically empty) page in all it’s (non-)glory can be found here:

http://t-machine.org/index.php/category/mmog-dev/

Over the coming weeks, I’ll be posting much more stuff to it. I hope.

So, what’s the connection between ES and MMO, that I’ve so tantalisingly been dangling in the title of the last three posts? (start here if you haven’t read them yet).

The short answer is: it’s all about data. And data is a lot harder than most people think, whenever you have to deal with an entire system (like an MMO) instead of just one consumer of a system (like the game-client, which is the only part of an MMO that traditional games have).

Obviously, we need to look at it in a lot more detail than that. First, some background…

Massively Multiplayer Game Development 101

There’s a few key things you need to be aware of in MMO development. Many professional game developers know some or all of these already, but seeing as I’ve even met MMO developers who didn’t know them all, and much of what I’m going to say later on won’t make sense without it, I’m going to do a quick recap. Bear with me; I’ll have to do some gross generalization to keep this short (so it won’t be 100% true or accurate).

1. The vast majority of the cost of MMO development is content

Content is:
– quests (missions, storylines, scripted events)
– 3D areas (meshes, textures and logic for: zones, dungeons, towns, landscapes)
– loot (item graphics, drop rates, item stats, item abilities)

Content is NOT:
– Fancy 3D graphics
– Physics engines
– AI
– Core game rules

…which tend to make up the bulk of non-MMO games.

As Raph Koster (amongst others, but IIRC Raph was one of the first to come up with a clear concise description) pointed out close to 10 years ago, the rate at which players consume content vastly excedes the rate at which developers generate it – you have perhaps 50 people working purely on content generation on a modern MMO, and you have perhaps 500,000 people consuming it. The problem is, those 500,000 use Thottbot, so they *share* their discoveries, and consume at a rate proportional to the square of their number, whereas it’s generally produced at a rate more directly proportional to the number of developers.

There are many ways to work around this issue, but most of them end up producing or managing vast amounts of content (they just get more cunning in exactly how that content is generated).

2. The vast majority of the development of an MMO takes place AFTER launch

The ten-year-old MMO’s have been releasing expansions and content updates every 6 to 18 months; modern MMO’s release new content every 3 to 12 months. “New content” in this context generally means entirely new 3D areas with entirely new quests and entirely new items, not to mention entirely new special abilities and new plotlines. i.e. complete “miniature MMOs”.
The only bits you don’t need to keep rewriting are the technology, although nearly all MMOs have done minor updates throughout their lifetime, usually when a new expansion pack is released as a retail boxed product (if you buy and install the expansion, not only do you get the bonus content, but all the “old” content becomes higher resolution / more pretty / added sparkly bits). Although … I long wished there was more updating of tech, and three major MMOs are currently doing huge updates: Ultima Online (1997) recently massively renovated their 10-year old graphics as Kingdom Reborn, Eve Online (2001) just completed their “shininess and detailed spaceships with added humanoids” update known as Trinity, and Anarchy Online (2001) recently previewed massive sweeping changes to their very poor detail and draw distance client (they say “the current engine was designed before there really were GPUs to utilize” but that is blatantly false as the GeForce had been on sale for almost 2 years when they launched, and the TNT,Voodoo, et al for years before that. Yes, I was bitter that when it launched the most recent hardware it supported was 5 years old :)). All represent major changes to the look and feel of the games – it’s hard to overestimate the visual impact of these. Runescape did their mega update (“Runescape 2″) a couple of years ago.

However … all are bringing games that were NOT cutting edge at launch up to a standard that is still well short of what is cutting edge now. This is important to understand: graphical quality is (according to what the successful MMO’s deploy) definitely not the main selling point for these games, unlike almost all other successful games you can possibly buy.

So … what is that “development” that takes place after launch? It’s content, not engine. And even when it’s engine (better graphics, better physics) it’s always a minimal improvement by the standards of standalone PC and console titles of the time.

For reference, many MMO’s after launch end up with LARGER development teams than they had in the 3-5 years of development leading up to the launch. All that extra content requires a lot of people (not to mention keeping all the old content working, updating it where it conflicts with new content, etc).

On top of all that, though, there is the issue of customer support. When you launch a traditional game, that’s it. Done. Over the last 15 years there’s been a trend towards launching occasional “patches” and even minor content updates – but mostly bugfixes – which, incidentally, I suspect has been largely driven by publishers learning from the MMOs: releasing a content patch for an old game is a good way to get extra marketing / publicity and increase some sales. Bethesda’s TES4/Oblivion even went as far as to charge for the patches.

MMO’s require a lot more work than that: since MMO’s are all monetized off continuous play (i.e. the more each player plays, the more money the publisher makes), either through monthly subscriptions or through in-game commerce, it is imperative to keep all the players playing as long as possible. If a player plays your MMO for 12 months instead of 6, you’ll make twice as much profit; if they play your console title for 12 months instead of 6, you’ll make no more money and will probably cannibalize your sales of the sequel.

So. Total lifetime development cost is a lot more important, financially, to an MMO than the pre-launch development cost.

3. The way the core logic works now is not how it will work forever

Sooner or later, you nerf.

Usually because you(r players) discover a disproportionately powerful game tactic in PvP that you didn’t think of which is unbalancing and ruining the game, and so you “have” to fix it.

Sooner or later, you choose to break every absolute rule of your game logic, because a new expansion wants to slightly redefine and “freshen” the core game.

And that means you have to be able to break every rule you ever hardcoded into the game-logic. Worse, it means you have to be able to handle the fallout (all the bugs that the change introduces, plus all the invisible bugs that were originally there but had no noticeable effect and now suddenly have visible effect, the new security holes, etc).

Since the ongoing development of the game (post launch) is greater than the initial development in scope and cost, over the lifetime of the game it is more important to make it easy to change the core logic – and to react to side effects of that change – than to be able to make the core logic easily in the first place.

4. Even a moderately successful MMO generates gigabytes of data every 24 hours

The progress of every player of the MMO has to be uniquely independently tracked, so that the game can remember for each player which quests they’ve completed, which equipment they have, etc. With 100,000 players, that data racks up quickly. A lot of it can be thrown away every day (for instance, you don’t generally need the history of how many hitpoints you had at the end of every day), but a lot of it has to be kept forever (whether you completed a given quest, and for some quests how exactly you completed it).

However, the trend at the moment is towards keeping ALL this data, in the form of historical records of your personal game experience – e.g. the automatic “blog” that is created whilst you play Vanguard, which e.g. updates every time you level up, or kill a monster, or die, etc.

This latter kind of data alone is generated at the rate of around 30Gb every 24 hours.

And any of that data that gets used in-game (e.g. even if just referenced in a conversation with an NPC) has to be programmatically accessible.

Ultimately, MMOs have to store, retrieve, and update vast databases. There is a huge decades-old industry that specializes in providing software to manage databases of this size and larger, but MMO developers don’t like SQL and Relational programming. This is a real problem – relational databases can handle the load, but relational programming is fundamentally incompatible with Object-Oriented programming. The net effect is that programmers who have to write their data to disk find it boring, difficult, and irritating to do – and that leads to mistakes and greatly increased bugs, as well as reduced functionality (because no-one wants to write or add to the code to add the new features more than they absolutely have to).

MMO Development Priorities

The net effect of these aspects of MMO development is that the long term profitability of an MMO can be greatly affected by a bunch of simple issues:

• How much effort is required to change the data used to describe an in-game item
• How much effort is required to add (or remove) new (old) types of in-game item
• How re-usable is the content
• How easy it is in practice (i.e. how much coding is needed) to re-use re-usable content
• How much specialist knowledge of the game code, overall game design, and details of the game systems is needed to modify the content and logic (can new team members be effective as the team who originally wrote the game pre-launch?)
• How exportable is the data generated by playing the game (progress, achievements, player-history, etc)
• How analysable is the game-data generated by playing (to make decisions about what to change, and check that game improvements have improed things)
• How modifiable the core-content is
• How easy it is to change individual rules and check the side-effects of the changes
• How many of all the above changes can be done WITHOUT requiring a programmer (can designers change code themselves? can artists?)
• How much can third-party specialist systems be utilised for server-side service provision

Leading to a few rules of thumb that are often adopted by commercial MMOs:

1. Use commercial databases for all persistent data-storage
2. Use Relational Databases and allow arbitrary general data requests
3. Implement as much as possible of the game-logic as raw data rather than as compiled data (and preferably as data rather than source code)
4. Use standard but simple to learn scripting languages where data isn’t expressive enough
5. Do lots and lots of testing during development
6. Do even more testing post-launch EVERY SINGLE TIME you consider changing ANYTHING on the live servers

Some of those aren’t very effective (too vague), others are horrifically expensive (it takes a lot of people to “test EVERYTHING” every time you make a change to a live game). None of them are great for performance in and of themselves (although they can be made highly performant, that requires extra work), and none of them are “programmer friendly” – in fact, they’re all deliberately programmer UNfriendly, being in there for the benefit of non-programmers.

NB: this is in no way a comprehensive or exhaustive list; I’m trying to give merely a flavour of the thing here for people who aren’t familiar with typical MMO dev processes. I don’t know any real MMO that faces only those challenges and uses only those rules of thumb, but they are indicative of what is used, and very very roughly explain why those things are used.

And so on to Entity Systems, and how they can affect the typical MMO development process by helping with problems or making the current solutions more programmer-friendly… (which is going to be Part 5. This is getting long enough already).

NEXT: http://t-machine.org/index.php/2008/02/27/central-tech-essential-strategy-or-complete-waste-of-time/ I work at a company where managing and directing a subset of the global core-tech is part of my day-to-day job, and I’ve been trying to find good, positive ways forwards. After a year in this role, I’m looking back and wondering whether it’s working for us, and where it’s been good and where it’s been bad. I don’t know what the perfect solutions are, so I’m just going to document the issues I’ve been wrestling with, and my current thoughts on what causes them, and my current ideas on what can be done to fix or avoid them. Personally, although I’m confident there is at least some role for some form of central technology in many games companies, I’m not sure yet what that role is.

Most (not all!) of the experiences I’ve personally had in games companies have been failures, some narrowly (managing to achieve good things, but failing at what they were originally setup for), some spectacularly (failing to provide anything good at all, and even dragging down one or more game-projects with them). Some flailed about and killed projects in their wailing death-throes, others gorged on the corpses of the projects they’d killed and rose again from the bones of what was left, like some undead creature (which is a worrying precedent: “how do you kill that which has no life?”). It’s fair to say my experiences have been mostly bad ones :) … BUT: until it was called “central tech” or “core tech” the *same kind of initiatives* (re-using code, centralising decision-making, making powerful proprietary libraries) seemed to work well, generally, in teams that I was in or working with. The failure cases seem (at least in my small amount of experience) to be a relatively modern problem.

What’s the point?

Speaking to people throughout the company where I currently work, and working with some of our pre-existing core-tech teams, I’ve found myself more and more asking a simple question: “why do we have Core Tech?” The answers have been illuminating…

“to write everything-including-the-kitchen sink, then on each game project only use the bits you need”
“recycle code from an existing project”
“I don’t know”
“write code that can be used and re-used in multiple projects”
“share the really cool and difficult stuff that one team did with other teams so they can benefit from its brilliance”

This has made me realise / remember a couple of things:
– non-programmers, and many inexperienced programmers, grossly underestimate the ratio in development time/cost between “making code that solves one problem” and “making a generic solution to a whole class of problems of which the current problem is just one example”. The first explanation described above is usually prohibitively expensive by *a large factor, maybe even an order of magnitude*, but many people seem unaware of this
– CT is sufficiently ill-defined and *poorly marketed internally* at most games companies that many internal dev staff genuinely don’t know why it exists or what purpose it’s supposed to be solving, let alone have the opportunity to rate the success or failure at that purpose
– many people notice that game development often involves re-solving or re-performing similar tasks, and wish for a perfect world in which you only ever have to do one “kind” of solution once, and then you get to re-use it forever after. That is largely true in mainstream IT, but sadly this seems to be far from the truth with games development. Personally, I suspect that the day the technology becomes derivative is the same day you realise your game itself is derivative, and has lost most of it’s fun.

I also wonder: if we don’t have CT, are you saying we wouldn’t re-use code? As a programmer, do you avoid finding code from old teams and finding ways to use it in your current project? Isn’t this a *basic part* of your job, part of what we pay you to do?

Is CT something that gives game-programmers an excuse to be less diligent than they would otherwise willingly be, thereby setting the scene for them to later complain that it’s “not as good as if I’d written it myself” (which is one of the cruelest things I’ve seen game-teams do to their CT teams, even if it’s true).

Other side of the coin: why is central tech hard?

Project teams steal the best hires
– Project teams, especially Producers and Leads, have a hugely difficult problem (ship a game on time, and make it “fun”, so it sells well) and a huge amount of personal buy-in (if it fails, they have no second chance: their sole responsibility is that one project), and a huge amount of measurability of success (how many sales does the game make?). They have a vested interest in finding the best of the best people, and they have – ultimately – large budgets to recruit them with. If someone really good at developing code for games is in central tech, they usually get stolen for a game-team sooner or later. Central tech departments have too little visibility (how do you measure the brilliance or crapness of your coretech? there’s no sales figures!), no concrete deadlines (they randomly invent them, or borrow a game-project’s deadlines, but they really don’t sink-or-swim on that project-deadline like the game-team itself does!), and usually restricted budgets (if not, well, that’s when the unlimited-spending-without-reason problem kicks in…). They also are rarely as cool as working on a “real game”: most people joined the industry to Make Games, not to Write Random Code For Someone-else’s Game. CT finds it hard to retain the best people…

Writing code a second time is generally a lot cheaper than modularising the original version
– …I’m hoping this is self-obvious. But sadly it means that it is ALWAYS easier for any individual game-team to re-write *from scratch* the code they wrote for their last game than it is for CT to make the generic version that they’re usually asked for. Game-teams then get pissed that the CT team seems incompetent compared to them (they notice that the time they have to wait for CT to do something is MORE than it would take them to do it themselves).

CT is always setup as an “ongoing” task: there is no defined beginning or end to what they do. Games companies (developers and publishers) are very specifically geared towards “project-based” tasks.
– From the funding, to the project management, to the staff rewards schemes, everything about the games-industry is setup for project-by-project operation. We know (sort-of) how to make project-based development work (reasonably) well. We don’t really have ANY experience in non-project work – certainly anyone who’s only ever worked in the games industry (who are strongly self-selected by current standard recruitment procedures) will have no experience of non-project development. Is it any surprise we often screw it up? Moreover, is it any surprise that we have such difficulty making the non-project CT team be respected and understood by the purely project-based game-teams?

Ideas

I’ve been kicking around some ideas on new directions to pursue. These may not be that great, but at the moment I don’t have to make any decisions, I just have to provide options, so I’m interested in any suggestions or alternatives…

Think of CT as a human resource, not a code repository.

This is one thing I’ve been trying: what *I* needed most urgently out of CT was the *expertise* component, and actually I didn’t really care about the code re-use etc. I knew I would, but I had a more pressing problem of two projects that lacked the necessary expertise in Network Programming to make an MMO. So … I created a tiny team dedicated to doing the network code for any game, and explicitly NOT to writing reusable network technology (NB: many years ago I ran a failed network middleware company, so I had a healthy apprecaition of how difficult and time-consuming it is to write generic network systems; I knew we simply did not have the time to write that stuff when we had projects already in full production that needed network code NOW).

Because of that, we can’t share the technology they create. Not yet, anyway. But we *can* share the expertise of the people in the team. They have to do a fair amount of international travel (bummer). So far, it seems to be working OK, but … it’s too small a sample and it’s been running too short a time (do you have any idea how hard it is to recruit really good network coders? :( ) to make any firm measurements of the success or failure of the approach.

Architect all your CT initiatives on a project-basis, not a service basis

I’ve seen a couple of examples of this, usually where senior people were frustrated by the progress of CT but needed some new example of the kind of stuff it was providing – so they went and started their own new pet-project to provide that, but outside the remit of the CT group. I’ve seen examples of this that were positive, and examples that were negative. Politically, it’s likely to cause a shitstorm as far as I can see, no matter how you sell it (unless this is blessed by CT itself, and none of the examples I’ve seen were their idea, although IIRC one of them did get CT’s agreement – they just knew they didn’t have BOTH the expertise AND the budget to provide it, so were happy to not have to take responsibility for it).

Certainly, it’s been the fastest way to get real significant progress on anything major: by doing it as a project, and going through the “standard” greenlight – pre-production – greenlight2 – production – ship cycle that all games companies are accustomed to for their games, large amounts of money have been secured quickly, and the projects have got going quickly.

Embed CT teams into the game-teams

The idea with this one is to overcome both the antipathy between game-teams and CT (by making them part of the same team), and also to ensure “automagically” that the CT team-members are working stuff that is guaranteed directly applicable to a real game – because they’re reporting to the Producer of a given game-team all the time.

If it works out well, this could solve all the problems. Although I’m not yet sure I understand how – in the long term – this is any different from having no CT at all. As noted earlier, this is closest to how I remember things working well before the idea of CT gained widespread popularity. OTOH, I’m not sure if that same situation will even work nowadays – the industry has changed a lot since then.

History of CT

I’m not sure about this, but my vague memory is that CT grew in popularity and – more importantly – commonness (i.e. how many companies were doing it) in parallel with the axing of internal game-engine teams and the move towards licensing external game-engines.

I have suspicions that a lot of the confusion of “what do we have central technology for?” comes from this, two-fold:
– a) those companies that didn’t entirely get rid of their engine teams had people left a bit stranded, who ended up being folded into CT, and then tended to carry on working much as they had inside the engine team, despite the different overall task of the dept (and this happened partly because of the lack of clear direction from management about what the new team was for anyway)
– b) many of the companies that dispensed with their substantial proprietary engines they’d been using for years then jumped to Unreal et al found that actually the licensable engines weren’t as good as they’d been lead to (or had lead themselves to) believe: they couldn’t go back to having an engine team, but they needed a way to reduce the huge budgets they were spending on just basic modification and fixing of the external tech, that weren’t easily justifiable on the game-budget (because they weren’t forseen, and were adding NOTHING to the game’s end feature list), so they ended up creating a CT team to hide / provide the budget to do that kind of work

I could be completely and utterly wrong here – this is just some ideas that have been kicking around in my head recently.

EDIT: I’d also like to point to this good post about game-engines, and why you shouldn’t build them (not necessarily a “never do it” argument, but an explanation of why you might want not to).

Last year I would have ranted about how retarded it is that game data is either entirely inaccessible to the web, or only accessible to an “official” website. …This year, however, is all about the positive. Rather than rant about no one providing such a feed, this is an un-rant about someone providing such a feed.

Dungeon Runners!

I found this via a news post about character sheets being viewable online at 3rd-party sites, making the assumption this meant an XML feed was available, and then digging through the forums until I found the post with a link.

This is really cool, I should tell you, just in case you don’t get that.

Receptions like this help keep us motivated to keep doing more of them :).

(because I’m now deviating from the original schedule I outlined in Part 1; what the heck, it was only a rough agenda anyway…)

Questions, questions…

First of all, there’s a bunch of good questions that have been raised in response to the first two posts:

• what data and methods are stored in the OOP implementation of an entity?
• where does the data “live”?
• how do you do object initialization?
• what does the ES bring that cannot be accomplished with an AOP framework?
• what’s the link between entity systems and SQL/Relational Databases? (OK, so that one’s my own question from last time)
• what, exactly, is an entity?

Let’s start with that last one first.

What, exactly, is an entity?

Obviously, I’ve already covered this from the conceptual perspective in the last post, where I defined them as:

For every discernible thing in your game-world, you have one Entity. Entities have no data and no methods

Great. But a couple of people were wondering what ACTUALLY is it, when you start implementing the thing? Is it a class? Is it an array of data? Is it a list of Components?

The last statement is particularly important: actually, entities have NO data and NO methods – an entity is not a stub for a class, it is not a struct of data, and – ideally – it’s not a list of Components. An entity is really just a name. Last post I also wrote that entities “do little more than to tag every gameobject as a separate item” – and that’s the key thing here, an entity is just a unique label for an in-game object.

I was a bit non-specific, because I didn’t want to say empirically that an entity cannot or should not be a list of components – sure, you CAN implement it that way – because many people do implement entities like that, and who’s to say what’s right or wrong here?

Although…I actually think that *is* the wrong way, and think that by the time you get to the last of these Entity Systems posts, you’ll probably agree with me ;). The problem is, the right way is usually too slow, so you’ll probably end up *to some extent* implementing entities as lists anyway, but if you do so as a performance optimization (e.g. by hiding that detail from the rest of the system) rather than as a logical implementation, it’ll cause fewer problems with your code in the long term.

So, to answer the question directly:

What is an entity?

An Entity is a number (a globally unique number)

Remembering, of course, that a String – or any other kind of unique label – is merely a fancy kind of number, not just in terms of how computers implement them, but in terms of how Mathematicians think of them. People tend to think that strings have extra features, for instance you can encode trees within strings easily (“root.node1.leaf”, for instance – as used in most OOP languages derived from C to navigate the namespace of classes, methods, etc) – although of course you can encode things like that in numbers, too, for instance “123045607″ (using 0 to stand for the dot… i.e. “123.456.7″). But … you *really* don’t want to do this!

Gotcha number 1: only OOP programmers want to give entities hierarchical names. It can tunr out a pretty dumb idea, if you think about it: it’s usually an implicit refusal to let go of those OOP ways we’re so used to thinking in, when those OOP ways are exactly what caused most of the problems we’re trying to fix by using ES’s.

So, don’t be tempted into hierarchical encoding, and definitely don’t do ANY encoding in the entity names – there’s a much BETTER place to do metadata for entities (trust me).

And I’ll be calling the implementation of an entity a GUID from now on (“Globally Unique IDentifier”).

What data and methods are stored in the OOP implementation of an entity?

The answer for the previous question makes this one easy to answer: none. The entity is merely a label, and whilst you DO want to store and manipulate some metadata around that label (e.g. it’s kind of handy to have a list of all the entity names you’ve currently got in memory, and be able to rename them without breaking everything), you should be thinking of them as nothing more or less than a GUID (String. Number. Whatever!).

Where does the data “live”?

The first time I made an ES, I implemented the individual Entities as classes, and although I didn’t put any methods in those classes (that would just be bog-standard OOP!), I did put the data into them. It didn’t occur to me that I’d need to worry about where the data was living, so I just stuck it somewhere natural – in the Entity.

Of course, I’d made a cardinal error, because whatever feels “natural” to an OOP programmer is usually “completely wrong” in ES programming.

Anyway, the last time I did an ES I was a lot wiser, and so we put the data inside the Components. All of it.

Gotcha 2: ALL the data goes into the Components. ALL of it. Think you can take some “really common” data, e.g. the x/y/z co-ords of the in-game object, and put it into the Entity itself? Nope. Don’t go there. As soon as you start migrating data into the Entity, you’ve lost. BY DEFINITION the only valid place for the data is inside the Component

How does that work?

Well, you need to start thinking about your Components a bit more carefully here. I previously said that:

A Component labels the Entity as possessing this particular aspect

So, again, a Component is merely another “label”, just like the Entity itself. Right? Wrong. This is my fault – in the last post, I was trying to Keep It Simple for you, and glossed over quite a few things; because the very first ES I made, I treated the components as mere labels, that description came out most easily as “the simple version” of what they are. In more detail:

A Component is the chunk of stuff that provides a particular aspect to any Entity

…where “provides” means “does all the things that are necesssary to make this work”. Which means that all your Components certainly *could* be implemented as standard OOP objects. To define an Entity, you just provide a name (GUID) and a list of Component classes that it needs, and to instantiate that Entity, you just instantiate one object from each class (and then you need to somehow attach those in-memory objects to your GUID, probably by implementing the Entity as an empty class that just contains a GUID and a list-of-component-instances (which I said to avoid, but we’ll come back to that later)).

But, as you may have noticed by now, I’m vigourously against using OOP anywhere inside an ES implementation. This isn’t some strange dislike of OOP itself, it’s just that in my experience with ES development it’s all too easy for experienced OOP programmers to keep trying to sneak some OOP back in where it’s inappropriate, and it’s all too easy to delude yourself into thinking this is a Good Idea. And, worse, from the teams I’ve worked on in the past, it has consistently seemed that the better you are at OOP programming, the harder this is to resist…

You COULD implement your Components as OOP objects. But, really, if you go back to the definition of a “System” from the last post, you’ll see that the methods for acting upon Components should all live inside the Systems, not the Components. In fact, re-reading that last post, I notice that I explicitly described Systems as doing exactly this: “a System essentially provides the method-implementation for Components”.

One tiny exception to this rule: getters and setters are really useful. If you’re implementing your ES within an OOP language, you might allow yourself to implement Components as objects just to get the benefits of get/set and e.g. multiple different versions of each get/set that do basic type conversion, or integrity checking on data, etc. Be warned, though, that if you do you MUST implement them as flyweight objects (go google the Flyweight Pattern if you don’t know it) or else you’ll lose lots of the memory-management and efficiency advantages of a full ES.

Where does the data “live”?

Each Component is merely a classification label (e.g. “Renderable”) and a struct/array/list/whatever of the data relating to it’s purpose; all data lives in Components; ALL data.

How do you do gameobject initialization?

Here’s a fun one – I haven’t mentioned ANYTHING about object/entity intialization. It was really only from trying to use entity systems that I finally realized just how bizarre OOP is when it comes to initialization: you have some “magic methods” (constructor, destructor, finalizer, etc) that have nothing to do with the main description of OOP, but instead are part of a meta-programming that allows you to get your OOP system up and running, and to keep it running, and then to tear it all down again at the end. Until then, I’d not noticed how out-of-place those methods are…

Hats-off to the OOP folks: they integrated their meta-programming so neatly into OOP that in most languages it’s unnoticeable. But I’ve not yet seen the same trick done for an ES – so, get used to the fact that initialization is going to be a bit … “different” …

There’s really a bunch of issues here:

• what’s the technical term for an entity’s archetype?
• how do you define the archetypes for your entities?
• how do you instantiate multiple new entities from a single archetype?
• how do you STORE in-memory entities so that they can be re-instantiated later on?

What’s the technical term for an entity’s archetype?

There doesn’t seem to be one. Really. There’s a couple floating around, influenced by people’s own backgrounds but I’ve heard quite a few used interchangeably.

My favourites are:

• template
• model
• assemblage

(I actually picked the last one from a thesaurus – looking for equivalents of “class” ;)).

The problem with both “template” and “model” is that they’re both very frequently used generic terms in programming. Entity is pretty good as a term, because it’s hardly used for anything else (and the main obvious competitor is now widely called a “gameobject” by game programmers). I’ve used both, in the past. Interchangeably. (cackle!)

So. “Assemblage”, anyone?

how do you define the archetypes for your entities?
how do you instantiate multiple new entities from a single archetype?
how do you STORE in-memory entities so that they can be re-instantiated later on?

Big topics. Really big. Too big to go into here (ah. Now I know what the next post is going to be about…).

What does the ES bring that cannot be accomplished with an AOP framework?

The simple, facetious, answer is: almost everything.

AOP (Aspect-Oriented Programming) doesn’t really help much to solve the problems that an Entity System solves (although it does solve similar ones that ALSO tend to be present when the ES-related ones are present), and doesn’t provide the new features that you get from an ES (e.g. the improved memory/data-performance).

An ES takes a system where there are many things going on that are all independent, which OOP tangles up together, and pulls them apart and lets them live on their own. It also highly efficiently manages simple interoperation between those disparate aspects of the program, and allows them to be disconnected, reconnected, at will.

AOP takes a system where there are things going on that are fundamentally DEPENDENT, which OOP tangles up together, and allows them to be reasoned about “separately, but together” – you can view the code for logging independent of the code which is being logged, but you have to still reason about them at the same time, you cannot merely ignore one or the other. By definition, different aspects (vested as Components) of an Entity are INdependent, and this whole “together” thing is an irrelevance.

However … AOP certainly helps if you have implemented with OOP something you ideally should have done with an ES, and you want to add new features to it, because the tangled OOP system is going to be a pain to modify without breaking stuff accidentally, and AOP will help you more precisely focus the impact of your changes. But it’s a poor replacement for just implementing the ES you wanted in the first place.

There is also a fairly large area of crossover between “some things you can do with entity systems” and “some things you can do with aspect oriented programming”, mainly because they are both fundamentally data-driven at the function-despatch level (which I’ll come back to later). However, they each use this data-drivenness in very different ways – AOP uses it to react to, and wrap itself around, the code of the program, whereas an ES uses it to react to the data of the program.

Using both together could be really exciting – but I’ve not been brave enough to try that yet myself.

What’s the link between entity systems and SQL/Relational Databases?

You might have guessed this by now – it’s been the theme of most of the answers above, starting from the very beginning of this post.

Mathematically-speaking, an Entity is a database Key, just as you’d see in any RDBMS.

Likewise, from a purely abstracted point of view, the “set of component-instances that comprise Entity #7742″ is, literally, a database Query.

THIS is why I said at the start that, ideally, you do NOT want to store your component-instances as a list inside a struct/class representation of each Entity. Fundamentally, that set is NOT defined as a list (a list is a static thing, it’s members don’t change without you changing them), it’s defined as a query (whose members are always changing, whether you want them to or not), and using one where really you wanted the other tends to cause problems in the long term.

Obviously, you can use Lists as a simple form of query (that query being “what things are currently in this list”), but that’s a rather poor and inflexible kind of query. Life is much more interesting if you embrace the dynamicity of the query, and fetch the components (and, hence, the VALUES of the data…) by running a query every time you want one or more of them.

This is a fairly star-gazing view of how to make games: the game is just one giant database, running dynamic queries all the time. Current performance, even of fast RDBMS’s, is just way too slow to be running thousands of queries per frame on a home PC.

However, these articles are about ES’s for massively multiplayer online game development, not just standard game development, and that changes two things in particular:

1. Most of the game is NOT running on home PC’s, it’s running on arbitrarily beefy server-farms that are already running bajillions of QL queries (nearly always SQL these days, but any of MS SQL Server, Oracle, or MySQL).

2. The data is much much more important than rendering speed; we have almost no problem making incredibly beautiful 3D rendered landscapes, but managing the data to make those landscapes act and react as if they were real (i.e. world-logic, game-logic, etc) is something we still find rather hard

A new thought for the day…

Think what you can do with an ES if you deploy it on an MMO server farm, and go the full monty by making all your entity/component aggregations stored merely as SQL queries. Here’s some to start you off…

1. No more pain from “Relational vs OOP”

One of the slowest parts of MMO server performance is translating OOP objects into RDBMS tables so that the database can save them, and then translating them back again when the game needs to read them. This also wastes large amounts of programmer time on boring, annoying, tricky to maintain code to handle the serialization processes.

2. Massive increase in parallelization for free

If you make all processing data-driven, and you have no data-structures that have to remain synchronized, then it becomes a lot easier to “just add more CPU’s” to increase runtime performance…

3. Where else could you use SELECT instead of data structures?

When each System has to do it’s processing, it’s probably going to iterate over “all Entities that have my Component”. Is that something you can store easily in a standard data-structure? Or is it better off as a QL query? And what else can you achieve if you start doing your processing this way? (hint: there’s some nice benefits for the low-level network programming here…)

NEXT: Part 4

(Part 1 is here)

Sadly, there’s some disagreement about what exactly an Entity System (ES) is. For some people, it’s the same as a Component System (CS) and even Component-Oriented Programming (COP). For others, it means something substantially different. To keep things clear, I’m only going to talk about Entity Systems, which IMHO are a particular subset of COP. Personally, I think Component System is probably a more accurate name given the term COP, but then COP is improperly named and is so confusing that I find if you call these things Component Systems they miss the point entirely. The best would be Aspect Systems, but then AOP has already taken ownership of the Aspect word.

An entity system is simply a part of your program that uses a particular way of breaking up the logic and variables of your program into source code.

For the most part, it does this using the Component Oriented Programming paradigm instead of the OOP paradigm – but there’s subtleties that we’ll go into later.

We refer to an entity “system” instead of entity “programming” because ES’s are usually found embedded in a larger, OOP, program. They are used to solve a subset of problems that OOP is poor at dealing with.

So, it is generally a replacement for Object Oriented Programming. The two are mutually exclusive: you have to choose one or the other (although any complex program may have multiple levels of abstraction, and you could mix and match OOP and COP/ES at different layers, any given layer is going to be one or the other (or, of course, any other alternative way of structuring your program).

Common misconception #1 – Entity Systems are part of OOP
Common misconception #2 – Entity == Class, Component == Object

Where OOP has Classes and Objects, an ES has Entities and Components – but it is very important to note that an Entity is NOT equivalent to a class, and a Component is NOT equivalent to an Object, although superficially they seem to share some characteristics. If you think of Entities/Component’s in OOP terms, you will never understand the ES, and you will screw up any attempt to program with it.

The whole point of ES’s using a different programming *paradigm* is that this means there CAN NEVER BE a direct equivalent between Classes and anything, or between Objects and anything – if there were, it would actually be the same paradigm, and just be a variant of OOP. It’s not. It’s fundamentally different and incompatible.

What’s an Entity?

An Entity System is so named because Entities are the fundamental conceptual building block of your system, with each entity representing a different concrete in-game object. For every discernible “thing” in your game-world, you have one Entity. Entities have no data and no methods.

In terms of number of instances at runtime, they are like Objects from OOP (if you have 100 identical tanks, you have 100 Entities, not 1).

However, in terms of behaviour, they are like classes (Entities indirectly define all the behaviour of the in-game object – we’ll see how in a second).

So, entities on their own are pretty much entirely useless – they are coarse-grained, and they serve to do little more than to tag every coarse gameobject as a separate item. This is where Components come in.

What’s a Component?

Every in-game item has multiple facets, or aspects, that explain what it is and how it interacts with the world. A bicycle, for instance, is:

• Made of metal
• Can be used by a human
• A means of transportation
• Something that can be bought and sold
• A popular birthday present
• Man-made

At this point, please note: OOP does not have this concept of multiple aspects; it hard-codes all the aspects into one glob, and declares that the set of behaviours and values of data to be the single-aspected Class of an Object. C++’s Multiple inheritance and Java’s Interfaces can get you a small part of the way towards the aspects of an ES, but they quickly collapse under the strain in fundamental ways that cannot be fixed within OOP.

The way that an Entity for an item represents the different aspects of the item is to have one Component for each aspect. The Component does one really important thing:

• Labels the Entity as possessing this particular aspect

It may do several other things, depending upon your implementation and the design of your ES. However, with an ES, this is the most important, because this is the basis of all method execution and processing.

What’s a System? (or subsystem)

An ES goes further than OOP in how it splits up the universe of code and data. In OOP, you make everything into Objects. With an ES, you have two different things that you split code and data into: the first is “Entity + Components”, and the second is “Systems”.

This is the source of a lot of the value of an ES. OOP is very good at implementing the part of any program that has lots of data floating around and lots of methods floating around which need to be executed only on a small, instanced, subset of the data in the program. OOP is very poor at implementing the “global” parts of a program, which have to operate “on everything”, or have to be invoked “from everywhere”. An Es solves this by explicitly dealing with all the global stuff using Systems, which are outside the realm of Entity/Component.

Each System runs continuously (as though each System had it’s own private thread) and performs global actions on every Entity that possesses a Component of the same aspect as that System.

There is a one to one relationship between Systems and AVAILABLE aspects (i.e. types of Component). A System essentially provides the method-implementation for Components of a given aspect, but it does it back-to-front compared to OOP. OOP style would be for each Component to have zero or more methods, that some external thing has to invoke at some point. ES style is for each Component to have no methods but instead for the continuously running system to run it’s own internal methods against different Components one at a time.

Typical Systems in a game would be: Rendering System, Animation System, Input System, etc.

The Rendering system wakes up every 16 milliseconds, renders every Entity that has the Renderable Component, and then goes back to sleep.

The Animation system constantly looks out for anything that would trigger a new animation, or cause an existing animation to change (e.g. player changes direction mid-step), and updates the data of each affected Entity (of course, only those that actually have the Animatable Component, i.e. are animations).

The Input system polls the gamepad, mouse, etc, and changes the state of whichever Entities with the Inputable Component are currently marked as being controlled by the player.

In tradtional OOP, if you have 100 units on a battlefield, and each is represented by an object, then you theoretically have 100 copies of each method that can be invoked on a unit. In practice, most OOP languages use runtime and compiler optimizations to share those methods behind the scenes and avoid wasting memory – although scripting languages for instance often don’t do any sharing, allowing all the methods to be independently changed on each and every instance.

By contrast, in an ES, if you have 100 units on a battlefield, each represented by an Entity, then you have zero copies of each method that can be invoked on a unit – because Entities do not contain methods. Nor do Components contain methods. Instead, you have an external system for each aspect, and that external system contains all the methods that can be invoked on any Entity that possess the Component that marks it as being compatible with this system.

That’s not an Entity System!

Well … at least, that’s what *most* people I’ve worked with consider to be an Entity System. But even within a single project, I’ve found multiple different ideas of what an ES is, some of which are compatible, others of which are incompatible. Here are some of the other ideas I’ve come across that people thought were ES’s.

Alternative definitions of entity systems:

1. An ES provides a different model of class-inheritance to OOP, especially w.r.t inheritance of fields. For any given entity-class you declare which aspects that class possesses, and the actual class is looked-up at runtime by piecing together all the different aspects.

2. The same as 1., but taken literally for methods as well as fields, i.e. this provides an interesting way to mix-n-match inheritance of functions/methods. So long as methods are attached to the fields upon which they operate, and/or a “pre-requisite” system is implemented such that “having aspect X automatically causes you to also have aspect Y”, then this works quite smoothly. NB: this is implementing “modular objects”: the “aspects” are complete objects from the standard theory of OOP.

3. A variant on 1. or 2. where the aspect data is pre-compiled into a large number of OOP classes at compile time, so that at runtime you are just running “normal” classes, without any lookup cost of the dynamic “I’ve got an entity E which claims to have aspect X; just hold on a sec while I build a function table to find out what the heck to do now someone just tried to invoke function Z…”

4. A way of compiling standard OOP classes into runtime-data (note: disassociating them from their OOP-ness) specifically so that you can treat them as streamed data instead of chunked binary + data. The main value of this is to play nicely with hardware that has tiny tiny amounts of memory available in the code cache and/or very very slow access to fetch extra code, but has vast bandwidth to stream SEQUENTIAL data through the CPU. PlayStation consoles especially like this approach, as opposed to the typical PC approaches. Note that in this case, the idea is that the game-programmers and game-designers are UNAWARE that there’s an entity system – it’s a runtime/compile time performance optimization, not a design-feature.

5. A system design that revolves around excessive use of the Observer pattern, but does so using standard OOP techniques. I mention this because it’s a common (ab)use of the concepts of ES’s, but lacks much of the advantages. For what it’s worth … that’s the situation I was in when I first discovered ES’s. So, personally, I really don’t count these as ES’s. These are not an ES, but an: “OMG, I need an ES!”, but many people persist in thinking (and claiming) they’re a true ES.

NB: IMHO this is one of those “you really shouldn’t do this” classes of Entity System usage. Why? Because you end up just creating MASSIVE OOP classes with tonnes of independent implemented Observer methods tacked onto them. That’s fine as a general approach, it buys you something whilst remaining ENTIRELY OOP, but it also loses most of what ES’s have to offer, especially in the realm of flexibility and compound inheritance.

Who cares about ES’s and why? (where did they invent these from?)

1. Coders who are used to writing a graphics/rendering engine and trying to keep it isolated / separate from the rest of the computer game. They learnt that they can write the entire graphics engine using only PARTIAL information (a couple of aspects – no more than 2 or 3) of every OOP object that exists in the game-universe. They need to use: Renderable (do I have to paint it to the screen every frame? [it will have all the methods necessary to do that painting generically]), Updateable (does it potentially change in response to one or more game-loop-ticks? [animations are a great example of this - their internal state changes independently of how many times they're rendered]),

2. Multi-threaded coders who are looking at the concept of aspects per se (i.e. Renderable as opposed to Updateable) to allow for coarse-grained multi-threading: you are allowed one thread per aspect, and this is guaranteed safe simply because the aspects are – by definition! – independent from one another. Since a typical ES will have a dozen or more aspects, you’ve just bought yourself reasonably effective performance scaling of up to 12 separate CPU cores, for free.

Also … going to the next level of difficulty and performance improvement, they’re looking for the largest possible individual (literally: things that “cannot be divided”) things that can be made the atomic level of multi-threading. The default position is that only manually-hard-coded atomic operations count, and OOP objects certainly are full of all sorts of multi-threading individual pieces so are far too coarse, but maybe components within entities are small enough to be the unit of atomicity.

3. Game designers who want to enable “everything to become anything”.

Do Cameras shoot people?

Do Bullets accept input from the player?

No?

Well … have you played Unreal Tournament? If you have but didn’t answer yes to the earlier questions then you need to find someone to show you a Redeemer being used in Alternate Fire mode (hint: you get to manually “fly by wire” the rocket from a first-person view inside the warhead itself).

ES’s allow everything and anything to be used as anything and everything, interchangeably, with no re-coding. That’s the beauty for designers – all the artifical constraints on design that were previously enforced by coders having to, you know, place SOME constraints just to write their own damn code and have it be reasonably performant and bug-free are suddenly lifted.

It’s true – you really CAN have strongly-typed safe code and yet have total freedom to pass any object to any method; at least, you can with Entity Systems.

4. Coders who go a bit overboard with the Observer pattern.

If you’re unsure about this one, and have used Observers a lot, ask yourself this: have you ever had that problem with Observers where you found that “just one base class isn’t enough”? i.e. one base class that implemented a handful of different Observers worked great, but as your system / game / application expanded you found you needed more variety in that base class, you needed multiple different versions, and you needed *every possible combination* of those different versions, because base-classes cannot/will not “stack” nicely on top of each other.

Thought for the day

Programming *well* with Entity Systems is very close to programming with a Relational Database. It would not be unreasonable to call ES’s a form of “Relation Oriented Programming”.

This needs a followup post, but you might want to think about that :) both in terms of how the above statement could be true (hint: think about what most of the code you write for each System is going to look like, and what exactly it is doing), and also in terms of what the knock-on effects of this are if it is true.

Bear in mind that no-one’s yet managed to do OOP fast with Relations; the excessively high conversion cost between RDBMS and OOP – both at coding-time and at runtime – is still one of the biggest single development problems with writing an MMO.

NEXT: Part 3

Anyway, to get you thinking (I’m not providing definite answers here, but just some stuff to make you think about more carefully about how you’re doing your networking), here are some common rules that perhaps no longer apply the way they used to:

I’ve been interviewing candidates recently for Network Programmer roles at NCsoft, one of the world’s largest developers of online games, so we look for the best people (and are usually quite good at spotting them). As an experiment, among the normal interview questions I ask, I started asking some ultra simple questions which have famous “correct” answers that are no longer true. Interestingly, this showed a great many applicants were still parroting answers that either they read from their undergraduate course-notes or in old books, or that they discovered themselves 5 years ago but apparently hadn’t done any real programming in the last few years, because they hadn’t noticed how things have already changed.

What does this mean? Well, for a start, some extremely powerful techniques in network development which people could actually use today are probably being ignored by many developers because “received wisdom” suggests they simply cannot work. It also means I’m going to have start thinking up some new interview questions, of course, because now everyone coming for interview is going to Google this page and change their answers accordingly…

Myth #1: Client bandwidth is the bottleneck

I moved house at the start of this year. I tried to get a cheap, relatively fast, internet connection. 1 Mbit would be fine, I wasn’t going to use it much at home (I already had mobile internet).

The slowest anyone was willing to sell me was 8 Mbits, and the fastest 24 Mbits. Overkill? Well … those DVD’s everyone’s downloading these days are pretty good at soaking up all the bandwidth you can throw at them, I guess…

Of course, I could pay more, and get less bandwidth (there’s a few sharks out there), but that’s just stupid.

And yet … most multiplayer games from the late 90′s through to the mid 2000′s were carefully optimized to get their maximum bandwidth usage per client down to around 1-2 KB/s, i.e. 8-16 Kbits. Or, to put it another way, around 1/3000th of the bandwidth I was offered for the grand sum of £24 a month.

Myth #2: The network is orders of magnitude “slower” than the rest of the computer: the server itself will never be the bottleneck

Another one that crops up often as a rule of thumb – you’ll be lucky to be getting 100 messages a second from your network connection, whilst your PC’s processor runs at 1 Ghz, which means that in the same period of time it can process 10 billion individual commands. Ten billion. That’s quite a lot more than one hundred.

Of course, over the LAN you’ll get a lot more than a hundred messages, but even with a decent fast hub and 100 Mbit ethernet you’ll probably max out at somewhere around ten thousand or so per second. That’s more than a billion times fewer (“slower”) than your CPU.

Now at this point you’ll have spotted the deliberate mistake there, and be saying “but no-one uses hubs these days, switches are now so cheap you can’t buy them,” (clever you) ” and we’ve got gigabit ethernet as standard on all the motherboards – you probably can’t even BUY a 100 Mbit network card any more!”. Right. So … the network can handle theoretically ten times as much traffic (wow!) and in practice probably more like twenty times as much now we’ve got rid of those shoddy hubs. Although, then again … nowadays it’s not the crapness of Ethernet that makes it struggle to hit its rated speeds, it’s usually the fact that your network card is “too” cheap, and you can’t buy a decent one because there’s no market left for them (oh, how ironic!).

Still, we’ve definitely got an order of magnitude more bandwidth, and improvements to the OS IP stacks mean we’ve probably got a fairly good improvement in latencies and the handling of vast numbers of packets on the wire. But we were talking about a factor of 1 million difference, so even a few orders of magnitude is still a total irrelevance.

Hmm. Seems I’m arguing that the network really IS a lot “slower” than you computer. Except … the suspicious amongst you, Dear Reader, will have noticed the quotes I keep putting around the word slower. Yeah. There’s a reason for that.

When you’re writing games people always talk about the number of messages they want to send per second, and ask if it’s reasonable. And in any online game, you’re never going to care about total bandwidth because you’re only sending tiny amounts of information about (c.f. the previous point about games getting down to a few kilobytes of traffic per second). But that’s the problem: mixing the two concepts of latency and bandwidth into one rating of “speed”.

When it comes to the server, the “total” latency it’s experiencing does NOT increase or decrease when you increase or decrease the number of connected clients (well, it does, a very very small amount), but the total bandwidth it’s experiencing DOES increase – linearly, in fact. And if we’ve taken any advantage of Myth #1 up above, and started letting our players transfer stuff like streaming movies of themselves to each other in-game, then potentially every new client is going to add their total ISP-provided bandwidth to the mix.

For a while now there has been something of a problem here with bandwidth usage per client, which is just the raw cost of the ISP bandwidth bill for your server. e.g. back in 2002 even with just 10,000 players the typical small MMOG could be paying as much as a six-figure sum on annual bandwidth bills. But costs have come down, especially if you buy in bulk, so that’s not such a big problem now.

The problem nowadays is not “how much can the internet handle?”, and not always “how much will it cost you to get that much internet traffic?”, but sometimes just “can your server cope, physically, with the volume of data you’re trying to pump in and out of it every second?”.

1,000 players each saturating their 24Mbit home broadband connections?

That’s 3 gigabytes (bytes, not bits) of data. Every second.

Phew. Well…that means we’ll need to stick 6 x 4-way gigabit ethernet cards in the server. That’s probably OK, they only cost a few hundred dollars each these days.

But … um … how much bandwidth is there INSIDE the server? Have you looked at the bandwidth of modern RAM these days? Peaks out at around 10 GB/s – enough that the sheer volume of your network traffic is seriously interfering with the PC’s ability just to function as a processing system. But what about the peripheral bus, which is going to be between you and the CPU/RAM – a nice fast PCIe x16 slot manages a mere 4 GB/s, only just enough merely to do the basic data-transfer (and this assumes the computer isn’t doing anything else except transferring data – obviously we want it to do actual processing too!).

Myth #3: Server Clusters should use TCP internally

Pretty much all multiplayer games these days do some clustering on the server side, and it’s a strange world of it’s own.

(do you think that because you’re not doing an MMO you don’t care about clusters? Well, tell me: exactly HOW is your multiplayer racing game (or FPS) going to allow people to start new games, hmm? A lobby-server, perhaps? Maybe even a login-server, that stores stats for each individual player? Maybe those need to talk to each other, and to the game-server itself?)

Although I hate it when people (usually forum posters ;)) insist that games should all use UDP “because it’s faster” or some such rubbish (define “faster”, please…), the opposite seems to be true with server clusters. Time and again people go with TCP, apparently without considering whether UDP might be better for them. (see? I said “might be” there. You won’t find any gross sweeping generalizations here! Um. Ahem.)

It’s easy to see why people go for TCP: you want the thing that most simplifies your server architecture, because, Damn it!, it’s hard enough making the bloody thing work in the first place – you have more than your fair share of headaches to deal with already, what with Race conditions, Object-Persistence, Transactional integrity, Security holes, etc.

Actually, I suspect it’s as much because cluster-focussed middleware is usually TCP based (think of all the little network tools and OS features that go together to make up your cluster, and how many of them are TCP based as opposed to UDP? How many of them give you a choice what to use as the stream provider?) and that developers just go with the flow.

A lot of the UDP vs TCP debate evaporates inside a server cluster. In particular, there’s no routers, no DSL modems, no switches all doing weird and horrible things to your traffic (TCP header compression? UDP delayed by higher QoS traffic? etc). So it’s a bit simpler.

But, most obviously, *there’s no such thing as packet loss*. So … UDP is suddenly a “reliable protocol” (whoa!). But, also, TCP is suddenly a “no artificial delays on received packets due to out-of-order reception” (double-Whoa!). Then again, that means that UDP is ALSO going to receive packets in-order.

(NB: that’s not really true, unless your routers and switches and OS’s behave themselves. But … with the cluster, you get to buy those bits yourself, so you have – in theory – the option to MAKE it true)

So … what’s wrong with TCP? Well, the one bit that’s really wrong with it is this:

i.e. UDP does “nothing special”, it just sends data, but TCP has loads of “extra stuff” that’s going on behind the scenes, including a 22-state FSM (Finite State Machine), that’s just sitting there hogging up resources and adding extra delay onto things like opening and closing connections. That’s all wasted effort – You Ain’t Gonna Need It (so don’t use it).

Sadly, poor old TCP suffers from the exact same problem it always suffers from in games development – nothing is optional. Which means you HAVE to use all that stuff, even though it’s doing you no benefit.

So, TCP is simply “at least as or more expensive” for clusters (uses more resource) than UDP, without really helping. And in a lot of cases, that causes problems as you scale up your number of clients, or number of simultaneous matches.

But something else is…

Because … when you think about it, there is a kernel of truth in the claim. It is based around the concept that source code is the ULTIMATE arbiter of “what the system actually does, in reality”. Unlike comments, and written documents, and diagrams, source code by definition can never get out of date, no matter how time-poor, ill-trained, lazy, or even just unfortunate your programming team is.

(im)Perfection

Advocates of “code is the best documentation” tend to point to the fact that in reality there is no such thing as the perfect human programmer who never makes a mistake, and so manual comments are automatically doomed to failure. Worse, any given piece of software spends more of its lifetime in an altered/updated state than it does in its original virgin “as-it-was-originally-typed-in” state. Therefore, on any non trivial project, the non-source-code forms of documentation are guaranteed, mathematically, to become more and more out of date at an equal or greater rate than you can fix through periodic (or even continuous) manual updating of them.

Source code is NOT the perfect documentation … but unit tests are. Just as source is – by definition – always up to date, and manual comments are effectively guaranteed over time to become less and less up to date, unit tests always remain up to date *or they fail*, since they in no way rely upon or pay any attention to the source, and instead are tightly bound to the *usages* of the source.

(an aside: if you use unit tests, but don’t run Continuous Integration (CI) or a similar build process that automatically runs all unit tests on the order of once a day, and reports in big flashing emails when any fail, then it’s time for you to re-read the manual. I add this only because there are people who don’t do this, and they could legitimately point holes in the above, but you guys – you need to realise you’re in a small minority of people who are “missing the point” :P)

What does this buy you?

(subtext: Why did I bother writing this blog post anyway? ;))

Because it has one very very important knock-on effect: if you have no documentation for your system and you’ve decided you really need some (and, let’s face it, the war over whether documentation was “actually needed” was won a long long time ago), then I have a piece of advice:

Don’t write any documentation; write unit tests instead.

This is more important than merely arguing that unit tests are “better” documentation (personally, I think they aren’t – they are only perfect in a provable sense, not in a “actually solve all the problems that documentation is needed for” sense) – this is really about the practical realities of your situation.

If you already have the system, and you have no documentation, you will never have the time to write it.

(this is something of a truism that is commonly accepted for reasons of sheer practicality)

But … you will have the time to write unit tests. And it’ll be a heck of a lot cheaper and easier to write docs. To write even your first sentence of your first document, you need full knowledge of the ENTIRE system. To finish writing the docs you need absolute knowledge of the system. That means you have to do 100% of the work (reading the source) at any given level of abstraction in order to do even 1% of the documentation.

But – by definition – you can start writing unit tests the moment you’ve even just tried running the application once. Unit tests require extremely limited extremely partial information – the less knowledge you have about the SYSTEM when writing tests for the UNITs, the better. In the long run, you can write unit tests to cover the key 25% of your system functionality with only doing as little as perhaps 5% of the work (reading the source).

So, in the real world: “when the system already exists, don’t write documentation – write unit tests”

Final thought: yes, this works in the real world … but what about for really LARGE and COMPLEX systems? Are tests the best form of documentation for them too?

Epilogue: what are the roles of documentation?

Just to give a flavour, they include:

• educating the user on the abstract concepts that are enshrined in the application
  defining the problems that the binary code is attempting to solve
• telling the user where to “start” with trying to use the binary code, whether as library or app
• defining a contract of what the code will / will not do that is isolated against future updates of the source (source only says what it does now, in this (possibly bugged!) version; documentation tells you what it will CONTINUE to do in future versions, even if bugfixes change the current behaviour))

But now … they’re back! (and I’ll be adding more followup stuff in the coming weeks/months)

If you’re interested in the article itself, there’s a pretty good writeup here (scroll halfway down to find it) that covers the main core points (although understandably leaves out several sections, and doesn’t include any of the diagrams from the book – if it were much longer they might get into trouble over copyright issues ;)).

These are the most useful IMHO links from the references in my chapter. If you want the full text, you’ll have to buy the book (you can get it fairly cheaply these days – and you get more than 60 other gems as well, so it’s probably well worth it), but these refs themselves are pretty good reading.

…but if you want the 144-slide version (!), see below. No extra content.

EDIT: use this link instead, link below is 404ing: http://www.slidefinder.net/C/Caching_Web_2_0/465867

Caching for Web 2.0 – How and Why – NB: these slides are the actual ones I used at the talk, which won’t make so much sense as the “official” ones available from the AGDC website, which I sanitized a bit to be easier to read. Unless you were actually at the talk, I recommend you grab the ones from the AGDC website.

My talk on Caching for Web 2.0 went pretty well, after some nightmare setup issues (big thanks to the A/V guys at the Austin convention center!). Note to CMP and AGDC organizers: whilst it’s great being offered not one, not two, but SIX different 15-pin VGA cables to plug your laptop into the projector, many modern laptops are DVI only. It would be nice to have, you know, just ONE DVI cable too? (and make sure its simple DVI, not the one with extra pins, or it simply won’t work).

Anyway, the best part was how often I’d be speaking to people during the rest of the conference and they’d say they’d spoken to other people who’d told them how great the talk was :). Of course, I forgot to ask people to fill out the feedback forms, so it’s going to be hit and miss whether I get a good rating or not :(.

We also had a really good informal / birds-of-a-feather / roundtable meetup on the Wednesday evening about Web 2.0 and Games – but more on that later, once I get the photos off my camera.

• No programmer required for designers to modify game logic
• Circumvents the “impossible” problem of hard-coding all entity relationships at start of project
• Allows for easy implementation of game-design ideas that cross-cut traditional OOP objects
• Much faster compile/test/debug cycles
• Much more agile way to develop code

I first started using entity systems in anger back in 2001-2003, when I was working on MMOG server middleware. We were targetting the few most painful problems in MMOG development, one of which was the difficulty of constantly changing your game logic after launch, which led us to entity systems. I learnt then that entity systems were an almost perfect solution to massively speeding up the development time for most MMOG’s, and for also allowing almost unrestrained re-writing of fundamental game features post-launch with very little effort.

That last issue is critical to the success of an MMOG: once an MMOG is launched successfully, its long term success or failure depends more upon the ability of the dev team to evolve that game into a better game month after month than upon anything else.

I learned a lot from that first run-in with them, more about the things that go wrong and what makes developing with entity systems particularly hard than about the things that went right. We also discovered that performance could easily become a major issue – although they are very flexible and dynamic, the lack of pre-compiled lookups and optimizations can make runtime performance disappointingly (unacceptably) poor. Why? Mainly because of the amount of indirection and checks needed to run even a single method call (but I’ll go into detail on that problem, and how to fix it, a bit later).

I moved on, and didn’t think about them again, until last year. In 2006 I joined the Operation Flashpoint 2 team as the lead network programmer, where we trying to make an MMO-FPS on an unprecedented scale, and I discovered that the programming team was considering an entity-system to drive the whole game. The attractions for the OFP2 team were different – mainly around the improvements to memory management they could get from it, and the stream-oriented coding (which is essential for PS3 development) – but it turned out to be something of a silver bullet for the “MM” and “O” parts of the MMOFPS. As the network programmer, discovering that an entity system was going to be the interconnect for all other subsystems was a huge relief: the entity system meant I could implement the complex latency hiding and prediction techniques with minimal interference with the coding of all the rest of the game systems. When you’ve got 20+ programmers on a team, you really don’t want to be placing yourself in a position where you’re going to have to redesign code for all of them just to make it “network friendly”.

whilst I was working on OFP2, I got in touch with Scott, and exchanged ideas on what the future might hold for entity systems, including additional uses for them, and on how to share this knowledge more widely. He encouraged me to start a blog (and, actually, a year later that was one of the main reasons I even started this blog, T=Machine), so I figured now was a good time to start (finally) writing about those entities :).

And to blame Scott for the existence of this blog…

Since then, I’ve been thinking a lot off and on about entity systems and their appropriateness for use in MMOG development (again!). Only this time around – thanks to OFP2 and some of the issues it threw up – I had a much better idea how to use them to increase performance rather than reduce it, and had come across some other major problems that they conveniently solve. Most obviously, they work wonders for PS3 development. Knowing how PS3′s fundamental architecture works, I can’t immediately see how you’d want to use anything other than a full entity system for game development. There’s so much horsepower in that beast that you can certainly write games many different ways, but it’s particularly well-suited to this approach.

As it stands, I’m beginning to think that next-generation MMOG’s are going to be practically impossible to develop unless based heavily around a core entity-system. “practically” being the key word – unless you’re willing to spend $100 million for your development…

“Anything is possible”, of course, so there’ll be many exceptions to that – but I think any team that doesn’t go this route is going to suffer a lot because of it.

I think most people would agree. But … recent discussions on game-industry mailing lists – and the experiences I had within games companies with programmers who were much better at programming than I was – made me realise that there’s a lot of ignorance over these systems, and many people aren’t getting anywhere near the full potential of them. So, if you’re interested, read on…

I’ll do this in a series of posts (it’s going to take some time to write it all up!), but it’ll go something approximately like this:

• Part 1 – intro (you’re reading it)
• Part 2 – what is an entity system
• Part 3 – more on what is, and isn’t, an entity system
• Part 4 – Introduction to MMO Development Practices
• Part 5 – initialization, storage, and data
• serialization, de-serialization, and distributed state management

ADDENDUM:

• Entity System on Android phones – a concrete example using Java

EDIT, December 2007 – You may also want to take a look at Mick West’s introduction to entity systems for game development, it’s a shorter read than my posts, but narrower in scope.

NB: as I post the other parts, I’ll update the list above to link to them.