T=Machine

Entity Systems are the Future of MMOs Part 5

October 26th, 2009 by adam

(Start by reading Entity Systems are the Future of MMOs Part 1)

It’s been a long time since my last post on this topic. Last year, I stopped working for a big MMO publisher, and since then I’ve been having fun doing MMO Consultancy (helping other teams write their games), and iPhone development (learning how to design and write great iPhone apps).

Previously, I posed some questions and said I’d answer them later:

how do you define the archetypes for your entities?
how do you instantiate multiple new entities from a single archetype?
how do you STORE in-memory entities so that they can be re-instantiated later on?

Let’s answer those first.

A quick warning…

I’m going to write this post using Relational terminology. This is deliberate, for several reasons:

It’s the most-correct practical way of describing runtime Entities
It’s fairly trivial to see how to implement this using static and dynamic arrays – the reverse is not so obvious
If you’re working on MMO’s, you should be using SQL for your persistence / back-end – which means you should already be thinking in Relations.

…and a quick introduction to Relational

If you know literally nothing about Relational data, RDBMS’s, and/or SQL (which is true of most game programmers, sadly), then here’s the idiot’s guide:

Everything is stored either in arrays, or in 2-dimensional arrays (“arrays-of-arrays”)
The index into the array is explicitly given a name, some text ending in “_id”; but it’s still just an array-index: an increasing list of integers starting with 0, 1, 2, 3 … etc
Since you can’t have Dictionaries / HashMaps, you have to use 3 arrays-of-arrays to simulate one Dictionary. This is very very typical, and so obvious you should be able to understand it easily when you see it below. I only do it twice in this whole blog post.
Where I say “table, with N columns”, I mean “a variable-length array, with each element containing another array: a fixed-size array of N items”
Where I say “row”, I mean “one of the fixed-size arrays of N items”
Rather than index the fixed-size arrays by integer from 0…N, we give a unique name (“column name”) to each index. It makes writing code much much clearer. Since the arrays are fixed-size, and we know all these column names before we write the program, this is no problem.

Beyond that … well, go google “SQL Tutorial” – most of them are just 1 page long, and take no more than 5 minutes to read through.

How do you store all your data? Part 2: Runtime Entities + Components (“Objects”)

We’re doing part 2 first, because it’s the bit most of us think of first. When I go onto part 1 later, you’ll see why it’s “theoretically” the first part (and I called it “1″), even though when you write your game, you’ll probably write it second.

Table 3: all components

(yes, I’m starting at 3. You’ll see why later ;))

Table 3: components
component_id	official name	human-readable description	table-name

There are N additional tables, one for each row in the Components table. Each row has a unique value of “table-name”, telling you which table to look at for this component. This is optional: you could instead use an algorithmic name based on some criteria like the official_name, or the component_id – but if you ever change the name of a component, or delete one and re-use the id, you’ll get problems.

Table 4: all entities / entity names

Table 4 : entities
entity_id	human-readable label FOR DEBUGGING ONLY

(really,you should only have 1 column in this table – but the second column is really useful when debuggin your own ES implementation itself!)

…which combines with:…

Table 5: entity/component mapping

Table 5 : entity_components
entity_id	component_id	component_data_id

…to tell you which components are in which entity.

Technically, you could decide not to bother with Table 4; just look up the “unique values of entity_id from table 5″ whenever you want to deal with Table 4. But there are performance advantages for it – and you get to avoid some multi-threading issues (e.g. when creating a new entity, just create a blank entity in the entity table first, and that fast atomic action “reserves” an entity_id; without Table 4, you have to create ALL the components inside a Synchronized block of code, which is not good practice for MT code).

Tables 6,7,8…N+5: data for each component for each Entity

Table N+5 : component_data_table_N
component_data_id	[1..M columns, one column for each piece of data in your component]

These N tables store all the live, runtime, data for all the entity/component pairs.

How do you store all your data? Part 1: Assemblages (“Classes”)

So … you want to instantiate 10 new tanks into your game.

How?

Well, you could write code that says:

int newTank() { int new_id = createNewEntity();
// Attach components to the entity; they will have DEFAULT values createComponentAndAddTo( TRACKED_COMPONENT, new_id ); createComponentAndAddTo( RENDERABLE_COMPONENT, new_id ); createComponentAndAddTo( PHYSICS_COMPONENT, new_id ); createComponentAndAddTo( GUN_COMPONENT, new_id ); // Setup code that EDITS the data in each component, e.g: float[] gunData = getComponentDataForEntity( GUN_COMPONENT, new_id ); gunData[ GUN_SIZE ] = 500; gunData[ GUN_DAMAGE ] = 10000; gunData[ GUN_FIRE_RATE ] = 0.001; setComponentDataForEntity( GUN_COMPONENT, new_id, gunData );
return new_id; }

…and this is absolutely fine, so long as you remember ONE important thing: the above code is NOT inside a method “because you wanted it in an OOP class”. It’s inside a method “because you didn’t want to type it out every time you have a place in your code where you instantiate tanks”.

i.e. IT IS NOT OOP CODE! (the use of “methods” or “functions” is an idea that predates OOP by decades – it is coincidence that OOP *also* uses methods).

Or, in other words, if you do the above:

NEVER put the above code into a Class on its own; especially NEVER NEVER split the above code into multiple methods, and use OOP inheritance to nest the calls to “createComponet” etc.

But … it means that when you decide to split one Component into 2 Components, you’ll have to go through the source code for EVERY kind of game-object in your game, and change the source, then re-compile.

A neater way to handle this is to extend the ES to not only define “the components in each entity” but also “templates for creating new entities of a given human-readable type”. I previously referred to these templates as “assemblages” to avoid using the confusing term “template” which means many things already in OOP programming…

An assemblage needs:

Table 1: all Assemblages

Table 1 : assemblages
assemblage_id	default human-readable label (if you’re using that label in Table 1 above)	official name	human-readable description

Table 2: assemblage/component mapping

Table 2 : assemblage_components
assemblage_id	component_id

This table is cut-down version of Table 5 (entity/component mapping). This table provides the “template” for instantiating a new Entity: you pick an assemblage_id, find out all the component_id’s that exist for it, and then create a new Entity and instantiate one of each of those components and add it to the entity.

Table 3: all components

Table 3: components
component_id	official name	human-readable description	table-name

This is the same table from earlier (hence the silly numbering, just to make sure you noticed ;)) – it MUST be the same data, for obvious reasons.

Things to note

DataForEntity( (entity-id) ) – fast lookup

If you know the entity-id, you may only need one table lookup to get the data for an entire component (Table5 is highly cacheable – it’s small, doesn’t change, and has fixed-size rows).

Splitting Table 5 for performance or parallelization

When your SQL DB is too slow and you want to split to multiple DB servers, OR you’re not using SQL (doing it all in RAM) and want to fit inside your CPU cache, then you’ll split table 5 usually into N sub-tables, where N = number of unique component_id’s.

Why?

Because you run one System at a time, and each System needs all the components with the same component_id – but none of the components without that id.

Isolation

The entire data for any given system is fully isolated into its own table. It’s easy to print to screen (for debugging), serialize (for saving / bug reports), parallelize (different components on different physical DB servers)

Metadata for editing your Assemblages and Entities

a.k.a. “Programmer/Designers: take note…”

It can be tempting to add extra columns to the Entity and Assemblage tables. Really, you shouldn’t be doing this. If you feel tempted to do that, add the extra data as more COMPONENTS – even if the data is NOTHING to do with your game (e.g. “name_of_designer_who_wrote_this_assemblage”).

Here’s a great feature of Entity Systems: it is (literally) trivial for the game to “remove” un-needed information at startup. If, for instance, you have vast amounts of metadata on each entity (e.g. “name of author”, “time of creation”, “what I had for lunch on the day when I wrote this entity”) – then it can all be included and AUTOMATICALLY be stripped-out at runtime. It can even be included in the application, but “not loaded” at startup – so you get the benefits of keeping all the debug data hanging around, with no performance overhead.

You can’t do that with OOP: you can get some *similar* benefits by doing C-Header-File Voodoo, and writing lots of proprietary code … but … so much is dependent upon your header files that unless you really know what you’re doing you probably shouldn’t go there.

Another great example is Subversion / Git / CVS / etc metadata: you can attach to each Entity the full Subversion metadata for that Entity, by creating a “SubversionInformation” System / Component. Then at runtime, if something crashes, load up the SubversionInformation system, and include it in the crash log. Of course, the Components for the SubversionInformation system aren’t actually loader yet – because the system wasn’t used inside the main game. No problem – now you’ve started the system (in your crash-handler code), it’ll pull in its own data from disk, attach it to whatever entities are in-memory, and all works beautifully.

Wrapping up…

I wanted to cover other things – like transmitting all this stuff over the network (and maybe cover how to do so both fast and efficiently) – but I realise now that this post is going to be long enough as it is.

Next time, I hope to talk about that (binary serialization / loading), and editors (how do you make it easy to edit / design your own game?).

(17 votes, average: 5.00 out of 5)

Loading ...

Filed under : design, games design, massively multiplayer, programming

123 Responses to “Entity Systems are the Future of MMOs Part 5”

Tom H. October 26th, 2009 at 1:09 am

Until recently I worked for a company that was doing something that I think is close to your ‘NEVER NEVER’ – but it isn’t quite clear to me. Can you expand on that, and talk about why you think it’s a bad idea?

FWIW, their software worked, but there were some concerns about scalability.
Atridas October 26th, 2009 at 9:20 am

I started reading your blog because of the first part of this, and I’m yet interested to see “how it ends”.

So, interesting post :D Someday I’ll put it in practice…
adam October 26th, 2009 at 1:15 pm

@Tom H

If you do that, you no longer have an Entity System, you only have half of one. You *may* feel that you still have the most important half (designers would disagree), but it’s pretty dangerous to do this because it almost certainly means you’ve lost your way, and you didn’t mean to do that.

I highlight it because if you ever notice yourself doing it, you should stop wht you’re doing, and think very carefully about what you’ve just done – if you just did this, you may also have recently broken the WHOLE of your ES.

So, since this series of posts is about how to make and use an ES … don’t do that! :)

c.f. Part 2, where I pointed out how OOP programmers frequently get halfway through building an ES, and then – accidentally! – turn it back into OOP. They they complain that the ES is a lot of effort and doesn’t help much. Well … duh.
Tom H. October 26th, 2009 at 3:54 pm

Ah, OK, here’s the difference:

In my former employer’s system, but trying to use your terminology,

1. Every component can be an assemblage
2. Assemblages mix in components, so there’s an OOP-inheritance-like relation
3. Entities are statically typed to a particular assemblage
4. There is a set of Systems (implemented in C++), but you can also attach scripts to assemblages. Script execution is handled by a System, but this leads to another OOP-like phenomenon, where scripts override or extend one another.

It’s got incredible tool-time flexibility, but doesn’t have the runtime flex that you’re also advocating, and I don’t think it’s engineered to go flyweight or get the other PS3 streaming advantages. Which is a very interesting thought, but I think the architect was coming from a PC/MMO background into a cross-platform situation and didn’t know enough about consoles.

I’m also curious about your apparent flat component model. It seems like there’s a tendency to have rather “fat” components? Their engine has a maybe a couple of dozen component types, a dozen more for physics, a dozen more for the trigger system – but then when you get to gameplay you’re defining custom components right and left to store whatever assemblage-specific data you need for this particular assemblage’s scripted activities.
Adam October 26th, 2009 at 4:39 pm

Ah, I see. Sorry, I haven’t been very clear on those areas.

Entities MUST NOT have their components defined by OOP; an Assembalge is just an uninstantiated / “not yet unified” entity … So the same goes for them.

Entities MUST have their components defined by aggregation – ditto assemblages.

But … Assemblages can also be defined by “nested” aggregation, which is just like multiple inheritance in C++. That’s safe. You don’t want it for runtime entities because it doesn’t help and it makes the performance lower (more queries required to read / write data per entitty).

off the top of my head: I like editors that let you use nesting to add/remove features from an assemblage – but they MUST also automatically unify the sub assembalg with its parent if you remove indiviudal components from the parent that were only there because of the sub.

(eg create Orc chieftain, add the Orc assemblage, and the Human Chieftain assemblage. Now, you want to be able to easily delete all the “humanity” you’ve accidentally picked up, without losing the “chieftainship”).

Or better: make refactoring assemblages easy: the above example would ideally result in creating a new Chieftain assemblage, splitting it out of human chieftain, embedding it back in, and addig it to Orc chieftain – all as a single atomic refactoring.
This leads into a bigger area though: how do you record the initial values for components?

Eg do you allow a component value to be defined relative to oter component values?
Matthew Weigel October 26th, 2009 at 8:28 pm

“(e.g. when creating a new entity, just create a blank entity in the entity table first, and that fast atomic action “reserves” an entity_id; without Table 4, you have to create ALL the components inside a Synchronized block of code, which is not good practice for MT code).”

For some databases, like PostgreSQL and Oracle, you don’t need a table whose sole purpose is providing that benefit – you can use and create sequences of unique IDs independent of columns and tables. I’m curious what performance advantages you’re talking about – it seems that an index on entity_components.entity_id would be sufficient.

I don’t have much else to say on the general subject at the moment – I’ve been giving various approaches some serious thought lately, but I’m not convinced of any of them. :-)
adam October 26th, 2009 at 10:00 pm

@Matthew

Yep, I tried to stick to the most basic form of Relational here – that way there’s less to explain to people who don’t know it already :). Also means it’s easier to conceptually move to [platform of choice] and not worry what built-ins it does/doesn’t have.

re: performance … I just meant that you’re likely to have places where you want to iterate over “all entities” with minimal lookup. It’s convenient to have that as an instantaneous lookup. For instance: if you’re caching entity data, you want to know the complete list of entities, and how many there are, stuff like that.

(incidentally…stuff like caching is likely to end up adding more columns/metadata to the Entity table – but since it’s metadata for activities that live “outside” the Entity System itself, i.e. it’s not game-related, that’s probably OK. I think :)).
adam October 26th, 2009 at 10:01 pm

@Matthew

So … what are these approaches you’re considering, and why haven’t any been chosen yet? :)
Andrew Crystall October 27th, 2009 at 12:16 am

I freely admit you’ve both lost me on the technicalities, and that I’m very interested in the designer tools of anyone doing this…
Andrew Hanson October 27th, 2009 at 1:43 am

First I just want to say entries like this series are why I love reading your blog.

I’m having problem buying/understanding the System division. How do systems communicate between each other? Lets take the Rendering system and the Animation system for example. When something happens and the Animation system decides it needs to move my player form point A to point B, how does the rendering system get this information? Which component inside my player has what information?

The follow up to that (assuming the systems must communicate some how, either though shared data or messages) is how does this scale? Esp once you’ve reached the point where you’d split table 5 onto different servers.
Matthew Weigel October 27th, 2009 at 9:11 am

@adam – the usual suspects, really; binary blobs, entity systems like you describe (and/or simpler ones), and a stronger / more manual approach directly mapping game entities to tables.

I haven’t settled on one because for the last year our team has been focused on prototyping gameplay for something other than your standard Diku/EQ/WoW clone. I was just starting on some aspects of world persistence the last couple of weeks, and now I don’t have to worry about that. A big part of my interest in exploring approaches is balancing production performance and data analysis, and none of the standard approaches do very well at one or the other.
adam October 27th, 2009 at 10:29 am

Where do you find ES to be less appropriate than the other approaches?
adam October 27th, 2009 at 12:31 pm

@andrew

it depends a bit uponhow your rendering layer works, obviously…

Beyond that, it depends how you choose to implement the ES internals of your systems.

I’ll think about this properly, and try to include a better response later, but off the top of my head (nb: I reserve the right to later declare some or all of this stupid and wrong ;))…

For instance: if your renderer requires polygon arrays, then your renderable component may include a list of polygons to render. Your animation system might mutate those polygons directly – either by modifying them in-situ, or by Reading eg a bones list from it’s own component – the animation component – moving the bones, regenerating the entire polygon list, and overwriting it.

The rendering system would pick up the polygons each frame, and probably pick up the world-co-ords too, and translate the polys into position.
Andrew Hanson October 27th, 2009 at 2:28 pm

@adam

So taking the example where the Animation system just modifies the polygons in the Render component. How is this best handled when the two systems are running on different servers? (I know its not likely that a desktop game with an Animation and Rendering system would be large enough to take two servers, but the problem still holds for larger Systems that need to interact)
Aengus October 27th, 2009 at 6:27 pm

@Andrew

This is just a thought but one way this might be done is if all of the different Components are stored on a separate Database Server and all of the System Servers are connected by a message system then the animation process could work like this:
First, the Animation System Server activates it’s Animation and updates the Animation Component on the database.
Then it sends a message to the Render System Server that a animation has been activated (this message would probably hold things like the Entity-id).
And finally the Render System Server will use the information in the Animation Message to fetch the animation data from the Database and use that during render time.
Hope this helps.
Arni Arent October 27th, 2009 at 9:56 pm

Good article!

Maybe you can readdress this issue that has been nagging me:
How do systems interact with each other?
Are the systems oblivious of each other?

Let’s imagine two systems:
“ShootSystem”
“RegenerateHealthSystem”
- But the condition of some game unit (tank) is that it cannot shoot while regenerating.

1. Can I not make the ShootSystem somehow query the RegenerateHealthSystem if that entity is regenerating?

2. Should these two Systems actually be one System? Something like “RegenerateHealthAndShootingSystem” ? That doesn’t make sense to me, I thought you said this was supposed to be modular!? :)

3. If these systems can communicate, then what is the significance of this when you have each of those systems running on it’s own server?
Jason Beardsley October 27th, 2009 at 11:05 pm

Good stuff. A couple of random comments:

1) If you’re willing to live with the limitation that a given entity can only have a single instance of any specific component (which may not be a limitation at all, depends on design), then the component_data_id columns are unnecessary, and the component data tables would use entity_id as their primary key.

Even if you really needed multiple components of the same type attached to an entity, I don’t know if I’d necessarily use a separate id space to store them. An “instance_id” column (unique only to a given entity_id) would also work, and component data tables would have be keyed on entity_id + instance_id.

Why do this? Well, I think it’s clearer to have component data indexed on entity_id anyway, but it also means that component data tables will be sorted based on entity_id, which is a good thing for performance.

2) I generally prefer to not store table names in other tables, e.g. the components.table_name column, and would opt for an algorithmic method. My knowledge of SQL is far from complete, but as far as I know, you can’t get a column value and then use that value in a later “static” query as the table name. Instead you have to build the query strings dynamically, which reduces performance (or at least, forces the database to re-analyze/optimize such queries).

Hard to describe exactly what I mean, but let’s assume that SQL included a unix-like backtick operator. The following is what I’d theoretically like to write, but (to my knowledge) cannot:

SELECT * FROM `SELECT table_name FROM components WHERE component_id = $2` WHERE entity_id = $1;

The above query is meant to return (arbitrary) component data, given an entity id and a component id. Then again, I’m not sure how the db optimizer could do much with the above, even if I could legally write it.

I guess my main issue is that I prefer using stored procedures – exclusively – from the application, for a number of reasons. I don’t know that I could do that with the “dynamic” table names utilized in your approach – it would depend on what the database procedural language supported.

And yes, using “known” table names does mean that entity loading queries would hard-code those tables (perhaps in a huge join, even), but in the interests of performance that is something you may ultimately find necessary. I look forward to the next article in this series, and how you might avoid the “N + c database queries for an entity with N components” problem.
Matthew Weigel October 28th, 2009 at 3:12 am

@adam: entity systems are really awful at data analysis, because so little of the data model is encoded in it (arguably the data model is just very loose, but the point is that at launch time the game has a data model implied by the data that is more concrete and useful than the entity system’s). They are also not very impressive from a performance standpoint, with lots of iterated queries just to build one assembly.

@Jason: Transact-SQL has ExecuteSQL(), which is about as bad as you imagine. I saw a case where it was useful – an automated database update script had some lines of SQL that referred to non-existent tables and columns if that portion of the script didn’t need to be run. The script was smart enough to check for that condition, but if you left the ALTER TABLE commands as bare SQL they’d be parsed and SQL Server would error on them before anything was actually run. Wrapping them in ExecuteSQL() let us check for whether they were going to be valid, and THEN get the database to parse/execute them.

But yeah, it’s the database equivalent of self-modifying code, with equivalent big angry warning signs flashing around it. :-)
adam October 28th, 2009 at 1:08 pm

@Jason

re: 1 – good point. As you say, assuming only one instance of a given component is needed per entity … I think that’s a better way of handling it.

re: “N + c database queries for an entity with N components” problem.

…each system is typically only going to need access to a handful of components. If you need all the data for an entity, it’s either for offline / human-readable purposes (load it on screen to show to a player), so that latency isn’t an issue any more, or it’s as part of a mass serialization, in which case you’re probably going to sidestep it and just start dumping tables en masse.

Ideally: rather than fetch entities on-deman, you pre-select all the data needed by a particular system for ALL entities (should be < 10% of the total data of all entities – as per above, you only pick up the components that this system actually uses), and pass it that as input parameter on each tick of the game-loop. There are obvious optimizations here, e.g. splitting that pre-select into batches, but the use-cases for how much you want to pass to the system are going to depend a lot on where / how you're using the ES (is this a PS3 client? a multi-server cluster? etc).
adam October 28th, 2009 at 1:18 pm

@Matthew

I might have misunderstood what kind of data you’re talking about, but…

“entity systems are really awful at data analysis” – I’d argue the opposite :), although I agree that what you say is true “by default”.

The difference for me is that any real world ES is going to fall flat on its face as a data-maintenance nightmare if *all* you implement is the runtime system. Practically, you *must* build toolchain features / plugins that support the ES directly in your game-editors.

As soon as you do that, all the data model is once again back – only this time it’s even more explicit (and easier to read and write) than in the alternatives (such as OOP): it’s purer, because the *only* thing affecting the structures you have is the metadata you added to track those structures. In, say, traditional OOP, you’ll “additionally” have occurences of subclassing and non-subclassing that had to be done for reasons other than the core data structure.

FYI – in my earlist conversations with Scott Bilas about ES’s for Dungeon Siege etc, IIRC … he picked on the editor-support as the biggest single win/lose criteria for an ES being deemed a success at the project / final game level. For offline games like DS, the ES being implemented well or badly alters the output of the progrmming team, and the final runtime performance and bugginess. BUt the editor support alters the volume and quality of game-content – which for an RPG (obviously!) is the biggest single factor.

So … yeah … very important topic. One I haven’t gone into in any detail (yet), because with modern games (larger scale and/or multi-threaded and/or online), the ES implementation itself becomes a lot more critical. And it’s easier to screw up, IMHO :). But one I certainly want to cover in detail at some point – although I have less knowledge on how to do that part “right” – I have ideas and wishes on how to do it “better than I did before”, but haven’t tried them out yet.
Jason Beardsley October 28th, 2009 at 7:16 pm

@Adam

As usual (since it’s all I do), I’m considering how these systems work in an MMO, and in particular on the server side. Loading all (dynamic) entities, even into a caching server, is probably not feasible. (Though for entity templates, this is exactly what you’d want to do.) And the typical MMO architecture has a “world simulation” process that needs access to most, if not all, components of an entity, so partial entity loading isn’t applicable. So, it’s vital to be able to load an entire entity from the database in an efficient manner. Making multiple queries, one per component, does not strike me as satisfying that goal.

In a previous project of mine, we faced this issue. It turned out, after much experimentation, that the fastest way to load an entity from the database (MS SQL Server, fwiw) was to do a join across all of the component-specific instance data tables. If an entity didn’t have a row in one of those tables, then the result would have NULLs in those column positions, which had to be handled appropriately at the application level. Not pretty, and definitely not the easiest thing to maintain, but sometimes that’s how it goes.

It’s worth noting is that we only had about 5 components that had any instance data; add a few more, maybe a different approach would be better. If I were starting with a new technical design, I’d probably stick with a simpler method (i.e. query each component table individually), until it became necessary to improve performance.
adam October 28th, 2009 at 10:53 pm

@Jason

“the typical MMO architecture has a “world simulation” process that needs access to most, if not all, components of an entity”

If it’s working correctly, I don’t see how that would happen?

If you’re accessing most/all components of the entity, then you’ve failed to build ES Systems – instead, you’ve built a monolithic “doEverthingPlease()” method.

Why would you do such a thing? What in your simulation algorithm reqires doing everything in a single process in a single place?
Matthew Weigel October 29th, 2009 at 12:48 am

@adam: re: what Jason is talking about, yeah. The problem – and this is where e.g. Darkstar falls down too – is that pretty much every piece of data has a good chance of affecting any particular step in the simulation. Are you at least level 35? Do you have a frobitz in your inventory that gives you a bonus to [whatever is happening]? Does your guild have a special power that gives you a bonus? “Relevant stats” at every step of the simulation easily balloons over time until any hard and fast demarcation you make… fails.

It is, in my opinion, a matter of designing the architecture of the game to fit the designers’ and players’ vision as much as possible, rather than restricting the vision to what architecture you have. The latter certainly comes into play quite a bit, but I think our job as programmers is to fight against it as much as possible… which is the challenging part that makes being a game programmer so fun and fulfilling, personally.

Now, re: data analysis on the entity system through tools, that’s precisely the same analysis model as e.g. binary blobs. Except that binary blobs are ridiculously fast to store and load compared to an entity system, require much less database hardware, and so on.
Jason Beardsley October 29th, 2009 at 5:41 pm

@adam

Just because there is a world simulation process doesn’t mean that the individual game systems within that process aren’t largely independent, and only work with a subset of entity components. For example, the movement system should (ideally) only need to know about an entity’s physics model – collision geometry, position/orientation, velocity, etc. But, if you have decided to distribute the server load based on geography (e.g. one server handles all processing for a portion of the world), you still end up with a process that needs the whole entity.

And, as Matt pointed out, those crazy game designers have ways of creating systems that do require access to numerous components. Combat systems, in particular, tend to grow in that direction. (For this reason, I’d argue, using geographical distribution is still the “best” way to spread the load on the server…but that is a whole other discussion.)
Olivier October 29th, 2009 at 7:00 pm

“If you’re working on MMO’s, you should be using SQL for your persistence / back-end”

Should we? Don’t you think it depends on how one wants to store and navigate one’s data? SQL/relational isn’t the golden hammer. I wouldn’t dismiss other storage/databases solutions (object, map/reduce inspired systems), especially for a game object entity system before thinking about a game’s specificity.
Olivier October 29th, 2009 at 7:06 pm

… Or maybe do you mean “you should” in the sense that “most of you are”?
adam October 29th, 2009 at 7:48 pm

@Olivier

Yes – sorry about that – I meant:

- EITHER: you’re already using it, or have used it, and at least know how to use it (most people)
- OR: you’re not using it – but that’s because you’re not using ANYTHING yet, you haven’t got that far (you probably WILL end up using it, and if you’re this new to MMO’s, you really SHOULD use it)

For the record: Alternative persistence backends are a lovely idea and I’m all for them.

OTOH … at AGDC, on our panel, Marty was claiming that only map/reduce was worth using, because “every other DB backend is too slow at writes” … and my response was twofold:

1. Use more functional code
2. Use Entity Systems

I believe you can make a lot of better-known techs (like OOP and SQL) work “well enough” if you modify them by using an ES. I also believe that if you really know what you’re doing, and you can find a robust enough language, and you coudl find qualified programmers, you’d write everything on the server in a functional language. But I also consider that totally unrealistic for 99% of us.

(if anyone tries it, let me know – I’ll be cheering for you, but don’t ask me to *bet* on you ;))
adam October 29th, 2009 at 7:58 pm

(Incidentally: 27 comments and still going … this is a big part of why writing anything declarative about ES design takes a lot of time and effort: so many details and edge cases to think about, that it’s worth thinking about, before you open your mouth and say something that implies stuff you didn’t intend :))

@Jason

I have no problem with a single process having everything – but it never needs it all at once. It only needs it sequentially – bringing me back to: when would you ever need all that data at once? (other than for data-dumps to go to a foreign system – offsite backup, etc)

[actuallly, I do have a problem with monolithic processes – unless it’s got a decent built-in scheduler and built-in threading system, and is micro-managing a load of mini-processes, which would be fine by me. But at that point I’d still like to know what was so wrong with the various OSS and commercial scheduling systems that you felt “I can do better; writing a multi-threaded OS ain’t so hard”; are the alternatives unusable?)
Olivier October 29th, 2009 at 8:14 pm

@adam

“I also believe that if you really know what you’re doing, and you can find a robust enough language, and you coudl find qualified programmers, you’d write everything on the server in a functional language.”

Would be nice if you could elaborate on that… One of these days when you get the time.
I think one of the main challenges of that is the use of external libraries and frameworks; F# should be faring better than erlang in that matter for instance… But most of the great frameworks one could use for an MMO backend I could think of are either exclusive or available on the java platform… Is there a solid functional language running on the Java platform?
adam October 29th, 2009 at 8:23 pm

@Matthew

“data analysis on the entity system through tools, that’s precisely the same analysis model as e.g. binary blobs. Except that binary blobs are ridiculously fast to store and load compared to an entity system, require much less database hardware, and so on.”

You’ve lost me here. Maybe I’m thinking of a slightly different concept when you say “binary blobs”, because the blobs I’m used to do not exhibit the features you describe.

Firstly, access:

A. ES’s provide direct access – and direct cross-entity joins – to all data.

B. Blobs literally *prevent* direct access – and all cross-entity joins.

The data analysis you can do offline on ES’s can also be done online – and it requires zero setup. To do the same analysis on blobs requires vast amounts of pre-processing (counted in hours or days to convert from the blobs into a usable format)

The inability to join across entities/game-objects is the main reason I would avoid blobs here – IMHO it’s like cutting both your DBMS’s arms off, blinding it, and then asking it to carry a cup of water without spilling a drop.

Secondly, speed:

If you have a crap DBMS, with poor locking (e.g. MySQL about 10 years ago), then yes – blobs significantly increase write speed by circumventing lock-management / consistency maintenance (at the expense that you’re not allowed any strong references between objects; ouch – that causes more performance problems elsewhere). But if you’ve got a good DBMS, circa 2009, that shouldn’t be the case.

You get some minor speed increases because you don’t have to do any CPU work. But you also get some minor speed decreases because you have to send vastly more data than you actually wanted / needed – and you cannot start processing on the received data until you’ve received the lot (unless you want to write some clever code to change your entire semantics of accessing data in RAM).

So … where am I going wrong here? Have there been some big changes in blobs in recent years that passed me by? (oops)
adam October 29th, 2009 at 8:25 pm

@Olivier

Side-effect free
Trivially (and automatically!) parallelizable
Trivially (and automatically!) cacheable
No more dupe bugs

What’s not to love? ;)
Olivier October 29th, 2009 at 8:27 pm

@adam

Sure… But… That means closing the door on a lot of nice/essential things….
Cameron October 31st, 2009 at 3:19 am

Hey Adam,

I’ve read and enjoyed all of the previous posts on Entity Systems (including yours), and they all seem to be targeted towards end-users / clients that run ON fast hardware, e.g. PS3, or gaming PCs running games coded in C/C++.

I know very little about the Cell Architecture, except that it is much faster than a PC, and probably an order of magnitude faster than a mobile device (like the iPhone). So my question is, what do you suggest for slower devices/platforms; mobile, flash, java?

For instance, flash isn’t multithreaded…would you simulate threads for the Systems? Also, forgive my naivete, but don’t languages like Java and AS3 *force* OOP?
adam October 31st, 2009 at 7:16 pm

@Cameron

Beneath all the other reasons for using an ES, the core reason is simply “to do things in an OOP environment that we cannot otherwise do” – so Java and AS3 in no way undermine usage of an ES.

Well … except that without access to C/C++ structs, it’s hard to do the “streaming” part fast. Various people have reproduced structs in Java (it’s only mildly tricky), and it works fine now that java has direct (low-level) access to RAM. I have no idea what equivalents are available in AS3 today. You could still use an ES without that, it’s just that *some* of the performance improvements would no longer work.

re: mobile…

I’ve been concentrating on “big” platforms because … well, actually … I’ve been concentrating on “multiprocessor” platforms, where (generally speaking) you want to do as much data-driven programming as you can, and where some of the side-effects of ES’s have no negative side-effect on performance (because they’re “multiprocessor friendly”).

Again, there’s no particular reason to focus on those platforms: I got started with ES’s purely for the improvements they bring to the implementation of game-logic. That’s universal.

But I always intended to veer these blog posts towards “not ONLY are ES’s a better way to code, BUT ALSO they give you higher performance (in certain types of game)” – hence the title ;). A lot of techniques in game development make development easier/faster, but slow down the runtime game a lot. It’s nice to occasionally find some that are easier AND improve performance.

(NB: that’s not the only reason for the title)
Valentin November 7th, 2009 at 5:07 pm

Thanks for the articles, ES is really interesting topic and it has its points. But I fail to understand how “big” are system. I mean can one system operate on many components? If they can, how many? If I have a tank for example and two systems: shoot system and damage system, the first one needs access to data of the later one obviously. How should I handle that?
adam November 10th, 2009 at 9:54 am

@Valentin

Ideally, as many systems as possible. That makes each system as small as possible, and as easy to edit + maintain + improve as possible.

…but use discretion, obviously.

In the end, *depending upon hardware/OS*, additional systems carry little overhead. At some point, adding X additional systems adds signficant overhead as the cost of iterating over “all entities” N+X times rather than N times becomes significant.

Generally, though, it’s not something to worry about: just have lots of systems now, and merge them later as/when necessary.

NB: one system certainly *can* act on multiple components, in fact most systems *must* act on components from other systems, although often only in a “read only” capacity.

(e.g. the rendering system needs to read the world-co-ords for each object, and add that to the local-co-ords for each polygon, in order to render everything in the right place on screen :). The world-co-ords are being maintained by another system, one to do with movement, but the polygon co-ords are being maintained by the rendering system. Or the animatin system, if that’s a separate system (which it probably is))
Edward November 15th, 2009 at 12:42 am

In the database, the ComponentData tables would, I assume, contain columns of numerical, datetime, string and binary data. These columns would be named appropriately and everything would look perfectly normal (no pun intended).

When we cross over to our application code, typically in an OO language, the component data is the point at which your entity model loses its generic one-size-fits-all structure. Whilst the core entities and components remain 100% dynamic, is there a point at which we transition to a more static data model, in particular for ComponentData?

I would expect code within a GunComponent to be able to call…

if (data.GunRate > 1000) {

… or …

data.GunName = “Colt”

… accessing the data via strongly typed accessors rather than a bag of string data. Perpetual casting and parsing of string data would otherwise cause a hefty performance hit.

This is a general question, but inspired by the following line from your most excellent (dude) article:

float[] gunData = getComponentDataForEntity( GUN_COMPONENT, new_id );

^ So does getComponentDataForEntity always return a float array?
adam November 15th, 2009 at 1:59 am

@Edward

The facetious answer would be “use structs”.

My preferred approach is passing around big streams of byte data, and using a single instance of a (pertinent) struct to view that data one record-at-a-time from the OOP code. This allows for query-system-friendly “ask for everything you need at once, then slam it down the pipe in a single batch of bytes” … while also providing sane access semantics.

I tried googling for a link to a technique for a more comprehensive answer, and then found that the terminology I’ve been using for ten years apparently doesn’t exist. I’ll get back to you on that one :(.
zuii February 6th, 2010 at 12:42 pm

Any chance of having a part 6? I would be interested to see how an editor for this would look like.
jay February 6th, 2010 at 8:28 pm

Thankyou for your really cool series of articles… I really like the concept but one thing bothers me that concerns with how subsystems communicate with each other.

for eg – A simple game with 3 subsystems called renderer, physicsSim and input. The game has 2 entities floating about. Both entities have a physics component, a renderable component and input component set up in the database.

The work the subsystems do is just the renderer just renders the renderable components, the physics just simulates the physics components and the input just updates the input components with the state of input. But where is the part that the input subsystem has to tell the physics system to apply forces based on input or the physics system to tell the renderer the transform to render at.

Is it ok to do a querry in the renderer to look for a physics component with the given entity id to get the transform to render at, if so I cant see the benefit and think this will cause threading issues or am I just still thinking in the wrong way to solve this problem?

Many thanks and I look forward to the next article.
jay February 6th, 2010 at 9:14 pm

I should of read the previous posts before posting since I think my question is answered already.

Thanks again.. both article and following discussions everyone.
Das Gamesystem-Modul « Zeitalter3 – Browsergames Entwicklerblog February 14th, 2010 at 9:42 am

[...] Gamedata-Klasse nimmt sich die Datenhaltung eines Entity Systems zum Vorbild und implementiert diese in PHP. Für Details und Erklärungen der Funktionsweise [...]
Datenhaltung mit gamedata.class.php « Zeitalter3 – Browsergames Entwicklerblog February 16th, 2010 at 6:50 pm

[...] Klasse zur Datenhaltung (gamedata.class.php) geschrieben. Diese ist in Zeitgeist angelehnt an die Datenhaltung eines Entity-Systems. In den Kommentaren wurde mir von Gameplorer folgende Frage zu dem Prinzip gestellt: Wie gut [...]
ConcernedFan February 17th, 2010 at 8:18 pm

We want part 6!
Rick B June 8th, 2010 at 4:10 am

I know this is an older post but I’d thought I’d ask anyways. We have our ES system running fairly well and it’s very similar to what you have described even though I did not find your article until today. Our components are simply containers of data and are stored as separate tables in SQL Server with one table per component type.

We are able to load all the entities and populate the objects in memory by using projections. This all works great and is fairly efficient in terms of the queries involved (one per component table).

The tricky part, and the area I need some help with, is how to persist the data as it changes during the running of the game loop. For example, we have about 100,000 location components with x, y, z coordinates that are being updated once every ten seconds. Trying to save this data back to SQL Server is painfully slow. Serializing to disk is fast but has problems when you want to change the structure or perform maintenance on the data.

How are people handling persistence?

Has anyone used any of the main memory databases for their objects?

Thanks,

Rick
adam June 8th, 2010 at 8:16 am

@Rick

NEVER write-through live data to an SQL db, it’s generally far too slow. Instead, send it to an in-memory SQL db of some kind (there’s many available, from free to commercial). They work very well as an “intelligent cache that speaks SQL directly, and avoids you having to re-write you code to speak to the cache instead of speaknig SQL”.

If the in-memory DB”s you can afford are still too slow, get hold of an even more lightweight/minimal SQL implementation (or roil your own), and again run it purely in memory. Or try running it off a RAMDISK, and setup your “real” db as a replication slave or similar.

In general, the db features you need to run the data live are a small subset of the db features you need on the persistent copy of the data. There’s no reason why you have to run the same DB server (or even same DB vendor!) for the two different copies…
adam June 8th, 2010 at 8:18 am

(FYI I recently wrote an ES for Android mobile phones; I didn’t run SQL at all, I just approximated it with a lightweight in-mem abstraction layer. I can then serialize that to flash-mem using SQLite as an entirely separate chunk of code. With the new version of Android, I might try running it through SQLite as well – new Android is allegedly up to 5x faster than old android – could be intereting to try)
Rick B June 9th, 2010 at 5:11 pm

I have been looking at the various MMDBs but the choices are fairly limited for my platform (.Net & Windows Server).

We’re also trying to work out efficient ways of messaging the entities as they need to react to changes. We have hundreds of thousands of objects so we need to be careful.

Thank you for your thoughts.
.
Abhijeet P June 17th, 2010 at 5:32 am

@Rick B: Figured I’d see you here. I wonder if you have looked at MongoDB. Seems pretty blazing fast since it runs in memory using memory mapped files:
http://blog.mongodb.org/post/101911655/mongo-db-memory-usage
adam June 17th, 2010 at 9:40 am

We often see people trying to.do.an.ES and an event-driven system together, at the same.time. I’m usually not sure why people do this. I’ve no idea about your case, but in most cases it proves too much change at once – that’s TWO unusual paradigms that a team is new to, each with complex.performance and behavioral changes of their own.

Im not convinced ES and Event driven systems go.well together for”normal” situations.. e.g. They could work extremely parallel (both approaches are.inherently parallelisable), but few people have enough processors to make that worthwhile these days.

Out of interest, why are you doing both together? Where is it working? Where is it causing yiu headaches?
Nash July 13th, 2010 at 8:49 pm

I know I’m late to the part, but there’s a few bits that I’m still struggling to wrap my head around:

In the scope of an RPG, how would things like items be handled? For example, if we have a potion that restores health, is this an entity, or an assemblage that is instantiated when a character picks it up? I think it’s the latter, but I’m still not sure.

I imagine when the game loads its content, it would load a list of definitions for various items, and then create them as required. So a character would have an InventoryComponent object that would just list the ID of any object entities.

Basically, I think I’m having difficulty visualing the line between entities that do something (monsters, treasure chests, event trigger zones) and entities that describe something (like potions, swords etc).

I think I answered my own question, but any advice would be very welcome.
adam July 13th, 2010 at 10:17 pm

Obviously, this depends on the rest of your game – it depends on HOW MUCH DETAIL you choose to put into your different systems.

Off the top of my head, I’d start with:
– entity, made up of:
– position component
– one-use-special-action component, with an enum type with value of “HEAL_POTION”
– inventory component (as in: when it’s in your inventory, it has a component that gives a bitmap you can display, and says how much inventory space it takes up, is it stackable, etc)

Although, in some RPG’s, the potions design is quite complex, so you might well choose to split-out the “potion” bit into mulitple components, e.g. a single “item-use” component, speciying that it’s “one-use” and has “instant-effect” PLUS a set of “item-effect-component” instances:
– alter-stats component, with values “HEALTH” and “50″
– alter-stats componeont, with values “STAMINA-PERCENT” and “100″
– alter-stats-temporary componeont, with values “MAX-HEALTH” and “10″ and a variable “Fade-time” of value “60 seconds”
– …etc

(As much as is necessary to make the implementation of YOUR game’s particular potions-logic easier to manage…)

In an RPG, that split of components also starts to make it REALLY easy to create – say – a new magic sword that has a side effect of increaing your max-health; it’s actually re-using the componone from a heal potion.
adam July 13th, 2010 at 10:18 pm

@Nash – your question in your second-to-last paragraph suggests:

“I’m thinking too much”

Don’t. There is no such difference. Entities are just data. Nothing more. There is no “do something” part to an entity. You are thinking of OOP. Stop thinking of OOP. ES’s want your life to be simpler … don’t complicate matters, just let them be as simple as they possibly can be :).
Nash July 14th, 2010 at 1:12 am

Thanks for taking the time to reply, I really appreciate it.

These systems always look so elegant on paper, but I’ve always struggled with the implementation. I’ll take the simple approach and see what I come up with :)
Neo Liang August 7th, 2010 at 2:10 pm

it ‘s same that you just talk about the store and initialize the entities,but how those entities interact with each other? does you use the system based on message or something else?
adam August 7th, 2010 at 2:46 pm

@Neo

Think of an entity as a dictionary of arrays (each array being a component). In most programming languages, implemented efficiently, that’s all it is.

A dictionary doesn’t “interact” with anything, it just sits there. You write other code that gets given refecnes to multiple dictionaries, and operates upon them. Your “other code” is the System(s). The entities just sit there, doing nothing.
Neo Liang August 8th, 2010 at 2:26 pm

thanks for replay, you said just “other code” do the interactivity.but
1.how did the system work?
2.is every component has a specific system for it or there is a public system for all components to handle the interactivity.
3.how does the system collaborate with each other?

and i have other more questions:
1.how to reuse the assemblage?
2.how to reuse the components as we want to make one new component with the old one?
T=Machine » Entity Systems are the future of MMOG development – Part 1 August 30th, 2010 at 9:07 pm

[...] Part 5 – initialization, storage, and data [...]
Barakat September 1st, 2010 at 3:34 am

“If you know the entity-id, you may only need one table lookup to get the data for an entire component”

I have a work in progress ES based off of what I learned from your article. I am trying to make a function that loads an entity from the database including all its component data. My question is, can you give an example of how to do this with the least number of queries?

I have a table for each component’s data, with the table name being in the “component” table. Right now I have it so it does one query to get a list of the component IDs and their table names, and then it does a query for each component table to get all the data.
adam September 1st, 2010 at 7:44 am

You’re into the realms of SQL optimization with that question…

My flippant comment was assuming that you chose a deterministic naming scheme for your components, so that e.g. for component “Position” you knew (without having to look it up) that the table name would be “CPosition” (or maybe “Component_Position” or whatever).

Component-class names are going to be unique anyway (whether you “name” them with strings, or with integers, or some other scheme) – so you should in theory be able to invent a DB schema that avoids looking up the table-name for each component.
Mirv October 22nd, 2010 at 12:10 am

Hi Adam,

first of all, fantastic post. My profession is in developing web applications with database backends, which I’ve been doing for a while now (too long perhaps?) About a year ago I started my first serious effort at game development. I’m working in C# and XNA, and I intend to release the game under Xbox Indie Games (but just a single player experience, not an mmo).

Anyway, I’ve been attempting to research as much as I can regarding entity/component systems. In addition to your article, other articles I consider ‘top articles’ are: Mick West: Cowboy Programing – Evolve Your Hierarchy and Marcin Chady: Theory and Practice of Game Object Component Architecture (GCD Canada 2009).

Chady’s design uses the terminology “Behavior” for the logic part of the Entity, and “Attribute” for the data part of the Entity. I’ve adopted this design approach in my game. In my model, an Entity is represented as an EntityKey with a unique value. Each Behavior has an associated list of EntityKeys. For an Entity to adopt a certain Behavior, it registers its EntityKey with the given Behavior, adding it to the associated Behavior-EntityKey list. When processing its EntityKey list, the Behavior’s logic can reference one or more Attributes.

Taking inspiration from your article, I’m treating Attributes as records in a list (emulating something like a database table). The Attribute lists are keyed with an EntityKey. An When the Entity Factory creates an Entity, it looks at the Behaviors the Entity needs and then creates ‘records’ in the appropriate Attribute lists.

Also taking inspiration from your article, I looked at using LINQ queries on my Attribute lists. The version of C# used in XNA supports LINQ-to-Object queries. Apparently LINQ queries are too expensive to use in the game loop, but I have experimented with using LINQ queries for initializing level data behind a ‘loading’ screen.

Coming from the domain-driven-design philosophy we employ at work, I have run into a quandary in the above entity/component design. I must first mention that I am by no means an expert in software engineering. And I find domain-driven-design is not a panacea, and it may very well be ill-fitted for designing game architecture. Martin Fowler’s ‘anemic domain model’ makes an argument that, in migrating all the logic out of the entity, we are loosing (all) benefits of object oriented design. Though I do see myself continuing with the above entity/component design, the trade-off in separating logic from data makes me wonder about the nature of steering away from object oriented design in the implementation of game entities.

I guess there was no real question in my post, just wanted to say keep up the great work, and I found your article very enlightening.
ConcernedFan November 2nd, 2010 at 3:40 pm

Part 6 please! :)
Francois December 1st, 2010 at 6:46 am

Hi Adam,

I really enjoyed this series of articles and I did go through them a few times. It’s interesting to note that the comments and your answers really add a lot of value to the posts!

I think a lot of us would love to read the next article :)

I’m starting to write a text based mmorpg on my free time (a bit different from my .net consulting day job but it’s fun!) and I’m really excited about implementing an entity system that will be used by the server C#. The backend storage is MSSQL.

1) One of the thing that I cannot yet get my head around is how to respond to individual network requests efficiently using the ES. I think I understand how to system process entities in batches inside the server game loop but this seems a bit different.

What I mean is that when I get a request from a client saying Entity X is attacking Entity Y, I might have to do security checks, procces the fight and then send back a lot of info like the outcome, XP gained, eventual loot, remaining health, level up info, …

I may have multiple of these requests per seconds and I’m afraid that since most of this data might be scattered in a lot of different components, it might be slow. Do you see a single System in charge handling these requests, performing the action and returning the results + the additional data. I’m afraid that’s a lot of responsibility for just one system.

2) If I query/update mssql in realtime, I don’t have to worry too much about multithreading but if decide to have in-memory representation of the data, more challenges appear: synchronize access to the data (game loop systems vs user requests), consistency and also probably persistence.

3) I’m not very clear on how I can best use the ES to implement of containers (inventory has bags, bags have items, chest contains loot, …), do you have any suggestion that could get me started?

4) Any objections about having some systems running stored procedures to update some data directly in the DB? This could possibly prevent fetching + storing + saving.
Eldad February 25th, 2011 at 3:24 pm

Hi Adam,
The articles are great. I am looking forward for the next article – hopefully you will give us some more information about AAA performance and use cases.
Meanwhile, I’m wondering about the implementation of a collision detection system.
We will need a:
collisionDetectionSystem
collisionDetectionComponent (x,y, deltaX, deltaY, velocity, radius etc)
Entity
Let’s say that we have 1000 entities, all of them are bind to collisionDetectionComponents.
Assuming that we need to perform a simple sphere test, how would you suggest of checking if there was any collision between the objects?
IMO, since the entities are stored in a list, we will need to check them against each other (1000*1000) – which is bad.
Another suggestion is that the collisionDetectionSystem will have a 2-3D array/hash table for the X.Y,Z and on-top have linked list to hold more then 1 entity on the same X,Y,Z (array[x][y][z]->entitiy_1->entitiy_2)
Now, we can check for collision only on specific areas and reduce the amount of checks.
Any idea?
Thanks,
Eldad
adam February 26th, 2011 at 12:53 am

@Eldad

Just use any standard CD system, anything you find on google. The ES doesn’t change the algorithms you use, it just changes how the data is stored and fetched
Eldad February 26th, 2011 at 12:54 pm

@Adam
I asked that question in order to get a better understanding on how components interact with each other and how to deal with systems that need to consolidate the component data for better performance.

Problem:
Let’s image that we have 4 components that need to interact with each other:
renderComponent – render enmities.
invisableComponent – if the entity is on the move, its visible, else invisible.
collisionDetectionComponent – check collision between entities
movementComponent – used to me the entity

In other words, the interaction between the components is:
movementComponent –> invisibleComponent -> renderComponent
-> collisionDetectionComponent

In addition, the collisionDetectionComponent will have better performance if it will hold the position of his entities rather then comparing each entity to each other.

Solution I:
http://cmpmedia.vo.llnwd.net/o1/vault/gdccanada09/slides/marcinchadyGDCCanada.ppt
(Slides #17 and #21) A solution can be messaging between the entities; we send a message to entity and let it decide how to deal with it.
This will get dispatched to all systems that are interested in attach messages.
void HealthBehaviour::OnMessage(Message* m)
{
switch (m.type)
{
case APPLY_DAMAGE:
Attribute* healthAttr = GetAttribute(HEALTH_KEY);
healthAttr->value -= m.damage;
if (healthAttr->value SetLogicState(DEAD);
break;

case ATTR_UPDATED:
if (m.key == HEALTH_KEY)
{
Attribute* healthAttr = GetAttribute(HEALTH_KEY);
if (healthAttr->value SetLogicState(DEAD);
}
break;
}
}

Solution II:
(Slide #20) Another solution is a onUpdate( pass, delta ) function on each system, every time that a component get changed, the previews and current will be sent to the different systems (that have component attachment between the entity and system):

GameObject::OnUpdate(pass, delta)
for b in behaviours
b.OnUpdate(pass, delta)

I can see some problems by using those techniques:
1. Performance issue
2. The systems behaviors should be built with consideration with other system – something that we shouldn’t do in ES.

Any idea what is the best approach for the problem described above?
Eldad February 26th, 2011 at 1:03 pm

@Adam, please ignore the previous post, it got submitted by accident.

I asked that question in order to get a better understanding on how components interact with each other and how to deal with systems that need to consolidate the component data for better performance.

Problem:
Let’s imagine that we have 4 components that need to interact with each other:
renderComponent – render entities.
invisableComponent – if the entity is on the move, its visible, else invisible.
collisionDetectionComponent – check collision between entities
movementComponent – used to move entitles.

In other words, the interactions between the components are:
movementComponent –> invisibleComponent -> renderComponent
-> collisionDetectionComponent

In addition, the collisionDetectionSystem will have better performance if it will hold the position of his entities and compare only those that close enough – rather then comparing each entity positionComponent to the other entities(n^n).

Solution I:
http://cmpmedia.vo.llnwd.net/o1/vault/gdccanada09/slides/marcinchadyGDCCanada.ppt
(Slides #17 and #21) A solution can be messaging between the entities; we send a message to entity and let it decide how to deal with it.
This will get dispatched to all systems that are interested in attach messages.
void HealthBehaviour::OnMessage(Message* m)
{
switch (m.type)
{
case APPLY_DAMAGE:
Attribute* healthAttr = GetAttribute(HEALTH_KEY);
healthAttr->value -= m.damage;
if (healthAttr->value SetLogicState(DEAD);
break;

case ATTR_UPDATED:
if (m.key == HEALTH_KEY)
{
Attribute* healthAttr = GetAttribute(HEALTH_KEY);
if (healthAttr->value SetLogicState(DEAD);
}
break;
}
}

Solution II:
(Slide #20) Another solution is a onUpdate( pass, delta ) function on each system, every time that a component get changed, the previews and current will be sent to the different systems (that have component attachment between the entity and system):

GameObject::OnUpdate(pass, delta)
for b in behaviours
b.OnUpdate(pass, delta)

I can see some problems by using those techniques:
1. Performance issue
2. The different systems should be able to interact with other system – something that we shouldn’t do in ES, each system should be independent.
3. Can cause to infinity recursion.

Any idea what is the best approach for the problem described above?
Adam February 26th, 2011 at 8:02 pm

Neither – that’s an incompatible design, that loses most of the benefits of the es I’ve described. Entities are data only.
Jacob March 1st, 2011 at 7:40 pm

Why exactly do you feel that ES and Event-Driven messaging systems are incompatible? It seems to me that by using messages to handle the communications between multiple Systems/Components (in case your ES implementation allows Components to have methods, like described below), you abstract from the identity of the other Components of an Entity, and just send messages to anyone interested in them, and receive messages of the types you registered for. This results in truly individual Components which can easily be swapped out for others with the same functionality but with, for example, different attributes.

In the slides linked above by Eldad, Components have data AND methods, namely onUpdate() and onMessage(). You would use the onUpdate() to do tasks that are required in each frame, something like Systems would do by applying operations on all their Components. You would use the onMessage() function to handle messages you registered for, you don’t care where they come from and are thus independent of the sender Component/System.

Or what about defining multiple Interfaces for Components, and querying Entities for a Component that implements a certain Interface?

One other question: like you mentioned, a good editor with support for adding Components to Entities is crucial to the succes of using an ES in a project. Do you know of any such editors? I am looking for one for a 2D game, and I would like to be able to visually build up my levels out of Entities and give them certain Components and place them in the game world.
Eldad March 2nd, 2011 at 1:07 am

@adam
I’ll appreciate if you could explain how to build a “collision detection” system and a “movement” system in the same application in the ES way.
I feel like its the only missing link for me.

Thanks,
Eldad.
adam March 2nd, 2011 at 2:40 am

@Jacob

If components can received messages, then the ES is an over-complicated way of doing OOP – and you should just stop, and do normal OOP instead. There’s nothing wrong with that – but you’d lose many of the benefits I’ve previously described.

A lot of the benefits REQUIRE that components are pure data; the moment you let them “receive messages”, those benefits are literally impossible.
adam March 2nd, 2011 at 2:42 am

@Eldad

Start again from the beginning. Components are just data – nothing more, nothing less.

Re-think how you would approach the algorithms. You need to internalise this stuff, or you will never be able to understand your own ES. It’s not easy if you’re used to OOP – but when you get used to it, it’s very easy.
Tom H. March 2nd, 2011 at 3:22 am

@Jacob:

The commercial engine Gamebryo LightSpeed has/had the kind of design you’re advocating for its entity system. It works, more or less, but as Adam states, it doesn’t have several of the benefits a “pure” entity system is supposed to bring. Unity3D also has an entity system that reminds me of the approach you’re suggesting, although I think there are some subtle differences.

I think what did the most to sway me to this line of thinking, beyond lurking on t-machine, was reading Tony Albrecht, particularly “Pitfalls of Object Oriented Programming” – which is a bit PS3 specific. Or Noel Llopis, whose columns in Game Developer are repeated & expanded on his blog at gamesfromwithin.com. I’m hoping that my next job will give me a chance to work with (or, more likely, write) a purer entity system.
Jacob March 2nd, 2011 at 8:04 am

@adam:

Exactly WHAT benefits do you lose with the approach I described?

Would an alternative be sending the messages to the Systems with a certain ComponentID as parameter? This should still have the benefits of being able to create any kind of objects you want by composing them of Components, without getting a huge hierarchy or bloated classes. And what do you think of the idea about components implementing specific interfaces like I mentioned?
adam March 2nd, 2011 at 11:10 am

Sorry I don’t have time right now to go and dig out examples for you. There’s lots that have been hilighted and discussed in these many pages of blog posts and comments.
Eldad March 2nd, 2011 at 1:43 pm

@Adam,
I never said that my component won’t be pure data.
I was talking about system additional functionality (entitysystem).
To be more specific, I was talking about systems that need to be sync in some way (“collision detection” and “movable”).
In order to have a functional CD system – it should be affected by the movable system.
How can we achieve that?

I hope that my question is clear.
adam March 2nd, 2011 at 2:41 pm

@Eldad

Here’s what made me think you were doing code + data:

“invisableComponent – if the entity is on the move, its visible, else invisible.”

Well, that’s not a data component. That’s a piece of code.

“collisionDetectionComponent – check collision between entities”

Ditto.

“movementComponent – used to move entitles.”

Possibly ditto – the “used to move entities” suggests it has some code?
adam March 2nd, 2011 at 2:51 pm

@eldad

Say you have:

Position: { x, y, z, }
Movement: {dx, dy, dz}
CollisionInfo: { collisionClass, collisionVolume }

You might implement a collisionDetectionSystem that each frame iterates over every entity which possesess those three components, and calculates:

SweptVolume = sweep( collisionVolume, x,y,z, x+dx, y+dy, z+dz )

…then runs a CD across all the swept volumes, using the collisionClass to e.g. make somem objects not collide with other specific types (maybe a player avatar doesn’t collide with player’s own gunshots, for instance)
adam March 2nd, 2011 at 2:55 pm

@Eldad

…but optimising this is the same as optimizing any CD in any game.

e.g. you might do an AABB pre-filter. You could, if you choose, supplement your CollisionInfo component with an AABB var.

Or add a new component:

CollisionAABB: { x,y,z, width, height, length }

(I’d probably do that – so if you change to a CD that isn’t AABB-based, you wouldn’t need to change your main ColliosnINfo compononet at all)

if you defined your AABB boxes as relative to the object origin in Position component, your CD system could cache the AABB tree, and each frame use the values in Movement to modify the cached copy instead of rebuilding the entire tree from scratch

…but agian: this is all bog-standard CD. ES hasn’t changed the algorithm. And there are better CD approaches out there, but they shoudl be easy to port across.
Eldad March 2nd, 2011 at 4:00 pm

OK, I got my answer.
Basically what your saying is that systems can share data (components).
in addition we can have systems that uses 1 component while others can use more – Right?

Do you think that systems should/can have pub/sub or observer implementation?
For example, if you have “Movement” system, it can publish messages onMove and onStopMove etc.

Eldad
adam March 2nd, 2011 at 5:00 pm

Yep.

Re: pub/sub – I love that paradigm, but it doesn’t work well on modern computers. It requires edge-driven CPUs, not clocked CPUs.

You can fake it in clocked fashion, by havnig a set of components that just contain deltas, that only exist when they’re non-zero. e.g.

1. During one tick: each system creates + add a bunch of deltas
2. During next tick: some systems read the delta components, then delete them at end of tick

In general, it’s often easier to re-write your logic to be non-edge-triggered. e.g. instead of “onStop”, do:

- “iterate over all movement components for entities that have the EnemyShip component”
- “for any of those that have a value of 0, look at their associated delta component, and see if it’s non-zero (i.e. it moved in the last frame”
– “…do the processing…”

this approach has no internal dependencies, so it’s a lot easier to debug IME
Eldad March 2nd, 2011 at 11:23 pm

Adam,
I can think of another approach for implementing that, I want to know what you think of that.
Instead of having an even-odd ticks we can have an hierarchy of of systems, each level will have its “own” tick.
The first level will have only independent systems, that mean that the system can make calculation only by their own components.
The next levels will require components from the previews ticks and not just their own.
When the last level will end – all the deltas will be removed.

for example, we have the following components/systems:
1st level:
Position: { x, y, z, }
Clothing: {shirt:x, pants:y }
2nd level:
Movement: {dx, dy, dz, velocity }
3rd level:
CollisionDetection { collisionClass, collisionVolume }
last level:
Render

so as you can see, if a unit need to move, the Movement system will set a new dPosition, then it will be checked by the CollisionDetection (on error, remove the dPosition) and finally it will be render to the screen.

What do think of that?

BTW, I didn’t understand how you will implement the Collision Detection system. its seems like you will need to to compare all the entities position using your algorithm N^N.
I think that it will be better to store an associate array/vector/hash map for location->entity list, that way when you check for collision you can do it VS units that are close enough.
Any suggestion on that one?
Jacob March 3rd, 2011 at 1:46 pm

@adam

When using an ES in an *offline* game (in my case for iOS devices), are there still any advantages to using a database to store the entities, components and systems? I understand they are the most natural fit, and I also understand the advantages they bring for an MMO game, but as you mentioned in one of your (excellent) posts, hundreds/thousands of queries are probably way too slow for home PC’s, and mobile devices even more so?
adam March 3rd, 2011 at 1:53 pm

@Jacob I wrote an Objective-C version you can download from github – c.f. http://entity-systems.wikidot.com/es-approaches, which doesn’t use a DB, but might help if you’ve not started yet.

There’s also a blog-post I did on ES with Android – works fine for real-time action games. The limiting factor was the JVM’s poor implementation of HashMap – which I’d *expect* an in-memory database to avoid (using a much more clever / cunning data structure, probably with ByteBuffers to do it in contiguous memory areas, etc).

I would certainly experiment with an in-mem SQLDB if I were you – but please report back with your results :)
PatrickB March 11th, 2011 at 8:24 pm

Hey Adam, interesting article series, can’t wait for part 6 (nudge).

Let’s say I’ve drunk the Kool-Aid and have an ES system that uses only relational queries to fetch data in an MMO. Spiffy.

Simplest example, I want to add 5 to X coordinate of every entity that has a Position component, and I want to do this every simulated frame. I guess we’re playing “Conveyer Belt Online (TM)” or something. Additionally, let’s subscribe to the design that an entity may have only a single instance of a component associated with it.

In practice, we should just write an SQL statement to update every row by adding 5 to the X position. Okay, fine, but when we move to non-trivial examples, this isn’t appropriate, so let’s assume you do your logic in C++ and let the DB do the fetching.

What would the expected approach be here? Fetch all rows in CPosition table, modify the value, update the table? That would take 1 query to fetch all of them (sounds good), a fast C++ loop, then N ‘update … set ‘ queries. That seems really slow – is this really how it should be done?

—
A different question: You said that writing back to the DB that you initially queried would be much too slow, and sure, I understand why that might be true. Your suggestion was that another database should be used to write to, and that it could even be a different vendor’s DB, etc.
Now what the heck do you do when you want to run the same query as last time? It seems like it will just fetch the old values. How does one properly handle this “write to a different DB” situation.
adam March 11th, 2011 at 10:18 pm

Wordpress authors are stupid bastards sometimes. Their comment form is “ooh, how clever we are” AJAX – but they hacked the web browser to do it so badly that if you mis click while typing, WordPress deletes your entire comment. Stupid, stupid, stupid.

So, briefly, SECODN TIME: in practice, don’t expect to use SQL, just understand that’s the theoretically perfect way of doing it. SQL drivers may optimize writes in clever ways – make no assumptions. There’s always stored procedures too; could be a bizarre and stupid approach, could be genius.
Jacob May 12th, 2011 at 2:16 pm

One of the advantages of such an entity system would be, as you mention in the articles, that iterating over all components of a system can be done really fast if they are laid out consecutively in memory. However, this won’t be the case because different systems need to reference multiple components!

Example: both the rendering and audio system need postion information, along with a bunch of other systems. Where do you store this position information? There are 3 possibilities really:
1) Store in both the renderable and audible components. This causes duplication of data and issues with keeping them in sync.
2) Store in one of the components. Then the other components that also need this data must either have a reference to it or query for this information through some kind of messaging system, which complicates things and is probably too slow.
3) Store in a separate Positionable component. This would be the cleanest solution I think, however now both the Renderable and Audible components need references to this Positionable component, or another way to access it.

The point is, no matter which solution you choose, components will always need to refer to other components (or their respective systems do), and this makes it impossible to lay out components consecutively in memory for every system that needs to iterate over them. Also, because this kind of communication will always be nessecary, utilizing parallelization by putting each system in a separate thread is not as easy as you make it sound.

Any thoughts on this?
adam May 12th, 2011 at 2:57 pm

Well, for starters … where components are being accessed read-only, there’s no reason why there has to be only *one* copy of them in-memory…

But the wider point is simply: you have enormous flexibility here to implement any of many optimization schemes. The data is now independent, homogeneous, and indexed. That’s about as good as it gets.
Jacob May 12th, 2011 at 3:06 pm

Thanks for the quick reply.

My point is that it’s not independent, as I tried to explain. Also, position data is not being accessed read-only, for example a Moveable component will need to modify it, as well as a physics component. Could you name a few of those many optimization schemes you are talking about?
adam May 12th, 2011 at 3:11 pm

A given system will often access a given component read-only – something you know at design-time.

The data in one component is independent of the data in other components from the same entity – instead of all being in a single OOP object.

Re: optimizations … this is core Computer Science, so any book on algorithm optimization, memory optimization, CPU optimization, etc. Compiler design, maybe? CPU design (especially multicore? Operating Systems (especially kernel-design and thread scheduling)? I’m sure there are good websites too, but I don’t know them myself.
Jacob May 12th, 2011 at 3:36 pm

Ok, another example:
A bullet that travels through the air has a moveable and positionable component. The PhysicsSystem then takes the deltas in the moveable component and adds them to the absolute values in the positionable component. This results in accessing different memory locations for those 2 components. Putting these 2 components after eachother in memory is possible, but what if there are 2 other systems A and B, which both need their own component together with these 2? Do you put moveable, positionable and A after eachother in memory, or moveable, positionable and B? No matter which you choose, either system A or B won’t have the components it needs available consecutively in memory.

This is only a very simple example with perhaps negligible overhead, but I hope my point is clear.
Axis May 22nd, 2011 at 11:03 am

Good day. Your articles is a fresh air to my dimmed by OOP mind. I have a question about ES concept. How entities can be relate one to another? For example, I have an (adventurer) entities with inventory component, and a (sword) entity with item component. Can inventory component store GUID of entity with item component? Or there can be another way to implement this relation.
PsionMage May 24th, 2011 at 8:39 pm

@Axis
Think of it from a different approach. Here’s a OOP (and physical reality) setup:
Adventurers have inventories which contain more entities

But using this system you need to use components to describe everything. And if you’re adding multiple copies of the same component to an entity, you might want to approach it from the other end.

Here is a simplistic setup:
Sword has a PickUpableComponent with two values:
Weight, ContainerId

Adventurer has a HasInventoryComponent:
MaxInventoryWeight, MaxInventorySize

then we have a PickUpItem System which:
When Adventurer makes contact with something with a PickUpableComponent:
1. checks to see if the Adventurer’s MaxInventoryWeight can fit the Sword’s weight
2. checks MaxInventorySize…
3. then adds the item to the inventory by adding the component: InInventoryComponent which contains:
InventoryOffsetX, InventoryOffsetY, InBagNumber, etc…

You could either remove the PickupableComponent (and store Weight in the InInventoryComponent) or keep both

You find the list of items in your inventory by simply retrieving all entities with the InInventoryComponent

You’ll see quickly that components multiply fast, but that’s fine.

adam, hope i did it credit.
PsionMage May 24th, 2011 at 8:53 pm

Clarification on previous comment:
If you have only one thing that can have an inventory (normal 1-player game), you don’t need ContainerId in InInventoryComponent.

If you have multiple inventories, you could put either the guid of the entity or a unique identifier of the container (stored in HasInventoryComponent)

The idea being that the adventurer doesn’t know he has something (that requires one to many). It’s that the sword knows that it’s had by something (many to one). Which is my understanding of how it would optimally work.
Axis May 27th, 2011 at 6:03 am

@PsionMage
Thanks a lot for the answer! Now I see how it works.
I trying to apply entity system’s approach to a non game project with tricky and complex data structures. This structures changes all time and ES may dramatically simplify development process.
Wildfire June 7th, 2011 at 11:32 am

A question regarding the “only data” part of components:

I have a component called “Location”, which represents the current position of an entity. I now have two ways of representing the position:

Either, I add three int fields (x, y, z) to the component. This way the component is pure data. Or, can I use the data type “Point” (struct) to represents the position? This is a complex data type, which somewhat violates the “pure data” view, because it does contain (static) methods.

Now, do I use x,y,z and “upgrade” those to Point whenever needed, or would you think it OK to use a more complex data type?

From my point of view, using Point doesn’t add any code to the component, however the data type itself does contain methods (in regards to that same data type).

Speaking of .NET, the Int32 data type is also a struct and +, -, … etc. are static methods… so technically, pure data isn’t even possible.
adam June 7th, 2011 at 11:56 am

@Wildfire

Put Point in there if you like. Just be aware that it might make netwrokgin code harder to write later on, and that you might be losing some performance improvements (I don’t know what your Point object is, under the hood – a struct is (usually) by definition “only data”)

It’s the spirit of the thing you want to stick with – if your language/platform doesn’t support low-level efficient storage, then that’s fine. Just make sure *you* don’t code any methods/logic/etc into the things you put in your Components.

Although … if your language *does* support this stuff … use it. Store as x/y/z and pull-out a Point object when you need it. This is one of the cases where I like to use a sprinkling of OOP (not recommended unless you’re *absolutely* confident you’re not abusing it): not inside the core, but as icing that makes the code shorter to write.

e.g. make an Entity class, give it an entity-ID, and stick a method “locationPoint” on there. The class is never used at runtime except as a convenient place to store convenience code, i.e. you’d end up:

MyEntity = new MyEntity( id ) // this object is thrown away immediately at end of this loop
MyPoint = MyEntity.locationPoint(); // impl: “return new Point( EntityManager.getComponent( Location, self.id ).x, EntityManager.getComponent( Location, self.id ).y, EntityManager.getComponent( Location, self.id ).z )”
Wildfire June 7th, 2011 at 1:01 pm

@adam

Thanks for the answer. Good point about networking, I was thinking something similar in regards to storage. In the worst case each complex type needs custom code for serialization, wheres built-in types are supported by the language itself.

The more I think about it, the better it seems to go with x/y/z. Structs like Point and Vector add a lot of convenience, however not every system might need that. And conversion, if needed, is trivial.
wolfscaptain July 31st, 2011 at 6:35 pm

I really like your idea of “pure” entity systems, and while my game is very small I would like to use it.

Now implementing the entity manager is the easy part, but I got stuck at two things:
1) Getting a component to update another component.
2) Getting a custom script to run on a specific component in order to update it.

I am trying to make a user input component that states that the entity listens to events such as keyboard and mouse presses.
Now, I want to give my player this component, with a script that will tell it what to do when an event does occur (say “set velocity to something if you pressed WASD”).
What I do right now is to give a callback to the user input component when I create it. Then when the system runs, it calls the callback for each existing user input component.
This seems to be very hack-ish and not a good structure.

Now getting components to talk to each other – I do not care about efficiency at the moment, so I want to have the messaging method you described previously – sending a message to all the components owned by an entity, and if they are not listening to that message they will just silently ignore it.

I implemented this by adding a method to the entity manager called getEntityFromComponent (I think the signature describes what it does good enough :) ).
Every time I want to send a message from a component, I get its owner from the entity manager, then I get all the components owned by the entity, and finally I loop over them and call their “onmessage” method, which is the callback I assigned them at construction time I was talking about before.

This again doesn’t look to me like a very good design. It’s a bit odd that a component needs to go through the entity manager to get messages to its brothers.

To summarize, I want to be able to listen to user input, and update appropriate components (position, which will in turn update rendering) in a better structured way, preferably with custom scripts for components rather then hard-coded responses to messages.

Do you have perhaps a better way to implement these two things?

Thanks.
adam July 31st, 2011 at 6:49 pm

You’ve broken two of the cardinal rules/guidelines:

1. Adding methods to your components

2. Message-passing between components

Remove that, re-think “how else could I achieve the same outcome, without breaking those rules?”, and try again. I think you’ll find it becomes much easier/cleaner.

(lots of discussion of this in the comments already – you might want to read the complete comments to all 5 entity systems posts)
wolfscaptain July 31st, 2011 at 9:03 pm

Sorry for not explaining properly – the systems themselves handle passing messages through the entity manager when they iterate over the components they control.

About methods – I just can’t find any other way to do stuff with messages in a custom way per-component without sticking callbacks in the components.

I read most of the comments over a very long time, there are just so many.
This same question seems to be the one most asked, and I don’t remember reading a direct answer (but I didn’t actually use entity systems then, so I might have not noticed).
Martin September 17th, 2011 at 4:51 pm

I am currently exploring various game engine architectures and I really enjoy this idea about entities as passive property bags and components as singleton-like subsystems. That means, my entities are not heavy loaded with functions, overrides and what not, so I can store/load them to/from relational DB, send over network and so on. And I can manage all inter-component relations from the subsystems themselves.

I have some practical implementation questions for scripting support and dynamic compnent creation.

Let’s say, I would like to have a flexible system where “modders” or advanced users are allowed to create and distribute minigames through some sripts (Lua, for example).

I imagine the process like this:

- the modder posts to the server his script (let’s call it CriminalitySubsystem). Let’s imagine there is some security control to deny malicious or potentially dangerous scripts, maybe some automated software to test the script or the script runs in some sandboxed environment

- the CriminalitySubsystem script has some function call into API of my engine to somehow register the new Criminality property in the database. Now the question is how to do it safely? I would like to avoid giving the ALTER permissions to unknown modders who are not programmers of my team, so all new component (and their related properties) registrations should preferably be implemented as INSERTS)

- now it is up to the modder to advertise: “Hey, I have a minigame, so get it from my avatar page and attach it, if you’d like to play”

- other users attach this new property and get dynamically registered into the CriminalitySubsystem (which is already running on the server). Again, no database ALTERs should be allowed.

What would be the best way to implement such a dynamic component creation without granting too much database access for the modders/scripters? I have no idea, what kind of properties the advanced user may want to add (I do not know the name nor the datatype), so I cannot have a database template for those unknown components.
adam September 17th, 2011 at 5:12 pm

Perhaps make a separate database for each mod(der) and grant them access to it?

Setup some basic protections at the DB level (e.g. stop it going above a certain size).

“popular” mods you could then examine by hand, and work with the modder to elevate them to “approved” status and merge them into the main DB, e.g. give them their own table.
Martin September 18th, 2011 at 11:09 am

@adam
That sounds good, but involves much human work. I hoped to somehow make this process completely automatic (assuming that there is some strong security sandbox around scripts to prevent possibly harmful features).

Maybe I could completely avoid separate tables for such dynamic scripted components and I could organize it like that:

table
COMPONENT_REGISTRY (registers dynamic component at the first launch of the component subsystem script)
USER_ID COMPONENT_ID COMPONENT_NAME

table
PROPERTY_REGISTRY (registers each property needed for the dynamic component at the first launch of the component subsystem script)
PROP_ID COMPONENT_ID PROP_TYPE_ID <-so the game engine knows how to treat the value of this property

table
CHARACTER_PROPERTIES (adds relations at the moment when the user attaches the component to his game character)
CHARACTER_ID PROPERTY_ID PROPERTY_VALUE <- what type should it be? actually depends on TYPE_ID in PROPERTY_REGISTRY)

It's like exploding the field [1..M columns, one column for each piece of data in your component] into individual property records in the CHARACTER_PROPERTIES table and bind those properties to entities (and not components to entities like the article suggests).

Would such database structure slow my system down?

Anyway, I already see two drawbacks for my approach:
1) now component-component_data link is too loose, it is possible to forget to add some component properties to the entity and thus break the component
2) problem with storing component data for various datatypes

But the advantage is that I do not need to create component tables dynamically, so the modders do not need ALTER permissions.
jack mathew December 27th, 2011 at 7:43 pm

Hi!,

I have just read all your articles about entity systems and they look fantastic. I like the idea a lot but I have some questions I would be very happy if you could answer.

Main question is, in this ES you have explained how do you do things as simple as “playsound()”? I mean, You have components with data and systems processing this components. Thats ok, but what about direct actions as the playsound()?

This question is related to the previous one too. I have game objects that are added to layers in the scene. How would I do a change layer request? For example think that the player has a controllable component and when user hits a key the player avatar must be moved to a closer layer, should I access the render system and tell it to move graphic component for entity id XXXXX to layer X?

Sorry if my explanation is not clear, but it is a bit hard to get the grasp of all this.

Thanks in advance.
David SF February 18th, 2012 at 2:50 am

Thanks for the fascinating article series.

Do you have any advice about modeling lists of components?

In your example, you show adding a single ‘gun’ as:

createComponentAndAddTo( GUN_COMPONENT, new_id );

What happens when you need to support multiple guns per tank?

One option is to model multiple guns into a single GUNS_COMPONENT, but this seems to be breaking the encapsulation of the concept of entities, as now the System is just a system over multiple guns, and each gun becomes a polymorphic class hierarchy to enable different gun behaviors.

Another option seems to be to support adding multiple weapon components onto the same Entity, but now our entity processing system is thematically different, as it is processing each entity multiple times, one for each matching component.

Yet another option might be to make the guns themselves Entities, and link them to the tank with Components. Does each Entity get a link to the other? If so, this still requires allowing multiple occurances of the same Component per entity. Something like (forgive my OOP syntax, I think it’s just easier to read the psudocode this way):

tankEntity = new Entity();

weapon1Entity = new Entity();
weapon1Entity.addComponent(new GunComponent());
weapon1Entity.addComponent(new OwnerComponent(tankEntity));
tankEntity.addComponent(new WeaponSubsystemComponent(weapon1Entity));

weapon2Entity = new Entity();
weapon2Entity.addComponent(new GunComponent());
weapon2entity.addComponent(new OwnerComponent(tankEntity));
tankEntity.addComponent(new WeaponSubsystemComponent(weapon2Entity));

A similar question arises in managing containership and inventories. Where are inventory items stored in this Entity/Component decomposition?
adam February 18th, 2012 at 3:31 pm

This has been asked a lot of times in the previous comments on this post and some of the others in the series, I suggest you read back through the comment threads.
Garagoth February 26th, 2012 at 5:35 pm

Hi,

I am trying to implement ES as a core for multi-player game. Apart from usual problems with storing lists (like inventory) in fast to get way, one thing troubles me a bit more: networking code.

As networking code usually runs in response to events (net traffic: packet received, etc), how do I hookup this with ES?
Should NetworkSubsystem be invoked periodically and then check incoming queue to process it, then check which content should be published to clients (and more specific, which content should go to which client? This will probably require reading some other components of entity to decide this)?
Where and how should received data be stored for processing? As components in database as well?
And one final question: how to select which components were changed and therefore should be processed by network code without adding timestamp column to each component (which would add a lot of data, sometimes not used at all, to store and update)?

Regards,
Garagoth.
Oleh Novosad March 12th, 2012 at 4:45 pm

Hello,

First of all, thank you Adam for sharing your experience with ES.

And I have some thoughts about ES.

From OOP design (oops it sounds so wrong!) we know that we should:

1. Design to interfaces.
2. Favor aggregation over inheritance.
3. Find what varies and encapsulate it
(encapsulation means hiding not only data, but implementation, types, everything).

Lets see ES in this context.

From what I understand:

1. Only system can write data into their component (and only into their component).

So system will have two types of methods:
a) methods that work with one component and receive it as parameter
b) methods that work with many components (two or more of same type and other parameters)

In a) case methods of system will look like this:

some_return_type method_name(MyComponent* my_component, … other parameters);

and if they can only WRITE data to my_component , what kind of method is this?
This is instance method of MyComponent class.
Because instance methods that change only this class’ instance data are instance methods and
in C++ they look like this:

some_return_type method_name(MyComponent * this, … other parameters);
where *this – is hidden parameter.

So we can easily move these kind of methods from system to component class.
Because this is the same.

What about b) case? It is better for those type of methods to stay in separate “System” class.

So components now not only have “GET” and “transform before GET” methods but also methods that can
change component data.

2. Entity is just a name(id). But why name has separate class in programming language? Actually I
don’t see reason why you need entities at all. Because instances of components are entities
(or entities are just instances of components that can contain other components).
For example I have components: leg, arm, body, head. Using these components I can form “entity”
- human:
1 left leg – instance of leg component
1 right leg – instance of leg component
1 left arm – instance of arm component
1 right arm – instance of arm component
1 body – instance of body component
1 head – instance of head component

But if enemy throw granate to this human, and it will tear her apart,
left leg – can become separate “entity”.

Also that human “entity” actually can be component instance of Army “entity”
(component instance of army component). But I also can remove this human component instance from
army component instance and add it to civil people component instance or dead people
component instance.

In DB you don’t need table 3 (from tables you proposed).
And table 5 will look like this(renamed it to component_instances – to be clear):

Table 5 : component_instances
id parent_id component_id component_data_id

where id – autoincrement field, id of instance of component
parent_id – self link to parent instance component
component_id – link to “type” of instance of component
component_data_id – link to “real” data of this component

If component don’t have parent, then parent_id is NULL.

Why it useful? Because everything are hierarchies.
Instance of human leg is usually a part of human instance. Human instance component is
usually part of some community intance component, etc.

But not only this. You solve problem when you need system that uses data of few components
and maybe even “indirectly” update data of few components. How? Simple.
You create component Collision Detection that “contains” Position component and Collision Data
component and then create system Collision Detection System that works with this component.
Collision Detection System can query all Collision Detection component instances
and for each of them get access to Position component instance and Collision component instance.
But remember, only Position system and Position component can change data in Position component,
so Collision Detection System to change data should use other systems and components itself.

3) Now lets examine ES from OOP design.
Lets start from end, last rule:
“Find what varies and encapsulate it
(encapsualtion means hiding not only data, but implementation, types, everything).”

What is changing in big games? Parts of different fucntionality, parts of objects, etc.
And it can even happen in dynamic way – like leg instance is not part of human instance
anymore. So ES encapsulate these changes.

Next rule:
“Favor aggregation over inheritance.”

Sure, we do this. And if classic aggreagtion looks like this:

class Human {
Leg left_leg;
Leg right_leg;
Arm left_arm;
Arm right_arm;
Body body;
Head head;
};

we do aggregation like this
(here is very not optimized way, because actually list is not contained here,
but idea is the same):

class Component {
list components;
};

class Human : Component {

};

In previous case I cannot move leg out of Human and in ES you can. Also you can add third leg
(mutation???!!).

So yeah, we have aggregation over inheritance here, and not just simple aggregation – but
dynamic aggregation.

And now first rule:
“Design to interfaces.”
We see it here too, as components have common interfaces.
Your component don’t worry about types of other components it contains.

4) So what is difference between OOP and ES? None, ES is part of OOP but it applies some useful OOP rules:

a) Only component or it’s system can change data of component
(pure OOP rule: methods of object can change data only of it’s own object).

b) It gives us very useful OOP pattern to create games: I would call it ES pattern(or CS pattern?).
This pattern propose dynamic aggregation to hide variations of object parts. It does this not only
in programming language level but also in RDB level.
ES pattern (or CS pattern) is similar to Bridge pattern but ES pattern has dynamic aggregation
(of operations or components) and additional encapsulation that hides types of operations (components).

c) You work with instances of components of same type not with instances of
components as containers of other instances of components.

5) Can you use inheritance between components?
Yes you can, but you should avoid it as much as possible because you should favor aggregation
(in this case dynamic aggregation) over inheritance.

Consider you have 2 components – Rectangle and Circle (not sure in real world you
will use such components). Rectangle component looks like this:
class Rectangle : Component {
int x1, y1, x2, y2;
};

Circle component looks like this:
class Circle : Component {
int x1, y1, radius;
};

If we favor aggregation, then we can create Point component
class Point : Component {
int x, y;
};

And now Rectangle component will consist of 2 Point components instances.
Cicrcle component will consist of 1 Point component instance and radius.

But we see now that Circle and Rectangle share 1 Point component instance, can we make Shape
base class for them that contains that 1 Point component instance and then “inherit” Rectangle
and Circle components from it? Why not? You can. But you don’t need to. As creating of instance
of component is matter of writing creating function. For example:

Component* createPoint(string debug_name) {
return new PointComponent(debug_name);
}

createRectangle(debug_name) {
// rectangle doesn’t have it’s own data, so we don’t need class for it in C++ language
c = createComponent(debug_name);
c.add(createPoint(“point1″));
c.add(createPoint(“point2″));
return c;
};

createCircle(debug_name) {
// Circle has it’s own data – radius, so we need to have class for it in C++ language
c = new CircleComponent(debug_name); // class that contains radius only
c.add(createPoint(“point”));
return c;
};

So we don’t need Rectangle class (only Rectangle system) and we only need Circle class because
of radius field (yes, technically we can move radius to some component if that makes sense):

class Circle : Component {
int radius;
};

But in real world you will have all components preconfigured into assemblages somewhere in db
(tables 1 and 2).

So component creation will look like this:

// nothing changes for this component
Component* createPointComponent(string debug_name) {
return new PointComponent(debug_name);
}

createRectangle(debug_name) {
c = createFromAssemblage(“rectangle”, debug_name);
return c;
};

createCircle(debug_name) {
c = createFromAssemblage(“circle”, debug_name);
return c;
};

So inheritance is not needed.

What are default values for data in components? There are two ways, or you use default values
for columns set in DB or you create your own default data initialization layer (usually using some
config files).

6) Another interesting question is,
can same component instance be part of other two componont instances?

Why not? Same human can be part of two different organizations.

On RDB level, ES pattern allows creation of dynamic links between tables. And same component can
be used by 100 other components.

Best wishes,
adam March 12th, 2012 at 5:14 pm

@Oleh

You seem to have ignored the performance issues and the maintenance issues. 50% of ES is “making it possible to write programs that can’t realistically be written in OOP”. 50% is “writing code that runs faster than OOP code”.

If you don’t care about writing real programs, and you don’t care about performance, then … no, I see no particular advantage in ES.

That said, a few minor corrections and comments:

1. You have just taken the PRIVATE method out of a PRIVATE system and inserted it in an UNRELATED CLASS — i.e. you destroyed all encapsulation, and achieved … nothing. Why would you want to do this?

You also took a lot of HIGH PERFORMANCE code and wiped it out. Objects take up lots of memory, raw data doesn’t. Object graphs are slow to traverse, raw data is fast.

2. Entity doesn’t have a class. In some implementations, as a convenience, people add a class for it. But it’s not intended nor required.

Also, relational databases are incapable of storing tree hierarchies without an infinite drop in performance. Your modified table is unusable in any real world program – it would be too slow.

4. ES is fundamentally relational in nature. ES disallows “methods”. ES disallows “objects” and “classes” within the app design. How is this “part of OOP”? It’s not “object oriented” it’s “object anti-oriented” :)
Oleh Novosad March 12th, 2012 at 6:18 pm

Thanks for quick answer,

1) Private methods? So systems methods are private? Ok. But they have methods that works only with related components , like PositionSystems works with PositionComponent. For example method PositionSystem.reset(PositionComponent c). You can easily move this method to component and it will became PositionComponent.reset() method.

I mean PositionSystem.reset(PositionComponent c) == PositionComponent.reset(). In C++, I don’t see any performance overhead. Because compiler transforms PositionComponent.reset() to static PositionComponent.reset(PositionComponent* this) – same method as if it is in system. As PositionComponent is class anyway and has base class – Component.
You have to hold your data in some struct or class anyway.

2. Agree, building deep hierarchies will be bad. Especially for MMOs where you have a lot of players and a lot of hierarchies. Because it is too many operations for one table. There should be way to optimize it. I like idea that you can combine few components to create new and have new system for combined components.

My question: so I have Human entity, it consist of few components(leg, arms etc), where I put code that do read data from few components and do something?

From my point of view – I could consider Human as new component that consist of other components and write system for it. Or should I not use ES for that problem?

4. I came from ruby on rails background where we have FAT models and we don’t have any problems with idea that we are using relational db. ES uses Systems with static methods – but in fact many of those methods equals to instance methods (as I described in 1.)
adam March 12th, 2012 at 6:43 pm

1. But WHY!?? Why would you do this? What does it buy you?

We know it COSTS:
– it makes the code much harder to read, because now the code for ONE algorithm is split into lots of DIFFERENT classes / files
– it makes the code MUCH SLOWER to run, because you just converted a raw array into an object

2. What are you trying to do?

4. Yes, RoR runs slowly and is extremely poor at handling objects. I don’t see how that justifies using anti-OOP programming and claiming it’s OOP :)
Oleh Novosad March 12th, 2012 at 8:53 pm

1. So you just query data from memory or db and process it with system? You even don’t need to create instance of your component class for each record from db?

2. I am trying to create AI that uses data from few components (like position, health, weapon possessed, etc.). Should I create it outside ES and it’s rules? I don’t understand this.

4. RoR is comparably slow because it uses Ruby – and Ruby is not compiler but interpretator. Also Ruby is very dynamic language (like you can create new classes , add methods and properties to it and create instances of it during execution). If I will understand question 2., maybe I will understand why ES is not OOP.

thank you for answers Adam
adam March 12th, 2012 at 10:48 pm

1. Or from the network – multiplayer game-state, for instance

2. Give an example of what you’re actually trying to do. Right now, you seem to be worrying, but haven’t encountered an actual problem.

4. It doesn’t really matter if it is or isn’t. The point is (as I explained in the first blog post): if you think of it in OOP terms, you’ll probably never understand it. You need to learn to throw away OOP, and then you’ll find ES easy.

Otherwise, you’ll keep trying to turn it into OOP. It was designed to escape OOP, so if you keep trying to drag it back … you’ll find it really difficult to make a good game.

IMHO
Oleh Novosad March 13th, 2012 at 11:05 pm

From part 2 of article:

“Each System runs continuously (as though each System had it’s own private thread) and performs global actions on every Entity that possesses a Component of the same aspect as that System.”

So System can read and change data of entire Entity “that possesses a Component of the same aspect as that System”?

I mean, can System read data and write data to other(not of System’s aspect) components of Entity? Or only read? Suppose read mode on components of other aspects should be enough.

Suppose I have AISystem and AIComponent. Now I assign AIComponent to every Entity that should be processed by AISystem. Then I will query all Entities that have AIComponent, read data that I need for AISystem from other components of each Entity, process some ai algorithm and save some results (like decision what to do) in AIComponent of that Entity. Suppose AISystem decided to do action “move” and saved that decision into AIComponent for Entity.

Now I will process MovementSystem, this system queries all Entities that have MovementComponent (not all entities can move). Do for each entity cycle. If Entity has AIComponent and decision made by AISystem is to move, then MovementSystem processes movement for that component.

Sounds good.

But what if AISystem need information not only from Entity it processes but also from other entities that don’t have AIComponent component. For example, to decide if I should move to left I need to check if there is not some dangerous object (like flame Entity – which has not AI) ?

And second question, suppose I have Human Entity and that Entity has two legs. Each leg is component of same aspect. What to do in this situation? Should I support few components of same aspect for Entity?

Thank you, Adam
Oleh Novosad March 14th, 2012 at 5:10 pm

I think I already have answers for my questions.
And thank you again for great articles.
roka March 27th, 2012 at 11:10 am

Very interesting article(s) and comments. As the discussion keeps going this topic remains hot.

I have some questions though I plan building an Javabased ES for a mmo browser-game engine. As rendering happens on clients while non-presentational game logic resides on the server (multiple systems :D) I have a split of the systems in two layers located on different machines. Rendering though is actually not a problem as I focus on the server-side concept of the engine. How would you build a system not based on a game loop but on request/reply concept?

Think of a situaion like this.
Player A calls an action to attack Player B with an Army C in his base located at D.
-> we have 2 player entities, one army entity (which somehow needs to be a component of player a?!), and a base at a given location.
Player A fires an action with the given parameters to the server which has a method for handling this case (possibly the mehtod path is stored as component_data aswell?!). That method belongs to a system in the sense of an ES. As it’s an system belonging to an component it get’s all necessary component_data, calculates the results and both answers the client and store the possibly altered data. Would you think this is a good way to solve that?

How do you think about using NoSQL databases like Couchbase (formerly known as membase) for storage?

Client/Server-Communication is not a problem as I would use jWebSocket for communication between those layers and there for go for a request/reply concept with async-RPC to the relevant systems (e.g. CombatSystem, ShopSystem, …).

Francois had some questions as well, which you didn’t answer. I guess you just didn’t see it or forgot about it. Those questions answered would help me aswell.

thanks, roka.
adam March 27th, 2012 at 1:24 pm

There is always a game loop. Unless you are using some freaky hardware that isn’t manufactured by Intel/AMD/etc.

NoSQL databases are mostly “databases for people who have small or simple data sets, or are too lazy to learn SQL”. If you have an exceptionally simple game – e.g. tetris – then a NoSQL database would be less effort to program against – go for it. For any real game, you should be using SQL. Or not using a database at all.
roka March 27th, 2012 at 2:03 pm

The game loops main purpose is to update the view depending on the changed game data. However game data may change without some one actually sees it. Think of an attack of an army while both users are logged off, as it doesn’t happen instant because the army has to get to the other players base. In this case the event still is going to happen but no “client” actually sees that change to the game data.

That rendering stuff actually isn’t interesting to me in the first point as the “rendering part” of the engine will have to work on the clients machine and my goal is to really decouple client and server. That way you may build an specific mobile app using the same data (player account) and server while using a different client.

It’s really going to be more async RPC call like or even RRPC call like as jWebSocket for instance supports it.
adam March 27th, 2012 at 2:33 pm

Game loops are NOT about rendering.

They are about the entire game.

Generally, you have a game loop on server, a game loop on client, and your asynch comms injects timestamped data from one to the other.

Generally, the server then uses an algorithm to overwrite or re-write the client data, and the client accepts what the server tells it.

Generally, the client then uses an algorithm to extrapolate from the data the server gave it, in order to render more smoothly.

(either or both those algorithms can be trivial – seemingly non-existent – or arbitrarily complex – e.g. a full multi-way data merge on server, using multiple sliding windows of game-state deltas per-client-squared)
adam March 27th, 2012 at 2:35 pm

…and none of that should affect your ES. Your ES should be updated *by* those algorithms, and should carry on running locally as if none of this existed.

As it happens, the datastructures used in an ES generally make it very easy to implement the more advanced / tricky algorithms both for client and server. This is because they are nearly always data-driven algorithms, so the ES obsession with data-centric design fits neatly.

…but you wouldn’t expect an ES to know *anything* about networking.
roka March 28th, 2012 at 8:50 am

So those algorithmes should not reside in the ES-Systems, right? They only allow access to them to get data or change it? If I got you right, those ES-Systems are more like an abstraction “layer” between somehow organised data (through assemblages) and a kind of game-logic-layer that carrys the algorithms?
Mythgarr May 6th, 2012 at 7:36 pm

I posted a comment on a different post but adam suggested posting here would receive more replies. I am re-engineering a platformer game from a few years back to use the Artemis Entity System (primarily as a way to become familiar with ES development). I’m trying to figure out how best to implement a few features -
A player can only jump when they are standing on the ground. Player input is handled in a PlayerControlSystem, collision is handled in a CollisionSystem. What is the best way to deal with only allowing the player to jump when they are standing on the ground? Should the CollisionSystem set a property on a Component that the PlayerControlSystem looks at? Likewise, if I want an AI to react by turning around when they hit a wall, how should that be handled? I can keep adding flags for reacting to every specific situation but that would lead to Component bloat. I could handle every possible collision situation within the CollisionSystem (ie: having an AI turn around, damaging a player, checking if the player is trying to jump and reacting accordingly, etc) but that would lead to an ENORMOUS event handler within that system.

What is the ‘right’ way to do this?

Entity Systems are the Future of MMOs Part 5

A quick warning…

…and a quick introduction to Relational

How do you store all your data? Part 2: Runtime Entities + Components (“Objects”)

Table 3: all components

Table 4: all entities / entity names

Table 5: entity/component mapping

Tables 6,7,8…N+5: data for each component for each Entity

How do you store all your data? Part 1: Assemblages (“Classes”)

Table 1: all Assemblages

Table 2: assemblage/component mapping

Table 3: all components

Things to note

DataForEntity( (entity-id) ) – fast lookup

Splitting Table 5 for performance or parallelization

Isolation

Metadata for editing your Assemblages and Entities

Wrapping up…

123 Responses to “Entity Systems are the Future of MMOs Part 5”

Leave a Reply

Get updates automatically

Blogroll

network programming

Meta

Categories