Entity Systems are the Future of MMOs Part 5

(Start by reading Entity Systems are the Future of MMOs Part 1)

It’s been a long time since my last post on this topic. Last year, I stopped working for a big MMO publisher, and since then I’ve been having fun doing MMO Consultancy (helping other teams write their games), and iPhone development (learning how to design and write great iPhone apps).

Previously, I posed some questions and said I’d answer them later:

  • how do you define the archetypes for your entities?
  • how do you instantiate multiple new entities from a single archetype?
  • how do you STORE in-memory entities so that they can be re-instantiated later on?

Let’s answer those first.

A quick warning…

I’m going to write this post using Relational terminology. This is deliberate, for several reasons:

  • It’s the most-correct practical way of describing runtime Entities
  • It’s fairly trivial to see how to implement this using static and dynamic arrays – the reverse is not so obvious
  • If you’re working on MMO’s, you should be using SQL for your persistence / back-end – which means you should already be thinking in Relations.
…and a quick introduction to Relational

If you know literally nothing about Relational data, RDBMS’s, and/or SQL (which is true of most game programmers, sadly), then here’s the idiot’s guide:

  1. Everything is stored either in arrays, or in 2-dimensional arrays (“arrays-of-arrays”)
  2. The index into the array is explicitly given a name, some text ending in “_id”; but it’s still just an array-index: an increasing list of integers starting with 0, 1, 2, 3 … etc
  3. Since you can’t have Dictionaries / HashMaps, you have to use 3 arrays-of-arrays to simulate one Dictionary. This is very very typical, and so obvious you should be able to understand it easily when you see it below. I only do it twice in this whole blog post.
  4. Where I say “table, with N columns”, I mean “a variable-length array, with each element containing another array: a fixed-size array of N items”
  5. Where I say “row”, I mean “one of the fixed-size arrays of N items”
  6. Rather than index the fixed-size arrays by integer from 0…N, we give a unique name (“column name”) to each index. It makes writing code much much clearer. Since the arrays are fixed-size, and we know all these column names before we write the program, this is no problem.

Beyond that … well, go google “SQL Tutorial” – most of them are just 1 page long, and take no more than 5 minutes to read through.

How do you store all your data? Part 2: Runtime Entities + Components (“Objects”)

We’re doing part 2 first, because it’s the bit most of us think of first. When I go onto part 1 later, you’ll see why it’s “theoretically” the first part (and I called it “1″), even though when you write your game, you’ll probably write it second.

Table 3: all components

(yes, I’m starting at 3. You’ll see why later ;))

Table 3: components
component_id official name human-readable description table-name

There are N additional tables, one for each row in the Components table. Each row has a unique value of “table-name”, telling you which table to look at for this component. This is optional: you could instead use an algorithmic name based on some criteria like the official_name, or the component_id – but if you ever change the name of a component, or delete one and re-use the id, you’ll get problems.

Table 4: all entities / entity names
Table 4 : entities
entity_id human-readable label FOR DEBUGGING ONLY

(really,you should only have 1 column in this table – but the second column is really useful when debuggin your own ES implementation itself!)

…which combines with:…

Table 5: entity/component mapping
Table 5 : entity_components
entity_id component_id component_data_id

…to tell you which components are in which entity.

Technically, you could decide not to bother with Table 4; just look up the “unique values of entity_id from table 5″ whenever you want to deal with Table 4. But there are performance advantages for it – and you get to avoid some multi-threading issues (e.g. when creating a new entity, just create a blank entity in the entity table first, and that fast atomic action “reserves” an entity_id; without Table 4, you have to create ALL the components inside a Synchronized block of code, which is not good practice for MT code).

Tables 6,7,8…N+5: data for each component for each Entity
Table N+5 : component_data_table_N
component_data_id [1..M columns, one column for each piece of data in your component]

These N tables store all the live, runtime, data for all the entity/component pairs.

How do you store all your data? Part 1: Assemblages (“Classes”)

So … you want to instantiate 10 new tanks into your game.

How?

Well, you could write code that says:


int newTank()
{
int new_id = createNewEntity();

// Attach components to the entity; they will have DEFAULT values

createComponentAndAddTo( TRACKED_COMPONENT, new_id );
createComponentAndAddTo( RENDERABLE_COMPONENT, new_id );
createComponentAndAddTo( PHYSICS_COMPONENT, new_id );
createComponentAndAddTo( GUN_COMPONENT, new_id );

// Setup code that EDITS the data in each component, e.g:
float[] gunData = getComponentDataForEntity( GUN_COMPONENT, new_id );
gunData[ GUN_SIZE ] = 500;
gunData[ GUN_DAMAGE ] = 10000;
gunData[ GUN_FIRE_RATE ] = 0.001;
setComponentDataForEntity( GUN_COMPONENT, new_id, gunData );

return new_id;
}

…and this is absolutely fine, so long as you remember ONE important thing: the above code is NOT inside a method “because you wanted it in an OOP class”. It’s inside a method “because you didn’t want to type it out every time you have a place in your code where you instantiate tanks”.

i.e. IT IS NOT OOP CODE! (the use of “methods” or “functions” is an idea that predates OOP by decades – it is coincidence that OOP *also* uses methods).

Or, in other words, if you do the above:

NEVER put the above code into a Class on its own; especially NEVER NEVER split the above code into multiple methods, and use OOP inheritance to nest the calls to “createComponet” etc.

But … it means that when you decide to split one Component into 2 Components, you’ll have to go through the source code for EVERY kind of game-object in your game, and change the source, then re-compile.

A neater way to handle this is to extend the ES to not only define “the components in each entity” but also “templates for creating new entities of a given human-readable type”. I previously referred to these templates as “assemblages” to avoid using the confusing term “template” which means many things already in OOP programming…

An assemblage needs:

Table 1: all Assemblages
Table 1 : assemblages
assemblage_id default human-readable label (if you’re using that label in Table 1 above) official name human-readable description
Table 2: assemblage/component mapping
Table 2 : assemblage_components
assemblage_id component_id

This table is cut-down version of Table 5 (entity/component mapping). This table provides the “template” for instantiating a new Entity: you pick an assemblage_id, find out all the component_id’s that exist for it, and then create a new Entity and instantiate one of each of those components and add it to the entity.

Table 3: all components
Table 3: components
component_id official name human-readable description table-name

This is the same table from earlier (hence the silly numbering, just to make sure you noticed ;)) – it MUST be the same data, for obvious reasons.

Things to note

DataForEntity( (entity-id) ) – fast lookup

If you know the entity-id, you may only need one table lookup to get the data for an entire component (Table5 is highly cacheable – it’s small, doesn’t change, and has fixed-size rows).

Splitting Table 5 for performance or parallelization

When your SQL DB is too slow and you want to split to multiple DB servers, OR you’re not using SQL (doing it all in RAM) and want to fit inside your CPU cache, then you’ll split table 5 usually into N sub-tables, where N = number of unique component_id’s.

Why?

Because you run one System at a time, and each System needs all the components with the same component_id – but none of the components without that id.

Isolation

The entire data for any given system is fully isolated into its own table. It’s easy to print to screen (for debugging), serialize (for saving / bug reports), parallelize (different components on different physical DB servers)

Metadata for editing your Assemblages and Entities

a.k.a. “Programmer/Designers: take note…”

It can be tempting to add extra columns to the Entity and Assemblage tables. Really, you shouldn’t be doing this. If you feel tempted to do that, add the extra data as more COMPONENTS – even if the data is NOTHING to do with your game (e.g. “name_of_designer_who_wrote_this_assemblage”).

Here’s a great feature of Entity Systems: it is (literally) trivial for the game to “remove” un-needed information at startup. If, for instance, you have vast amounts of metadata on each entity (e.g. “name of author”, “time of creation”, “what I had for lunch on the day when I wrote this entity”) – then it can all be included and AUTOMATICALLY be stripped-out at runtime. It can even be included in the application, but “not loaded” at startup – so you get the benefits of keeping all the debug data hanging around, with no performance overhead.

You can’t do that with OOP: you can get some *similar* benefits by doing C-Header-File Voodoo, and writing lots of proprietary code … but … so much is dependent upon your header files that unless you really know what you’re doing you probably shouldn’t go there.

Another great example is Subversion / Git / CVS / etc metadata: you can attach to each Entity the full Subversion metadata for that Entity, by creating a “SubversionInformation” System / Component. Then at runtime, if something crashes, load up the SubversionInformation system, and include it in the crash log. Of course, the Components for the SubversionInformation system aren’t actually loader yet – because the system wasn’t used inside the main game. No problem – now you’ve started the system (in your crash-handler code), it’ll pull in its own data from disk, attach it to whatever entities are in-memory, and all works beautifully.

Wrapping up…

I wanted to cover other things – like transmitting all this stuff over the network (and maybe cover how to do so both fast and efficiently) – but I realise now that this post is going to be long enough as it is.

Next time, I hope to talk about that (binary serialization / loading), and editors (how do you make it easy to edit / design your own game?).

Did this post help you?

You can say “thank you” by giving me your email address, and letting me contact you next time I make a game of my own:

148 thoughts on “Entity Systems are the Future of MMOs Part 5

  1. wolfscaptain

    Sorry for not explaining properly – the systems themselves handle passing messages through the entity manager when they iterate over the components they control.

    About methods – I just can’t find any other way to do stuff with messages in a custom way per-component without sticking callbacks in the components.

    I read most of the comments over a very long time, there are just so many.
    This same question seems to be the one most asked, and I don’t remember reading a direct answer (but I didn’t actually use entity systems then, so I might have not noticed).

  2. Martin

    I am currently exploring various game engine architectures and I really enjoy this idea about entities as passive property bags and components as singleton-like subsystems. That means, my entities are not heavy loaded with functions, overrides and what not, so I can store/load them to/from relational DB, send over network and so on. And I can manage all inter-component relations from the subsystems themselves.

    I have some practical implementation questions for scripting support and dynamic compnent creation.

    Let’s say, I would like to have a flexible system where “modders” or advanced users are allowed to create and distribute minigames through some sripts (Lua, for example).

    I imagine the process like this:

    - the modder posts to the server his script (let’s call it CriminalitySubsystem). Let’s imagine there is some security control to deny malicious or potentially dangerous scripts, maybe some automated software to test the script or the script runs in some sandboxed environment

    - the CriminalitySubsystem script has some function call into API of my engine to somehow register the new Criminality property in the database. Now the question is how to do it safely? I would like to avoid giving the ALTER permissions to unknown modders who are not programmers of my team, so all new component (and their related properties) registrations should preferably be implemented as INSERTS)

    - now it is up to the modder to advertise: “Hey, I have a minigame, so get it from my avatar page and attach it, if you’d like to play”

    - other users attach this new property and get dynamically registered into the CriminalitySubsystem (which is already running on the server). Again, no database ALTERs should be allowed.

    What would be the best way to implement such a dynamic component creation without granting too much database access for the modders/scripters? I have no idea, what kind of properties the advanced user may want to add (I do not know the name nor the datatype), so I cannot have a database template for those unknown components.

  3. adam Post author

    Perhaps make a separate database for each mod(der) and grant them access to it?

    Setup some basic protections at the DB level (e.g. stop it going above a certain size).

    “popular” mods you could then examine by hand, and work with the modder to elevate them to “approved” status and merge them into the main DB, e.g. give them their own table.

  4. Martin

    @adam
    That sounds good, but involves much human work. I hoped to somehow make this process completely automatic (assuming that there is some strong security sandbox around scripts to prevent possibly harmful features).

    Maybe I could completely avoid separate tables for such dynamic scripted components and I could organize it like that:

    table
    COMPONENT_REGISTRY (registers dynamic component at the first launch of the component subsystem script)
    USER_ID COMPONENT_ID COMPONENT_NAME

    table
    PROPERTY_REGISTRY (registers each property needed for the dynamic component at the first launch of the component subsystem script)
    PROP_ID COMPONENT_ID PROP_TYPE_ID <-so the game engine knows how to treat the value of this property

    table
    CHARACTER_PROPERTIES (adds relations at the moment when the user attaches the component to his game character)
    CHARACTER_ID PROPERTY_ID PROPERTY_VALUE <- what type should it be? actually depends on TYPE_ID in PROPERTY_REGISTRY)

    It's like exploding the field [1..M columns, one column for each piece of data in your component] into individual property records in the CHARACTER_PROPERTIES table and bind those properties to entities (and not components to entities like the article suggests).

    Would such database structure slow my system down?

    Anyway, I already see two drawbacks for my approach:
    1) now component-component_data link is too loose, it is possible to forget to add some component properties to the entity and thus break the component
    2) problem with storing component data for various datatypes

    But the advantage is that I do not need to create component tables dynamically, so the modders do not need ALTER permissions.

  5. jack mathew

    Hi!,

    I have just read all your articles about entity systems and they look fantastic. I like the idea a lot but I have some questions I would be very happy if you could answer.

    Main question is, in this ES you have explained how do you do things as simple as “playsound()”? I mean, You have components with data and systems processing this components. Thats ok, but what about direct actions as the playsound()?

    This question is related to the previous one too. I have game objects that are added to layers in the scene. How would I do a change layer request? For example think that the player has a controllable component and when user hits a key the player avatar must be moved to a closer layer, should I access the render system and tell it to move graphic component for entity id XXXXX to layer X?

    Sorry if my explanation is not clear, but it is a bit hard to get the grasp of all this.

    Thanks in advance.

  6. David SF

    Thanks for the fascinating article series.

    Do you have any advice about modeling lists of components?

    In your example, you show adding a single ‘gun’ as:

    createComponentAndAddTo( GUN_COMPONENT, new_id );

    What happens when you need to support multiple guns per tank?

    One option is to model multiple guns into a single GUNS_COMPONENT, but this seems to be breaking the encapsulation of the concept of entities, as now the System is just a system over multiple guns, and each gun becomes a polymorphic class hierarchy to enable different gun behaviors.

    Another option seems to be to support adding multiple weapon components onto the same Entity, but now our entity processing system is thematically different, as it is processing each entity multiple times, one for each matching component.

    Yet another option might be to make the guns themselves Entities, and link them to the tank with Components. Does each Entity get a link to the other? If so, this still requires allowing multiple occurances of the same Component per entity. Something like (forgive my OOP syntax, I think it’s just easier to read the psudocode this way):

    tankEntity = new Entity();

    weapon1Entity = new Entity();
    weapon1Entity.addComponent(new GunComponent());
    weapon1Entity.addComponent(new OwnerComponent(tankEntity));
    tankEntity.addComponent(new WeaponSubsystemComponent(weapon1Entity));

    weapon2Entity = new Entity();
    weapon2Entity.addComponent(new GunComponent());
    weapon2entity.addComponent(new OwnerComponent(tankEntity));
    tankEntity.addComponent(new WeaponSubsystemComponent(weapon2Entity));

    A similar question arises in managing containership and inventories. Where are inventory items stored in this Entity/Component decomposition?

  7. adam Post author

    This has been asked a lot of times in the previous comments on this post and some of the others in the series, I suggest you read back through the comment threads.

  8. Garagoth

    Hi,

    I am trying to implement ES as a core for multi-player game. Apart from usual problems with storing lists (like inventory) in fast to get way, one thing troubles me a bit more: networking code.

    As networking code usually runs in response to events (net traffic: packet received, etc), how do I hookup this with ES?
    Should NetworkSubsystem be invoked periodically and then check incoming queue to process it, then check which content should be published to clients (and more specific, which content should go to which client? This will probably require reading some other components of entity to decide this)?
    Where and how should received data be stored for processing? As components in database as well?
    And one final question: how to select which components were changed and therefore should be processed by network code without adding timestamp column to each component (which would add a lot of data, sometimes not used at all, to store and update)?

    Regards,
    Garagoth.

  9. Oleh Novosad

    Hello,

    First of all, thank you Adam for sharing your experience with ES.

    And I have some thoughts about ES.

    From OOP design (oops it sounds so wrong!) we know that we should:

    1. Design to interfaces.
    2. Favor aggregation over inheritance.
    3. Find what varies and encapsulate it
    (encapsulation means hiding not only data, but implementation, types, everything).

    Lets see ES in this context.

    From what I understand:

    1. Only system can write data into their component (and only into their component).

    So system will have two types of methods:
    a) methods that work with one component and receive it as parameter
    b) methods that work with many components (two or more of same type and other parameters)

    In a) case methods of system will look like this:

    some_return_type method_name(MyComponent* my_component, … other parameters);

    and if they can only WRITE data to my_component , what kind of method is this?
    This is instance method of MyComponent class.
    Because instance methods that change only this class’ instance data are instance methods and
    in C++ they look like this:

    some_return_type method_name(MyComponent * this, … other parameters);
    where *this – is hidden parameter.

    So we can easily move these kind of methods from system to component class.
    Because this is the same.

    What about b) case? It is better for those type of methods to stay in separate “System” class.

    So components now not only have “GET” and “transform before GET” methods but also methods that can
    change component data.

    2. Entity is just a name(id). But why name has separate class in programming language? Actually I
    don’t see reason why you need entities at all. Because instances of components are entities
    (or entities are just instances of components that can contain other components).
    For example I have components: leg, arm, body, head. Using these components I can form “entity”
    - human:
    1 left leg – instance of leg component
    1 right leg – instance of leg component
    1 left arm – instance of arm component
    1 right arm – instance of arm component
    1 body – instance of body component
    1 head – instance of head component

    But if enemy throw granate to this human, and it will tear her apart,
    left leg – can become separate “entity”.

    Also that human “entity” actually can be component instance of Army “entity”
    (component instance of army component). But I also can remove this human component instance from
    army component instance and add it to civil people component instance or dead people
    component instance.

    In DB you don’t need table 3 (from tables you proposed).
    And table 5 will look like this(renamed it to component_instances – to be clear):

    Table 5 : component_instances
    id parent_id component_id component_data_id

    where id – autoincrement field, id of instance of component
    parent_id – self link to parent instance component
    component_id – link to “type” of instance of component
    component_data_id – link to “real” data of this component

    If component don’t have parent, then parent_id is NULL.

    Why it useful? Because everything are hierarchies.
    Instance of human leg is usually a part of human instance. Human instance component is
    usually part of some community intance component, etc.

    But not only this. You solve problem when you need system that uses data of few components
    and maybe even “indirectly” update data of few components. How? Simple.
    You create component Collision Detection that “contains” Position component and Collision Data
    component and then create system Collision Detection System that works with this component.
    Collision Detection System can query all Collision Detection component instances
    and for each of them get access to Position component instance and Collision component instance.
    But remember, only Position system and Position component can change data in Position component,
    so Collision Detection System to change data should use other systems and components itself.

    3) Now lets examine ES from OOP design.
    Lets start from end, last rule:
    “Find what varies and encapsulate it
    (encapsualtion means hiding not only data, but implementation, types, everything).”

    What is changing in big games? Parts of different fucntionality, parts of objects, etc.
    And it can even happen in dynamic way – like leg instance is not part of human instance
    anymore. So ES encapsulate these changes.

    Next rule:
    “Favor aggregation over inheritance.”

    Sure, we do this. And if classic aggreagtion looks like this:

    class Human {
    Leg left_leg;
    Leg right_leg;
    Arm left_arm;
    Arm right_arm;
    Body body;
    Head head;
    };

    we do aggregation like this
    (here is very not optimized way, because actually list is not contained here,
    but idea is the same):

    class Component {
    list components;
    };

    class Human : Component {

    };

    In previous case I cannot move leg out of Human and in ES you can. Also you can add third leg
    (mutation???!!).

    So yeah, we have aggregation over inheritance here, and not just simple aggregation – but
    dynamic aggregation.

    And now first rule:
    “Design to interfaces.”
    We see it here too, as components have common interfaces.
    Your component don’t worry about types of other components it contains.

    4) So what is difference between OOP and ES? None, ES is part of OOP but it applies some useful OOP rules:

    a) Only component or it’s system can change data of component
    (pure OOP rule: methods of object can change data only of it’s own object).

    b) It gives us very useful OOP pattern to create games: I would call it ES pattern(or CS pattern?).
    This pattern propose dynamic aggregation to hide variations of object parts. It does this not only
    in programming language level but also in RDB level.
    ES pattern (or CS pattern) is similar to Bridge pattern but ES pattern has dynamic aggregation
    (of operations or components) and additional encapsulation that hides types of operations (components).

    c) You work with instances of components of same type not with instances of
    components as containers of other instances of components.

    5) Can you use inheritance between components?
    Yes you can, but you should avoid it as much as possible because you should favor aggregation
    (in this case dynamic aggregation) over inheritance.

    Consider you have 2 components – Rectangle and Circle (not sure in real world you
    will use such components). Rectangle component looks like this:
    class Rectangle : Component {
    int x1, y1, x2, y2;
    };

    Circle component looks like this:
    class Circle : Component {
    int x1, y1, radius;
    };

    If we favor aggregation, then we can create Point component
    class Point : Component {
    int x, y;
    };

    And now Rectangle component will consist of 2 Point components instances.
    Cicrcle component will consist of 1 Point component instance and radius.

    But we see now that Circle and Rectangle share 1 Point component instance, can we make Shape
    base class for them that contains that 1 Point component instance and then “inherit” Rectangle
    and Circle components from it? Why not? You can. But you don’t need to. As creating of instance
    of component is matter of writing creating function. For example:

    Component* createPoint(string debug_name) {
    return new PointComponent(debug_name);
    }

    createRectangle(debug_name) {
    // rectangle doesn’t have it’s own data, so we don’t need class for it in C++ language
    c = createComponent(debug_name);
    c.add(createPoint(“point1″));
    c.add(createPoint(“point2″));
    return c;
    };

    createCircle(debug_name) {
    // Circle has it’s own data – radius, so we need to have class for it in C++ language
    c = new CircleComponent(debug_name); // class that contains radius only
    c.add(createPoint(“point”));
    return c;
    };

    So we don’t need Rectangle class (only Rectangle system) and we only need Circle class because
    of radius field (yes, technically we can move radius to some component if that makes sense):

    class Circle : Component {
    int radius;
    };

    But in real world you will have all components preconfigured into assemblages somewhere in db
    (tables 1 and 2).

    So component creation will look like this:

    // nothing changes for this component
    Component* createPointComponent(string debug_name) {
    return new PointComponent(debug_name);
    }

    createRectangle(debug_name) {
    c = createFromAssemblage(“rectangle”, debug_name);
    return c;
    };

    createCircle(debug_name) {
    c = createFromAssemblage(“circle”, debug_name);
    return c;
    };

    So inheritance is not needed.

    What are default values for data in components? There are two ways, or you use default values
    for columns set in DB or you create your own default data initialization layer (usually using some
    config files).

    6) Another interesting question is,
    can same component instance be part of other two componont instances?

    Why not? Same human can be part of two different organizations.

    On RDB level, ES pattern allows creation of dynamic links between tables. And same component can
    be used by 100 other components.

    Best wishes,

  10. adam Post author

    @Oleh

    You seem to have ignored the performance issues and the maintenance issues. 50% of ES is “making it possible to write programs that can’t realistically be written in OOP”. 50% is “writing code that runs faster than OOP code”.

    If you don’t care about writing real programs, and you don’t care about performance, then … no, I see no particular advantage in ES.

    That said, a few minor corrections and comments:

    1. You have just taken the PRIVATE method out of a PRIVATE system and inserted it in an UNRELATED CLASS — i.e. you destroyed all encapsulation, and achieved … nothing. Why would you want to do this?

    You also took a lot of HIGH PERFORMANCE code and wiped it out. Objects take up lots of memory, raw data doesn’t. Object graphs are slow to traverse, raw data is fast.

    2. Entity doesn’t have a class. In some implementations, as a convenience, people add a class for it. But it’s not intended nor required.

    Also, relational databases are incapable of storing tree hierarchies without an infinite drop in performance. Your modified table is unusable in any real world program – it would be too slow.

    4. ES is fundamentally relational in nature. ES disallows “methods”. ES disallows “objects” and “classes” within the app design. How is this “part of OOP”? It’s not “object oriented” it’s “object anti-oriented” :)

  11. Oleh Novosad

    Thanks for quick answer,

    1) Private methods? So systems methods are private? Ok. But they have methods that works only with related components , like PositionSystems works with PositionComponent. For example method PositionSystem.reset(PositionComponent c). You can easily move this method to component and it will became PositionComponent.reset() method.

    I mean PositionSystem.reset(PositionComponent c) == PositionComponent.reset(). In C++, I don’t see any performance overhead. Because compiler transforms PositionComponent.reset() to static PositionComponent.reset(PositionComponent* this) – same method as if it is in system. As PositionComponent is class anyway and has base class – Component.
    You have to hold your data in some struct or class anyway.

    2. Agree, building deep hierarchies will be bad. Especially for MMOs where you have a lot of players and a lot of hierarchies. Because it is too many operations for one table. There should be way to optimize it. I like idea that you can combine few components to create new and have new system for combined components.

    My question: so I have Human entity, it consist of few components(leg, arms etc), where I put code that do read data from few components and do something?

    From my point of view – I could consider Human as new component that consist of other components and write system for it. Or should I not use ES for that problem?

    4. I came from ruby on rails background where we have FAT models and we don’t have any problems with idea that we are using relational db. ES uses Systems with static methods – but in fact many of those methods equals to instance methods (as I described in 1.)

  12. adam Post author

    1. But WHY!?? Why would you do this? What does it buy you?

    We know it COSTS:
    – it makes the code much harder to read, because now the code for ONE algorithm is split into lots of DIFFERENT classes / files
    – it makes the code MUCH SLOWER to run, because you just converted a raw array into an object

    2. What are you trying to do?

    4. Yes, RoR runs slowly and is extremely poor at handling objects. I don’t see how that justifies using anti-OOP programming and claiming it’s OOP :)

  13. Oleh Novosad

    1. So you just query data from memory or db and process it with system? You even don’t need to create instance of your component class for each record from db?

    2. I am trying to create AI that uses data from few components (like position, health, weapon possessed, etc.). Should I create it outside ES and it’s rules? I don’t understand this.

    4. RoR is comparably slow because it uses Ruby – and Ruby is not compiler but interpretator. Also Ruby is very dynamic language (like you can create new classes , add methods and properties to it and create instances of it during execution). If I will understand question 2., maybe I will understand why ES is not OOP.

    thank you for answers Adam

  14. adam

    1. Or from the network – multiplayer game-state, for instance

    2. Give an example of what you’re actually trying to do. Right now, you seem to be worrying, but haven’t encountered an actual problem.

    4. It doesn’t really matter if it is or isn’t. The point is (as I explained in the first blog post): if you think of it in OOP terms, you’ll probably never understand it. You need to learn to throw away OOP, and then you’ll find ES easy.

    Otherwise, you’ll keep trying to turn it into OOP. It was designed to escape OOP, so if you keep trying to drag it back … you’ll find it really difficult to make a good game.

    IMHO

  15. Oleh Novosad

    From part 2 of article:

    “Each System runs continuously (as though each System had it’s own private thread) and performs global actions on every Entity that possesses a Component of the same aspect as that System.”

    So System can read and change data of entire Entity “that possesses a Component of the same aspect as that System”?

    I mean, can System read data and write data to other(not of System’s aspect) components of Entity? Or only read? Suppose read mode on components of other aspects should be enough.

    Suppose I have AISystem and AIComponent. Now I assign AIComponent to every Entity that should be processed by AISystem. Then I will query all Entities that have AIComponent, read data that I need for AISystem from other components of each Entity, process some ai algorithm and save some results (like decision what to do) in AIComponent of that Entity. Suppose AISystem decided to do action “move” and saved that decision into AIComponent for Entity.

    Now I will process MovementSystem, this system queries all Entities that have MovementComponent (not all entities can move). Do for each entity cycle. If Entity has AIComponent and decision made by AISystem is to move, then MovementSystem processes movement for that component.

    Sounds good.

    But what if AISystem need information not only from Entity it processes but also from other entities that don’t have AIComponent component. For example, to decide if I should move to left I need to check if there is not some dangerous object (like flame Entity – which has not AI) ?

    And second question, suppose I have Human Entity and that Entity has two legs. Each leg is component of same aspect. What to do in this situation? Should I support few components of same aspect for Entity?

    Thank you, Adam

  16. Oleh Novosad

    I think I already have answers for my questions.
    And thank you again for great articles.

  17. roka

    Very interesting article(s) and comments. As the discussion keeps going this topic remains hot.

    I have some questions though I plan building an Javabased ES for a mmo browser-game engine. As rendering happens on clients while non-presentational game logic resides on the server (multiple systems :D) I have a split of the systems in two layers located on different machines. Rendering though is actually not a problem as I focus on the server-side concept of the engine. How would you build a system not based on a game loop but on request/reply concept?

    Think of a situaion like this.
    Player A calls an action to attack Player B with an Army C in his base located at D.
    -> we have 2 player entities, one army entity (which somehow needs to be a component of player a?!), and a base at a given location.
    Player A fires an action with the given parameters to the server which has a method for handling this case (possibly the mehtod path is stored as component_data aswell?!). That method belongs to a system in the sense of an ES. As it’s an system belonging to an component it get’s all necessary component_data, calculates the results and both answers the client and store the possibly altered data. Would you think this is a good way to solve that?

    How do you think about using NoSQL databases like Couchbase (formerly known as membase) for storage?

    Client/Server-Communication is not a problem as I would use jWebSocket for communication between those layers and there for go for a request/reply concept with async-RPC to the relevant systems (e.g. CombatSystem, ShopSystem, …).

    Francois had some questions as well, which you didn’t answer. I guess you just didn’t see it or forgot about it. Those questions answered would help me aswell.

    thanks, roka.

  18. adam Post author

    There is always a game loop. Unless you are using some freaky hardware that isn’t manufactured by Intel/AMD/etc.

    NoSQL databases are mostly “databases for people who have small or simple data sets, or are too lazy to learn SQL”. If you have an exceptionally simple game – e.g. tetris – then a NoSQL database would be less effort to program against – go for it. For any real game, you should be using SQL. Or not using a database at all.

  19. roka

    The game loops main purpose is to update the view depending on the changed game data. However game data may change without some one actually sees it. Think of an attack of an army while both users are logged off, as it doesn’t happen instant because the army has to get to the other players base. In this case the event still is going to happen but no “client” actually sees that change to the game data.

    That rendering stuff actually isn’t interesting to me in the first point as the “rendering part” of the engine will have to work on the clients machine and my goal is to really decouple client and server. That way you may build an specific mobile app using the same data (player account) and server while using a different client.

    It’s really going to be more async RPC call like or even RRPC call like as jWebSocket for instance supports it.

  20. adam

    Game loops are NOT about rendering.

    They are about the entire game.

    Generally, you have a game loop on server, a game loop on client, and your asynch comms injects timestamped data from one to the other.

    Generally, the server then uses an algorithm to overwrite or re-write the client data, and the client accepts what the server tells it.

    Generally, the client then uses an algorithm to extrapolate from the data the server gave it, in order to render more smoothly.

    (either or both those algorithms can be trivial – seemingly non-existent – or arbitrarily complex – e.g. a full multi-way data merge on server, using multiple sliding windows of game-state deltas per-client-squared)

  21. adam

    …and none of that should affect your ES. Your ES should be updated *by* those algorithms, and should carry on running locally as if none of this existed.

    As it happens, the datastructures used in an ES generally make it very easy to implement the more advanced / tricky algorithms both for client and server. This is because they are nearly always data-driven algorithms, so the ES obsession with data-centric design fits neatly.

    …but you wouldn’t expect an ES to know *anything* about networking.

  22. roka

    So those algorithmes should not reside in the ES-Systems, right? They only allow access to them to get data or change it? If I got you right, those ES-Systems are more like an abstraction “layer” between somehow organised data (through assemblages) and a kind of game-logic-layer that carrys the algorithms?

  23. Mythgarr

    I posted a comment on a different post but adam suggested posting here would receive more replies. I am re-engineering a platformer game from a few years back to use the Artemis Entity System (primarily as a way to become familiar with ES development). I’m trying to figure out how best to implement a few features -
    A player can only jump when they are standing on the ground. Player input is handled in a PlayerControlSystem, collision is handled in a CollisionSystem. What is the best way to deal with only allowing the player to jump when they are standing on the ground? Should the CollisionSystem set a property on a Component that the PlayerControlSystem looks at? Likewise, if I want an AI to react by turning around when they hit a wall, how should that be handled? I can keep adding flags for reacting to every specific situation but that would lead to Component bloat. I could handle every possible collision situation within the CollisionSystem (ie: having an AI turn around, damaging a player, checking if the player is trying to jump and reacting accordingly, etc) but that would lead to an ENORMOUS event handler within that system.

    What is the ‘right’ way to do this?

  24. Yannbane

    I’m very inyerested in roka’s and Mythgarr’s last posts, can anyone reply?

    For example, how should people using the engine implement logic in their games? Making custom systems? What if the functionality conflicts?

  25. adam

    re: collisions.

    Create a “current collision” component with an enum of current case + any other data you need. This is essentially how all physics engines work already! (although usually they have more than one possible situation at once, per object)

    when the player is in mid-air, remove/delete the collision component until they hit something.

    Then other systems can look for and react to the presence and data of that component.

  26. adam

    I don’t have much time. If I’m too busy, or a question is too incomprehensible for me, or I can’t see a short way of answering it, I’ll generally leave it for others to answer. If no-one answers, it’s probably because the question was too hard to understand – or required too much work *not specific to ES* to answer.

  27. Chris Ochs

    @adam

    - You hinted that if it wasn’t for the performance gains, ES might not be the best solution. Wouldn’t the data-driven aspect still be enough, or do you see a better way to build a data driven game with reusable components that is better then ES (performance not being a factor) ?

    - Design of systems… Seems like there has to be a systematic way to manage system dependencies and order of operations within systems. In RPG like games everything ends up touching everything else. Without some declarative way to manage this, the system becomes very hard to follow. I’ve thought of using state machines, or some type of declarative syntax so it’s obvious by looking at the system how the logic flows.

  28. adam Post author

    Personally, I think it’s still the best solution for almost every game I make, because of the benefits to designers. But it’s hard for me to measure how much that’s coloured by the specific games I tend to work on :)

    For system/system dependencies, there’s been a few suggestions in the comments in here and the other posts on ES’s. Personally, I like the meta-declaration system where each system has:

    – a set of pre-conditions (systems that must run first)
    – a set of post-conditions (systems that must run after)

    It’s trivial to write some code to load those from e.g. an XML config file, and to automatically pick a runtime order for you – and to error if you ever accidentally specify impossible conditions.

  29. Tom H.

    Yep, I’d agree that you really want to use something like Adam’s “meta-declaration system”. A former employer built an entity system around assigning a static priority number to each system, and ran into many problems because that was hard to maintain in practice, and very conducive to performance problems.

  30. AMCerasoli

    Ok, and what if create a ‘map’ for each component in my videogame? That way each system knows already what components are going to be used, after all, each system works for each component.
    So, each component has its separated std::map container with the key being an integer and the content being a component to a specific entity and when the system is checking for example to check collision detection using all the “positions” in the component ‘map’, in case it needs to perform special operations on a specific entity, what it has to do is just take the id for that specific entity and search for other components using the normal “query” way.
    This is because maybe I haven’t understood this approach completely, but it seems too expensive to make those “queries” all the time 60 FPS and knowing that many of them don’t even need to know about other entities.

  31. adam Post author

    Prove it before making changes or else every change you make will be a stupid one.

  32. Chris Ochs

    Our team has been hashing out how to represent relational data when using an entity/component system. Pretty much all of the examples and discussions on the net are about rather simple systems. But real games get a lot more complicated.

    The best thing I’ve been able to come up with so far is to allow components to reference other entities by asking ‘give me the entities that have these components’, or ‘give me entities that have these components with these values’.

    As an example of something where you need this type of relation, imagine a cost structure, where the cost of an entity is some currency and another entity or entities, and/or it might require you to already own a specific entity to make the purchase.

    Or you have some type of effect that gets applied to a player, that itself can apply other effects, cause the player to gain/lose items (entities), etc..

    Most of the time these relations look like a tree or graph.

    This seems to work fairly well to represent almost any type of related data.

  33. Adam

    @chris – please write a blog post about that, with the concrete examples you’re using to test the design – eg your purchase example.

    From the brief description it sounds like you MIGHT be overcomplicati g things – but without more detail its impossible for the rest of us to fully appreciatewhat youre running into.

    Also, it would be a very useful “worked example” for a lot of people, i think, to help them understand.

    Fundamentally … Standard databases are appallingly bad at storing tree structures, so id expect an ES to fail horribly (in performance terms). I dont know what the state of the art is in tree databases, but last time i looked (5+ years ago), there was none, outside of extremely expensive proprietary systems.

    Most people seem to have gone even further away from tree structures – eg the recent excitement aboit mongodb and key/value stores?

  34. Pingback: Drafting up an Engine Framework « Programmatical Musings

  35. Ronald

    Adam, thanks for everything, it’s really something :-) And not just for game development i think.

    One question i’m struggling with: do you think it’s ok for Systems to have a sort of (simple) messaging/event System to pass (for instance) environment events (window size changed, mouse click, network disconnection, etc). Sometimes i feel that i need a little glue between the Operating System and the game, and sometimes even between Systems. I thought that maybe a simple event system would still make Systems not too dependant, and each System can decide for itself if it should handle an event or not.

    Thanks in advance!

  36. adam Post author

    Do whatever you want inside the Systems. In particular: typically, different systems are different problem-domains, so it’s a great point to use a different paradigm (even a different programming language, if you’re using e.g. C++ with Lua, or C with Python, etc.). For instance, the design needs of a physics system (perhaps: easiest to use low-level hand-optimized C with careful memory management) could be radically different from those of a in-game “triggered events” system (perhaps: everything written in Python, and loading user-created scripts at runtime).

    They are especially good for wrapping all the OS-interfacing, hardware-interfacing, etc. All that non-logic-stuff.

    Just ensure that all your *game logic* is done without message parsing, without OOP method/data coupling, etc.

  37. Ronald

    Alright, that is very understandable and clear! It’s perhaps very useful to have like 2 or 3 Systems in a separate folder, which serve as OS or network wrappers, to keep good overview for porting to other environments.

    One other small question, then i’ll stop and resume programming: Is it possible to add two same component types to one entity? Like a tank having two guns. Or is there a better advice to solve this (ie make two component types)?

    Thanks for your quick answer. Very excited about this, even for web development.

    Cheers,

    Ronald

  38. Ronald

    One little sidenote: I kind of predict that as a project grows larger, an ES would probably get a little messy. A pure OOP based engine had a little advantage on that maybe…. So i decided to add good human-friendly descriptions to all (meta)types, and i’ll try to organize scripts as clear and logical as possible. I know it’s not a problem at this moment, and i shouldnt worry about it… It’s just something i can imagine to be important at some point.

    Best regards,

    Ronald (The Netherlands)

  39. Anthony Pigeot

    Hey,
    I read all your post and that seems all great.
    I’m trying to make an implementation of this system in Javascript (using SQLike, a SQL clone for JS – http://www.thomasfrank.se/sqlike.html ).
    Even though I read all your numerous warning about not using classes and not mixing OOP and ES, I still can’t get to understand why.

    I mean, I know you said components are not classes and entities are not objects, which is true, but what would keep us to implement them in such forms ? Why shouldn’t I create an entity class that I could instantiate, that had an addComponent methods, and whose components would also be classes instances, giving me the benefits of adding methods, setters and getters ?

    I’m writing my ES system, and working with SQL call each time I want to modify data is a pain. And I have the feeling that I’m still accessing an object’s property, but just instead of creating a real object, I created an SQL table, complicating the process of accessing it. Then, next to it, I’ve got a bunch of big functions that I call systems, which will act on the components. That gives me the impression of just giving a bunch of singletons (say, RenderingSystem using all the Renderers), which ends up not really different from actual OOP.
    The only benefit I see is how easy it will be to use an SQL database with my server. But on this topic I’m wondering, what about noSQL ? I like the json, it is a great structure, and your ES system could be implemented this way too, would that be bad ?

    So do you have some examples of the bad consequences this would bring ? I’m not a very experienced programmer (only a few years, still student) so maybe I lack some of the “coding maturity” required, but I just don’t get it, and it is sort of killing my motivation to implement it.

    For example when I look at the Unity’s system, it is the “bad” way of doing ES you mentionned : Unity’s components are classes with methods and properties which are just put inside a gameobject, and that is all running pretty well.

    Sorry for so much questions but I think I’m missing something here.

  40. adam Post author

    It sounds like you should follow the warnings, then.

    Why are you trying to use an ES in the first place?

    If you’re struggling to understand it, maybe you’d be better to not use an ES, instead write a few projects using other approaches, and come back to this when you understand better what you’re trying to avoid.

  41. Anthony Pigeot

    Well I’ve had a lot of problems with managing lots of data for online games (or any sort of content based games actually) with OOP, mostly for objects initialization and real-time data save/loading to/from server, so I kinda see why another system is needed. What I don’t exactly see is how using these entities won’t make me front the same kind of problems, just presented differently…

    Anyway I will try to make a basic implementation of this to see how it ends up, maybe I’ll understand its benefits by then.

  42. adam

    “with OOP, mostly for objects initialization and real-time data save/loading to/from server, so I kinda see why another system is needed”

    In that case, you should see that an ES makes this part literally *trivial* – you can write the full client/server code in a couple of hours — or write an extreme, AAA engine version, in a couple of days, future proof for (almost) all future uses.

    With an ES, suddenly all your comms code became boilerplate, handled by a small number of for-loops that cover *every possible* situation.

    (unless you want to implement complex algorithms – but even then, your code is pretty simple, since it’s acting on pure data)

  43. faris

    Adam,

    Regarding your last message, do you actually suggest writing the client using an ES? Most client tasks, e.g loading assets into memory, sending network packets, rendering, physics prediction etc are all distinct layers that don’t really fit into the style of an ES. With a single scene graph intended for physics/rendering there actually isn’t much data to process on the client.

    The power of the ES, as you have stated many times, comes from the ability to iterate over large sets of data quickly and parallelize your code, which I only see being able to take true advantage of on the server.

    Aside from keeping a common ES codebase between the client/server, is there a compelling reasonthat I am missing?

  44. Anthony

    Hi,

    I’ve been building my Entity System on node.js and I have a few things I am wondering:

    My database is MySQL. I have made a few performance tests to see how it runs, and considering a 10,000 components DB I get select/update times of more than 10 ms. Considering a game loop runs in 16 MS (or 32 for 30 FPS), and that many different requests may be needed by each system, it seems extremely unrealistic to me that I could run a complete game loop in such a short time.
    These tests were run on my macbook, not on a dedicated server.

    Here are the results of my perf test:

    ===== PERFORMANCE =====
    Time to create 1000 entities: 7
    Time to select from a 10,000 entries table: 34
    =====Entity creation and modification cycle====
    ․Time for creating one entity : 1
    ․Time for creating a component : 1
    ․Time for getting a component : 11
    ․Time for setting a component : 15
    .Cycle time : 32

    Just the get/set requests alone take an entire game frame. So I know you said in your posts that this is meant to be run on servers, but what type of servers do you think would be able to run these sort of requests in a reasonable amount of time?

    I just want to do this as a personal project so I started developing my ES system, assuming I would be able to run this correctly in a standard affordable server, but seeing the results on my computer I am worried.
    I don’t know much about MySQL, so I have no idea if my performance are usual of if there is a way to optimize all this.

    I am also worried about the iteration process. If my computer isn’t powerful enough to run the server, it means that if I am working on the server I have to deploy to my distant, powerful enough server each time I want to test. Isn’t that a big loss of time?

    Also, it seems to me that a huge amount of requests will be made in a game. Each system has to get data to process, sometimes it may have to query the database multiple times (for example a physics component getting only a certain category of the data, then another, etc). Is it really possible to have low enough SQL access time to support these constant requests?

  45. adam Post author

    Mysql is a disk-based DB. For what you describe, you almost certainly want a memory-base sql server instead. You (probably)dont need/want to write every change every frame to disk – maybe not even once per minute.

    If you dont know how to optimize mysql, and dont want to learn it, either use a memory-only db, or do t use sql until you actively want to save. Maybe write copies to sql when key evnets happen (player gets loot, player dies), and everything else only wrote every five minutes or so, as a background process on a different thread.

  46. Anthony

    That’s what I figured out when I ran the server on my computer that had an old hard drive and I heard it make so much noise. I initially thought that MySQL kept in memory the last used tables and stored in the hard drive automatically in background, but it seems not.

    Thanks for your answer, I will rewrite this with a on-memory SQL server to see how it runs

  47. Pingback: Alternatives to storing all my entities in an SQL database | Question and Answer

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

Current month ye@r day *