Category: system architecture

I work at a company where managing and directing a subset of the global core-tech is part of my day-to-day job, and I’ve been trying to find good, positive ways forwards. After a year in this role, I’m looking back and wondering whether it’s working for us, and where it’s been good and where it’s been bad. I don’t know what the perfect solutions are, so I’m just going to document the issues I’ve been wrestling with, and my current thoughts on what causes them, and my current ideas on what can be done to fix or avoid them. Personally, although I’m confident there is at least some role for some form of central technology in many games companies, I’m not sure yet what that role is.

Definition

Interestingly, there are no clear definitions on Google/Wikipedia for either “central technology” or “core technology” (the two main names I’ve heard used throughout the games industry). I see that as a warning that this is pretty under-explored territory I’m heading in. Cool.

The vague definition is: “sharing technology between multiple game-project teams and taking any new technology created during a game project and re-using it afterwards for other projects”.

There appear to be two major concepts wrapped up in the central-tech teams that I’ve seen. Some teams have only one of these, some have both. Usually, there are substantial examples of both, mixed within the same team. Often, the team has one focus in particular, but people outside the team in the rest of the company pick one or the other to *believe* the team has as their focus, depending upon what they personally most want out of the team. When this is divergent from the team’s own internal focus, things are off to bad start already.

Centralised technology (control, direction, authority, mandating):
– control: deciding what technology to create for “general use” by game-project teams
– direction: researching and reporting to the teams what tech is needed over the coming years by all game teams (yes, including telling teams themselves what they need)
– authority: deciding all general and long-term technology choices for ALL game teams, and making decisions on what tech teams are allowed to create and use
– mandating: forcing game-teams to use the centralised technology

Shared technology (re-use of common code, standing on shoulders of others, increased stability, decreased startup costs)
– re-use: finding code that gets written the same way on successive projects, and turning it into a library then never re-writing it again
– standing on shoulders: getting a domain expert to write difficult code once, turning it into a library, so that people who don’t understand enough of the tech to write it themselves can leverage it into future games without having to learn it themselves
– stability: using code in multiple successive projects so that eventually, after a few projects, it is extremely heavily tested and bug-fixed, much more so than any code written and used for the first time in a given project
– decreased startup: reduce the amount of time after a project goes into production that it takes to get the “first playable” version of the game running, because a lot of “typically useful” code already exists; unlike “re-use”, this is NOT necessarily code you will keep in the long-term – it’s just a leg-up to get you started quickly

Problems (real or perceived)

In some cases, Central Technology / Core Technology teams have been growing on game development companies like mould on a piece of bread in recent years – gradually expanding their remits (and team sizes) until they end up affecting (usually negatively, despite the best of intentions!) every part of game development. I’ve had lots of experiences with centralised tech depts, although all of them in the games industry – interestingly, I never saw any software company even consider doing this when I worked in mainstream IT (across various industries), only games companies. And it seems a relatively recent phenomenon, too – at least at the scale that we see nowadays.

Typical problems cited by game-teams and third-parties affected by coretech (NB: these may not be fair!):
– a bunch of non-game programmers inventing arbitrary (undirected by user) solutions for potential (not verified because no concrete use-case) problems for not-yet-existent (haven’t started production yet) projects
– forced upon game-teams who have to twist and force their game into the inflexible (already built, without knowing the game-teams requirements because the game-team wasn’t around at the time) mould of the now-built core-tech (ask any EA employee about centralised technology initiatives…)
– generates pointless tools that are always in search of a problem to solve

Ultimately, the more bitter and cynical end of the spectrum of those who’ve experienced coretech come up with something that veers towards one of:
– it’s an excuse to spend infinite amounts of money forever without clear benefit and no oversight or restraints on feature set or spending
– it’s an excuse to take freedom and power away from game-teams and centralise it under an authority who has no reason to care if the game ships on time or with most appropriate feature-set

I suspect those two ultimate, generic criticisms map to the two typical definitions – the former to “shared tech” and the latter to “centralised tech”, although I’ve not looked at that in detail.

Most (not all!) of the experiences I’ve personally had in games companies have been failures, some narrowly (managing to achieve good things, but failing at what they were originally setup for), some spectacularly (failing to provide anything good at all, and even dragging down one or more game-projects with them). Some flailed about and killed projects in their wailing death-throes, others gorged on the corpses of the projects they’d killed and rose again from the bones of what was left, like some undead creature (which is a worrying precedent: “how do you kill that which has no life?”). It’s fair to say my experiences have been mostly bad ones :) … BUT: until it was called “central tech” or “core tech” the *same kind of initiatives* (re-using code, centralising decision-making, making powerful proprietary libraries) seemed to work well, generally, in teams that I was in or working with. The failure cases seem (at least in my small amount of experience) to be a relatively modern problem.

What’s the point?

Speaking to people throughout the company where I currently work, and working with some of our pre-existing core-tech teams, I’ve found myself more and more asking a simple question: “why do we have Core Tech?” The answers have been illuminating…

“to write everything-including-the-kitchen sink, then on each game project only use the bits you need”
“recycle code from an existing project”
“I don’t know”
“write code that can be used and re-used in multiple projects”
“share the really cool and difficult stuff that one team did with other teams so they can benefit from its brilliance”

This has made me realise / remember a couple of things:
– non-programmers, and many inexperienced programmers, grossly underestimate the ratio in development time/cost between “making code that solves one problem” and “making a generic solution to a whole class of problems of which the current problem is just one example”. The first explanation described above is usually prohibitively expensive by *a large factor, maybe even an order of magnitude*, but many people seem unaware of this
– CT is sufficiently ill-defined and *poorly marketed internally* at most games companies that many internal dev staff genuinely don’t know why it exists or what purpose it’s supposed to be solving, let alone have the opportunity to rate the success or failure at that purpose
– many people notice that game development often involves re-solving or re-performing similar tasks, and wish for a perfect world in which you only ever have to do one “kind” of solution once, and then you get to re-use it forever after. That is largely true in mainstream IT, but sadly this seems to be far from the truth with games development. Personally, I suspect that the day the technology becomes derivative is the same day you realise your game itself is derivative, and has lost most of it’s fun.

I also wonder: if we don’t have CT, are you saying we wouldn’t re-use code? As a programmer, do you avoid finding code from old teams and finding ways to use it in your current project? Isn’t this a *basic part* of your job, part of what we pay you to do?

Is CT something that gives game-programmers an excuse to be less diligent than they would otherwise willingly be, thereby setting the scene for them to later complain that it’s “not as good as if I’d written it myself” (which is one of the cruelest things I’ve seen game-teams do to their CT teams, even if it’s true).

Other side of the coin: why is central tech hard?

Project teams steal the best hires
– Project teams, especially Producers and Leads, have a hugely difficult problem (ship a game on time, and make it “fun”, so it sells well) and a huge amount of personal buy-in (if it fails, they have no second chance: their sole responsibility is that one project), and a huge amount of measurability of success (how many sales does the game make?). They have a vested interest in finding the best of the best people, and they have – ultimately – large budgets to recruit them with. If someone really good at developing code for games is in central tech, they usually get stolen for a game-team sooner or later. Central tech departments have too little visibility (how do you measure the brilliance or crapness of your coretech? there’s no sales figures!), no concrete deadlines (they randomly invent them, or borrow a game-project’s deadlines, but they really don’t sink-or-swim on that project-deadline like the game-team itself does!), and usually restricted budgets (if not, well, that’s when the unlimited-spending-without-reason problem kicks in…). They also are rarely as cool as working on a “real game”: most people joined the industry to Make Games, not to Write Random Code For Someone-else’s Game. CT finds it hard to retain the best people…

Writing code a second time is generally a lot cheaper than modularising the original version
– …I’m hoping this is self-obvious. But sadly it means that it is ALWAYS easier for any individual game-team to re-write *from scratch* the code they wrote for their last game than it is for CT to make the generic version that they’re usually asked for. Game-teams then get pissed that the CT team seems incompetent compared to them (they notice that the time they have to wait for CT to do something is MORE than it would take them to do it themselves).

CT is always setup as an “ongoing” task: there is no defined beginning or end to what they do. Games companies (developers and publishers) are very specifically geared towards “project-based” tasks.
– From the funding, to the project management, to the staff rewards schemes, everything about the games-industry is setup for project-by-project operation. We know (sort-of) how to make project-based development work (reasonably) well. We don’t really have ANY experience in non-project work – certainly anyone who’s only ever worked in the games industry (who are strongly self-selected by current standard recruitment procedures) will have no experience of non-project development. Is it any surprise we often screw it up? Moreover, is it any surprise that we have such difficulty making the non-project CT team be respected and understood by the purely project-based game-teams?

Ideas

I’ve been kicking around some ideas on new directions to pursue. These may not be that great, but at the moment I don’t have to make any decisions, I just have to provide options, so I’m interested in any suggestions or alternatives…

Think of CT as a human resource, not a code repository.

This is one thing I’ve been trying: what *I* needed most urgently out of CT was the *expertise* component, and actually I didn’t really care about the code re-use etc. I knew I would, but I had a more pressing problem of two projects that lacked the necessary expertise in Network Programming to make an MMO. So … I created a tiny team dedicated to doing the network code for any game, and explicitly NOT to writing reusable network technology (NB: many years ago I ran a failed network middleware company, so I had a healthy apprecaition of how difficult and time-consuming it is to write generic network systems; I knew we simply did not have the time to write that stuff when we had projects already in full production that needed network code NOW).

Because of that, we can’t share the technology they create. Not yet, anyway. But we *can* share the expertise of the people in the team. They have to do a fair amount of international travel (bummer). So far, it seems to be working OK, but … it’s too small a sample and it’s been running too short a time (do you have any idea how hard it is to recruit really good network coders? :( ) to make any firm measurements of the success or failure of the approach.

Architect all your CT initiatives on a project-basis, not a service basis

I’ve seen a couple of examples of this, usually where senior people were frustrated by the progress of CT but needed some new example of the kind of stuff it was providing – so they went and started their own new pet-project to provide that, but outside the remit of the CT group. I’ve seen examples of this that were positive, and examples that were negative. Politically, it’s likely to cause a shitstorm as far as I can see, no matter how you sell it (unless this is blessed by CT itself, and none of the examples I’ve seen were their idea, although IIRC one of them did get CT’s agreement – they just knew they didn’t have BOTH the expertise AND the budget to provide it, so were happy to not have to take responsibility for it).

Certainly, it’s been the fastest way to get real significant progress on anything major: by doing it as a project, and going through the “standard” greenlight – pre-production – greenlight2 – production – ship cycle that all games companies are accustomed to for their games, large amounts of money have been secured quickly, and the projects have got going quickly.

Embed CT teams into the game-teams

The idea with this one is to overcome both the antipathy between game-teams and CT (by making them part of the same team), and also to ensure “automagically” that the CT team-members are working stuff that is guaranteed directly applicable to a real game – because they’re reporting to the Producer of a given game-team all the time.

If it works out well, this could solve all the problems. Although I’m not yet sure I understand how – in the long term – this is any different from having no CT at all. As noted earlier, this is closest to how I remember things working well before the idea of CT gained widespread popularity. OTOH, I’m not sure if that same situation will even work nowadays – the industry has changed a lot since then.

History of CT

I’m not sure about this, but my vague memory is that CT grew in popularity and – more importantly – commonness (i.e. how many companies were doing it) in parallel with the axing of internal game-engine teams and the move towards licensing external game-engines.

I have suspicions that a lot of the confusion of “what do we have central technology for?” comes from this, two-fold:
– a) those companies that didn’t entirely get rid of their engine teams had people left a bit stranded, who ended up being folded into CT, and then tended to carry on working much as they had inside the engine team, despite the different overall task of the dept (and this happened partly because of the lack of clear direction from management about what the new team was for anyway)
– b) many of the companies that dispensed with their substantial proprietary engines they’d been using for years then jumped to Unreal et al found that actually the licensable engines weren’t as good as they’d been lead to (or had lead themselves to) believe: they couldn’t go back to having an engine team, but they needed a way to reduce the huge budgets they were spending on just basic modification and fixing of the external tech, that weren’t easily justifiable on the game-budget (because they weren’t forseen, and were adding NOTHING to the game’s end feature list), so they ended up creating a CT team to hide / provide the budget to do that kind of work

I could be completely and utterly wrong here – this is just some ideas that have been kicking around in my head recently.

EDIT: I’d also like to point to this good post about game-engines, and why you shouldn’t build them (not necessarily a “never do it” argument, but an explanation of why you might want not to).