Central Tech: Essential strategy, or complete waste of time?

I work at a company where managing and directing a subset of the global core-tech is part of my day-to-day job, and I’ve been trying to find good, positive ways forwards. After a year in this role, I’m looking back and wondering whether it’s working for us, and where it’s been good and where it’s been bad. I don’t know what the perfect solutions are, so I’m just going to document the issues I’ve been wrestling with, and my current thoughts on what causes them, and my current ideas on what can be done to fix or avoid them. Personally, although I’m confident there is at least some role for some form of central technology in many games companies, I’m not sure yet what that role is.

Definition

Interestingly, there are no clear definitions on Google/Wikipedia for either “central technology” or “core technology” (the two main names I’ve heard used throughout the games industry). I see that as a warning that this is pretty under-explored territory I’m heading in. Cool.

The vague definition is: “sharing technology between multiple game-project teams and taking any new technology created during a game project and re-using it afterwards for other projects”.

There appear to be two major concepts wrapped up in the central-tech teams that I’ve seen. Some teams have only one of these, some have both. Usually, there are substantial examples of both, mixed within the same team. Often, the team has one focus in particular, but people outside the team in the rest of the company pick one or the other to *believe* the team has as their focus, depending upon what they personally most want out of the team. When this is divergent from the team’s own internal focus, things are off to bad start already.

Centralised technology (control, direction, authority, mandating):
– control: deciding what technology to create for “general use” by game-project teams
– direction: researching and reporting to the teams what tech is needed over the coming years by all game teams (yes, including telling teams themselves what they need)
– authority: deciding all general and long-term technology choices for ALL game teams, and making decisions on what tech teams are allowed to create and use
– mandating: forcing game-teams to use the centralised technology

Shared technology (re-use of common code, standing on shoulders of others, increased stability, decreased startup costs)
– re-use: finding code that gets written the same way on successive projects, and turning it into a library then never re-writing it again
– standing on shoulders: getting a domain expert to write difficult code once, turning it into a library, so that people who don’t understand enough of the tech to write it themselves can leverage it into future games without having to learn it themselves
– stability: using code in multiple successive projects so that eventually, after a few projects, it is extremely heavily tested and bug-fixed, much more so than any code written and used for the first time in a given project
– decreased startup: reduce the amount of time after a project goes into production that it takes to get the “first playable” version of the game running, because a lot of “typically useful” code already exists; unlike “re-use”, this is NOT necessarily code you will keep in the long-term – it’s just a leg-up to get you started quickly

Problems (real or perceived)

In some cases, Central Technology / Core Technology teams have been growing on game development companies like mould on a piece of bread in recent years – gradually expanding their remits (and team sizes) until they end up affecting (usually negatively, despite the best of intentions!) every part of game development. I’ve had lots of experiences with centralised tech depts, although all of them in the games industry – interestingly, I never saw any software company even consider doing this when I worked in mainstream IT (across various industries), only games companies. And it seems a relatively recent phenomenon, too – at least at the scale that we see nowadays.

Typical problems cited by game-teams and third-parties affected by coretech (NB: these may not be fair!):
– a bunch of non-game programmers inventing arbitrary (undirected by user) solutions for potential (not verified because no concrete use-case) problems for not-yet-existent (haven’t started production yet) projects
– forced upon game-teams who have to twist and force their game into the inflexible (already built, without knowing the game-teams requirements because the game-team wasn’t around at the time) mould of the now-built core-tech (ask any EA employee about centralised technology initiatives…)
– generates pointless tools that are always in search of a problem to solve

Ultimately, the more bitter and cynical end of the spectrum of those who’ve experienced coretech come up with something that veers towards one of:
– it’s an excuse to spend infinite amounts of money forever without clear benefit and no oversight or restraints on feature set or spending
– it’s an excuse to take freedom and power away from game-teams and centralise it under an authority who has no reason to care if the game ships on time or with most appropriate feature-set

I suspect those two ultimate, generic criticisms map to the two typical definitions – the former to “shared tech” and the latter to “centralised tech”, although I’ve not looked at that in detail.

Most (not all!) of the experiences I’ve personally had in games companies have been failures, some narrowly (managing to achieve good things, but failing at what they were originally setup for), some spectacularly (failing to provide anything good at all, and even dragging down one or more game-projects with them). Some flailed about and killed projects in their wailing death-throes, others gorged on the corpses of the projects they’d killed and rose again from the bones of what was left, like some undead creature (which is a worrying precedent: “how do you kill that which has no life?”). It’s fair to say my experiences have been mostly bad ones :) … BUT: until it was called “central tech” or “core tech” the *same kind of initiatives* (re-using code, centralising decision-making, making powerful proprietary libraries) seemed to work well, generally, in teams that I was in or working with. The failure cases seem (at least in my small amount of experience) to be a relatively modern problem.

What’s the point?

Speaking to people throughout the company where I currently work, and working with some of our pre-existing core-tech teams, I’ve found myself more and more asking a simple question: “why do we have Core Tech?” The answers have been illuminating…

“to write everything-including-the-kitchen sink, then on each game project only use the bits you need”
“recycle code from an existing project”
“I don’t know”
“write code that can be used and re-used in multiple projects”
“share the really cool and difficult stuff that one team did with other teams so they can benefit from its brilliance”

This has made me realise / remember a couple of things:
– non-programmers, and many inexperienced programmers, grossly underestimate the ratio in development time/cost between “making code that solves one problem” and “making a generic solution to a whole class of problems of which the current problem is just one example”. The first explanation described above is usually prohibitively expensive by *a large factor, maybe even an order of magnitude*, but many people seem unaware of this
– CT is sufficiently ill-defined and *poorly marketed internally* at most games companies that many internal dev staff genuinely don’t know why it exists or what purpose it’s supposed to be solving, let alone have the opportunity to rate the success or failure at that purpose
– many people notice that game development often involves re-solving or re-performing similar tasks, and wish for a perfect world in which you only ever have to do one “kind” of solution once, and then you get to re-use it forever after. That is largely true in mainstream IT, but sadly this seems to be far from the truth with games development. Personally, I suspect that the day the technology becomes derivative is the same day you realise your game itself is derivative, and has lost most of it’s fun.

I also wonder: if we don’t have CT, are you saying we wouldn’t re-use code? As a programmer, do you avoid finding code from old teams and finding ways to use it in your current project? Isn’t this a *basic part* of your job, part of what we pay you to do?

Is CT something that gives game-programmers an excuse to be less diligent than they would otherwise willingly be, thereby setting the scene for them to later complain that it’s “not as good as if I’d written it myself” (which is one of the cruelest things I’ve seen game-teams do to their CT teams, even if it’s true).

Other side of the coin: why is central tech hard?

Project teams steal the best hires
– Project teams, especially Producers and Leads, have a hugely difficult problem (ship a game on time, and make it “fun”, so it sells well) and a huge amount of personal buy-in (if it fails, they have no second chance: their sole responsibility is that one project), and a huge amount of measurability of success (how many sales does the game make?). They have a vested interest in finding the best of the best people, and they have – ultimately – large budgets to recruit them with. If someone really good at developing code for games is in central tech, they usually get stolen for a game-team sooner or later. Central tech departments have too little visibility (how do you measure the brilliance or crapness of your coretech? there’s no sales figures!), no concrete deadlines (they randomly invent them, or borrow a game-project’s deadlines, but they really don’t sink-or-swim on that project-deadline like the game-team itself does!), and usually restricted budgets (if not, well, that’s when the unlimited-spending-without-reason problem kicks in…). They also are rarely as cool as working on a “real game”: most people joined the industry to Make Games, not to Write Random Code For Someone-else’s Game. CT finds it hard to retain the best people…

Writing code a second time is generally a lot cheaper than modularising the original version
– …I’m hoping this is self-obvious. But sadly it means that it is ALWAYS easier for any individual game-team to re-write *from scratch* the code they wrote for their last game than it is for CT to make the generic version that they’re usually asked for. Game-teams then get pissed that the CT team seems incompetent compared to them (they notice that the time they have to wait for CT to do something is MORE than it would take them to do it themselves).

CT is always setup as an “ongoing” task: there is no defined beginning or end to what they do. Games companies (developers and publishers) are very specifically geared towards “project-based” tasks.
– From the funding, to the project management, to the staff rewards schemes, everything about the games-industry is setup for project-by-project operation. We know (sort-of) how to make project-based development work (reasonably) well. We don’t really have ANY experience in non-project work – certainly anyone who’s only ever worked in the games industry (who are strongly self-selected by current standard recruitment procedures) will have no experience of non-project development. Is it any surprise we often screw it up? Moreover, is it any surprise that we have such difficulty making the non-project CT team be respected and understood by the purely project-based game-teams?

Ideas

I’ve been kicking around some ideas on new directions to pursue. These may not be that great, but at the moment I don’t have to make any decisions, I just have to provide options, so I’m interested in any suggestions or alternatives…

Think of CT as a human resource, not a code repository.

This is one thing I’ve been trying: what *I* needed most urgently out of CT was the *expertise* component, and actually I didn’t really care about the code re-use etc. I knew I would, but I had a more pressing problem of two projects that lacked the necessary expertise in Network Programming to make an MMO. So … I created a tiny team dedicated to doing the network code for any game, and explicitly NOT to writing reusable network technology (NB: many years ago I ran a failed network middleware company, so I had a healthy apprecaition of how difficult and time-consuming it is to write generic network systems; I knew we simply did not have the time to write that stuff when we had projects already in full production that needed network code NOW).

Because of that, we can’t share the technology they create. Not yet, anyway. But we *can* share the expertise of the people in the team. They have to do a fair amount of international travel (bummer). So far, it seems to be working OK, but … it’s too small a sample and it’s been running too short a time (do you have any idea how hard it is to recruit really good network coders? :( ) to make any firm measurements of the success or failure of the approach.

Architect all your CT initiatives on a project-basis, not a service basis

I’ve seen a couple of examples of this, usually where senior people were frustrated by the progress of CT but needed some new example of the kind of stuff it was providing – so they went and started their own new pet-project to provide that, but outside the remit of the CT group. I’ve seen examples of this that were positive, and examples that were negative. Politically, it’s likely to cause a shitstorm as far as I can see, no matter how you sell it (unless this is blessed by CT itself, and none of the examples I’ve seen were their idea, although IIRC one of them did get CT’s agreement – they just knew they didn’t have BOTH the expertise AND the budget to provide it, so were happy to not have to take responsibility for it).

Certainly, it’s been the fastest way to get real significant progress on anything major: by doing it as a project, and going through the “standard” greenlight – pre-production – greenlight2 – production – ship cycle that all games companies are accustomed to for their games, large amounts of money have been secured quickly, and the projects have got going quickly.

Embed CT teams into the game-teams

The idea with this one is to overcome both the antipathy between game-teams and CT (by making them part of the same team), and also to ensure “automagically” that the CT team-members are working stuff that is guaranteed directly applicable to a real game – because they’re reporting to the Producer of a given game-team all the time.

If it works out well, this could solve all the problems. Although I’m not yet sure I understand how – in the long term – this is any different from having no CT at all. As noted earlier, this is closest to how I remember things working well before the idea of CT gained widespread popularity. OTOH, I’m not sure if that same situation will even work nowadays – the industry has changed a lot since then.

History of CT

I’m not sure about this, but my vague memory is that CT grew in popularity and – more importantly – commonness (i.e. how many companies were doing it) in parallel with the axing of internal game-engine teams and the move towards licensing external game-engines.

I have suspicions that a lot of the confusion of “what do we have central technology for?” comes from this, two-fold:
– a) those companies that didn’t entirely get rid of their engine teams had people left a bit stranded, who ended up being folded into CT, and then tended to carry on working much as they had inside the engine team, despite the different overall task of the dept (and this happened partly because of the lack of clear direction from management about what the new team was for anyway)
– b) many of the companies that dispensed with their substantial proprietary engines they’d been using for years then jumped to Unreal et al found that actually the licensable engines weren’t as good as they’d been lead to (or had lead themselves to) believe: they couldn’t go back to having an engine team, but they needed a way to reduce the huge budgets they were spending on just basic modification and fixing of the external tech, that weren’t easily justifiable on the game-budget (because they weren’t forseen, and were adding NOTHING to the game’s end feature list), so they ended up creating a CT team to hide / provide the budget to do that kind of work

I could be completely and utterly wrong here – this is just some ideas that have been kicking around in my head recently.

EDIT: I’d also like to point to this good post about game-engines, and why you shouldn’t build them (not necessarily a “never do it” argument, but an explanation of why you might want not to).

MMOGchart.com gets first update in 8 months

http://www.mmogchart.com/charts/

At long last, the new version of my charts and analysis is now online! I don’t have numbers for every game, but there are quite a few updates, most notably to World of Warcraft, RuneScape, Dofus, Tibia, and NCSoft’s various titles. Also included are preliminary numbers for Vanguard: Saga of Heroes, Lord of the Rings Online, and Tabula Rasa.

A new look and feel to the site as well, I noticed. I’m glad to see the site back and running, and just in time for GDC too :). I can’t vouch for the accuracy of the numbers, but it’s a hugely valuable service to all of us in and around MMO development to have a detailed comparison like this and to see historically how each of the games fares against its predecessors. Even if 95% of the opinions people form from this analysis seem to be wrong … (remember any of the “no game will go above a couple of million subscribers because the charts show them all stealing from each other and none growing the total market” arguments of yesteryear?).

What Web 2.0 means for the games industry

“In reality, one size has never fit all, but when players didn’t have so many choices, they had to put up with it. Now they don’t. No longer can publishers rely on retailing strategies designed to make money by forcing players to buy what the publishers want them to buy, when and where the publishers want them to buy it. These strategies are aimed more at wooing retailers with slotting and promotion allowances than at wooing customers, and they just won’t fly anymore. In the future, retailing strategies are going to have to be like those of Amazon.com or the one-hour eyeglass shops, which are designed to sell the consumers what they want to buy. And they do it by making it easier, better, less cumbersome to do so.”

Continue reading “What Web 2.0 means for the games industry”

Games industry conferences versus blogging

I’m not happy with the direction games industry conferences are going in; I specialize in online games, and I’ve worked at the forefront of monetizing online entertainment, I’ve actually *made money* out of Web 2.0 – so I have real expertise in making use of the internet – and I really think we (as an industry) are missing a trick with our conferences. There’s an opportunity to do something really valuable and re-invigorate the conferences.

The previous entry outlined what conferences profess to offer their speakers, and what it costs the speakers to attend. Now I’m going to talk about the real, untapped, value of conferences to the speakers, and what we as speakers should be demanding, and how in the end it benefits all of us, including the organizers just trying to turn a healthy profit.

Continue reading “Games industry conferences versus blogging”

Problems of speaking at games industry conferences

I go to GDC every year, and also to 2-3 other conferences, but apart from GDC I vary which exact ones from year to year. These days, I’m a speaker at nearly every conference I go to, and I’ve never yet been paid for speaking, so it’s fair to say I have a pretty big time investment in each of them. I don’t make the choice to go to a conference lightly (especially given how long I’m out of the office for a typical conference, and how exhausted I am by the learning, the networking, the partying, and the international travel).

But I’m getting increasingly dissatisfied with the conferences themselves, especially as a speaker. And it seems to be getting worse, not better – and that’s particularly worrying. The conferences are still great, but the problems are significant.

First up, the costs of speaking, and the ever shrinking advertised benefits…

Continue reading “Problems of speaking at games industry conferences”

Creating a New Game-Development Studio

How do you create a new game studio from scratch?

The topic came up recently when someone asked me what I’d do if I had the responsibility to create a new dev studio – what would I do, how would I go about it, etc? I’ve founded a couple of small startups before (both as CTO and CEO), and worked as everything from a secretary to a CTO (Chief Technical Officer) at other people’s companies, so I have some idea of how to do this on a practical level.

I also used to co-run a very successful business-plan competition, where we gave away hundreds of thousands of pounds of startup capital every year. Many of the finalists became good friends of mine, and we continued to work with and help some of the companies for a good few years after they won or were finalists as part of our ongoing support programme.

So, here’s some initial thoughts. I don’t know if this is right or wrong, good or bad, but it’s something I’d like to work out better because I may well find myself trying to do it someday.

Continue reading “Creating a New Game-Development Studio”

How to become an ARG designer (Design a game to cure cancer)

(ARG as in Alternate Reality Game, of course…)

http://www.letschangethegame.org/

A new ARG project from Adrian and Dan, in aid of Cancer Research UK. Great to hear they’ve got this off the ground, it looks like it’s going to be fantastic. But it could also become the biggest event in helping new ARG designers a chance to get their feet wet since the start of Unfiction and ARGN:

People always used to ask me
how they could become ARG designers, and I would always say that they
should try and gain experience – but with such a small field, the
only way to do that is through grassroots games. While people might
have plenty of time to volunteer, grassroots games still cost *some*
money which people often can’t spare. This is a way to give lots of
people experience in thinking about game design, and one team the
opportunity to make a really significant game. — Adrian Hon, Six to Start

RPG Vault – Online Worlds Roundtable #15

Inteviewed for RPG Vault’s latest Online Worlds Roundtable – http://rpgvault.ign.com/articles/817/817490p3.html

“Web 2.0 businesses compete directly with the games industry on multiple levels, and anyone who doesn’t spot that and act accordingly will suffer. If your game doesn’t embrace – or deliberately and carefully reject – Web 2.0, you’ll that your users have created the space you *didn’t*, and someone else is monetizing it. And they’re probably making more revenue than YOU are from providing the core service in the first place!” – Adam Martin

AGDC 2007: Web 2.0 + Games meetup

I’m talking at Austin GDC on “Caching for Web 2.0”, and I’ll be having a small dig at the games industry and the obsession some people have with “game 1.0/2.0/3.0” on the side, but I’d really like to meet up with anyone interested in how best we can capitalize on the lessons from Web 2.0.

I think the vast majority of people in games still don’t “get it” when it comes to understanding web 2.0, and are going to make some really stupid egg-on-face mistakes and miss a few more big opportunities. But’s that just my opinion… :)

What’s yours?

UPDATE:
Venue: Halcyon
Time: 19:00
Day: Wednesday 5th

UPDATE 2: a couple of photos here

Computer Games Industry Careers – Programmers

I’ve been in the games industry for some years now, and one of the things I find slightly annoying is that there’s very little discussion of “career progression”, and that what little there is typically focusses on “how to get your first job in the industry”.

Career Charts

Many years ago, I used to work at IBM’s R&D labs, and discovered that they had whole tables of “possible career paths” that you could download internally and consult when trying to decide what to do with yourself. They pressed hard on everyone to take some time out every 3-6 months to evaluate how their personal career was progressing, think about what they wanted to be doing in the future, and decide what long-term decisions they should be making in the present and near-future to help them get there.

The emphasis was on the individual, and of taking control of (and responsibility for!) your own future. The guides were there to help you find out what a potential future boss would expect of you if you wanted to apply for a job years in the future – when you’re a junior programmer, you may have no idea what a senior manager is supposed to do. No problem – except if you wanted to become one, in which case you wouldn’t know what to concentrate on learning for the next 10 years.

Big companies in the games industry, like EA, certainly do a similar thing internally. But I’ve so far failed to find any decent external, public, guides. So, for the benefit of anyone else who’s ever wondered what they should be doing to further their games-industry career, I’m going to start publishing my own take on this. I’d really really like input and feedback from other people, although there are big problems with conflicting definitions e.g. of what, exactly, a “development director” is. But at least I can start…

Chart 1: Programmers

Typical career progression for Programmers

This is a basic chart showing the main flow of career progression for programmers.

The most important thing to point out is that it only shows internal progression – later posts will show how you can easily move sideways from some of these positions into different disciplines, especially Design and Production. But I’ll be covering each core discipline in it’s “plain” form first, and showing the links between them later.

Programmer levels

Junior – recent graduates and/or people with no games industry experience and not enough years of hard-core C++ coding to jump straight to the Programmer level. Note that most Juniors pick up some specialism – not enough to become an expert, but enough that later they can re-specialise in that role as a Senior.

Programmer (normal) – anyone who’s passed their apprenticeship as a junior, typically with 1-3 years of experience programming on games projects, and credited on 1-2 published titles.

Senior – a programmer who has decided to specialize in an area of programming, becoming an “expert” over and above a normal programmer. Usually someone who chose to avoid a managerial position – although note that on larger teams most Senior’s end up managing Juniors in their area of expertise. However, the management is mostly mentoring, as opposed to Lead’s who do much more project-management-esque roles. Seniors typically start off as a Junior in the same area of expertise, and refine their skills and gain lots more experience whilst a normal Programmer.

Lead – a programmer who manages a team of programmers. There are two types of Lead: the mentoring lead who still programs day to day, who is an expert programmer who could have been a Senior, and the project-managing lead who hardly ever programs (if at all) and spends more time arranging the workload for all the other programmers and helping out the Project Manager(s) / Producer(s) with scheduling and delivery. Both types act primarily as go-between, interfacing between the entire programming team and the rest of the world (design team, art team, and producer). Most Leads are experimenting to see how much they like project management, and may switch to being a Producer or Project Manager later on.

Technical Director – ultra-experienced / skilled experts, usually troubleshooters floating amongst all dev teams, or attached to a particularly large team. They take on all the non-direct-programming tasks that require substantial technical expertise. This often means a big role in hiring, solving long-term problems, and architecting large systems for complex games.

Development Director – this one’s very vague, because more than any other Programming role there’s huge variety in the actual responsibilities of this role from company to company. I’m defining it here as “the person who is in overall charge of all direct creation of all games: programming, art, and design”. Their role is entirely strategic – they may be a skilled technical person (there are many other routes to this role – see upcoming charts in future posts), but they delegate ALL technical issues to their one or more Technical Directors. However, they typically make the final decision on anything affecting the development process or the overall studio and how it develops.

Next … Design (probably. Or maybe Production…)