Month: May 2009

The audacity to believe

Do you live in San Francisco? Or, have you ever been there, for a conference, perhaps, or a holiday? (since the games industry’s biggest annual conference takes place in downtown SF, literally adjacent to and physically underneath the memorial)

Have you been to the Martin Luther King memorial?

No, not the famous one(s) elsewhere that are all over the web in arguments and rantings about costs etc. I mean the small, quiet, semi-secret one hidden in the heart of San Francisco, in the Yerba Buena Gardens.

conferences games industry international iphone massively multiplayer

5-year predictions (2009 to 2014) for the MMO/Online Games industry

Post author By adam
Post date May 18, 2009

Last week at the LOGIN conference I sat on a panel with three far more smart/successful/famous people than myself entitled “Online Games 2014: Twelve Spoilers for the Future” (I think I was there as “the argumentative one” ;)). The real value of the panel was the four of us arguing^H^H^H^H^H^H^Hdiscussing each other’s predictions, and the audience suggestions afterwards, but the predictions themselves were pretty interesting alone, just to compare and contrast.

I couldn’t liveblog this session (obviously) and it looks like no-one else did, so – until the slides go up on the conference website – here’s what I can remember of the predictions (I may get some of these wrong, apologies!):

There will be tonnes of cheap rackspace; anything that uses Cloud Computing will be very successful thanks to low cost base
iPhone will become the dominant gaming platform
We’re heading into a big recession that may do well for the MMO industry
Browser-based MMOs will disappear in favour of iPhone/SmartPhone-based MMOs
South Korean MMO Publishers will vanish as a major player in the MMO industry, eclipsed and swallowed by Chinese and SEA MMO Operators (“non-publishing” backgrounds)
Europe will get its first successful Europe-wide MMO publisher, and that company will quickly rise to dominance over the more fractured USA MMO publisher market
Advertiser-sponsored Virtual Worlds will be huge in number and variety
A small percentage of advertiser-sponsored VWs will succeed – but will dominate the mainstream MMO market, since for them “profit is optional”
Traditional game developers will be blindsided by the advertiser-sponsored MMOs
Most PC MMOs (IIRC “90% or more”) will become F2P
Console development-studios will become dominant in the MMO market since they are best at “polish” and very high quality user-experiences
…and one more I’ve forgotten (!). Actually, some of the above I suspect I’ve misinterpreted – have to wait for the slides to be posted to check…

I’ve never before engaged in these kinds of generic future predictions, because I have so little confidence in either my own ability to describe them, or in my ability to understand other people’s ones in a useful fashion. I joined this session because the opportunity to argue them against other people was a lot more interesting. As stated above, I think our conversations on the panel were a lot more valuable than the actual predictions themselves.

Of course, when it comes to more narrow, specific predictions, well … if I really knew the answers there, I wouldn’t be telling you, I’d be making billions out of knowing :). And anyway, at that point you’re effectively asking me what the precise strategy is of my current employer (whoever that may be), which I’m generally not going to be able to reveal :).

FYI the speakers on the panel:

David Edery, previously Worldwide Games Portfolio Manager for XBLA
Charlie Stross, author of Halting State, Accelerando, etc
Tarrney Williams, previously General Manager of Relic Entertainment
+ me, of course

computer games design games design

Bartle explains himself

Richard has often been accused of being “arrogant”, “insane”, and even, simply, “wrong” for his comments along the lines of:

He hasn’t “played” an MMO in decades (possibly “ever”) … because he can’t stop himself from interpreting as he uses them
Surprisingly many MMOs are just WoW by another name
He only needs to play the first few levels of a new MMO to see if it’s really new; he needn’t bother with the rest of it

He’s explained before, in abstract terms, why this is all true *to him*, and pretty much left people to stew if they don’t understand that (while always actively engaging them in conversation to try and explain further).

Now he’s blogged a concrete example of what goes through his head when playing a particular quest-chain / zone: WoW’s STV. If you’ve ever wondered, and/or been confused/horrified/dismayed/insulted by Richard’s statements online and haven’t had the chance to speak to him about it all in person, then I’d highly recommend reading it. I suspect that this concrete analysis will elucidate to a lot more people most of the meaning that the abstract explanations failed to convey. Well, we’ll see…

Uncategorized

LOGIN 2009: Middleware Roundup

Post author By adam
Post date May 13, 2009
1 Comment on LOGIN 2009: Middleware Roundup

Moderator: Joe Ludwig (JL)
Patricio Jutard (PJ), Three Melons
Chris Dyl (CD), Turbine
Rick Lambright (RL), Gazillion
Mitch Ferguson (MF), Carbine

Summary

A lot of nice little anecdotes about MMO middleware experiences – but it would have been nicer to have had a lot more detail on them (not easy to do in panel format).

The panellists briefly touched upon several few sacred cows – such as “Is it absolutely essential to have the full source code to exernally-licensed middleware?” (which I’m glad to hear being questioned – with server middleware especially the developers are hopelessly underqualified to understand huge chunks of source, even at a debugging level (e.g. the many cases where the devs simply cant practically debug distributed-system source)). There are good reasons for having source (and the other sacred cows) but more questioning and more critical thinking from developers about what they *really* need and especially what they’re going to *do* with it would be good for everyone, I think.

Any mistakes understandings and misattributions etc my own fault. My personal comments in [square brackets]

Talk

If you were going to make a new engine/MMO today, what middleware would you use?

PJ: we focussed on web based games when we started. We decided to use Unity, no royalty sharing made it a very attractive pricing model.

CD: hypothetically … I’d make substantial use of middleware, although we didn’t really at Turbine because it wasn’t there when we started.

RL: we look at what’s going to be unique about the game design we’re starting, and then look for middleware that matches well with the uniqueness of the individual game, whatever gets us a good chunk of the way towards that.

MF: we’ve decided each time to just do it ourselves.

How do you deal with the risk of using middleware code that’s never been used in a shipped MMO, that you’re the first major user of / the other major users haven’t shipped yet?

PJ: make sure you have the source code (although we couldn’t afford the more-expensive source-code license for Unity), make sure you have good support and responsiveness from vendor Tech Support.

CD: make sure the vendor is committed to the vertical business space, that your success is part of their core product strategy

JL: being first shipping user can get you a big price discount from the vendor

RL: with BigWorld we don’t have all the source. You have a limited number of engineers who’ll be skilled enough to do anything meaningful with that source anyway, and they could quit, leaving you in trouble. We have a commitment that if something exceptional comes up then we have the *possibility* of the vendor giving us source access temporarily, with their engineers physically present at all times, but we hope not to need that.

MF: any middleware that includes game design / game-logic elements is highest risk. I’ve had low-risk pieces of middleware fail, but that’s quite easy to workaround because it’s got relatively little effect on the overall project. I don’t want to name vendors in particular examples, because it’s not necesssarily their fault.

JL: we used SpeedTree for a couple of years, but eventually didn’t ship with it, because it couldn’t integrate with our art pipeline. Not a bug in that, just unfortunate accident of how it works and how we worked.

CD: we had the same problem with ST.

CD: the two scariest middlewares we used at Turbine were having Physics, and Kynapse for path-planning. Do we de-stabilise our live servers by integrating this cutting edge middleware new version in? As an MMO operator you really want to be able to keep an old version in situ for a long time – maybe a lot longer than the middleware vendor actually wants you to have it there.

[ADAM: that’s an extremely interesting problem…]

We had a big problem with a middleware engine where the provider just went out of business. We got some support from our publisher there to work through it, but what do you do about that?

CD: source code escrow is a good thing, for the absolute disaster scenario. But it’s really no different from other catastrophic business risks. Make sure you have one (more than one) person who’s been over the source in detail with the vendor at least once, so understands it well enough to pick up if they vanish, etc.

RL: escrow is mandatory, including all the required libraries, correct verisons etc.

[ADAM: nb: same speaker who doesnt have full source for some aspects of BigWorld]

JL: we didn’t escrow Alchemy, too naive to realise we should, but we got someone to send it to us as they were going out of business. Just getting it to build was a huge problem, we found.

[ADAM: our industry is still cursed by the fact that almost no-one has a great build system in place :) ]

MF: with Renderware we had source for the main thing, but there were dev tools that we had no source for, and so we couldn’t fix bugs in the tools, it was a nightmare. Very fine Modularity of your middleware – which doesn’t really exist yet – would be a real saver in this area. A hyper-modular middlware, I think that would make the risk much lower. Then if the company did go out of business, you could replace those building-blocks one at a time. I think that modularity is very important, or as close as you can get.

[ADAM: ha. I fought hard at the tech level to keep exactly that as one of the top 3 features of our server middleware back in 2002, but unfortunately back then (my naivety in assuming developers understood more than they did) no game developer had anywhere near enough understanding of the problems to realise how valuable that was. Ultimately, it was a waste of money for us as a company, because no customer understood it enough to place the value on it that would pay for how much it was costing us to achieve and maintain it. I’m not convinced that’s changed enough to make it profitable enough for MMO providers – even if developers now see the theoretical value, I don’t think they are able to really afford it]

Audience: Given how heavy the sales process is … is there a role for open source (OSS) in this?

MF: I think OSS is actually necessary for our industry to remain profitable, to reduce the cost of development. If I say bad things about middleware it’s not because I want it to fail (I really want it to succeed, I think it’s critical for us), but because I’m so disappointed with the current state of it.

RL: how many games have Python in them? Well, that’s an example of OSS that’s now heavily in use. We have an enormous amount of OSS we use inside the backend systems / servers.

What are the packages are you using for service side?

CD: we use a 3rd party billing system, that I wouldn’t actually recommend. We chose it many years ago, and it isn’t really right for what we do now.

(audience): there’s big advnatages to using external billing provider, e.g. benefit of having someone else as the merchant of record, isn’t there.

[ADAM: personally, I think that’s short-sighted, because in many ways the last thing in the world you want to have happen, as a service provider, is be disengaged from the customer’s wallet. Never let someone else come between “you” and “your customer’s money”; they can handle it, facilitate it, but not take over the CRM]

What about internal middleware? A lot of companies form Core Technology teams, creating a culture of making MW for internal use, re-use in other titles, etc

CD: D&D and LoTRO have the exact same codebase on the platform level because of our internal middleware. We ported the LoTRO DirectX 10 layer as a full working initial Dx10 directly into D&D, without changing it.

RL: the question comes down to whether you have a very specific target for what you’re doing. If you try to make one engine for all your games, and/or all your studios (if you’re multi-studio) then that’s a route to huge difficulties and disasters. At the same time I’ve seen companies that have huge in-house expertise in their internal wide-use engine and are able to churn out new games easily. But I think it’s a really really difficult problem to solve that way.

CD: we can have a full game prototype from zero to running in 3 months because we have all this internal re-usable code. For a relatively low investment we can build a small portion of a world to try out new game concepts.

[ADAM: I’m guessing the caveats there are simple that the content-cretion costs are of course still directly proportional to how much you want to show – hence “build a SMALL portion” etc]

MF: It was painful sometimes when the company was going down that they weren’t helping on the games themselves directly, but when Perpetual folded, the core tech group, the reusable code, was the one part that they managed to sell off. Clearly it has a lot of value, even if it’s painful getting there.

PJ: I think it makes sense to have one team on gameplay and building things, and one focussed on technology, because it’s different expertise for the staff to have.

What about MMO Middleware on consoles? Will midleware bring MMOs to consoles finally?

CD: it’s absolutely helped the transition, you’ll see stuff very soon where it’s facilitated console MMOs.

RL: stuff that’s provided to you multi-platform is a huge win from using middleware.

Questions from Audience

Q: sounds from what you’ve said here today like the state of the middleware industry today is fragile and embryonic?

MF: I agree. I think it’s a difficult industry. Business model is tough, it’s sales heavy, hard to get your cash in when people go under –

JL: – that’s what happened with Intrinsic, they had agreements with lots of companies paying a small amount each month towards the license fee, and when a few of those went out of business, the sudden cashflow damage killed Intrinsic

Q: what are your experiences with the multi-platform engines?

RJ: we tried most/all of the flash stuff. We’ve found Unity to provide the most-productive pipeline, fast and easy to prototype with, works nicely with our dev methodology (scrum).

RL: the more MMO development you do, the more you realise it’s all about the tools. The content-creation tools etc become more important, ultimately, than the engine itself.

CD: the cost of developing an MMO is never the technology, four or five times that cost is the cost of the content-generation. So the more efficient your tools are, the quicker you can get final game-content being made, that’s key.

Q: (from MF) does middleware “help” or “hurt” innovation?

MF: I feel it could hurt quite a lot. e.g. Some years ago, I could not find any middleware that would support instances, so we couldn’t use it.

CD: agreed. I think something monolothic like BigWorld could be particularly limiting, but the piecemeal approach of lots of little parts, increased productivity, actually I think helps innovation because it helps on your dev-resource costs a lot.

Q: are any of your perspectives on middleware non-technology centric, it’s been very tech-centric so far? I’ve seen a lot of recurring frustration with middleware. What kind of heuristics do you use to establish whether middleware is the right choice?

RL: you should never take a fully-staffed team and then dump in middleware on them. The two should be built hand-in-hand; middleware changes your hiring needs, and vice versa.

CD: Tech is always both an enabler and a limiter.

Q: as a vendor looking-in … we see particular teams (art vs code vs design) a lot more vocal than others, and the decision to use/not-use it gets skewed.

RL: the purchase decision is often lead by the lead client engineer, server stuff gets ignored, which is clearly wrong.

computer games games design games industry marketing massively multiplayer

A better way to review video games

Post author By adam
Post date May 10, 2009
5 Comments on A better way to review video games

Reviewing video games is hard. In some ways, it’s an impossible mission: a reviewer has too many conflicting interests:

please the publishers or else be denied access to the materials they seek to review
please their editor or else don’t get paid; but the editor’s primary source of capital is often advertising … from the publishers
answer the consumer’s main question in a way that earns their trust: should they purchase this game or not?
stand out from the crowd of a million game players who decide to write about their hobby

Who’s your Daddy?

This has been a problem for as long as I can remember (20+ years of game playing and reading game reviews); the consumer *believes* that the reviewer is answerable to them – but it has been a very long time (10 years now?) since consumers were the paymaster of reviewers; nowadays, it’s advertisers (which usually means: game-publishers).

Of course, consumers still wield huge power. The virtuous value circle – the only circle that matters – is driven by consumers:

A reviewer has a “readership” of consumers who are influenced in their purchasing decisions by those reviews
Publishers therefore court the reviewer to try and curry favour with the consumers and increase sales of the publishers’ products (to those readers, and anyone they themselves influence – friends, family, colleagues, etc)
Reviewers earn more money, and get deeper access to development teams (courtesy of the publishers), so produce more reviews

But that power is – clearly – both indirect and hard to quantify. A consumer – even many of them – threatening to “stop reading a reviewer’s reviews” is not particularly effective.

Publications like Edge helped along the indirection of consumer-power when they decided to go out of their way to obscure the identities of their individual reviewers, turning reviews into as much of a crap-shoot as buying games was in the first place. Since the web rose to prominence, it’s been eroded at the other end – there’s now so many reviewers around that, well … who has the time to remember who any individual reviewer is?

Qui custodet custodes?

But if journalists/reviewers are supposedly there as a watchdog on the publishers’ marketing depts, supposedly helping the consumer determine which are the (non-refundable) purchases they ought to be making, then who’s checking that the journalists themselves are honest?

No-one, really. And that’s where the rot begins. The storms of outraged public opinion are nothing new: examples of journalists writing reviews of games (reviews both scathing and rejoicing) they hadn’t even played go way back into the 1980’s.

A case study in lies, damn lies, and video game journalism

In case you hadn’t heard, this week a “staff writer” from Eurogamer (a games review / news site) ripped to pieces one of the most recently-released MMOs – Darkfall. At which point Aventurine, the developer of Darkfall, responded with increasing anger and dismay.

But the really interesting thing here is that Aventurine didn’t merely rant “you bastards! Our game is Teh Awesum!!!111! STFU, Beotch!” (well, they did that as well) … no, they dropped a little A-bomb in the middle of their reply:

“We checked the logs for the 2 accounts we gave Eurogamer and we found that one of them had around 3 minutes playtime, and the other had less than 2 hours spread out in 13 sessions. Most of these 2 hours were spent in the character creator”

Pwned. MMO developers *actually know whether your journalist played the game before reviewing it*. What’s more … they have proof…

The EG reviewer (whose “references and background are immaculate”, according to the editor – but from reading his only two EG reviews, I’m afraid it does rather sound like he knows little about MMOs), responded (via his editor) with the claim:

“the logs miss out two crucial days and understate others, … and he insists he played the game for at least nine hours”

It would seem that someone is lying (and it could be either party). Worse, someone is being particularly stupid. Because the journalist is claiming “your computers lie”, and the developer is claiming “your journalist is a lier”; either way, it’s not a subtle, small, mistake – whoever is wrong, if they get discovered, they’re going to create themself a good amount of long-term trouble (bad reputation).

Lots of MMO developers write shitty server code, and honestly don’t know what the hell is going-on inside their own game-world (but fondly imagine that they do – and proudly boast to the press (in the vaguest terms) that they do). But the rule of thumb is that devs who don’t know … don’t even know what it is they ought to be claiming that they know. The specificity of Aventurine’s claims suggests that they do have the stats, and those stats are mostly correct.

(I say “mostly” because there is a bit of vagueness about what – precisely – the reviewer was doing in-game. That reeks of holes in their metrics/logging. They clearly know when the player was logged-in, and what they did/said in chat, and how many characters were created – but apparently not what they were doing in the client, e.g. how long did they spend in character creation? Implicitly: unlogged; unknown)

Whereas it’s quite likely that a non-knowledgeable journalist, accustomed to buggy games, would assume that they could safely claim “your server is buggy, those figures are wrong”.

Unfortunately for any such journalist, server logs are generally either correct, or absent entirely – there’s rarely any middle-ground. If he knew a bit more about MMO tech he might know this; very few journos (any of them?) know that much about the games they review, though.

So … based on nothing but casual observation and intimate knowledge of the tech issues (and several decades of reading game reviews…), I’m leaning in favour of Adventurine and against Ed Zelton. My guess (pure *guess*) is that he’s been caught out being either incompetent or perhaps a bit lazy as a reviewer, and he’s thought he could get away with blaming it on buggy code. From reading the review, I get the impression he wishes he were Ben “Yahtzee” Croshaw (from Zero Punctuation) – although he clearly isn’t funny enough – but he seems to like saying “it’s shit; you’re shit; you’re all shit; STFU” instead of reviewing the game, and seems to think that’s good enough. As an MMO player, my feeling was that the review was, well … useless – without even playing the game, there is so much more I would want to hear in a review, and so much of his wanky whining that I couldn’t care less about. As an MMO developer, it felt downright insulting, as if he’d made no effort at all to play the game as a game. Actually, it felt like he’d hardly played MMOs in his life, and didn’t really know what they were.

(NB, from the review: his apparent ignorance of some of the most important *and best-selling* RPG + MMORPG games of all time – the Ultima series – suggests that he really isn’t much good as a game reviewer. YMMV.)

Reviewing the reviewers

Up-front I’m going to point out that I don’t believe all MMO developers are currently capable of doing this – many people would be amazed to discover the true state of metrics collection in this industry – although *all* modern MMO developers ought to, and it’s not too hard to add-on later (add it to the list of “things MMO developers ought to do as standard practice, but many of them don’t do”). But it’s a general thing that I think we should move towards.

MMO developers (well, actually, the Operators, but that’s getting pedantic) are in an excellent position to help guard journalistic honesty, in a way that traditional game developers have never been able to. I would like to start seeing the following published by *every* MMO developer each time their game is reviewed:

What level the account(s) started at
What level the account(s) peaked at
How many hours the reviewer spent at the lowest levels, levelling-up manually
How many hours the reviewer spent at the highest levels
What percentage of time was spent on each of the different primary character classes and factions
Which areas of the game / aspects the reviewer actually engaged in (hours of combat, hours of crafting, hours of chat, etc)

…but, honestly, this isn’t so much about “journalistic honesty” (I used that phrase tongue-in-cheek above) as it is about starting a virtuous cycle of developers being more cognizant of what, actually, players “do” in their games – preferably *before* gold launch. In particular, if publishers (developers) started supplementing reviews with this info (as a matter of course), I think we’d see a sea-change in industry staff appreciating three key things about metrics:

How little metrics they’re actually collecting compared to how much they think they’re collecting
What metrics actually matter, and/or are useful?
How players actually play the game; by extension: how fun is the game, really, and which parts suck horribly?

Does this work / matter?

At NCsoft, I got into the habit of asking prospective partners, hires/employees, and external studios which MMO’s they played (fair enough) … and how many characters they’d got to the level-cap with / what level their characters had reached. It started as an innocent question, but I quickly noticed how often it gave early warning of failures of honesty among individuals, and how much it presaged the problems they would have in the future.

The two worst problems were “complete ignorance of the MMO industry (either of pre-existing design practices, or tech practices)” and “personal self-deceit about what the person knows, and what they don’t know”. The latter tended to be a far worse problem: when someone is deceiving *themself*, it’s doubly hard to re-educate them, because first you have to get them to accept their own deception.

Of course, it turned out to lead to a lot of defensive responses and a spew of self-justification, which made us both uncomfortable. In those situations, it can easily lead to making assumptions that certain people’s opinions are “worth less” because, say, you know for a fact they’ve never really played an MMO – at least, not in the way that most of that MMO’s players would/will/do play it. I hate that tendency, since it’s part of a snobbishness that lies at the root of a lot of oyster-like, head-in-sand behaviour in our industry. On the other hand, it’s important and useful to know when someone’s ideas are random conjecture and when they’re based on fact (and very few people in a design meeting or publisher/developer meeting will honestly tell you their ideas are conjecture :)).

On the whole, though, it turned out to be a really useful line of questioning – even bearing in mind the additional (smaller) problems it created. There are obvious problems that come from the statistical supplementing of free-form prose game-reviews – but I’m confident that these will be outweighed by the advantages (and the problems that will be shrunk).

PS:

Despite the TLC of good friends, I’m still weak and sapped of all energy from my month of illness. I’m triaging like mad to deal with urgent issues, but there’s plenty of highly important stuff that’s been pending on me for a while that I still haven’t had the time + energy to deal with. So, if you’re still waiting … I’m sorry.

games industry iphone

Free stats on iPhone developers: who are they?

Post author By adam
Post date May 4, 2009

I’ve been looking around the web and it seems no-one has any good, FREE stats on who the different iPhone developers are, and what they’re doing.

So … I’ve made a 40-question survey (its all yes/no or multiple-choice-answer questions, hopefully it shouldnt take too long to fill out – sorry), and I’ll send the results to everyone who fills out the survey :) (remember to fill in a valid email address if you want a copy!). The idea is to get some aggregate stats on the different developers, how big they are, what they’re doing, etc:

http://www.surveymethods.com/EndUser.aspx?BB9FF3EEB9FAECEB

I’m hoping to get at least a few hundred people to fill this out, so we can get some meaningful information (without having to pay market research firms to get it). As well as sending the detailed stats to anyone who replies, I’ll post the main highlights publically, so anyone can see them.

I’ll be posting it in a couple of forums and sites, and hopefully I’ll get a wide range of responses. Feel free to share the link with any other iPhone developers you know!