Day: November 23, 2008

Now that I no longer work for a large MMO publisher, I no longer have access to all the juicy numerical goodness, research, and stats that they had on their games and everyone else’s. A chance email recently suggesting I take a look at Xfire’s gamestats led to some quick experiments that came out surprisingly well. It’s given me a new predictor for player numbers for any MMO that’s available in English which is sufficiently accurate that I’m going to use it going forwards. Take it or leave it :).

(this is rather the opposite end of interpretation to “Over 1 billion people play online games” – and make sure you read Raph Koster’s thoughts before trying to interpret these figures)

What are these used for

Even though there is NO audited, trustable source for these figures, we already know that the public “guesstimates” like MMOGchart.com are routinely used:

• in audited (!) company annual reports as a reference point (especially in China and South Korea)
• by publishers, when deciding which game projects to fund (used directly in projections of potential market-size – and hence how much cash funding to provide!)

These numbers are *seriously important* to the industry (like it or not!).

What’s out there – official figures

There are three types of official figures for player numbers for online games:

1. Very precise figures included in the quarterly or annual audited company accounts, and legally-required to be accurate
2. Detailed figures included in press-releases and/or conference presentations
3. Vague figures cited in public interviews

Public companies whose primary business is online games are often expected (required, perhaps?) to publish precise figures (a side-effect of the rules on what they have to stick in their annual reports). Not all do (?), but noteworthy examples include:

• NCsoft (one of the best-known publishers to do this, and the one with most “global” data, covering USA, Europe, and Asia)
• CJ Internet (South-Korea + Asia only)
• Giant Interactive (China only)
• NetEase (China only)
• Shanda (China only)

You … may well note a trend there. These figures are useful, and aid businesses operating in Asia, but by comparison life is somewhat harder for anyone wanting to sell into America or Europe. In all fairness, there are American and European companies that chose to (usually irregularly) make official statements via Press Releases, but this is an order of magnitude less detailed and usually less accurate than what would go in an annual report for a public company.

(NB: IMHO, the American and European economies and industries suffer for this lack of transparency – business models are more fragile, staff are less well-informed, decision-making is weaker, etc).

What’s out there – estimated figures

1. Bruce Woodcock’s MMOGchart.com – guestimates extrapolated from superficially similar games with official figres
2. mmogdata.voig.com – guestimates from a private methodology
3. Vague figures cited in public interviews
4. Independently measured figures

Bruce started out by taking as many of the official figures as he could find, modelling graph-based trends, and then re-applying those trends to missing data to try and extrapolate or interpolate the missing items. Where a game has never had ANY official figures, he took estimates based on a wide variety of inputs, everything from unsubstantiated rumours through to unofficial figures “leaked” by employees of the companies that were running the games.

Good points: (mostly) documented estimation process, started with accurate data, includes data for many games, includes detailed writeups explaining which figures are “accuate” and which are “guesses”, and ascribes an estimate of the amount of error in each individual estimate
Criticisms: assumes all games behave similarly in growth/shrinkage, updated very infrequently (every 4-12 months)

Phil‘s VOIG was started apparently in frustration with the slowness of updates to Bruce’s figures (originally he updated frequently, but over time updates got less and less frequent). Phil doesn’t divulge his methodology, and you cannot download their figures (although you could read the website visually and type down each individual number. Umm. No, thanks).

Good points: *still* more frequently updated than Bruce even though Bruce has tried to speed up again
Criticisms: unknown methodology, unknown error-margins, poor data format, no download of figures available

Lots of games industry staff believe in sharing their figures more openly than their managers are willing to. On top of that, it’s often difficult or very difficult to answer a journalist’s question in an interview – or to explain a decision made during a post-mortem or conference talk – whent the audience have no idea what the underlying figures are. So, we often see individuals from games companies making public statements as to player figures for various of their games.

Good points: effectively these are “official” figures
Criticisms: not just vague as to numbers (usually they are only quoted to 2 sig.figs) but also vague as to *meaning* (registered players? active? paying?), very irregular publication times, often non-specific about what *date* they apply to (and people often quote figures that are a year or more out of date!)

A few organizations try to independently measure figures. It has long (ten years) been a complaint in the industry that no organization of high reputation in the traditional Media sphere (e.g. ABC for printed publication circulations) has started auditing online games. Recently, there have been huge efforts by a handful of companies to measure website traffic specifically – e.g. Quantcast, Compete, comScore – and for some online games those figures are often extremely good (games where people have to use a website each time they play the game, for instance).

Good points: stringent accounting standards (they hope to become ABC equivalents), strong expertise with web properties generally (so accustomed to the many tricks that black-hat website owners use to try and inflate their figures), very frequently updated (in some cases as frequently as per-day, taking them almost into real-time status)
Criticisms: mostly useless for non-web games

…but this final type – independently-measured figures – is the one we need more of. Because we need something that:

• updates frequently, giving us “up to date” figures whenever we consult the source
• uses a common reporting standard across ALL games (doesn’t compare “registered” from one game against “active” from another)
• requires little effort to maintain (likely to stick around long term and become a reliable resource)
• uses an open algorithm that is easily verfiable by anyone (the maintainers cannot deliberately write-up or write-down individual games without detection)

Xfire

Xfire is one of several companies trying to make “a social network for video game players” by creating a custom chat client that you keep open while playing the game. This allows them to track who is playing what games, when, for how long. For some time now they’ve been publishing (openly, for free), stats on how many hours each game is being played for per day in total. That figure gives some idea of the total “attention” that particular games are receiving, both individually and comparitively, but it’s useless for anything else.

I’d looked at the Xfire stats before, but only used them for very high-level comparitive judgements, since in most cases I work with games that have wildly varying “average number of hours of play per player per month”, and so the Xfire stats could not be used to judge games.

I had an email from one of the Xfire guys, suggesting I look at the stats again, and I noticed that they currently have a “number of Xfire users playing each game” stat too. Interesting…

A stupidly simple Methodology

Xfire has far too few users for those users-playing-today figures to be even close to the actual Concurrent Users figures, let alone number of players.

But I have a lot of high quality data on a wide variety of games (through official and unofficial channels), and I have most of the “official” figures, so I wondered what would happen if I tried using some well-known and accurate figures to look for a correlation with the daily users figures on Xfire. Pretty obvious. NCsoft sells directly into US and Europe and has established subs games in both western-developed MMORPG (City of Heroes/Villains (CoH/CoV) – known as “CoX”) and eastern-developed MMORPG imported into USA/Europe (Lineage 2 – known as L2).

I chose these two games because:

• They’re from the same publisher, so counting algorithm OUGHT to be about as similar as we’ll ever get for different games
• They’re both subscription based, so we get a relatively non-ambiguous figure
• (most important of all) NCsoft releases precise figures for both these games *every single quarter*

The ratio of “Xfire activity” : “actual subs” is very different for those two games – but I wondered how well they predict the ratios for other games I had the figures for? I tried classifying each game simple as “eastern import” or “western”.

In each case, I looked for the following success / fail / anomaly criteria:

• (any game), L2 and CoX are approximately equal multiples of known figures = fail
• (any game, true figure unknown), L2 and CoX are both much bigger or much smaller than the estimated figure = anomaly
• Eastern game, L2 is a smaller multiple of the known figure than CoX = success
• Western game, CoX is a smaller multiple of the known figure than L2 = success

The “anomaly” result allowed me to run this against all the games where we only have “generally-accepted estimates”, and then decide in each case whether it was a breakdown in the methodology, or if it pointed to the “generally-accepted estimate” being wrong.

I had 4 types of number to compare against, FYI:

• Official figures
• Personal estimate (sometimes based on insider-knowledge, sometimes based on industry “common knowledge”, sometimes on odd bits of public data that indirectly confirms or predicts for a particular game)
• Public estimates
• Private official figures

Because Bruce gives you a downloadable spreadsheet of his data – and because you can read his own commentary on how (in)accurate each individual figure is – I used his data as the “public estimate” figures.

East vs West – Some example data

Name	Official/trusted	MMOGchart	Best-Guess	Xfire	Xf-v-NC-CoX	% NC-CoX	Xf-v-NC-L2	% NC-L2
2Moons			n/a	448	74,567	n/a	314,633	n/a
9Dragons			n/a	211	35,120	n/a	148,187	n/a
Age of Conan	415000		415000	1032	171,771	41.39%	724,780	174.65%
Anarchy Online		12000	12000	164	27,297	227.47%	115,178	959.82%
Archlord			n/a	1330	221,372	n/a	934,067	n/a
Audition			n/a	473	78,728	n/a	332,191	n/a
City of Heroes / Villains	125000	136250	125000	751	125,000	100.00%	527,432	421.95%
Dance Online			n/a	107	17,810	n/a	75,147	n/a
Dark Age of Camelot		45000	45000	144	23,968	53.26%	101,132	224.74%
Dofus	10000000	452000	10000000	1433	238,515	2.39%	1,006,405	10.06%
Dungeon Runners			n/a	95	15,812	n/a	66,719	n/a
Dungeons & Dragons Online		45000	45000	159	26,465	58.81%	111,667	248.15%
EVE Online	250000	236510	250000	3429	570,739	228.30%	2,408,208	963.28%
EverQuest		175000	175000	109	18,142	10.37%	76,551	43.74%
EverQuest II		200000	200000	440	73,236	36.62%	309,015	154.51%
Exteel			n/a	202	33,622	n/a	141,866	n/a
Final Fantasy XI		500000	500000	509	84,720	16.94%	357,474	71.49%
Granado Espada			n/a	222	36,951	n/a	155,912	n/a
Hellgate: London			n/a	542	90,213	n/a	380,650	n/a
Hero Online			n/a	269	44,774	n/a	188,920	n/a
Horizons		5000	5000	58	9,654	193.08%	40,734	814.68%
Kal Online			n/a	207	34,454	n/a	145,377	n/a
Kart Rider			n/a	10	1,664	n/a	7,023	n/a
Legends of Mir			n/a	0	0	n/a	0	n/a
Legends of Mir 2			n/a	0	0	n/a	0	n/a
Legends of Mir 3			n/a	0	0	n/a	0	n/a
Lineage	1100000		1100000	2	333	0.03%	1,405	0.13%
Lineage II	1005000	1006556	1005000	1431	238,182	23.70%	1,005,000	100.00%
MapleStory	15000000		15000000	4042	672,770	4.49%	2,838,721	18.92%
Mu Online			n/a	56	9,321	n/a	39,329	n/a
Neopets			n/a		0	n/a	0	n/a
Perfect World			n/a	1472	245,007	n/a	1,033,795	n/a
Pirates of the Burning Sea		65000	65000	64	10,652	16.39%	44,948	69.15%
Pirates of the Caribbean Online		10000	10000	443	73,735	737.35%	311,122	3111.22%
Ragnarok Online			n/a	173	28,795	n/a	121,499	n/a
Regnum Online			n/a	236	39,281	n/a	165,744	n/a
RF Online			n/a	347	57,756	n/a	243,700	n/a
ROSE Online			n/a	83	13,815	n/a	58,291	n/a
RuneScape	6000000	1200000	6000000	2535	421,937	7.03%	1,780,346	29.67%
Seafight			n/a	151	25,133	n/a	106,048	n/a
Second Life		91531	91531	4398	732,024	799.76%	3,088,742	3374.53%
Secret Online	10000000		10000000		0	0.00%	0	0.00%
Silkroad Online			n/a	4980	828,895	n/a	3,497,484	n/a
Special Force			n/a		0	n/a	0	n/a
Star Wars Galaxies		100000	100000	644	107,190	107.19%	452,285	452.29%
Tabula Rasa		75000	75000	184	30,626	40.83%	129,224	172.30%
The Lord of the Rings Online		150000	150000	2282	379,827	253.22%	1,602,662	1068.44%
Toontown Online		100000	100000	172	28,628	28.63%	120,797	120.80%
Twelve Sky			n/a	797	132,656	n/a	559,738	n/a
Ultima Online		75000	75000	192	31,957	42.61%	134,843	179.79%
Vanguard: Saga of Heroes		40000	40000	583	97,037	242.59%	409,444	1023.61%
Warhammer Online	800000		800000	5621	935,586	116.95%	3,947,662	493.46%
Wonderland Online			n/a	202	33,622	n/a	141,866	n/a
World of Warcraft	12000000	10000000	12000000	112784	18,772,304	156.44%	79,208,889	660.07%
World War II Online		12000	12000	50	8,322	69.35%	35,115	292.63%
Yohoho! Puzzle Pirates	200000	34000	200000		0	0.00%	0	0.00%

East vs West – Does this work?

It’s not as bad as I thought it would be – there’s at least *some* correlation here :). It’s good enough that if you assume an inherent error margin of +/- 20% you can feel confident you’re getting good numbers.

It also works quite well with the private figures I have which (because of their sources) I consider to be pretty good.

Ah, but … statistically, does it work?

Well, running some simple Pearson correlation tests over the public numbers, I get a small increase in correlation (about 0.6 instead of 0.53) for using this method instead of just using a single comparator. That’s actually pretty good although I’d hoped for better. It does get a little better if I add in some private figures and/or replace some of the public estimates with private info I have.

I’d like to get hold of more data, either more things tracked by Xfire or more “public, official” figures, to check the correlation better. At the moment, there are a *lot* of holes in the public “Best Guess” column :(.

East vs. West: Interesting correlations and anomalies

9Dragons, 2Moons, Dance Online – The L2 predictor would put these at 150k, 310k, and 75k respectively. Acclaim has announced that they have a total of 750k active players across their 9 games, of which these three have been running for Acclaim longest and are the most mainstream of their games. At a total of 535k between them via L2 predictor, that fits reasonably well with the published figure.

Age of Conan – methodology breaks here. CoX predicts a mere 40% of the figure that Funcom has “officially unofficially officially” released recently. Sorry.

Anarchy Online – my estimate is 27k players; Bruce’s estimate is probably only counting subscribers, whereas the game has been F2P for several years now.

Archlord – L2 predicts a whopping 1 million players. Unless this is a truly huge hit in Asia, this has got to be wrong: even though the game has gone F2P in the west, the number of players in the west are generally thought to be in the region of less than 25k.

Audition Online – L2 predicts 300k, I believe it’s more like 3 million. c.f. the Maple Story notes.

DAoC – I’m happy with the CoX estimate of 25k players

Dofus – RS prediction is 1/3 of the reported number by Ankama. However … Ankama’s number *appears* to be “registered accounts” (the number I’ve used for RS is an estimate of active monthlys – the RS registered accounts figure is twice as big).

Also, Dofus is very much a French-language product. Although they’ve internationalized, their French start is still clear in that 30% of the playerbase is French (according to Ankama).

Therefore, I suspect that this may ALSO be being negatively affected by Xfire’s bias against non-English-language users (non English speakers tend to avoid English products if they can get equivalent native products).

Proportion of French users for internationalized American MMOs normally runs to around 30% of the European players, who normally represent around 50%-100% of the American players, which would suggest that Dofus has between twice as many and four times as many French players as an English-language MMO. Given the relative lack of interest advertising it in America, I’ll go for the four times.

Therefore, Xfire would only be counting around 75% of the playerbase it ought to be counting, and we’d get (via the RS predictor) around 4.5m players. I’m happy with that (until I get a phone call from Ankama. Salut?)

Dungeons and Dragons Online – again, a big drop, not so big as EQ2, but then I’ve heard that D&DO has had some uptick thanks to cross-selling to Lord of the Rings Online players.

Who knows? I can certainly *believe* the 25k predicted by CoX, but I’d ask some Turbine people if I were you, see if you can ferret out some more precise info…

EVE Online – methodology breaks here. The 250k figure comes from CCP themselves (give or take up to 10k). Based on the unique game-design and marketing, maybe EVE is just special (yeah, I know – I’m just making excuses here :)).

Everquest 2 – it’s a big drop from Bruce’s last-reported figure, but I think the CoX predictor of around 80k players is probably closer than Bruce’s.

FF XI – A while back I’d have thought the L2 estimate of 350k players was about right – nowadays I’m not sure, it’s been a long time since I looked into FF XI numbers in detail?

Horizons – CoX predictor says twice Bruce’s last estimate. Believable, but unconvincing.

Lineage – Epic fail. I can only guess: Xfire isn’t tracking Lineage 1 players. Off the top of my head, it’s hardly played outside Asia – Lineage 2 gets all the marketing love etc in the west. c.f. notes on other Asia-only games.

Lord of the Rings Online – CoX predicts a HUGE increase vs. Bruce’s estimate here. Turbine have never officially released figures IIRC, so maybe Bruce’s estimate should be treated as a shot in the dark anyway. Given that there’s been no new servers added to LOTRO since they were making noises about hoping to reach circa 500k – but also no server-merges – I could accept the 380k estimate from CoX predictor. But it’s just a guess.

Maple Story – L2 predictor breaks, suggesting just 2.8m players, less than a fifth of the best estimate I could find. This leads to a suggestion for a more precise prediction routine, see below…

Perfect World – L2 predicts 1m. Looking at companies with similar revenues to Perfect World, “active players” come in at approximately 1m-3m. Thoughts?

Pirates of the Burning Sea – CoX is predicting a mere 10k players – one sixth of the Bruce estimate. I believe it, because 7 months ago they shut down 7 of their 11 servers, leaving only 4 servers with a maximum concurrent playerbase of around 8k between them. In practice, I would have estimated around 15k-20k players based simply on the number of servers they’re running, but I suspect they may have kept extra around to keep some variety in the server populations, and because running just 1 or 2 extra servers is not that expensive, but gives you good fast response for all players.

Pirates of the Caribbean Online – Again, CoX predicts a whopping 7 times as many players as Bruce’s estimates. But then this game is one of those that went F2P, so, again, I can believe the CoX predictor when it says 75k players.

Seafight – CoX predicts 25k players, but this is way short of my own estimates based on the publisher’s aggregate player numbers. c.f. the notes on Maple Story, and the new analysis below.

Silkroad Online – L2 predictor comes close to my personal rough estimate based on taking their quoted number of registered players and dividing by 4, but its still short by about 30%. Again, see notes on MapleStory.

SL – isn’t a game, I’ve included it simply because both sources were counting it. The fact that it bears no resemblance to ANY of the games is no surprise considering it really has very little in common with them.

Star Wars: Galaxies – CoX’s predictor is within 10% of Bruce’s number. Cool.

Tabula Rasa – CoX predicts 30k. Estimates by industry consultants like Jessica Mulligan (see the comments) put it at around 30k.

Toontown Online – methodology broken; I’m sure the CoX estimate is wrong, and that TTO’s audience (young children) doesn’t overlap with Xfire’s audience.

Vanguard – The CoX predictor is suggesting more than twice the number of players that Bruce estimates. I have no idea what the correct number is – I haven’t bothered tracking Vanguard since its inexcusably poor launch. I’d love some independent confirmation of one number or the other being closer?

Warhammer Online / WAR – CoX predicts 935k, EA recently stated 800k. Not bad…

World of Warcraft – CoX predictor gets it wrong by a factor of 1.5 … you could take that as an indicator of the amount of error in the predictor :).

WW2 Online – I’m happy with the CoX estimate of 8k players

Some notable MIA games

Habbo Hotel – definitely millions of active players, but not tracked by Xfire. c.f. notes on ToonTown Online w.r.t. Xfire’s poor demographic tracking.

Neopets – definitely millions of active players, but not tracked by Xfire. c.f. notes on ToonTown Online w.r.t. Xfire’s poor demographic tracking.

Puzzle Pirates – 200k active players from the last number they put out publically, IIRC, but not tracked by Xfire.

Secret Online – 10 million players (“active”, IIRC) in China, announced in US/EU 7 months ago, not tracked by Xfire.

Special Force – not tracked by Xfire.

East vs. West – Problems

So we see three major problems here:

• Xfire doesn’t appear to track non-western players at all, tracks European-but-primarily-non-English players (Dofus, Seafight) noticeably poorly
• Xfire doesn’t appear to track younger users at all (all the games for young children / parents come out “untracked”)
• Basing all eastern game estimates off a subscription-only game (L2) works for a lot of things, but for the few really massive F2P (free to play) eastern games it fails

Can’t do anything about the first two problems, since those are flaws in Xfire itself, but I thought I’d have a quick look at the third problem and see if adding additional predictors (specifically for F2P games, both east and west) would help.

Analysis: Subscription vs F2P (Free to play)

The eastern F2P games fail dramatically when judged purely by the sucess of Lineage 2. Unfortunately, the mighty NCsoft “doesn’t do” F2P games, so we’re going to have to look at other sources of comparison.

Maple Story (eastern, localized), Runescape (Western) seem like good starting points, although in both cases there is only mediocre “official” data.

Subs vs. F2P: notable failures from East vs. West

Seafight – RS gets only 25% over my guesstimate based on the number of games Bigpoint publishes and the total number of active users they have. Serious guesswork – although Seafight is one of the slightly more popular of the BP games, so I would expect it to have more than that many users, closer to the RS predictor. But I suspect it could be MUCH higher, as much as 3-5 times higher, since we have no data on how much overlap there is between BP players of different games, and this assumes zero overlap.

Audition – Even using MS as a predictor, we get barely half of my last estimate for Audition’s playerbase. I’ve heard rumours it has been eroding a great deal in the past few years (it is an old game now – and with little or no design updates, it shows!), so I guess this is possible?

Subs vs. F2P – Does this work?

I’d have to say … no. And at that point, you start getting into using dozens of different predictors, split by genre + revenue model + country of origin + age of game, etc … and there aren’t enough MMOs in the world for that level of detail to be worth it (you’re into fantasy land by that point – and it’s too much effort :)).

Analysis: WoW

NB: this one I don’t take seriously, it’s just for fun; I think it’s meaningless until/unless I get hold of an Asian equivalent of Xfire, and come up with a WoW equivalent from Asia (probably Maple Story – huge locally, and large globally), and we can do the WoW-western-based-global vs. MapleStory-eastern-based-global comparison.

It would be interesting to see how well that worked as a predictor – does the “global success” dominate, or does the “subscription vs. F2P” dominate?

Follow-up ideas

1: Correlate “hours played”

Xfire’s preferred stat is “hours played per day” not “number of people playing per day”. This stat varies massively by Genre in fairly obvious ways. Doing a similar correlation to the above one for mapping “hours played + genre” to “number of people playing per day” would be relatively easy and possibly even more valuable.

2: Cross-correlate “hours played” with the above-inferred “number of players”

Especially useful would be to take the results of 1 above, and combine them with the work done in this article.

That would give you a basis for inferring “number of players” directly from Xfire’s primary free published-statistic.

3: Xfire to penetrate Asia

Well, we can wish…

4: Xfire to track younger children / older parents

My guess is that they too would love it if they could do this…

Exclusions

Kart Rider got closed down in USA, and although it’s still one of the most played games in the world, Xfire shows a mere “10 people playing” – so I guess those are the few who’ve braved non-localized versions?

Legend of Mir – Xfire is tracking it, but saying 0 for all games. IIRC they shut down the old LoM games as they open new ones – has a LoM 4 just come out?

PS….

WordPress is *still* corrupting raw HTML source – if you see big blocks of whitespace in this blog, it’s something odd in WP not liking inline style declarations. Sorry.