Now that I no longer work for a large MMO publisher, I no longer have access to all the juicy numerical goodness, research, and stats that they had on their games and everyone else’s. A chance email recently suggesting I take a look at Xfire’s gamestats led to some quick experiments that came out surprisingly well. It’s given me a new predictor for player numbers for any MMO that’s available in English which is sufficiently accurate that I’m going to use it going forwards. Take it or leave it :).
(this is rather the opposite end of interpretation to “Over 1 billion people play online games” – and make sure you read Raph Koster’s thoughts before trying to interpret these figures)
What are these used for
Even though there is NO audited, trustable source for these figures, we already know that the public “guesstimates” like MMOGchart.com are routinely used:
- in audited (!) company annual reports as a reference point (especially in China and South Korea)
- by publishers, when deciding which game projects to fund (used directly in projections of potential market-size – and hence how much cash funding to provide!)
These numbers are *seriously important* to the industry (like it or not!).
What’s out there – official figures
There are three types of official figures for player numbers for online games:
- Very precise figures included in the quarterly or annual audited company accounts, and legally-required to be accurate
- Detailed figures included in press-releases and/or conference presentations
- Vague figures cited in public interviews
Public companies whose primary business is online games are often expected (required, perhaps?) to publish precise figures (a side-effect of the rules on what they have to stick in their annual reports). Not all do (?), but noteworthy examples include:
- NCsoft (one of the best-known publishers to do this, and the one with most “global” data, covering USA, Europe, and Asia)
- CJ Internet (South-Korea + Asia only)
- Giant Interactive (China only)
- NetEase (China only)
- Shanda (China only)
You … may well note a trend there. These figures are useful, and aid businesses operating in Asia, but by comparison life is somewhat harder for anyone wanting to sell into America or Europe. In all fairness, there are American and European companies that chose to (usually irregularly) make official statements via Press Releases, but this is an order of magnitude less detailed and usually less accurate than what would go in an annual report for a public company.
(NB: IMHO, the American and European economies and industries suffer for this lack of transparency – business models are more fragile, staff are less well-informed, decision-making is weaker, etc).
What’s out there – estimated figures
- Bruce Woodcock’s MMOGchart.com – guestimates extrapolated from superficially similar games with official figres
- mmogdata.voig.com – guestimates from a private methodology
- Vague figures cited in public interviews
- Independently measured figures
Bruce started out by taking as many of the official figures as he could find, modelling graph-based trends, and then re-applying those trends to missing data to try and extrapolate or interpolate the missing items. Where a game has never had ANY official figures, he took estimates based on a wide variety of inputs, everything from unsubstantiated rumours through to unofficial figures “leaked” by employees of the companies that were running the games.
Good points: (mostly) documented estimation process, started with accurate data, includes data for many games, includes detailed writeups explaining which figures are “accuate” and which are “guesses”, and ascribes an estimate of the amount of error in each individual estimate
Criticisms: assumes all games behave similarly in growth/shrinkage, updated very infrequently (every 4-12 months)
Phil‘s VOIG was started apparently in frustration with the slowness of updates to Bruce’s figures (originally he updated frequently, but over time updates got less and less frequent). Phil doesn’t divulge his methodology, and you cannot download their figures (although you could read the website visually and type down each individual number. Umm. No, thanks).
Good points: *still* more frequently updated than Bruce even though Bruce has tried to speed up again
Criticisms: unknown methodology, unknown error-margins, poor data format, no download of figures available
Lots of games industry staff believe in sharing their figures more openly than their managers are willing to. On top of that, it’s often difficult or very difficult to answer a journalist’s question in an interview – or to explain a decision made during a post-mortem or conference talk – whent the audience have no idea what the underlying figures are. So, we often see individuals from games companies making public statements as to player figures for various of their games.
Good points: effectively these are “official” figures
Criticisms: not just vague as to numbers (usually they are only quoted to 2 sig.figs) but also vague as to *meaning* (registered players? active? paying?), very irregular publication times, often non-specific about what *date* they apply to (and people often quote figures that are a year or more out of date!)
A few organizations try to independently measure figures. It has long (ten years) been a complaint in the industry that no organization of high reputation in the traditional Media sphere (e.g. ABC for printed publication circulations) has started auditing online games. Recently, there have been huge efforts by a handful of companies to measure website traffic specifically – e.g. Quantcast, Compete, comScore – and for some online games those figures are often extremely good (games where people have to use a website each time they play the game, for instance).
Good points: stringent accounting standards (they hope to become ABC equivalents), strong expertise with web properties generally (so accustomed to the many tricks that black-hat website owners use to try and inflate their figures), very frequently updated (in some cases as frequently as per-day, taking them almost into real-time status)
Criticisms: mostly useless for non-web games
…but this final type – independently-measured figures – is the one we need more of. Because we need something that:
- updates frequently, giving us “up to date” figures whenever we consult the source
- uses a common reporting standard across ALL games (doesn’t compare “registered” from one game against “active” from another)
- requires little effort to maintain (likely to stick around long term and become a reliable resource)
- uses an open algorithm that is easily verfiable by anyone (the maintainers cannot deliberately write-up or write-down individual games without detection)
Xfire is one of several companies trying to make “a social network for video game players” by creating a custom chat client that you keep open while playing the game. This allows them to track who is playing what games, when, for how long. For some time now they’ve been publishing (openly, for free), stats on how many hours each game is being played for per day in total. That figure gives some idea of the total “attention” that particular games are receiving, both individually and comparitively, but it’s useless for anything else.
I’d looked at the Xfire stats before, but only used them for very high-level comparitive judgements, since in most cases I work with games that have wildly varying “average number of hours of play per player per month”, and so the Xfire stats could not be used to judge games.
I had an email from one of the Xfire guys, suggesting I look at the stats again, and I noticed that they currently have a “number of Xfire users playing each game” stat too. Interesting…
A stupidly simple Methodology
Xfire has far too few users for those users-playing-today figures to be even close to the actual Concurrent Users figures, let alone number of players.
But I have a lot of high quality data on a wide variety of games (through official and unofficial channels), and I have most of the “official” figures, so I wondered what would happen if I tried using some well-known and accurate figures to look for a correlation with the daily users figures on Xfire. Pretty obvious. NCsoft sells directly into US and Europe and has established subs games in both western-developed MMORPG (City of Heroes/Villains (CoH/CoV) – known as “CoX”) and eastern-developed MMORPG imported into USA/Europe (Lineage 2 – known as L2).
I chose these two games because:
- They’re from the same publisher, so counting algorithm OUGHT to be about as similar as we’ll ever get for different games
- They’re both subscription based, so we get a relatively non-ambiguous figure
- (most important of all) NCsoft releases precise figures for both these games *every single quarter*
The ratio of “Xfire activity” : “actual subs” is very different for those two games – but I wondered how well they predict the ratios for other games I had the figures for? I tried classifying each game simple as “eastern import” or “western”.
In each case, I looked for the following success / fail / anomaly criteria:
- (any game), L2 and CoX are approximately equal multiples of known figures = fail
- (any game, true figure unknown), L2 and CoX are both much bigger or much smaller than the estimated figure = anomaly
- Eastern game, L2 is a smaller multiple of the known figure than CoX = success
- Western game, CoX is a smaller multiple of the known figure than L2 = success
The “anomaly” result allowed me to run this against all the games where we only have “generally-accepted estimates”, and then decide in each case whether it was a breakdown in the methodology, or if it pointed to the “generally-accepted estimate” being wrong.
I had 4 types of number to compare against, FYI:
- Official figures
- Personal estimate (sometimes based on insider-knowledge, sometimes based on industry “common knowledge”, sometimes on odd bits of public data that indirectly confirms or predicts for a particular game)
- Public estimates
- Private official figures
Because Bruce gives you a downloadable spreadsheet of his data – and because you can read his own commentary on how (in)accurate each individual figure is – I used his data as the “public estimate” figures.
East vs West – Some example data
|Name||Official/trusted||MMOGchart||Best-Guess||Xfire||Xf-v-NC-CoX||% NC-CoX||Xf-v-NC-L2||% NC-L2|
|Age of Conan||415000||415000||1032||171,771||41.39%||724,780||
|City of Heroes / Villains||125000||136250||125000||751||
|Dark Age of Camelot||45000||45000||
Dungeons & Dragons Online
|Final Fantasy XI||500000||500000||
|Legends of Mir||
|Legends of Mir 2||
|Legends of Mir 3||
|Pirates of the Burning Sea||65000||65000||64||10,652||16.39%||44,948||
|Pirates of the Caribbean Online||10000||10000||443||73,735||737.35%||
|Star Wars Galaxies||100000||100000||
|The Lord of the Rings Online||
|Vanguard: Saga of Heroes||40000||40000||583||97,037||242.59%||409,444||
|World of Warcraft||12000000||10000000||12000000||112784||
|World War II Online||12000||12000||
|Yohoho! Puzzle Pirates||200000||