But they should.

So … here are the evaluations (from the audience) for our panel session at AGDC 09.

Judge for yourself whether you want to attend any future sessions featuring us again (Adam Martin, Bill Dalton, Rick Lambright, Joe Ludwig, Marty Poulin).

Head Count: 74; Evaluations: 32 (43% response rate)

• Overall rating of the presentation – 88% (AVG: 86%)
• How relevant was the topic to you? – 86% (AVG: 84%)
• How well did this class meet your expectations? – 94% (AVG: 84%)
• Would you recommend this session to a colleague? – 90% (AVG: 84%)
• Evaluate the speakers’ ability to communicate – 94% (AVG: 86%)
• If there were visual aids (slides) how were they? – 74% (AVG: 60%)

All of those are above average, and I’m glad that a particularly high number would recommend the session to their colleagues.

It seems that we did particularly well on fulfilling the remit (very high number for “met expectations”), and that our speakers had an awesome ability to communicate (almost 10% higher than average for the other speakers at the conference).

Audience Comments

1. The most entertaining session I attended, but didn’t sacrifice information value.
2. Interesting format, I wouldn’t mind seeing more of this, but it is time consuming
3. Good stuff
4. Slow, confused start lost valuable time for Q&A
5. Should have done middleware
6. Only 3 topics covered. Expected others

Comment 4 – yeah, something I’m unhappy about too, (it wasn’t our fault, it was the people running the conference), but there was nothing for it but to grin and carry on. Someone screwed-up the radio microphones, and we lost a lot of time at the start waiting for them to fix it. There was nothing we could do – they had connected the mics from a different room to *our* speakers. We didn’t find out until the person in the other room started talking, and it all came out through our speakers :(.

Comment 6 – we covered 4 topics (oops, audience can’t count :P). We all wanted to do more, but at GDC conferences, the organizers only give us 1-hour slots. With 4 speakers + moderator, I think that was pretty good, especially considering the time we lost at the start.

Perhaps someone will clone this format for a future conference (seems a good idea), and try to get a 2-hour slot for it?

1. Costs stupid amounts of money
2. Isn’t very good
3. Is only available on windows
4. …but “usually” works

OpenOffice:

1. Is completely free
2. Is an almost exact clone of Microsoft Office circa Office 2000/XP
3. Is open-source (so that sometimes you can easily fix it yourself, quite surprisingly!)
4. …but apart from the Microsoft Word part, “often doesn’t work”

I’ve been using OpenOffice’s Word clone as a complete replacement for Word for the past 3 years, and it’s been perfect. Previously, I used to use Word *and* OpenOffice, because the latter had some big bugs left in it.

Sadly, OpenOffice’s Excel clone is … often shockingly buggy. I won’t go into the details. But this post is about one missing/broken feature in particular: OpenOffice by default saves / exports XML which (for most people, and all simple uses) is unusable/unreadable – and is very hard to convert with XSLT. Read on for a script that will fix this for you…

An Obsession, with Compression…

In a well-intentioned-but-actually-stupid attempt to make the filesize slightly smaller, it implements a trivial form of compression that – in any imperative programming language – would be easy to decompress.

Sadly, the primary language for XML processing is XSLT – which is a functional language. Functional languages fundamentally (I keep using this word, but it’s true – this is a core part of the theory of the basic precepts of the language!) *cannot* handle that form of decompression. By design. By mathematical proof.

NB: you can workaround this, of course, but doing so within a functional language is difficult for most people.

There is no option in OO to “not use this optimization”. The file format for OO requires that the output XML file be compressed as a ZIP file anyway! So this compression trick is pointless!, maybe? (unless you are using non-XSLT code to process it. Which is pretty sad, since XSLT/XML are designed to be used together)

Several people have tried to provide exporters for OO to get around this (hmm. It’s a tad ironic that the built-in file-format is so hard to work with that people have to write custom exporters instead). Unfortunately, most of these exporters WILL FAIL on any real spreadsheet and WILL CORRUPT THE DATA, because the exporter-authors didn’t look carefully enough at the incoming data.

I’ve found one instance of an otherwise quite nice exporter that has the data corruption bug here – http://digitalimprint.com/misc/oooexport/ (NB: I’ve emailed my fixed version to the author of that page – hopefully they will replace their version with the fixed one)

I’ve found one instance of a “fixed” exporter that worksaround some of the data corruption, BUT … is missing the basic features everyone wants of “use the column label as the XML tag”.

And I’ve found loads and loads of broken attempts :(.

FIXED: an XML exporter that seems to work even on large, complex spreadsheets

So … after dusting off my XSLT skills, I’ve taken the former one, rewritten most of it, and here’s a fixed exporter for OpenOffice that will write your spreadsheet out as simple XML. I’ve been using this to convert XL spreadsheets to various things, including custom SQL databases (XLS can be opened directly in OpenOffice; OO exports as XML using my script; I then run a final XSLT from the command line using SAXON which converts the XML into a series of SQL INSERT/CREATE/UPDATE statements, as appropriate).

To install this, you need to follow the instructions at the http://digitalimprint.com/misc/oooexport/ site, but overwrite the “generic_xml.xslt” file that creates with the one provided here: (sorry, I haven’t got around to packaging it myself yet – I’m hoping this gets adopted by the digitalimprint folks!)

Download this file, save it as “generic_xml.xslt”, and copy it over the top of the one that oooexport installed; NB: different Operating Systems will install that file in different locations
generic_xml.xslt

I have some unit tests, but I’m trying to work out where to put them online for easy download (I’m almost tempted to start an SF.net project page just to hold them. Unless there’s somewhere better, I’ll probably do that)

Oh, BTW – here’s a small design flaw in OO: when you run an export in OO, any error messages are ignored and replaced with a useless error dialog that just says “writing to file failed” (it doesn’t even give the filename!). Those error messages are hiding on the serial console – if you’re a sufficiently advanced unix sysadmin to know what a serial console is, and how to read it.

Anyway, if you try running the exporter and it fails, now you know where to look…

So it’s time to call-out some of the interesting things to be found in the blogging world of MMO people.

By the way … you can tell who’s working on uber-secret or personally exciting projects these days because they’ve suspiciously stopped blogging for months at a time. Lazy slackers, the lot of them. The more you do, the more you should blog! :P

There are some that should be on the blogroll but aren’t (yet), and some other bloggers I should mention (but I’m sticking to the blogroll only for this post – I’ll go through others next time). Feel free to add your own recommended reading in the comments.

Blogs to read:
Brinking (Nabeel Hyatt)
* Who? serial entrepreneur, raised funding and sold companies
* What? currently running a funk-tastic social / music / games company with a bunch of Harmonix guys
* Why? big commentator on the games/apps/making money/predictions parts of All Things Facebook

Broken Toys (Scott Jennings / LTM)
* Who? became infamous in the early days of MMOs as a player of Ultima Online who ranted publically, amusingly, and sometimes even insightfully
* What? ex-NCsoft, now doing intriguing web games at John Galt Games
* Why? In his heart Scott’s still a player, and more than anyone else I’ve seen he interprets the world of MMO design, development, and playing through the players’ eyes. Interesting point: he’s mostly concerned with life-after-launch. Funny that. Players kind of find that bit the most interesting. Also keeps a close eye on community-management screw-ups, and WoW generally

Bruce Everiss
* Who? ex-head of marketing for Codemasters
* What? um, I’m not sure what he’s doing these days, apart from becoming a “professional blogger”
* Why? he aims to comment on every single interesting piece of news in the mainstream games industry. That’s a lot of commentary. Always something to read! IMHO he is often completely wrong about anything online-games, and a lot of business and “future of industry” stuff – Bruce is from an older age of the industry. But … he says a lot of interesting things and sparks a lot of interesting debates in the process. Worth reading. Just remember he is extremely (deliberately, I’m sure) provocative, and don’t take it too seriously.

Coke and Code (Kevin Glass)
* Who? A programmer working in mainstream IT
* What? An insanely prolific author of casual games “in his free time, as a hobby”
* Why? Because he’s better at making games than many professionals I’ve met, and he is very very prolific, making new libraries, toolsets, editors, games, game engines – and commenting on it all as he goes, and throwing up new games for you to play all the time

Erik Bethke
* Who? ex-Producer for Interplay
* What? CEO of GoPets, an online casual virtual world that’s especially big in Asia (and based in South Korea)
* Why? A hardcore WoW player who analyses the game-design as he goes, and relates very honestly a stream of both emotional experiences and seminal events in the game that should give you lots of things to be thinking about, especially if you’re a designer, business person, or product manager.

Extenuating Circumstances (Dan Hon)
* Who? ex-MindCandy, current CEO of SixToStart
* What? one of the first Bloggers (on the whole of the internet!) in the UK, and an awe-inspiring assimilator of “everything happening on the internet, with technology, with media, with entertainment and the future of the world” for all of the ten years I’ve known him.
* Why? He’s still an excellent tracker of all those things, and finds memes very quickly. Nowadays he just auto-posts links (lots of them, every day) with a few words of commentary scattered here and there (del.icio.us descriptions) – making his blog a ready-made news filter for you :)

Fishpool (Osma Ahvenlampi)
* Who? CTO of Sulake (makers of Habbo Hotel)
* What? a very technical commentator, often in great detail, on the issues of running a 100-million user virtual world, with observations about Habbo’s community, business, and culture thrown in
* Why? He posts very rarely, but when he does, they’re usually full of yummy detail

Futuristic Play (Andrew Chen)
* Who? ex-VC (Mohr-Davidow Ventures)
* What? entrepreneur with a web-background who’s come into the games industry and bringing lots of useful stuff with him
* Why? blogs a LOT on advertising (and how to make money out of it in games and web and casual), and on metrics, and how you can use them to run you games or web business better. Also has a long fascination with what are the best parts of the games industry, and the best of the web industry, and how we can each put those best bits together to be even better

Off the Record – Scott Hartsman
* Who? ex-Everquest, ex-Simutronics
* What? Senior Producer for MMOs – but previously an MMO lead developer, and once (apparently) a Game Designer.
* Why? he’s funny, he knows his stuff, and he’s worked on some of the most important MMO projects outside Asia, so he’s got an interesting perspective going there.

Orbus Gameworks (Darius Kazemi)
* Who? ex-Turbine, now CEO of Orbus (a games-metrics middleware company)
* What? Likes the colour orange *a lot*, infamous for networking his ass off at games conferences (*everyone* knows Darius), very friendly, generous – and mildly obssessed with the use of metrics and stats to improve the creativity and success of game design (in a good way)
* Why? If you liked the Halo heatmaps when they came out, you’ll love some of the stuff they post on the Orbus company blog. A year ago they were posting heatmaps-on-steroids. If you thought “metrics” equalled “spreadsheets of data” then prepare to have your view changed pretty thoroughly.

Prospect Magazine/First Drafts (Tom Chatfield)
* Who? section-editor of the highly respected socio-political print magaine Prospect
* What? a highly-accomplished English Literature post-grad (bear with me here) … who also happens to have been a lifelong hardcore game player, I think the only person I know who got a hardcore character to level 99 on Diablo2, and now plays WoW a lot.
* Why? although Prospect only very rarely (like, only a few times ever) covers games, it’s very interesting to see what the rest of the world – especially the highly educated and highly intelligent but non-technical, older generations – thinks of us. And a bit of culture in your blog reading is probably good for you, too.

Psychochild (Brian Green)
* Who? ex-3DO/M-59, now the owner and designer of the revamped, relaunched, more modern Meridian-59
* What? an MMO game designer who disingenuously describes himself as an indie MMO designer but like most of the others has probably spent too long doing this and knows too much (compared to many of the modern “mainstream” MMO designers) for that to be true any more
* Why? lots and lots of great design ideas and commentary here for anyone wanting to do MMO design

Scott Bilas
* Who? programmer on Duneon Siege
* What? …in particular, responsible for the Entity System (one of my main areas of interest)
* Why? Scott’s phased in and out of blogging, but when he does blog he tends to do good meaty programming posts that contain lots of source code and some useful lesson or algorithm.

Sulka’s Game (Sulka Haro)
* Who? lead designer for Sulake (Habbo Hotel)
* What? more of a Creative Director than game designer, more of a web background than games, but above all a community/product/creative person who knows his stuff. Also a big player of MMORPGs
* Why? are you cloning Club Penguin or Habbo Hotel and want some pointers about revenue models, community management, and how to be successful with virtual-item sales? You might want to read his posts ;)

The Creation Engine No.2 (Jim Purbrick)
* Who? ex-Codemasters, ex-Climax (both times working on MMO projects)
* What? originally a network / MMO academic researcher, then a network coder, and now the person who runs Linden Lab (Second Life) in the UK. Very big proponent of all things open-source, always doing interesting and innovative things with technology
* Why? Keep an eye on the more innovative technology things that are done with Second Life (stuff you don’t tend to read about in the news but – to a tech or games person – is a heck of a lot more interesting by a long long way), and get some insight into the life of serious open-source programmers who succeed in living and breathing this stuff inside commercial environments

The Forge (Matt Mihaly)
* Who? developer of one of the earliest commercially successful text MUDs, now CEO of Sparkplay Media
* What? spent many years running Achaea, a text-only MUD that made a healthy profit from pioneering the use of itemsales (virtual goods) – and the things weren’t even graphical – and has now finally (finally!) moved into graphical games with the MMO he’s developing
* Why? one of the few MMO professionals who talks a lot about his experiences playing on consoles (especially Xbox), which makes for a refreshing alternate view – especially from the perspective of an MMO person talking about social and community issues in those games. Just like Brian Green, claims to be an indie MMO designer, but probably knows far far too much for that to be even vaguely justifiable

Vex Appeal (Guy Parsons)
* Who? ex-MindCandy
* What? Guy is an extremely creative … guy … who had a small job title but a big part in inventing and rolling out a lot of the viral marketing stuff we did for Perplex City (online game / ARG from a couple of years ago)
* Why? Awesome place to go for ideas and info on the cutting edge of doing games stuff with social networks. Usually. Also … just makes for a fun blog to read

We Can Fix That with Data (Sara Jensen Schubert)
* Who? ex-Spacetime, currently SOE
* What? MMO designer, but like Lum / Scott Jennings, comes from a long background as player and commentator, and shorter background as actually in the industry. Like Darius Kazemi, spent a lot of time in doing metrics / data-mining for MMOs
* Why? Take Darius’s insight into metrics for MMOs, and Scott’s knowledge of what players like, don’t like, and ARE like, and you get a whole bunch of interesting posts wandering around the world of metrics-supported-game-design-and-community-management. Good stuff.

Zen of Design (Damion Schubert)
* Who? ex-EA (Ultima Online), currently at Bioware (MMO)
* What? MMO designer who’s been around for a long time (c.f. UO)
* Why? Damion writes long detailed posts about MMO design, what works, what doesn’t, practicalities of geting MMO development teams to work together, how the playerbase will react to things, etc. He also rather likes raiding in MMORPGs – which is fascinating to see (given his heavy background as a pro MMO *designer*)

[NC] Anson (Matthew Wiegel)
* Who? ex-NCsoft
* What? Dungeon Runners team
* Why? was doing lots of interesting and exciting things with data-mining/metrics in the free-to-play low-budget NCsoft casual MMO. Watch this space…

People with nothing to do with games, but you might want to watch just because they’re interesting:
Bard’s World (Joshua Slack)
* ex-NCsoft
* Josh is one of the key people behind Java’s free, hardware-accelearted, game engine (JME)
Janus Anderson
* Who? ex-NCsoft
* What? um, he’s been taking a lot of photos recently
* Why? watch this space
Mark Grant
* Who? non-Games industry
* What? an entrepreneur, web-developer, and Cambridge Engineer
* Why? very smart guy, and interesting posts on web development (no games tie-in)

Where to start with analysing data?

Analysis of data about what your consumers are doing is invaluable to any company looking to optimize their sales and profitability, especially so for online games. But today (2008) there is little or nothing in the way of credible products for doing analysis suitable for games. Many companies have built proprietary systems, with all the costs and horros that come with that. But that’s way out of the reach of startups and games studios.

So, where can we start? Well, website analysis is a mass-market use for these tools which has a lot in common, so there should be a good range of free software, and open-source stuff too.

Free webstats: Historical Perspective

(not *just* boring history :) – this briefly explains something fundamental about the different types and complexities of stats analysis tools that will be useful background knowledge for future posts)

A little over ten years ago, when I was still a student, and didn’t want to spend money on anything if I didn’t have to, I wrote my own web log analyser, that would run over my Apache and/or IIS logfiles and tell me lots of fascinating things about who was visiting my web site.

There were commercial alternatives at the time, varying from cheap ($50-$150) that lacked basic features I quickly wrote for myself, to expensive ($500-$1000) that had everything I could ever want and a lot more. I soon gave up maintaining my proprietary software (lack of time, real job – and the assumption that good analyser software would come down in price). I started using the open-source analysers, because they were “almost good enough” for very basic usage, and – in theory – they would improve over time.

The cheap stuff compared to free was mostly better just because it had pretty graphs and convenient user interfaces for “1st order” data. i.e. anything that could be determined merely by looking directly at raw data, e.g.

• total number of visitors
• which countries visitors came from
• sites that linked to yours

The expensive stuff compared to the cheap stuff was mostly better for having “2nd order” data, i.e. things that could only be calculated by looking at raw data BUT ALSO by using some pre-calculated 1st order data, e.g.

• percentage of visitors from each site (requires you to count number of visitors from each site AND ALSO to calculate the total number of visitors who came from any site)

…and also “higher order” data, i.e. things that could only be calculated by using 1st order and/or 2nd order data and combining it in new ways, e.g.

• number of visitors from each site who did NOT also come in from another site (requires you to first generate a list of visitors from each site, then start again from start of the logs checking for each one whether they ALSO came in from a site that was NOT the original one)

Most famously, the most desired piece of higher-order data was “find out where each user is going, the sequence of pages they click through whilst on the site, from the first page they visit to the last page”. None of the free stuff did that, and most of the cheap stuff didn’t do it, or did it very badly.

So … what is free today?

10 years later AWStats is not noticeably better now than it was then, despite being actively maintained. It’s picked up some – IMHO – relatively frivolous features and still hasn’t gained the most basic of analysis features from the commercial products of 10 years ago: it still can’t/won’t track for you the progress of a user through the site.

Analog and Webalizer, the two other free analysers I tried around the time I stopped doing my own, both of which were vastly inferior even to AWStats, don’t seem to have gone anywhere in that time either (although someone has forked Webalizer to make a slightly improved version)

Has *no-one* been playing with the source of these tools and adding basic features? I know a few sites- like the excellent InternetOfficer page on AWStats – have been adding and sharing basic hacks to vastly improve it, but these really just scratch the surface of what is needed. (if you’re using AWStats and you haven’t added the IO stuff, I highly recommend looking at them and cherrypicking some you like – although it’s a real PITA to add more than one hack because of the stupid config system used by AWStats – you have to remember to manually increment a unique ID for each module you add. ARGH!)

For the record: I have been using AWStats continuously for the last 6 or 7 years, and have hacked a lot of stuff to work with it. *I* don’t have problems with it, but it’s disappointingly lacking in areas where I need more.

So, I thought it was time to have a look around at what else is out there.

Google Analytics

When Google bought Urchin, I thought maybe this would mean we wouldn’t need to rely on AWstats any more. The truth turned out to be a bit different – Google Analytics is, in many ways, as “almost but not quite enough” as AWStats. In particular, getting meaningful Referrer analsysis out of GA is a nightmare (I have no idea why we’re still having to hack in custom regexps just to get one of the most fundamental pieces of info out of GA – and note that the manual regexp additions still don’t work for a lot of sites: I’ve sometimes set it up on a GA site and nothing happens, for no apparent reason).

GA is awesome for some things – like marketing-centric tracking – and is adaptable (as above) – but it’s still missing so much that it’s no surprise to me that other alternatives continue to be heavily used. Apart from the many things you need to make custom strings to track (like the referrers above), it:

• is several days behind “live” data (at least in Europe, it’s nearly always more than 24 hours behind)
• over-simplifies reports (very litle data is provided for most reports)
• provides no easy way to combine output of one report with output of another – no mashups allowed! – c.f. Yahoo Pipes for an example of what GA could trivially provide to the user to become totally awesome

Now, if there’s a chance GA might be “good enough” for you, then I suggest you take that route and run with it – GA “can do” a lot (if you muck around with it a lot), it’s owned by Google, and it’s very well-known. You can google for a lot of tips on using it, and I suggest reading things like Andrew Chen’s blog which has a lot of tips on what you should be looking for when doing your web metrics. I’ll be coming back to the topic of “what you should be looking for” in another post – but first I want to get the basic state of tools out of the way.

Free Alternatives – a future?

What’s on the scene today? Here’s 6 other free webstats analysers I found (in addition to the market-leader (AWStats) and the aforementioned Analog and Webaliser (which you really shouldn’t bother looking at).

Microsoft’s adCenter Analytics

This seems to be being pitched as a direct competitor to GA, right down to similar naming and presentation (as well as being free to use, and requiring the creation of a Microsoft account to be eligible for using it).

I tried signing up, but then I got this very disappointing response:

Thank you for registering for the Microsoft adCenter AnalyticsBeta project.
You will receive your adCenter Analytics invitation as capacity allows.

This is pretty fricking stupid: if you’re competing against Google, you shouldn’t go around offering users access to your program and then getting all high and mighty about how you might deign to allow them to use it at some non-specified future time of your choosing.

So, for now: Microsoft’s product is effectively vaporware. Sigh.

labsmedia’s ClickHeat

“ClickHeat is a visual heatmap of clicks on a HTML page, showing hot and cold click zones.” – i.e. it tracks exactly where the user clicked the mouse on your page, and then shows you an aggregate of “all clicks by all people”, with places that were clicked more often showing up in a lighter colour than places clicked less often. Heatmaps are a great visualization tool for aggregate data like this.

They have a nice live demo that you can try out, and see what happens on their site – use the username “demo” and password “demo” – although it defaults to showing clicks from “today” which for their site is too few to be interesting, you can just click on the “month” button in the navbar at the top to see an interesting map of their site.

In particular, the way you can change the transparency level in real time is awesome – if a map gets too bright in one area, and you can’t see what people were clicking on, change the transparency to get a better look.

Slimstats

Works fine, but … this is nothing more or less than a simplified view of the AWStats core data – it’s got less data than AWStats but makes it easier to read all in one place.

Oh, well.

BBClone

This is a first-order analyser only. That makes it a complete waste of time, IMHO. Any first order stats I want to track I can do *from the command line* in linux by typing something about this long:


cat "access*.log" | cut -d=" " -f7,9 | uniq -c


…which looks obscure and obtuse, but you can google to find premade ones that do what you want, and then you only need to change the numbers 7 and 9 in there to change what data summaries are provided. And when you use linux regularly, you can remember the whole command line off the top of your head easily, bung it in a script, and you’re done.

Roxr Software’s Clicky Analytics

EDIT: DECEIVED! This one isn’t free at all; it’s like a bunch of the commercial ones today that “pretend” to be free, but have absurdly low limits on the free usage; if my niche blog is enough to go over their daily limits (hint: yes, it does), then the service is clearly a waste of time

This one looks really good. The only problem I can see so far is that it won’t work for sites with “more than 100,000 daily page views” – that’s not going to be a problem for anyone here; when your site gets that popular, you should have the spare manpower to build/spare money to buy whatever you need.

I’ve only just started using this, so I can’t comment on it yet. But I do want to point out they are nice enough to provide a Wordpress plugin for you that automatically adds the tracking stuff to each page as required, so that makes life easier for anyone wanting to track their WP blogs.

Reinvigorate

This used to be a web stats analyser, I know a few people who used to swear by it, but apparently not any more – they’ve replaced it with a desktop application that is “powered by REinvigorate” but appears to be a lot less what we want here than the old Reinvigorate stats analysis.

There appears to be no way to get access to the *actual* Reinvigorate, the product we wanted to use; all links just go back to the download site for the desktop application instead. Oh, well.

Woopra

Looks promising – but (like the Microsoft product) it’s currently an invite-only beta, with a low limit on the number of daily pageviews, so although it *could become* totally awesome, for now it’s a case of “may work for you – IF you can get into the beta – and IF the final product doesn’t turn out too expensive”. Some big unknowns there. But worth a look, IMHO.

If you need something doing properly, you gotta do it yourself?

So, although this started off as a review of free web tools, now that I’ve got this far I’m considering digging out the source code for my old proprietary web server log analyser and starting to use it again. Maybe even share it with other people if anyone’s interested.

It was very fast (at least for some uses it was much faster than AWStats), although I think the latest version I was doing some slightly nasty and interesting-but-silly things with using the local file system as a dynamic database – not flat files, but on-disk hashes, to be able to process arbitrarily complex relationships (”show all users who did X after doing Y more than twice in the previous week, but only if they used Internet Explorer on their first visit”) large files in very low memory (hey, back then my server had about 64Mb RAM; memory was at a premium!).

This time, I think it would be interesting to do the whole thing in SQL instead, and run against an in-memory SQL DB like HSQLDB.

(really, though, I’m hoping that this absurd suggestion – that I might write a log analyser myself :) – will poke at least one person into pointing out how ignorant and unobservant I am for not noticing some open-source tool out there already which rocks and does the few things that GA doesn’t :))

Followups…

For another time, I want to cover some of this:

How is this useful for game developers (apart from the obvious)?
What other options are there for people doing online games?
If you’re going to roll your own metrics for games development, how should you do it?

So, what’s the connection between ES and MMO, that I’ve so tantalisingly been dangling in the title of the last three posts? (start here if you haven’t read them yet).

The short answer is: it’s all about data. And data is a lot harder than most people think, whenever you have to deal with an entire system (like an MMO) instead of just one consumer of a system (like the game-client, which is the only part of an MMO that traditional games have).

Obviously, we need to look at it in a lot more detail than that. First, some background…

Massively Multiplayer Game Development 101

There’s a few key things you need to be aware of in MMO development. Many professional game developers know some or all of these already, but seeing as I’ve even met MMO developers who didn’t know them all, and much of what I’m going to say later on won’t make sense without it, I’m going to do a quick recap. Bear with me; I’ll have to do some gross generalization to keep this short (so it won’t be 100% true or accurate).

1. The vast majority of the cost of MMO development is content

Content is:
– quests (missions, storylines, scripted events)
– 3D areas (meshes, textures and logic for: zones, dungeons, towns, landscapes)
– loot (item graphics, drop rates, item stats, item abilities)

Content is NOT:
– Fancy 3D graphics
– Physics engines
– AI
– Core game rules

…which tend to make up the bulk of non-MMO games.

As Raph Koster (amongst others, but IIRC Raph was one of the first to come up with a clear concise description) pointed out close to 10 years ago, the rate at which players consume content vastly excedes the rate at which developers generate it – you have perhaps 50 people working purely on content generation on a modern MMO, and you have perhaps 500,000 people consuming it. The problem is, those 500,000 use Thottbot, so they *share* their discoveries, and consume at a rate proportional to the square of their number, whereas it’s generally produced at a rate more directly proportional to the number of developers.

There are many ways to work around this issue, but most of them end up producing or managing vast amounts of content (they just get more cunning in exactly how that content is generated).

2. The vast majority of the development of an MMO takes place AFTER launch

The ten-year-old MMO’s have been releasing expansions and content updates every 6 to 18 months; modern MMO’s release new content every 3 to 12 months. “New content” in this context generally means entirely new 3D areas with entirely new quests and entirely new items, not to mention entirely new special abilities and new plotlines. i.e. complete “miniature MMOs”.
The only bits you don’t need to keep rewriting are the technology, although nearly all MMOs have done minor updates throughout their lifetime, usually when a new expansion pack is released as a retail boxed product (if you buy and install the expansion, not only do you get the bonus content, but all the “old” content becomes higher resolution / more pretty / added sparkly bits). Although … I long wished there was more updating of tech, and three major MMOs are currently doing huge updates: Ultima Online (1997) recently massively renovated their 10-year old graphics as Kingdom Reborn, Eve Online (2001) just completed their “shininess and detailed spaceships with added humanoids” update known as Trinity, and Anarchy Online (2001) recently previewed massive sweeping changes to their very poor detail and draw distance client (they say “the current engine was designed before there really were GPUs to utilize” but that is blatantly false as the GeForce had been on sale for almost 2 years when they launched, and the TNT,Voodoo, et al for years before that. Yes, I was bitter that when it launched the most recent hardware it supported was 5 years old :)). All represent major changes to the look and feel of the games – it’s hard to overestimate the visual impact of these. Runescape did their mega update (”Runescape 2″) a couple of years ago.

However … all are bringing games that were NOT cutting edge at launch up to a standard that is still well short of what is cutting edge now. This is important to understand: graphical quality is (according to what the successful MMO’s deploy) definitely not the main selling point for these games, unlike almost all other successful games you can possibly buy.

So … what is that “development” that takes place after launch? It’s content, not engine. And even when it’s engine (better graphics, better physics) it’s always a minimal improvement by the standards of standalone PC and console titles of the time.

For reference, many MMO’s after launch end up with LARGER development teams than they had in the 3-5 years of development leading up to the launch. All that extra content requires a lot of people (not to mention keeping all the old content working, updating it where it conflicts with new content, etc).

On top of all that, though, there is the issue of customer support. When you launch a traditional game, that’s it. Done. Over the last 15 years there’s been a trend towards launching occasional “patches” and even minor content updates – but mostly bugfixes – which, incidentally, I suspect has been largely driven by publishers learning from the MMOs: releasing a content patch for an old game is a good way to get extra marketing / publicity and increase some sales. Bethesda’s TES4/Oblivion even went as far as to charge for the patches.

MMO’s require a lot more work than that: since MMO’s are all monetized off continuous play (i.e. the more each player plays, the more money the publisher makes), either through monthly subscriptions or through in-game commerce, it is imperative to keep all the players playing as long as possible. If a player plays your MMO for 12 months instead of 6, you’ll make twice as much profit; if they play your console title for 12 months instead of 6, you’ll make no more money and will probably cannibalize your sales of the sequel.

So. Total lifetime development cost is a lot more important, financially, to an MMO than the pre-launch development cost.

3. The way the core logic works now is not how it will work forever

Sooner or later, you nerf.

Usually because you(r players) discover a disproportionately powerful game tactic in PvP that you didn’t think of which is unbalancing and ruining the game, and so you “have” to fix it.

Sooner or later, you choose to break every absolute rule of your game logic, because a new expansion wants to slightly redefine and “freshen” the core game.

And that means you have to be able to break every rule you ever hardcoded into the game-logic. Worse, it means you have to be able to handle the fallout (all the bugs that the change introduces, plus all the invisible bugs that were originally there but had no noticeable effect and now suddenly have visible effect, the new security holes, etc).

Since the ongoing development of the game (post launch) is greater than the initial development in scope and cost, over the lifetime of the game it is more important to make it easy to change the core logic – and to react to side effects of that change – than to be able to make the core logic easily in the first place.

4. Even a moderately successful MMO generates gigabytes of data every 24 hours

The progress of every player of the MMO has to be uniquely independently tracked, so that the game can remember for each player which quests they’ve completed, which equipment they have, etc. With 100,000 players, that data racks up quickly. A lot of it can be thrown away every day (for instance, you don’t generally need the history of how many hitpoints you had at the end of every day), but a lot of it has to be kept forever (whether you completed a given quest, and for some quests how exactly you completed it).

However, the trend at the moment is towards keeping ALL this data, in the form of historical records of your personal game experience – e.g. the automatic “blog” that is created whilst you play Vanguard, which e.g. updates every time you level up, or kill a monster, or die, etc.

This latter kind of data alone is generated at the rate of around 30Gb every 24 hours.

And any of that data that gets used in-game (e.g. even if just referenced in a conversation with an NPC) has to be programmatically accessible.

Ultimately, MMOs have to store, retrieve, and update vast databases. There is a huge decades-old industry that specializes in providing software to manage databases of this size and larger, but MMO developers don’t like SQL and Relational programming. This is a real problem – relational databases can handle the load, but relational programming is fundamentally incompatible with Object-Oriented programming. The net effect is that programmers who have to write their data to disk find it boring, difficult, and irritating to do – and that leads to mistakes and greatly increased bugs, as well as reduced functionality (because no-one wants to write or add to the code to add the new features more than they absolutely have to).

MMO Development Priorities

The net effect of these aspects of MMO development is that the long term profitability of an MMO can be greatly affected by a bunch of simple issues:

• How much effort is required to change the data used to describe an in-game item
• How much effort is required to add (or remove) new (old) types of in-game item
• How re-usable is the content
• How easy it is in practice (i.e. how much coding is needed) to re-use re-usable content
• How much specialist knowledge of the game code, overall game design, and details of the game systems is needed to modify the content and logic (can new team members be effective as the team who originally wrote the game pre-launch?)
• How exportable is the data generated by playing the game (progress, achievements, player-history, etc)
• How analysable is the game-data generated by playing (to make decisions about what to change, and check that game improvements have improed things)
• How modifiable the core-content is
• How easy it is to change individual rules and check the side-effects of the changes
• How many of all the above changes can be done WITHOUT requiring a programmer (can designers change code themselves? can artists?)
• How much can third-party specialist systems be utilised for server-side service provision

Leading to a few rules of thumb that are often adopted by commercial MMOs:

1. Use commercial databases for all persistent data-storage
2. Use Relational Databases and allow arbitrary general data requests
3. Implement as much as possible of the game-logic as raw data rather than as compiled data (and preferably as data rather than source code)
4. Use standard but simple to learn scripting languages where data isn’t expressive enough
5. Do lots and lots of testing during development
6. Do even more testing post-launch EVERY SINGLE TIME you consider changing ANYTHING on the live servers

Some of those aren’t very effective (too vague), others are horrifically expensive (it takes a lot of people to “test EVERYTHING” every time you make a change to a live game). None of them are great for performance in and of themselves (although they can be made highly performant, that requires extra work), and none of them are “programmer friendly” – in fact, they’re all deliberately programmer UNfriendly, being in there for the benefit of non-programmers.

NB: this is in no way a comprehensive or exhaustive list; I’m trying to give merely a flavour of the thing here for people who aren’t familiar with typical MMO dev processes. I don’t know any real MMO that faces only those challenges and uses only those rules of thumb, but they are indicative of what is used, and very very roughly explain why those things are used.

And so on to Entity Systems, and how they can affect the typical MMO development process by helping with problems or making the current solutions more programmer-friendly… (which is going to be Part 5. This is getting long enough already).

NEXT: http://t-machine.org/index.php/2008/03/02/improving-your-dev-process-automating-collection-of-game-crash-info/ (As spotted in last month’s Game Developer magazine’s Inner Product column. Good enough that I wanted to draw some attention to it, and hopefully add a bunch more of these good ideas as I come across them / remember them)

Most games, during development, automatically either popup – or display in a custom textarea – a message every time somethign unexpected happens, something that MIGHT be a symptom of a bug. There are even conference talks about how to avoid making your playtesters so sick of error messages that they stop bothering to read them, and hence the messages become mostly a waste of time.

How many games do a simple call to the SQL client library to send to the local SQL server a simple detailed report of what happened, when, why, and how bad it was? This information is ALL already available, but how hard is it to copy/past the following into your source code and execute it as an SQL command?

INSERT INTO reports.errors (time, problem, severity, stacktrace, user, IP, clientversion) values (TIMENOW(), “+assertion.getMessage()+”, “+assertion.getLevel()+”, “+assertion.getStacktrace()+”, “+assertion.getUsername()+”, “+system.getLocalIPAddress()+”, “+client.getVersion()” );

Nice idea – free, detailed, metrics on exactly why, when, and where your game exe is crashing, any unexpected problems, anyone who’s running an out-of-date version, etc.

And with the dev team no longer having to rely on users *actually bothering to tell them* when they see big flashing warnings and errors, we might avoid some of the programmer/artist antipathy (”[artist] hey, can you help me with something?” “[programmer] its got an error! no wonder it doesnt’ work!”, “oh, ignore that, it always does that”, “um, how long has it been ‘always’ doing that?”, “ever since around the time the game started running really slowly, 2 months ago. It sucks – now everything takes me twice as long to test. But that’s just life, I guess” [cue: programmer grinds teeth, smashes forehead on desk, or similar]).

NB: I have nothing against artists. They are just an easy scapegoat for a not-very-funny-but-actually-quite-painful-if-you’ve-been-there joke. No artists were harmed in the making of this post.

Speaker: Shannon Posniewski, Cryptic

I was expecting something shockingly naive and/or stupid from the title of the session. The first thing the speaker said was that the title was completely wrong, so I ran with that. With that out of the way, the talk was fine, although small things kept coming out during the talk that were hard to believe or worrying claims.

So it was going OK, until … right at the end, just before the Q&A, and partly during the Q&A, the speaker dropped some serious shockers:

– they’ve never actually used this custom DB: everything that the speaker says RDBMS’s can’t do he doesn’t yet know for sure that his proprietary DB server can do either
– they never bothered to try alternative SQL servers, they just used MS SQL Server
– they’d never heard of Versant until after they’d developed this proprietary DB – apparently they never did any evaulating of other possible DB engines
– they’ve created their own proprietary query language based on C and C syntax, and don’t find any value in using SQL

Disclaimer: I recently started to work for the company that published CoH/CoV, and apparently forced the speaker into using SQL and an RDBMS in the first place. I didn’t join the company until a long time after this had all happened, and I work in a different division in a different continent, so everything in this talk was new to me (although I admit I’d be fascinated to hear what NCsoft’s thoughts were on these development directions)

My own commentary in [ square brackets ], any mistakes/misunderstandings my own fault :).

Title

The title is deliberately wrong and provocative – it’s really that current vendor solutions are inappropriate for the needs of MMOs

Databases 101

[Adam: spent 15 mins explaining basics of modern database usage:

ACID explanation

Relational explanation

SQL explanation

Keys explanation]

SQL DBMSs are safe, tested, used by practically everyone, lots of experience of using them exists.

Why not use it [SQL + RDBMS]?

SQL is optimized for the general case, and is generally “OK” – it will never be terrible. It will always work..

Sometimes queries take a long time to setup – creating an index automatically, etc.

It self-optimizes on the fly with caching etc, but that introduces stalls at unpredictable times. Unpredictable latency caused by this is very bad because sometimes you e.g. try to pick up something in the game world from within the game client, and nothing happens, so user is confused – because of that latency.

[Adam: if this strikes you as very strange behaviour for an RDBMS, you are not alone. It might help to know that the end of the talk, the speaker revealed that they'd only ever used Microsoft SQL Server and hadn't bothered trying out any alternative RDBMS's, and didn't really know much about the others]

Inserts are extremely slow – just for each single row that you insert. Because of all the overhead of checking for consistency etc, inserts are slow.

Once you’ve got data into the database, the per-row overhead to update anything in the row is also fairly expensive (although you can do several changes per row at once for almost no extra penalty).

Object-Relational Impedance Mismatch

Trying to map OO systems into Relational databases is painful and shoehorning it comes with lots of performance loss.

Every class I’ve ever seen in database architecture talks about normalizing all your data. Doing it that way – the “right way” – can really backfire on you.

[Adam: ... apart from all the other classes in DBA, the ones that also talk about DE-normalizing data. I went to different database classes than the speaker, apparently :) ]

On COH we had this idea that we’d store statistics on each individual player – how many times did they pick stuff up, how many times did they kill people, how long they spent online, the number of times they trash-talked in chat, etc.

[Adam: i.e. gameplay metrics, a la the work done by Bungie with Halo heatmaps, or by various 3rd party companies at the moment in the games industry. Cool stuff]

We made a table where there was one row-insert per player PER piece of data that had changed. Some of those stats change all the time, like “how long have they been online”.

So the traffic to the database was extremely high, because each thing had to be updated constantly.

[Adam: Well ... IME this isn't the *normal* way of dealing with that use-case, so I'm not surprised it bombed. If they'd approached it in a more standard way, using simple multi-server database organization, I suspect it might have worked out fine from the start]

So … we threw out all the tranditional DB concepts, and made one column per stat, with no actual names for columns, instead just have “Stat 1 value”, “stat 2 value”, etc.

[Adam: I really don't get how this ever even seemed like a sensible idea, but there was a long queue of people asking questions at the end so I didn't manage to ask before I had to go to next session]

In the interests of performance, we were twisting our data, either in code, or in the DB, or in both places. This became a huge impedance to our development process.

Solution: Caching

Problems:
– unexpected unpredictable 1-second and multi-second latencies

[Adam: I'm really suspcipious about this problme - where is it coming from?]

Solution:
– created a server process that all DB calls from other server processes were indirected through
– it had a second thread that was busy writing stuff out to the DB
– appeared that all writes were instantaneous, but they were being buffered.
– i.e. write-through cache
– all changes you made in game-server memory would show instantly as having been committed

Problems2:
– throughput was still far too slow, although at least now there was no latency

Solution:
– instead of updating every stat every time it changes, we did “snapshots” and only occasionally save them
– …but means that there was no ACID properties being used any more except for Durability

Since that was shot anyway, we went whole-hog, basically. What we decided was we’d keep all the data for the players on a given map on that game server. The database was being used just as a storage engine instead of as a transaction engine. The business logic is now all inside the game-server, and we just periodically flush it all out to the DB.

Problems3:
– lost data
– server-server comms

Solution:
– flush data faster

Conclusion: we optimized the servers, the database engine, to the point that we lost all the benefits of RDBMSs. All the ACID stuff etc had just become a hindrance to us.

Final solution: Cryptic DB

Requirements
– regain ACID benefits
– low predictable latency
– high read/write throughput
– minimize code-db impedance (the schemas changed very often, and that was very painful)
– backup and restore easily

[Adam: suggests that hiring a professional DBA, and using a commercial OODB, would be the best option to start with]

Relaxations
– offline character access is unimportant
– domain integrity became the responsibility of the programmer
– general ad-hoc queries can be slow (we don’t use them)
– SQL not needed

Background – some stuff they’d already done

In CoX we took all static data (mission data, character class template etc) and didn’t put it into a DB. We stored it on disk and loaded it into memory manually.

To help us do this, we created a “StructParser” [just a serialization tool]. This was what we used to start building Cryptic DB. We wrote a pre-parser that looked at our C structs in the source code and wrote out parserss/serializers for it.

Using this for live data as well as static gave us benefits of not having two different storage systems in parallel.

[Adam: in general, that's a pretty good thing to achieve]

Allowed us to create generic mutators, because we know the datatypes and memory layout. So, now we have a generic mutator database. Only simple operations at this point, like mathematical addition of a single counter.

Complicated things like checking whether to allow a trade (lots of rules and things to check) need business logic. We decided to sepaaret the data and logic away from the database, and put it into our game server.

A particular game server owns the data for an entity. E.g. a team server would know about teams, and anything that has to do with teams will be held on that server. It knows how to modify those entities.

[architecture diagram]

Most transactional actions are now being done locally only, so execute extremely fast.

To make it easier to develop, we created auto-transactions. These instrument and change the code so that transactions are written in C. We did field-level locking on the items we were modifying.

[C code with a bunch of special macros implementing a basic transaction, but mostly written in C]

[Adam: here he gave a simple example of distributed transaction processing (although he didn't call it that), enough to convey the general idea, although he ommitted the details that make DTP hard / interesting technically]

Cryptic DB uses basic journal-based techniques for data consistency longterm.

Conclusion

We’ve tried stored procedures etc, doesn’t work. Other MMO devs have had to resort to using solid-state disks etc to make it work fast enough. So, don’t just believe what they say in database classes.

[Adam: the mention of SSD is disingenuous: only one MMO company has decided to try that, and their from Iceland so they're a bit odd anyway (/me ducks and runs for cover) - but seriously for them it's part of an overall game-design and/or technology strategy - they aimed for and achieved gameplay that Cryptic hasn't (single-server world)]

Q: Did you look at federated data as a solution?

We looked a little, didn’t try to implement it, because we’d worked out a proprietary solution already that we reckoned would work.

Latency would still be a problem anyway with it.

Q: How do you handle corruption of data in your DTP?

There’s certain things we relax – we don’t worry about cosmic rays flipping bits randomly.

Within the database, everything is checksummed.

Q: what’s the throughput of your cross-server transactions? How many of your servers are local versus distributed?

I would say at a guess we have a very small number of distributed TP’s. We have a zoned games, so almost everything is local, probably less than 5%.

I have no idea how much slower cross-server transactions are. I don’t think they’re particuarly slower, the biggest problem is just the overhead of having to wait for the locking of data. Probably a couple of times slower, but not an order of magnitude.

Q: you’re saying that with your proprietary “Cryptic DB” you keep all data in memory. What do you do when you hit the limit of RAM in a machine?

It’s a pretty straightforward task to virtualize your memory. If you’re WoW you’ve got no chance of fitting it all into memory, so you only keep in memory the working set of characters that are online right now.

If you still run out, then we make a second shard. We’re talking tens and hundreds of thousands of simultaneous players before we get to that problem.

Q: Do you cluster machines to make the DB?

It’s one giant machine with a lot of memory in it

[Adam: which would suggest you could benefit from SSD, maybe... :P]

Q: Have you considered co-locating the data cache with the servers themselves?

No. We haven’t needed to consider it yet.

You can compress the data in memory and other tricks, so it’s not a problem yet.

Q: how do you do backups?

A set of three servers work together to make sure they can handle the load and so that they can ensure they are being back up correctly.

These are effectively zipped flat files, for simplicity. There’s one server frontend to rest of games, responding to read requests and making in-memory changes for transactions that come through. The second server is slaved to it, and every once in a while writes out an image of it’s DB in a safe way. The last server picks up the things that have been changed and merges them into the master DB. Master DB is then copied back to first and second servers.

At any one spot in the change the data is consistent within that one machine. There may be a huge journal at some point, but it will be consistent.

It turns out that MMO players care more about not being able to play than about losing ten minutes of game data. By about an order of magnitude.

When someone claims they lost a Sword of Death, don’t worry too much, just give them a new sword, because you save a lot of pain and money in customer support costs.

Q: Sounds like you’ve made an OODB, how does that compare with Objectivity or Versant? And the disadvantage of an OODB seems to be the loss of querying ability, so you can’t research behaviour or do metrics to examine your gameplay etc – how do you sovle that?

When we started working on this we didn’t know that Versant existed. We only found out when we first spoke about it publically.

Other ones we knew about but we already had what we wanted from a previous game that was fully tested by us and used in produciton. We were going to have to worry about learning the vendor specifics, and converting our data and pipeline stuff.

I’m also not convinced that an OODB would have worked for the stuff we’re trying to do. We’re writing a lot of code, and the fact you can write AUTO_TRANSACT at the top of a function, and knowing it will Just Work is worht it’s weight in gold.

Re: adhoc queries, we have created a proprietary query language that looks a lot more like C than it does like SQL.

If for some reason we lack a feature we need in our DB, it’s so easy for the DB to interact with C code that you could just write the damn thing in C and have the DB execute that.

Q: Is this in production?

No

Q: How do you test it?

We eat our own dogfood. The true test will be the next time we have a game that goes live.

We have a zillion stress tests, and run that stuff routinely, of course. Right now we just have test-harnesses that fake it.

Q: Could you have made Cryptic DB from the start, or did you only learn it by using SQL from the start? Did you get anyt benefit from SQL?

We got nothing from SQL, it was forced upon us by our publisher who forced us to use SQL. NCsoft were doing network operations and backups etc, so they said look we already know how to use it, and go ahead with it.

I have no idea if we’d have used it given free chance.

We learnt from it that variant latency is a killer, and learnt where the major bottlenecks were in MMO data writing, so we did learn a lot by working with SQL that was transferable knowledge.

Q: How do you handle verisoning on the data, like schema changes?

The actual DB if it sees a field that it doesn’t know, it just ignores it silently. If we add a field that isn’t there, it silently creates it.

We set things to zero or null to create them.

Q: Do you have a central contorl of the DB design, the schema?

Every programmer can add whatever fields they want. We haven’t run into any problems with it yet.

There’s a certain data structure which is the character definition. At the start of the project someone sat down and designed that, and that doesn’t change often. What does change is the occasional adding of a new data field every now and then [i.e. no additional tables being added]

Q: How much did it cost in money and hours to build?

Whole system from start to finish was two people for maybe a year.

Q: Did you try multiple implementations of your SQL server? Different vendors?

No, we did not. There was no indication to us why different ones would ever be faster, so it didn’t seem worth investigating it. I think it is problems endemic to all databases in general.

unless, maybe, you just spent huge amounts of time and effort on database administration. We don’t know how many players we will have, so we can’t actually define optimal schemas and things in advance.

One of our main goals was to spend more time on game coding and less on database coding.

Q: did you spend a lot of time profiling your SQL server?

I think we tried profiling everything, automatically created indexes, stored procedures, etc. We got maybe 15%, 20% improvement, and DBA’s think that’s great, but we were looking for multiple hundreds of percent improvement in performance.

Q: Does your new DB store quests?

It stores state for how you’re progressing through, but not yet the actual quest template data.

Q: Have you looked at any vertical file DB systems?

No.

It’s entirely possible that it’s much faster than using Relational. We were already used to using SQL so we kept in that direction.

Q: will you be releasing this as a third-party DB?

Maybe – but not at least until we’ve put it into production for ourselves!