What’s with the obsession with having one unified table for all your stats? That’s insane. It’s a fucking RELATIONAL database.

I can only hope that the talks I have given on this have steered developers away from making those mistakes in the future.

Another note is that I probably overstated it when I said “Whole System in 2 programmer years”. What I meant was the Transaction and DB part of the system. It was built on top of a data serialization engine developed and in production in CoH. Basically, it was 2 programmer years to transition from a robust data parser to a high-performance DB and transaction engine.

This paper is an interesting take on how a specific OLTP engine could be designed:
http://www.vldb.org/conf/2007/papers/industrial/p1150-stonebraker.pdf

Summary: Current RDBMSs are provably useless for everything but OLTP and actually they’re rubbish at that too. Suggested replacement (called H-Store) OLTP design using distributed nodes with all data in main memory. No redo log (failed machines rebuild from good ones), undo log in main memory and not persisted beyond completion of transaction. Clustering gives availability and scalability advantages. Since OLTP transactions are light-weight and fast, make it single threaded for each node and lose multi-threading management overhead. Assumptions made about the types of transactions to be run (no ad-hoc queries, no analysis queries etc.) and query planning and execution simplified based on that. They’ve built a prototype that wins at TPC-C by a factor of 82.

It’s not actually that far from Crypic DB, if I understand it correctly.

Personally, having historical game metrics in SQL accessable relational databases seems like the most sensible approach. Whether the write-heavy ‘live’ game database should also be in a conventional disk based RDBMS is worth discussion I think.

I realize now that the talk should have emphasized the static data storage system we had more. It was the existence of this system which precipitated our approach to Cryptic DB. Since we already had a method of defining data structures with metadata and storing and retrieving them generically, we already had a big jump on the system. The DB impedance is practically zero, which is a huge boon.

Matthew: The write-through cache method operates well for City of Heroes/Villains and is still used today. Debugging it wasn’t fun, and we lost real ACID game transactions in the process. In our new engine, we need a lot more performance than that could provide (15x is the goal, IIRC) and we want game transactions (e.g. both sides of a trade) to be ACID to the DB. A write-through cache does not typically provide this.

I appreciate all your comments. If I ever give the talk again, I shall be certain to declare the shortcomings of the talk itself at the outset. :-)

[Adam: I’m really suspcipious about this problme - where is it coming from?]

We’ve run into the same thing on our SQL Server installations. The SQL Server collects statistics and uses them to change the optimization plan used for queries. On a busy database this can introduce a delay of up to 30 seconds at the moment the plan changes.

SQL Server 2005 introduced a new option to do this statistic updating in the background so queries don’t get queued up behind it. CoH came out in 2004, though, so was a little late to do them any good.

On the other hand, because we’re already using SQL Server as a storage engine, it’s fairly easy to mix in some relational data (in our case, the goal is to post-process saved data and make a second, non-authoritative copy in the database that is amenable to data mining) as we go along. The data mining can be done by pretty much anyone at the company with a little SQL experience, and because we get transactional replication for cheap (that is, it’s already coded), we can make offline copies of that data as well.

It’s really interesting to know what you tried, and I like that aspect, but a lot of people in the audience are going to take what you say as true, and a lot of what you did say flies in the face of standard practices that are easily researchable on the web these days.

I think if you focussed more on the “what we tried in order to solve the problems” BUT also went and researched other solutions to those kinds of problems, and presented the two sets of approaches side-by-side, and perhaps looked at why it was that you had so much trouble finding out that stuff in the first place, then most or all of my criticisms would disappear.

Every MMORPG developer I’ve spoken to has had major performance problems with their database. They have tried to solve this problem in several ways. I didn’t make a big enough point of this during the talk, I guess, but I was presenting what *Cryptic’s* experience was with a relational DB and what we decided to do to solve those problems. I wasn’t trying to sell anything or saying that “this is *the* solution.” It’s merely what we decided to do.

Another studio may have different needs (and competencies) from ours, and so their decision might be different. We did the best thing for us.

High volume of inserts is indeed a performance hot spot, and most generic relational database systems don’t deal with this all that well (relatively speaking). But then the nature of the data (for metrics, logs, and audits) is such that backing it with full relational model is not ideal anyway (most of it is either aggregate or sequential data which lends itself better to a “flat” record model).

There are significant industries that need to deal with this same issue (high level of concurrent inserts for logs and audits, with low and/or predictable latency) and major database vendors do offer products that are specifically tailored for this use case (MS SQL not being one of them).

Spending few weeks/months for research might have saved a year’s worth of implementation effort.

My god… they spent 2 man years developing a database system that they think will out-perform systems that have spent thousands of man years in development? I have no doubt this will crash and burn.