e.g. they are effectively unusable with Apple’s own SVN integration (Apple clearly doesn’t use SVN internally), unless you are the only person working on your project (”no teams allowed”). But the good news is … most of this can be fixed!

Step 1: Find the project templates

…usually somewhere buried inside /Library/Application Support/Developer/Shared/XCode

Step 2: Copy *all* project templates into your home directory

IF YOU DO NOT DO THIS, APPLE *WILL* DELETE YOUR CHANGES (every time you upgrade XCode, it sees your changes, and says to itself: “Oh. How cute! Something for me to destroy!” and promptly overwrites them)

Copy everything in:

/Library/Application Support/Developer/Shared/XCode/*

to:

/Users/[your user name]/Library/Application Support/Developer/Shared/XCode/[your company name]/*

replacing the bits in square brackets appropriately.

Step 3: Start un-f***ing the defaults

Now for the painful bit. Apple’s system for project templates is very basic, and what they’ve done is to create a separate template for EVERY combination of project type, platform (iPhone, iPad, and “both”), and tickbox options (with core data / without core data, etc).

You have to make the same changes in every single one of those projects. This is hours of work. But, once done … you are free of having to re-do this work on every new project you start.

Example fixes I like to do:

NOTE: Apple’s version of SVN requires that you split your project up into as many XCode groups as possible; only one person in an entire team is “allowed” to add/remove/rename files inside a single group. Anything else, and SVN trashes your XCode project file.

So, not only does organizing your project increase productivity (it’s quicker and easier to navigate), it means SVN will crash less often. Hooray!

1. Create a Resources directory

Apple – amazingly – didn’t bother to do this. They create a Resources group by default, that dumps all your (usually hundreds) of resources and images into the root directory.

So, delete the group, go to Finder (right-click on the Project itself in Groups&Files, and select “reveal in finder”), create a Resources directory, then drag/drop the directory into XCode.

Result: it appears *exactly the same* on screen (sigh), but at least your image files will now get organized into their own neat directory.

2. Create a Libraries directory

Most apps use lots of libraries. Apple refuses to allow Frameworks on iPhone, so we’re forced to use Libraries instead. Framworks are neatly bundled into a single item inside XCode; Libraries are a mess of files. Since Apple gives us no choice, let’s at least make it a little less painful.

As for Resources directory above, create a Libraries directory. Anything you drag-drop into XCode’s Libraries group will be copied into the Libraries sub-directory.

3. Fix the ObjectiveC Linker bug

Apple’s linker has a minor bug that causes most libraries to fail to run; if you use any 3rd party library that was written in ObjectiveC (that is: nearly all of them on iPhone), you will probably hit this bug.

Every one of your targets MUST include this in the build settings:

“Other Linker Flags” : “-ObjC -all_load”

…hopefully Apple will fix this in XCode 4 – or just make it the default! For now, you know what to do: put it in as a new default :).

4. Create 4 (or more) extra Targets

XCode has a *LOT* of critical showstopper bugs that Apple has been slow to fix, especially in the area of “building your app and running it”. I frequently encounter these on other people’s projects. There is only one sane workaround for this, and that’s to do what XCode’s designers intended – but which Apple doesn’t help you with – make LOTS of Targets.

In XCode, a “Target” means:

A version of the app built to run on a specific piece of hardware, or with a specific code-signing

The App Store requires a unique code-signing.
AdHoc copies require a unique code-signing.
Running your app in “test mode” on a device … requires a unique code-signing.
Running your app in the Simulator … is a different platform.
Running your app on the iPad … is a different platform.

So that’s 5 unique targets right there. They ALL REQUIRE separate Targets. If you read Apple’s docs closely, you’ll see that the Developer Portal does – quietly – tell you to do this. Many people just constantly – manually – edit their build files, changing them back and forth. Nightmare!

Worse, XCode has several bugs where changing build files gets ignored, either for an amount of time (say 10 seconds) or until a specific event (say, rebooting OS X. Yes, really!).

So, it’s imperative that you set up separate Targets in XCode, and that you use them. Apple provides a control that does a fast switch between different Targets, so it’s literally just two clicks to switch at any time.

Each target also has – potentially – unique:
– libraries that it includes
– compiler flags
– resources
…but by default, all the above are AUTOMATICALLY added to all Targets, so life is still easy.

Here’s the Targets that I find I need on every single iPhone app I write:

1. appname App Store: this is the one that is configured for uploading to the App Store, and is codesigned for it
2. appname Sim Local: ONLY compiles for the Simulator. Uses Compiler flags to use local resources rather than the internet. i.e. accesses our local Staging server for web fetches, so we can use test data (and so testing is much much faster). Anyone who writes internet-based apps and doesn’t use a Staging server is rather foolish.
3. appname Sim LIVE: as for above, except it connects to the live servers. Useful for doing fast testing against Live data; I usually use this for checking reported bugs (it’s faster than constant synching to the device)
4. appname iPhone: ONLY compiles for the iPhone
5. appname AdHoc: creates an AdHoc copy. NB: this actually requires several extra files (entitlements, iTunesArtwork) as well as using a different codesigner

…although I haven’t adapted that list yet for iPad development. With universal binaries, I’m not sure how many of those I can merge, and how many I’ll have to duplicate.

5. Replace Apple’s crappy source-code templates

Do you like having stupid, fake “copyright” notices at the top of your source files? Using C-style doc comments no less (i.e. incompatible with all source-code-documentation systems)? No, me neither. If they were Doxygen, Javadoc, or similar comments they might at least be useful, but Apple didn’t bother with that.

So, again, you have to fix it yourself. This is similar to steps 1 and 2 above, but you take DIFFERENT files and you put them in a DIFFERENT folder. See here for a simple guide on how to do that. Once you’ve made the copies, you can manually edit them and remove the cruft.

6. Set the defaults for NIB files

This is a bit trickier. If you edit the template for new NIB files, and hit save … Interface Builder corrupts the file.

You have to do this:

1. duplicate the template file
2. Edit the template file
3. Save it
4. Go to a command line, and run “diff NEWFILE OLDFILE”
5. Open the old template file in TextEdit, and manually insert the SPECIFIC changes you made, while ignoring the changes that you didn’t make

If you don’t do this, weird stuff happens, like new NIB files getting confused about who their “File’s Owner” is, and putting “<<NAMEOFOFILESOWNER>>” or similar rubbish in there.

Warnings

There’s a few things to note, where XCode’s badly-written build code can collapse. No reason why it should do this, but it does.

Don’t move plist files out of the root

XCode (like all OS X applications) “knows” when a file is renamed or moved. It updates everything correctly EXCEPT the app’s main plist file. Since every target has a unique plist file, you need to keep all of them in the root, or the build stage will crash.

Give each Target a unique “Product Name”

XCode has poor exec code, and it gets “confused” if you have two targets that both output applications with the same filename. It seems to work, then it will occasionally mis-compile / mis-build – or even refuse to run at all.

The easy workaround is to edit the build-settings and give a different suffix to each Product Name – e.g. if the default is “appname” then make the AdHoc target be “appname AH”.

Welcoming some new people to Red.Glasses (my iPhone development / agency company), and lots of projects all happening at once.

Given how many people I see out of work, I’m delighted to.be bucking the trend, but until.our new people get.up.to.speed, its a bit too much, and I’m looking forward to things calming down a bit.

PS: typos due to the nexus one and its broken touch sensor and badly designed keyboard. Don’t have time to get to a PC to.post: )

I had an idea; maybe if I deleted TB, then restarted it, and forced it to go offline first, THEN configured folders, THEN allowed it to connect, it might. just. work.

Of course, doing so discovered some obvious bugs in TB. Sigh. I got it mostly working in the end, after some false starts. So, if this is something you want to try, here’s how to do it:

1. Start TB, and give it your account details.

You want it to look like this (see note above; if you don’t do this correctly, you’ll get different settings, with no SSL/TLS)

2. QUICKLY go to offline mode (next step) or … unplug your internet cable – this may be easier

Why on *earth* do you have to do this?

Well … there’s a major bug in TB. If you do NOT create an account *the first time the app launches*, then you can never create a Gmail account. The code that configures email accounts *will not run* after that first launch.

Really, I’m serious: try it. It will – if you’re lucky – try to configure your gmail account with all encryption turned off, thereby sharing your emails with everyone on the network. Nothing you can do will make it accept SSL/TLS. Even typing the port number manually, it fails to work.

BUT … if you allow it to configure the account on first-run, it will correctly setup everything as SSL/TLS

3. Go to offline mode

4. Go to Synchronization settings (from the same menu – File menu)

Click the ADVANCED button (it’s not advanced, it’s basic, but this is a hangover from the Mozilla Mail days, when the GUI for configuring the app was poorly arranged)

5. De-select the folders that Thunderbird should never have pre-selected in the first place

NB: this is language dependent! Google “kindly” names some of these folders depending on your regional language. Great idea, unfortunately it makes config / instructions a bit more tricky.

In the UK (this is different in USA!), the folders you must untick are:

• Bin
• Trash
• [Google Mail]
• [Google Mail]Bin

Like this:

and this:

6. Are you finished ? No, you aren’t; There’s some “magic” folders left…

On the left, under Inbox, there are a couple of magic folders that don’t appear in the synch list – but will synch automatically, and kill your disk space. One of them in particular: “All Mail” (that is: every single email that Gmail has for you, ever, anywhere. All unsorted)

You have to right-click, and go to the 3rd tab, and deselect the checkbox, to make this safe:

You MUST also do a difficult one – the Trash folder.

Thunderbird will *not allow you* to deselect this one, but you have to expand it, and inside find the “Bin” and “Trash” folders, and manually deselect them as with “All Mail”.

NB: in the attached screenshot, I hadn’t realised this was needed, so you can see it’s downloaded my whole Trash folder from the server. Ugh.

NB: yes, in theory you have *already* deselected those folders. But with Gmail you can easily end up with two copies of the folders, both with the same name, but in different subfolders. Depends which clients you’ve used with Gmail in the past (ironically, in my case, they exist because I ran an earlier version of Thunderbird, where the Gmail integration wasn’t so good).

(you may also want to do the same for Sent Mail and Spam – depending on personal preference / need)

7. Finally (finally!) you can de-enable Work Offline, from the File menu

Now what?

Well, nothing happens. Because Thunderbird is “magic!”.

If you wait long enough, and you’re lucky, it will magically start to synch.

If you don’t like waiting, staring at a computer, you can force it to download – go to the File menu, and from the Offline submenu, select Download now.

HOWEVER once again there are issues: this will work outside the normal Activity Window system, so your download will be in the foreground, and doesn’t even appear as an “Activity”. Sigh. So, you might want to try waiting for the “magic” to happen instead, and hope you get lucky.

(in theory, it should be a 10 minute wait at most, but earlier versions of TB 3 used to have severe problems with this, sometimes never synching at all)

Summary

This is still imperfect. Thunderbird insists on “indexing” all 5,000 of my Sent Mail contents – although at least it’s no longer trying to download them all locally. It also insists on “indexing” the contents of things like “All Mail”, which is completely incorrect.

But I can find no way to remove the annoying “All Mail” folder, and if you ever click on it accidentally, then BAM! the expensive processing starts. The best you can do is use the little buttons at the top left hand side to switch mode to ”

Oh – and you aren’t allowed to stop it. The Activity Manager is missing a feature (which is already in bug reports from last year – I remember seeing people ask for it) : a “click to cancel” button for each activity.

Hopefully, though, so long as you remember to avoid clicking on those magic folders, you can at least make TB work normally…

Ditto for all the folders we chose not to download locally … and, ditto, I can’t find a way to remove them from the list, so there’s no way to stop myself from accidentally clicking one.

There are a few other things you should do too, to make up for the incorrect GMail setup. For instance, you should go to Thunderbird settings, and DISABLE the “save copy of messages in Sent folder”, because GMail will automatically do that on the server side anyway.

PS: … ARRRRRRRRRRGGGGGHHHHH!!!!!

I was just about to press the post button, when I noticed something horrific.

TB just started downloading the 5,000 emails in a “Bin” folder … somewhere … despite my very clear instructions not to.

This is a buggy load of crap. Try the workarounds listed in this post – see if they work for you. But, ultimately “Don’t use it” is my advice.

That came as a major shock. I didn’t set out to write about EE, I just wanted to have a look for myself, but this changed my mind. I’m probably not the target audience anyway, so take this with salt, but IMHO it raises some interesting points.

Alpha is the new closed-beta; closed beta is the new beta; beta is the new launch

The game is officially in Beta, I believe. Is that an excuse for what follows?

IMHO and IME: No. Within the context of “MMO’s”, and especially in the context of the web (look at the various Google products that are technically still in beta), you can no longer hide behind that as you once could.

I did experience a few small parts of the EE experience along the way, so here they are:

1. You are not allowed to play the game straight-off; you are required to provide personal info etc before even being shown a screenshot of the game, let alone running the game itself
2. Copy/Paste is disabled in the registration form. It works for the “name” field, but not for the email and password fields.
3. My main email address is on a domain name that “Email provider is not supported, try another email address.”

(in desperation, I tried bugmenot.com, but unfortunately Sparkplay has already deleted the only account on there)

For any mainstream PC MMO you’d call this a bizarre over-reaction on my part (most of them would already have taken my credit-card number, so few people will stop to quibble about email address issues). But this isn’t one of those games…

“Finally, a real MMO for your browser”

(that’s the Earth Eternal tagline)

O, RLY?

Let’s get this straight. 15 years after the web became popular, you’re telling me that “the browser-experience” is:

1. It’s Windows-only (!!!) [BTW: despite *already knowing* that I'm running a Mac, they take your personal info *before* telling you this]
2. I’m not allowed to try the product before I use it
3. I’m not allowed to use an email address of my choice, I have to use one that the website-ownser “approves” of
4. The website-owner specifically writes code to disable basic features of my web browser

In a world of Kongregate – where I can play several MMO’s *right now*, on any computer, some of them with no signup – you’ve got to be kidding me.

I’ve known Matt for years, and I’ve watched with interest the trajectory of Achaea / IRE / Sparkplay. It’s been an interesting journey, from the years of declaring that 3D graphics were great but unnecessary for commercial viability (with his own Achaea the poster-child), to today’s VC-funded 3D MMO.

There’s an irony there, especially if you dig out Matt’s frequent public statements on 3D games in the past – but in context it all makes sense. I mention it pre-emptively, so I can knock it down :). Yes, Matt said such things – but at the time he also actively proved that you could make a living off non-3D games. He showed that with controlled budgets, intelligent business models, and controlled scope … it worked. It was small revenues, small profits, but it was profitable. Many of the lessons that he shared have been picked up over the years and re-used by 3D games in the race to drive down their own costs, and adopt more competitive business models.

So … I don’t care about the change in stance (hey, the world has changed a lot in that time), but … I do care that someone who taught so many people how to avoid the high-barrier-to-entry business models manages to shut me out of playing his first graphical MMO.

Compare and contrast…

I’ve already mentioned Kongregate. The vast majority of games there require no login, including some small (but grind-tastic) MMOs. The most recent substantial MMO to appear on there *does* require that you already have a Kong account, sadly. (Although, incidentally, if you install the Kong Facebook app, it silently creates an invisible account + logs you in, so it’s possible you’d play that MMO without ever knowingly creating an account)

I suspect that Sherwood still doesn’t require reg – when I used to play, it certainly didn’t, and IIRC from conversations with Gene, this was something he actively chose to retain. Ironically, I couldn’t double-check this: Adobe’s current installer for Shockwave has a chunk of badly written code that attempts to shutdown “all web browsers” before it will run – even if I’m only installing in one of the browsers.

So … EE may be a great game … and it may be launchable from within a browser … but it’s still fighting the browser as much as complementing it.

I’m using this app as a personal experiment in designing educational and learning games. Unfortunately, this update took something like 4 months to get through submission (rejected twice, first time for a missing “cancel” button, second time for using private APIs) … so it’s been hugely delayed.

Last year, I spent a lot of time investigating a new idea for a startup that would make online games used solely for education. This is a clone of one of the demo games we made as part of the investor presentation.

The “experimental” part is that I’ve been trying out different variations on the social and online aspects. Most of that is missing in the current version – to get it through submission, I ended up ripping out most of it.

But now that it’s been accepted, I’ll be updating it a lot more, gradually adding features back in (e.g. Facebook integration, which I originally had working a month before Facebook Connect went live).

By the way … For the record, although the app is a paid one, and I’ve made a bunch of apps since, I’ve never earned a penny from this first one – it’s never hit the minimum threshold of payments (approx $250) before Apple actually pays you.

2 weeks ago, I started a small blog called App Rejections. Here’s the traffic graph for the first two weeks. The tallest peak over towards the right is 25,000 unique visitors hitting the site:

It could have been worse; being a die-hard pessimist when it comes to server management, I’d left the webserver “throttled” to a fraction of max connections. The jumps in traffic when I selectively removed that throttling suggest that overall perhaps 10% of visitors couldn’t get through (as high as 35% at peak times).

I *believe* there will be no performance problems now. I should be able to take those peaks again without the server dieing – I’ve put some simple cheats in place, like a pre-generated static front page that I manually cache. Although I’m also hoping to double the RAM in that server (it’s only got 1GB right now!).

What happened?

Well, apart from the obvious (lots of news sites picked up the site, and a few big-hitters on Twitter too, the biggest being someone with almost 1 million followers. Awesome! But also … Argh! Webserver death!).

Actually, what happened was my own laziness combined with a cluster**** between Apache2 and PHP5.

Basically … PHP is a fine language with a sometimes-dubious runtime and a terrible set of libraries, much of which remained incompatible with Apache2 for a very long time. Allegedly, some core parts of PHP’s sphere of libs are *still* only compatible with Apache1 (or the backwards-compatible part of Apache2 – i.e. none of the performance boost). (really? is this still true? hard to believe)

And while I *should* have been running Apache2 in modern mode, ditto PHP5, the last time I checked PHP and my PHP libs, they weren’t yet compatible with Apache2. There are workarounds, but they were enough effort (and this is just a small server) that I hadn’t bothered. In particular, I probably should have setup FastCGI to workaround the PHP incompatibilities. But it’s been a LONG time since I’ve used FastCGI…

Running Debian, I’d left this server with the standard PHP setup: Apache2 in backwards-compatibility mode, and PHP in a default config. This server does relatively little webserving; I figured it wouldn’t matter much. Especially with the “pessimistic” Apache2 settings.

Unfortunately … Apache2 in compatibility mode reserves RAM per HTTP connection, and never releases it, even after its no longer needed. Even with my concurrent connections throttled to a low sensible number (20 in this case), *some PHP process somewhere* was taking advantage of the oodles of RAM, and sucking up all the RAM in my server.

For no reason that I can tell. Which is worrying.

I’ve now put PHP back down to a hard limit of a few tens of megabytes, and – so far as I can tell – no part of any PHP app is failing. I’ve also set a 100-requests limit per process, which will force Apache2 to recycle *all* RAM every twenty minutes or so at high load.

I’ve de-throttled the concurrency, so the webserver can still use all available RAM if it really wants to, but this time only under high traffic – not “merely because some crappy PHP script somewhere is getting greedy”. There’s no complaints in the error logs. I removed Cacti from this server a few months ago (there were bugs in the Debian graphs that I got fed up with), so it’s not that.

Wordpress? Probably…

What I didn’t try, but should have

For some reason, I don’t have any PHP accelerator on that server. There was a reason, I can’t remember what it was.

If I had, then I believe I would still have had the same problems, mainly because the real problem was PHP using up too much RAM. But *maybe* things would have run a lot better – I don’t know, because I didn’t try.

The unfortunate pain of PHP and RAM

The thing that I find annoying in this (apart from the long slowness of getting full Apache2 compatibility) is the number of standard PHP apps that “demand” hundreds of megabytes of RAM each … per request. If you google the PHP error message for “script ran out of memory” (I can’t remember the exact text) for any particular webapp, you’ll find hundreds of hits from developers saying “edit php.ini to increase that limit to 200Mb” or similar.

At 200Mb, your 2Gb RAM webserver will thrash (and soon crash) the moment ten people try to view a webpage or download anything – CSS, JPG, GIF, etc – at the same time.

Ten people. Right. And that’s assuming you don’t have a database anywhere, or any other services running.

And the sad truth is that there’s nothing you can do about it (unless you have a *lot* of time to devote to rewriting the PHP apps yourself, or engaging in various convoluted sysadmin tricks like having two independent copies of the webserver – one for the app, one for the rest of the world).

A lot of modern CMS’s seem to “recommend/require” 70-120Mb. This is insane. What are they *doing* with all that RAM? More to the point … WHY?

In the end, it’s probably an example of my number 1 reason to avoid PHP:

99% of PHP programmers either have no idea what they’re doing … OR … so utterly over-engineer every PHP app that it does far too more than what PHP is meant for *and optimized for*

Sometimes I wonder whether as much as 30% of all PHP programmers secretly wish they were writing J2EE apps instead, but won’t admit it to themselves

The kind of things that a CMS is doing to soak up 100Mb per process make perfect sense inside a Java VM container – and, actually, will work fine without killing the server. They make no sense at all in a PHP project. You *need* a powerful VM to be doing that kind of thing.

Notes to anyone with similar problems

1. If performance gets really bad, use the very old technique of:
   1. restart the webserver (this kicks off all the current users, *and* cleans out RAM)
   2. reload the main pages of the site in your browser, ASAP
   3. save those pages to disk
   4. edit apache’s site config, and add an AliasMatch line for each saved file (saving the front page of the site is always a good start, that’s what most incoming traffic is probably pointing to – use “AliasMatch ^/$ /var/www/path-to-saved-front-page.html”)
   5. restart the webserver
   6. try reloading the affected pages, while the server is under load, to check you didn’t get the AliasMatch rule wrong or something equally stupid
2. Run “top”, and see how much RAM each apache2 process is using
3. Edit apache2.conf/httpd.conf, and make sure that MaxClients is less than ((RAM in this server) / (answer from top above))
4. Edit php.ini, find the line about max RAM usage, and laugh at the “recommended” default of 128Mb. You must be joking. Change it to something sensible – and then redo all previous steps above
5. Run “ab” to rapidly loadtest your server from the commandline and check that the above things are working. Especially useful for forcing lots of traffic so that you can see all the processes jumping to the top of the list in “top”

But, really, if you don’t already know all that, you should probably be finding yourself a qualified sysadmin who does … or doing a lot more detailed learning instead of relying on some throwaway comments I’ve put on my blog! :)

EDIT: Within a few hours of writing this, my gmail has reverted to normal behaviour. I *was* getting the HTML reply for all emails, now I’m getting it for none. I had wondered whether it was context specific – but now I’m getting normal emails even when replying to HTML ones, so apparently not.

GOOGLE IS WATCHING YOU!

I’ve no idea why this is, but here’s a tongue-in-cheek suggestion: Google *really really wants* to be evil. But they’ve made so much news out of their company motto that they don’t believe they can get away with that.

So, instead, they just act passive-aggressive.

This lets them beat down their users, while maintaining a modicum of plausible deniability.

This week’s backwards step for Gmail

You are no longer allowed to reply to emails directly, unless you’re willing to convert them to HTML and send an HTML page instead of a plain email.

Every single time you want to reply, you now have to:

1. click the reply buttton (this used to be all you had to do!)
2. Click another reply button
3. Wait for Gmail’s plodding DHTML to redraw the page
4. Dialog pops up: do you really want to reply? (you click yes)
5. Wait for more plodding DHTML, and eventually you get the email-compose pane

This is quite possibly the last straw for me and Google products. For them to break something so fundamental beggars belief. Can I ever trust any of their products not to self-destruct again?

(this one bug is currently costing me somewhere between 10 and 15 minutes EVERY DAY, because of the number of emails I write)

Why has Google done this?

Two possible answers.

1. The gmail team now employs some really stupid people who apparently know nothing about email clients.

(unlikely)

2. This is a subtle attempt to get a small increase in ad-revenue from Gmail.

By forcing everyone to do the above 5 steps, they are (passive-aggressively) irritating people as much as they can to “punish” them … until the users give in and write HTML instead of emails.

When people write HTML, they’re more likely to insert private rich content (images etc).

Google can make more money from advertising the more they know about you, and the richer the content you’ve included.

(bear in mind this is the same Google that used to try and steal your entire list of private phone calls from your cell phone if you installed the official gmail client – maybe they still do)

Far-fetched?

Maybe. But then … would you have believed me a year ago if I told you that Google would put real-time adverts INSIDE your email conversations, based on the words you typed?

(they added that a month or so ago. They seem to have removed it again now – it greatly increased the time it takes to compose emails, so I think they removed it until they can get it working at a sane speed)

Google Support: Don’t call us. Ever.

Quote from gmail support page:

If the steps provided don’t resolve the problem, you’ll be able to send us a report. While we do read each email, you’ll only hear from us (generally within a few days) if we need more information from you.

This is a lie. That used to work (I once managed to report a bug, a long time ago), but now the system refuses to let you write a report, instead it provides a link to the public forums and tells you to go whine at other users.

I wonder. Is it that they know they’ve broken the webapp, but it was a deliberate act, and they really don’t want to hear anyone’s opinions on the matter?

Even if you create an entirely separate user-account for them, with their own settings … if they happen to setup their keyboard differently, then:

1. It permanently alters the keyboard settings, and there is no “reset”, nor “choose a different setup” button
2. It also overwrites the keyboard settings for all other users of the computer, whether they want it or not
3. It doesn’t even require admin privs – the user is just allowed to overwrite everyone else’s profile without question (including admin profiles)

Sigh. Yet again, a classic example: Apple has no idea how to make software. Someone at Apple *deliberately* broke this (they chose to remove the setup button after it was first setup), so that not only is the software buggy, but there’s nothing the user can do to fix it :(.

EDIT: googling suggests that the two ways to fix this are 1) mess with a lot of low-level config that will probably lock you out of your keyboard entirely, or 2) format your Mac and re-install from scratch. Hilarious. I’ll try the local Apple store, but from previous experience their “Genius” staff don’t seem to know much beyond the obvious, so I’m not hopeful. Sigh.

Scroll down for detailed reviews/explanations of each venue. If you know of others, feel free to add a comment at the bottom of this page. Please use the same format (One line with: name/type/secure/rating/sockets, then your own text review). I’ll try to visit them myself, and add my own reviews to the main article when I get time.

Venues

Note: each place is listed with 5 pieces of info:
– Name of venue
– Type of venue (restaurant, pub, cafe, etc)
– Is their Wifi encrypted? (see this article for why/how this matters)
– Rating (1 = terrible, 5 = perfect)
– Tables that have Electric sockets (either single or double), for recharging your laptop – how many in the venue?

• The Florist – Pub – Secure 5/5 (E:3)
• The Eddy – Pub – Secure 4/5 (E: 2)
• Regency Tavern – Pub – Secure 4/5 (E: 1)
• Earth and Stars – Pub – Insecure 4/5 (E: 2)
• Hampton Arms – Pub – Insecure 3/5 (E:?)
• Windmill – Pub – secure – 3/5 (E: some)
• The Tin Drum – Restaurant/Pub – Insecure 3/5 (E: 2)
• The Duke of Wellington – Pub – Secure 3/5 (E: 3)
• Prince of Wales – Pub – Insecure 2/5 (E:?)
• Cafe 37 – Internet Cafe – Secure 1/5 (E: 0)
• Cafe Nero – Cafe – Secure 1/5 (E: ?)
• Cafe Nia – Restaurant – Secure 1/5 (E: 1 – pay!)

And, the odd one out:

• Taylor St Barista – Cafe – None 0/5 (E:0)

(read on for details + map links)

Reviews

The Florist – Pub – Secure 5/5 (E:3)

(map)
Decent pub, with a strong internet connection.

I give this one 5 because it has everything you need internet-wise, it’s close to the city centre, and it’s usually empty of people. That last bit may be a good thing or a bad thing depending upon what you’re looking for, but generally if you want to sit down and use wifi, quiet is a great thing (not so good for the landlord).

The Eddy – Pub – Secure 4/5 (E:2)

(map)

A popular but dark pub, often full of people (especially the 25-40 crowd). The layout makes it relatively quiet, even when standing-room only (unless there’s a football match on the big-screen TV).

Internet connection is usually solid and fast – but sometimes slow (I think some of the people living in flats in the streets around the corner use the wifi too).

Regency Tavern – Pub – Secure 4/5 (E: 1)

(map)

Quiet pub currently well-known for doing an especially good Sunday Roast lunch (although, this being Brighton, chefs change employers often, so that might go out of date suddenly). Allegedly the oldest pub in Brighton

The only power socket in the venue is beside the entrance to the Ladies toilets, a big 4-way extension cable above the cushioned wall.

Earth and Stars – Pub – Insecure 4/5 (E: 2)

(map)

Brighton’s only “organic” and “eco-friendly” pub, with solar panels on the roof, and a range of organic alcoholic drinks on offer. Has an upstairs room, where the wifi signal is weak, but you wouldn’t need to go up there unless the pub is completely crammed full. Landlord is usually happy for people to come in and use wifi even if they’re not eating/drinking (if the pub is relatively empty). Gets very busy on winter evenings, and thurs-sunday evenings all year round, but quiet most of the rest of the time.

Hampton Arms – Pub – Insecure 3/5 (E:?)

Like the Windmill, has an outdoors seating area in sunshine with wifi; unlike the Windmill, the signal here is strong. I tried in the middle of the day, and the wifi strength was fine, but the internet connection crashed every 30 seconds, losing all connections. The barstaff switched the router off and on again a couple of times, and eventually the internet connection held. Maybe just an ISP problem? Need to check again soon.

Windmill – Pub – secure – 3/5 (E: some)

You can sit outside, in the Sun, BUT … very Poor wifi, signal strength bounces between 20% and 100%, and even when wifi is up, the internet connection keeps giving up on websites like yahoo.com and google.com, randomly running very fast or very slow. Indoors, it worked a lot better, even though signal strength was almost the same. Amusingly, perhaps too optimistically, the wifi network is named “THE WINDMILL SUPER FAST FIBER OPTIC”.

The Tin Drum – Restaurant/Pub – Secure 3/5 (E:2)

(map)

3/5 – recently, the internet has been “on the blink” and on some days you can’t get a connection at all. Check you can get on before sitting down for food and drink

Great food, popular venue, very noisy coffee machine. Place gets noisy throughout the day: at lunch, mid-afternoon, and evenings.

The Duke of Wellington – Pub – Secure 3/5 (E:3)

(map)

3/5 – whenever someone pays using a debit or credit card, the PIN card reader causes the internet connection to crash. On a busy day, this can mean long periods with no internet. Also, during quiz night, they turn it off completely.

Just around the corner from the Eddie, very recently completely renovated (it used to be horrible dive of a pub, now it’s clean and modern). Very bright inside, free Pool table in the afternoons.

Prince of Wales – Pub – Insecure 2/5 (E:?)

(no map, address reported by google maps is impossible)

2/5 – unpleasant venue, wifi works fine.

A very small pub right on the western side of Churchill Square (the main shopping precinct + shopping area). Unfortunately, as it’s a tiny pub, and they’re in one of the busiest places in Brighton, it seems they’ve had to get aggressive with people wandering in from the street – you have to buy a minimum amount of food/drink and ask for the wifi details each time. It’s also dimmer, dingier, and smellier than most pubs in the city. Not a great atmosphere.

Cafe 37 – Internet Cafe – 1/5 (E:0)

(map)

2/5 – the only “internet cafe” if your own computer breaks. But …they may charge you extra (approx £1.00) for using the wifi

Standard english cafe, serving very cheap coffees + full english breakfast. Also doubles as an internet cafe, with 3 PC’s in the corner, charging £1.00 per hour for use.

In the early days, you could bring your own laptop, and so long as you were eating + drinking they didn’t mind you using the wifi for free. 6 months later, they started also charging for the wifi, which seemed over the top.

If the wifi were free, this would probably be the cheapest place in the whole of Brighton to get free wifi (in terms of cheap food/drink).

Cafe Nero – Cafe – Secure 1/5 (E: n/a)

(multiple around the city)

1/5 – Each coffee you purchase entitles you to an extra 15 minutes of free Wifi. This is pretty extreme, and not recommended, given how many places do free wifi with no strings attached.

You have to keep re-logging in with a 10-digit secret code (that changes each time you purchase something) each time your time runs out. A lot of hassle.

Only reason to use this place is that it usually has the best coffee of the places that have any wifi at all.

Cafe Nia – Restaurant – Secure 1/5 (E:1 – pay!)

(map)

1/5 – They now *charge* by the hour for electricity if you bring a laptop. This is irrespective of whether you eat or drink anything (I had a full meal, two coffees, and still got charged separately for “using our electricity”)

One of the few places you can get both Wifi and Chai Latte. Good food, a bit pricey, but … the internet crashes frequently, and has done so constantly for the past few years. There is a “magic table” near the front, between the two doors, where you are less likely to get pushed off the internet connection. But, now that they charge for electricity, would you really bother?

Taylor St Barista – Cafe – None 0/5 (E:0)

(map)

No wifi.

But the very best coffee in Brighton, bar none.

If these guys ever get wifi, I wouldn’t bother going anywhere else for coffee, drinks, and meetings.

Summary

A lot of nice little anecdotes about MMO middleware experiences – but it would have been nicer to have had a lot more detail on them (not easy to do in panel format).

The panellists briefly touched upon several few sacred cows – such as “Is it absolutely essential to have the full source code to exernally-licensed middleware?” (which I’m glad to hear being questioned – with server middleware especially the developers are hopelessly underqualified to understand huge chunks of source, even at a debugging level (e.g. the many cases where the devs simply cant practically debug distributed-system source)). There are good reasons for having source (and the other sacred cows) but more questioning and more critical thinking from developers about what they *really* need and especially what they’re going to *do* with it would be good for everyone, I think.

Any mistakes understandings and misattributions etc my own fault. My personal comments in [square brackets]

Talk

If you were going to make a new engine/MMO today, what middleware would you use?

PJ: we focussed on web based games when we started. We decided to use Unity, no royalty sharing made it a very attractive pricing model.

CD: hypothetically … I’d make substantial use of middleware, although we didn’t really at Turbine because it wasn’t there when we started.

RL: we look at what’s going to be unique about the game design we’re starting, and then look for middleware that matches well with the uniqueness of the individual game, whatever gets us a good chunk of the way towards that.

MF: we’ve decided each time to just do it ourselves.

How do you deal with the risk of using middleware code that’s never been used in a shipped MMO, that you’re the first major user of / the other major users haven’t shipped yet?

PJ: make sure you have the source code (although we couldn’t afford the more-expensive source-code license for Unity), make sure you have good support and responsiveness from vendor Tech Support.

CD: make sure the vendor is committed to the vertical business space, that your success is part of their core product strategy

JL: being first shipping user can get you a big price discount from the vendor

RL: with BigWorld we don’t have all the source. You have a limited number of engineers who’ll be skilled enough to do anything meaningful with that source anyway, and they could quit, leaving you in trouble. We have a commitment that if something exceptional comes up then we have the *possibility* of the vendor giving us source access temporarily, with their engineers physically present at all times, but we hope not to need that.

MF: any middleware that includes game design / game-logic elements is highest risk. I’ve had low-risk pieces of middleware fail, but that’s quite easy to workaround because it’s got relatively little effect on the overall project. I don’t want to name vendors in particular examples, because it’s not necesssarily their fault.

JL: we used SpeedTree for a couple of years, but eventually didn’t ship with it, because it couldn’t integrate with our art pipeline. Not a bug in that, just unfortunate accident of how it works and how we worked.

CD: we had the same problem with ST.

CD: the two scariest middlewares we used at Turbine were having Physics, and Kynapse for path-planning. Do we de-stabilise our live servers by integrating this cutting edge middleware new version in? As an MMO operator you really want to be able to keep an old version in situ for a long time – maybe a lot longer than the middleware vendor actually wants you to have it there.

[ADAM: that's an extremely interesting problem...]

We had a big problem with a middleware engine where the provider just went out of business. We got some support from our publisher there to work through it, but what do you do about that?

CD: source code escrow is a good thing, for the absolute disaster scenario. But it’s really no different from other catastrophic business risks. Make sure you have one (more than one) person who’s been over the source in detail with the vendor at least once, so understands it well enough to pick up if they vanish, etc.

RL: escrow is mandatory, including all the required libraries, correct verisons etc.

[ADAM: nb: same speaker who doesnt have full source for some aspects of BigWorld]

JL: we didn’t escrow Alchemy, too naive to realise we should, but we got someone to send it to us as they were going out of business. Just getting it to build was a huge problem, we found.

[ADAM: our industry is still cursed by the fact that almost no-one has a great build system in place :) ]

MF: with Renderware we had source for the main thing, but there were dev tools that we had no source for, and so we couldn’t fix bugs in the tools, it was a nightmare. Very fine Modularity of your middleware – which doesn’t really exist yet – would be a real saver in this area. A hyper-modular middlware, I think that would make the risk much lower. Then if the company did go out of business, you could replace those building-blocks one at a time. I think that modularity is very important, or as close as you can get.

[ADAM: ha. I fought hard at the tech level to keep exactly that as one of the top 3 features of our server middleware back in 2002, but unfortunately back then (my naivety in assuming developers understood more than they did) no game developer had anywhere near enough understanding of the problems to realise how valuable that was. Ultimately, it was a waste of money for us as a company, because no customer understood it enough to place the value on it that would pay for how much it was costing us to achieve and maintain it. I'm not convinced that's changed enough to make it profitable enough for MMO providers - even if developers now see the theoretical value, I don't think they are able to really afford it]

Audience: Given how heavy the sales process is … is there a role for open source (OSS) in this?

MF: I think OSS is actually necessary for our industry to remain profitable, to reduce the cost of development. If I say bad things about middleware it’s not because I want it to fail (I really want it to succeed, I think it’s critical for us), but because I’m so disappointed with the current state of it.

RL: how many games have Python in them? Well, that’s an example of OSS that’s now heavily in use. We have an enormous amount of OSS we use inside the backend systems / servers.

What are the packages are you using for service side?

CD: we use a 3rd party billing system, that I wouldn’t actually recommend. We chose it many years ago, and it isn’t really right for what we do now.

(audience): there’s big advnatages to using external billing provider, e.g. benefit of having someone else as the merchant of record, isn’t there.

[ADAM: personally, I think that's short-sighted, because in many ways the last thing in the world you want to have happen, as a service provider, is be disengaged from the customer's wallet. Never let someone else come between "you" and "your customer's money"; they can handle it, facilitate it, but not take over the CRM]

What about internal middleware? A lot of companies form Core Technology teams, creating a culture of making MW for internal use, re-use in other titles, etc

CD: D&D and LoTRO have the exact same codebase on the platform level because of our internal middleware. We ported the LoTRO DirectX 10 layer as a full working initial Dx10 directly into D&D, without changing it.

RL: the question comes down to whether you have a very specific target for what you’re doing. If you try to make one engine for all your games, and/or all your studios (if you’re multi-studio) then that’s a route to huge difficulties and disasters. At the same time I’ve seen companies that have huge in-house expertise in their internal wide-use engine and are able to churn out new games easily. But I think it’s a really really difficult problem to solve that way.

CD: we can have a full game prototype from zero to running in 3 months because we have all this internal re-usable code. For a relatively low investment we can build a small portion of a world to try out new game concepts.

[ADAM: I'm guessing the caveats there are simple that the content-cretion costs are of course still directly proportional to how much you want to show - hence "build a SMALL portion" etc]

MF: It was painful sometimes when the company was going down that they weren’t helping on the games themselves directly, but when Perpetual folded, the core tech group, the reusable code, was the one part that they managed to sell off. Clearly it has a lot of value, even if it’s painful getting there.

PJ: I think it makes sense to have one team on gameplay and building things, and one focussed on technology, because it’s different expertise for the staff to have.

What about MMO Middleware on consoles? Will midleware bring MMOs to consoles finally?

CD: it’s absolutely helped the transition, you’ll see stuff very soon where it’s facilitated console MMOs.

RL: stuff that’s provided to you multi-platform is a huge win from using middleware.

Questions from Audience

Q: sounds from what you’ve said here today like the state of the middleware industry today is fragile and embryonic?

MF: I agree. I think it’s a difficult industry. Business model is tough, it’s sales heavy, hard to get your cash in when people go under -

JL: – that’s what happened with Intrinsic, they had agreements with lots of companies paying a small amount each month towards the license fee, and when a few of those went out of business, the sudden cashflow damage killed Intrinsic

Q: what are your experiences with the multi-platform engines?

RJ: we tried most/all of the flash stuff. We’ve found Unity to provide the most-productive pipeline, fast and easy to prototype with, works nicely with our dev methodology (scrum).

RL: the more MMO development you do, the more you realise it’s all about the tools. The content-creation tools etc become more important, ultimately, than the engine itself.

CD: the cost of developing an MMO is never the technology, four or five times that cost is the cost of the content-generation. So the more efficient your tools are, the quicker you can get final game-content being made, that’s key.

Q: (from MF) does middleware “help” or “hurt” innovation?

MF: I feel it could hurt quite a lot. e.g. Some years ago, I could not find any middleware that would support instances, so we couldn’t use it.

CD: agreed. I think something monolothic like BigWorld could be particularly limiting, but the piecemeal approach of lots of little parts, increased productivity, actually I think helps innovation because it helps on your dev-resource costs a lot.

Q: are any of your perspectives on middleware non-technology centric, it’s been very tech-centric so far? I’ve seen a lot of recurring frustration with middleware. What kind of heuristics do you use to establish whether middleware is the right choice?

RL: you should never take a fully-staffed team and then dump in middleware on them. The two should be built hand-in-hand; middleware changes your hiring needs, and vice versa.

CD: Tech is always both an enabler and a limiter.

Q: as a vendor looking-in … we see particular teams (art vs code vs design) a lot more vocal than others, and the decision to use/not-use it gets skewed.

RL: the purchase decision is often lead by the lead client engineer, server stuff gets ignored, which is clearly wrong.

Nobody knows – Apple didn’t bother with release notes for this 1.75GB monster. Allegedly, there aren’t even release notes inside the download itself.

What apps will it break?

Nobody knows – Apple didn’t allow developers to test it before they put it live.

What does Apple have to say about it?

I don’t know – the official developer site has zero coverage other than the download link itself and a short note to say it is “mandatory” and that it might break your phone if wrongly installed.

More to come once I’ve got it working…

Bar one massive bug, IMHO it’s good enough for daily usage right now. Sadly, that bug is a showstoppper. It’s partly due to a big bug that the developers are aware of (so big they hilight it on the download page), but it seems to also be partly due to something else. The overall effect is that ThunderBird 3 Beta simply “doesn’t” download IMAP emails. As far as I can see, this makes it useless for the most common uses of a dedicated email client (but … it *is* a beta).

According to the bug report for the bug the developers are aware of, the inability to download IMAP email shouldn’t be a problem because they have a new super-secret invisible “automatic background download”. Obviously, this is a stupid feature for any finished app – there’s no way you can have something like that be non-inspectable – but they’re a long way away from a gold launch, and I’m happier to get early access to it as-is rather than wait ages for bells-and-whistles.

However, either Gmail has an extra bug (quite possible – Google broke the IMAP standard when they added IMAP to Gmail), or OS X has an extra bug, or the TB team simply don’t realise quite how bad the situation is – right now, as an IMAP email client, TB beta is practically useless with OS X + Gmail, and having used it for the past 3 weeks or so, I don’t see how you could think of it otherwise. Since it won’t batch download IMAP emails (the bug they hilight), and the alleged new feature of “background download” simply isn’t working (they dont mention this one :( ), you have to manually click every single one of your emails in every single folder in order to get them to download. How long does it take to click on 1,000 emails? Quite a long time…

(and, NB: you have to wait for each one to start downloading before clicking the next, or it appears to cancel the download)

I tried it out on a gmail account that had 5,000 or so emails – randomly a couple (about 10-20) got downloaded. The vast majority did not. I had a look at the traffic going through the network card, and after a few seconds it was idling, so Thunderbird wasn’t even attempting to download anything. If I go offline, all the messages have “this message hasn’t been downloaded” errors. Surprise! Nothing I do has any effect (unsurprising, given the bug report).

I wonder if it’s partly got by unnoticed because the developers already had most of their mail cached (successive installs of TB re-use the cache of downloaded IMAP email), and didn’t get to see what happens with a fresh install of this “new” mail client? Maybe the bug only shows up when you have more than, say, 10 new emails to download? (when you first install, any Gmail account is likely to have many thousands to download)

Sadly, since there’s not even a progress UI to tell you what’s happening, I can’t really tell what’s going wrong.

But *apart* from that it’s looking really good, and – assuming that IMHO critical bug doesn’t affect you (I suspect it doesn’t happen on Windows, perhaps?) – I think a lot of people will be doing a mail-client switch soon. And after a few weeks of usage it already seems to work much, much better on OS X than Apple’s own Mail application.

If you don’t need offline IMAP, give it a try: Downloads page for Thunderbird beta

One final note to OS X users: the beta still has at least one nasty bug from the alpha, so be warned: it interacts very badly with OS X’s window manager, and after about 10 minutes of running it suddenly takes about 50% of your CPU usage in OS X windowing libraries. So you probably dont want to leave the app running in the background permanently. Although your CPU has spare processor time, TB has basically taken exclusive control of the windowing library (perhaps they’re acquiring locks and failing to drop them?).

Once that happens, the whole OS grinds to a halt: you’ll find it very hard in *any* application to click on buttons, dialogues will take ten seconds to appear (instead of 0.01 seconds), etc. Kill TB, and your system goes back to normal. Check your CPU usage, and it’s being gobbled by the OS X Window Server, or related OS X internals. I am sure this is an OS-X only issue, since it seems to be the OS X libs themselves that are going AWOL.

• Any meeting, event, or party at an international conference; many attendees will be flying in a day or so beforehand
• Conference calls with people in more than one timezone; conference calls tend to be with multi-timezone, otherwise you’d just have a face-to-face meeting

I’ve tried lots of things, and although Outlook 2003 and beyond have some moderately handy tools for this (you can view “2 timezones simultaneously”, although IIRC *only* 2, sigh), 99% of the time I need something web-based that anyone/everyone can use. And today I found this piece of awesomeness:

http://www.timeanddate.com/worldclock/advmeeting.html

…which produces things like this:

(I would insert an image here, but unfortunately – despite the claims of the Debian maintainer of Wordpress – Debian Wordpress still can’t do file uploads, and Wordpress’s authors haven’t fixed their extremely poor approach to file uploads. Sigh)

BUT … it completely breaks online/offline operation, and refuses to download ANY emails unless you manually download EACH AND EVERY email by clicking on it.

This makes it, well, entirely useless for anyone using it as an offline mail client (which, I’m assuming, is the main use case for most people!)

So, don’t bother downloading it – stick with Shredder 2 / Thunderbird 3 alpha 2. It’s buggy, but at least it does download email when you tell it to! (usually).

NB: in case anyone wonders why I’m using an alpha/pre-alpha in production, here’s why: On OS X (and, yes, this is partly a reflection that OS X is a second-citizen when it comes to TB releases), Thunderbird 2 is so utterly useless that the 3.0 pre-alpha works better *across the board*. I was rather depressed to discover that TB 2, gold version, is missing many of the core features of Mozilla Mail, which TB 1 was based upon. TB 3 at least manages to bring back some of the basic functionality (although not all, not yet – I have fingers crossed!). In particular: where TB 2 cannot cope with Gmail (apparently because Gmail willfully ignores bits of the Internet Standards; sigh – BAD google!), for TB 3 the Mozilla devs hacked around Gmail’s bugs to make TB 3 work with Gmail *mostly* correctly.

NB2: Mozilla Mail was the best graphical mail client I’ve ever used, even today some years after it got replaced by Thunderbird. There is (was) some great usability stuff in there. So, much as it is frustrating to be stuck with an alpha build right now, it’s still pretty good, and I’ve got high hopes for the final release of TB 3!

VWFE 2008 (Virtual Worlds Forum Europe) got cancelled because the venue was taken away by the Police, so the organizers arranged an emergency Unconference for today instead. I decided – with only five minutes of prep (this is an Unconference…) – to do a session on “ways to monetize virtual worlds”.

I ran this session, so I took very few notes, sorry.

So … if you weren’t at the session, you probably missed most of the good bits :P.

Background

I’ve been in MMOs in some form or other for ten years, and I’ve seen VWs arise as a viable product, but the companies I meet seem to often have serious lack of understanding / plans of how they are going to make anything more than trivial revenues. Each year, the average technology of new companies (both presentation/graphics, and back-end/servers) seems to improve, but each year the monetization plans of new companies seem to get poorer, on average.

This is quite disappointing, so I wanted to do a session on ways to monetize VWs, and show that there’s no excuse to merely say “advertising”, or to stick to one or two plain derivative revenue models.

Reference points

How much revenue should we be talking about here?

2008 FaceBook – $300m
2007 FaceBook – $150m

2008 MySpace – $800m
2007 MySpace – $550m

So … if your monetization method tops out at a million dollars a year, forget it.

How can you monetize a VW or MMO?

Here I opened it up: in the few minutes before the presentation, I had thought up 9 different unique methods of monetization; how many could the audience come up with?

My ones (shared afterwards)

• virtual item sales
• dual-currency systems (currency-sales)
• physical merchandise
• lead-generation advertising / research
• sponsoring
• marketplace ownership
• subscription
• event fees
• consumable / rentable resource fees (LAND, also INGREDIENTS)

Audience ones:

• subcriptions
• virtual item sales
• ad-supported
• sponsorship
• merchandise
• event fees
• physical goods sales inside virtual worlds
• currency exchange fees
• be a deposits bank
• licensing in-world businesses
• tax

…I liked the “licensing” and “become a deposit bank” in particular :).

We then ran through them all and shared what our estimations of the total amount of money people across the industry were making using that particular monetization model. I can’t remember exactly what we said, and it was heavily influenced by me (because I’m quite up to date on figures for global public compnies for this – its something I’ve researched a lot and tracked for the past few years)

Something in the region of $5billion for subs, $3billion for virtual item sales, a guesstimated $1b or less for advertising (really really unsure of this one – I was guessing blind basing it mostly off CPM figures for ad-driven games – I hoped someone else present would know a real figure for this), a few billion for merchandise.

But … Villi shared a figure he had researched formally for a report last year: $2.1billion for the worldwide revenue from virtual-item sales. I reckoned that is looking like $3billion for 2008 (going off the projected revenues of the Chinese operators and how much they’ve gone up).

An interesting comment…

“Many VW’s don’t think about the problem of getting physical money into the world in the first place – the use of payment providers etc”

Think about that. Think about it until you understand what it means, what it implies for those “many VWs” – and how it relates to the current real-world financial crisis.

Summary

Companies are spoilt for choice when it comes to VW monetization, really.

And there are half a dozen forms of monetization that – straw poll – are already producing billions of dollars revenue across the industry.

And … forget “advertising”: there are better things out there, each of which is as good or better than advertising.

Last time I said that “good Developers are very similar to Valley Technology Startups”, which suggests one obvious way things could develop:

Publishers to become Venture Capitalists; Developers to become commodities

In this model, the Publishers spend their time “speculating” in Dev studios: instead of buying a studio in order to own the output of that studio, you buy it with the intent of later selling the studio itself at a profit.

This is a good model; as I first heard from Jack Lang, at one of the Cambridge Enterprise Conferences many years ago:

“The best way to get rich is by buying and selling things. Preferably companies”

IIRC it’s a quote that’s been around for a long time, but I can’t remember where the original quote came from, unfortunately.

(of course, this observation is why Publishers are box-shifters in the first place: simply buying and selling things is an easy, fast, stable, sustainable way of making a very large amount of money. It’s not particularly creative, perhaps, but it’s a very efficient way of generating profit, and lets you leverage your resources/cashflow way above the profit you would generate simply from manufacturing your own goods and selling them)

The real beauty of this is that – as Jack’s quote illustrates – a Publisher and a VC have very similar fundamental business models: they buy stuff that they have no intention of using themselves primarily to sell those things at a profit to other people. In both cases, the less time the company can hold onto the products, while getting the same price differential between purchase and sale, the better. What is a VC? A VC is just a higher-value version of a box-shifter. So, for a Publisher to diversify into being a VC may not be so difficult…

As someone who’s been through the mill of raising finance for startups before, I’d also like to add that “funding game development” is commonly thought of as having no equal in risk and unpredictability – save “providing venture capital for a new startup”. The hardest part of being a VC – the insanely high risks involved – is bread and butter for Publishers, who routinely spend tens of millions of dollars on stuff they don’t understand, cannot effectively control, and cannot reliably predict!

Studios would find that:

1. Publishers would look after them more – you don’t want to harm something you’re planning to sell
2. Funding and marketing decisions would be driven more by what was in the interests of the studio rather than the Publisher’s marketing dept or cashflow
3. Publishers would stop being stupid about trying to “reduce costs” of development purely for the sake of it
4. Publishers would be a LOT more interested in supporting and creating external partnerships for the studio, especially where those partnerships involved competing publishers or their subsidiaries (because that would make it easier in the future to sell the studio to that competing publisher), which would help reduce development costs (a little) and increase productivity / quality of working environment (probably a lot more – publishers usually consider this too little justification for allowing such things)
5. If the publisher got cold feet about publishing a certain game and it was far advanced, they would push for POSTPONING it rather than RUSHING it – they’d rather sell the studio BEFORE it publishes the game, and “price-in” the potential of the game than sell AFTER the game had launched and flopped
6. The publisher would push harder for maintaining quality standards of the games output by the studio – they have literally invested in the brand of the studio, a brand they are planning to build up in value as much as possible, before selling

Publishers would find that:

1. Development costs would no longer mar their balance sheets and make their financial performance look bad; the offset of being able to mark the studio as a sellable asset with a quantifiable value in excess of the money being poured into it would make it all smell sweeter to shareholders
2. There would be less friction with studios, leading to better communication, less frustration, and probably better overall quality of product – and hence, more profits
3. Studios could safely be given more leeway to make strategic business decisions that were “right for them”, offering the possibility of mega-wins for the publisher whenever those paid-off (e.g. the decision by early FPS developers to not only allow but encourage modding was enough to terrify publishers even today, and yet a massive win in sales and profits), but also to not have to take responsibility – and blame – when they failed; this would all be priced into the “value” of the studio as a separate, tradeable, entity
4. If a studio made some bad strategic decisions that led to commercial failures, that might actually INCREASE its tradeable value, if the market perceived that the studio had “learnt” significantly from the mistakes; potentially, such increased value could completely offset the actual financial losses incurred from the mistake

A practical example

I’m just pulling this out of thin air, trying to think of a studio that many years ago was worth something, got acquired, went internal, and now is probably worthless. When EA bought Westwood Studios, one of the things they paid for was the brand; how much value did they really extract from that brand? How much value does it have *today*? Today, it’s probably next to none – customers don’t care, and other games industry companies all know that the real meat of Westwood Studios (the staff, the equipment, the processes) was disbanded shortly after being bought by EA. Could they have made more money by promoting and protecting the WS as an owned-but-independent studio? If they’d taken that route, and even if they had made less money than they have with the route they chose, would it be more than made up for by the fact that they might be able to sell WS right now for, say, a couple of hundred million dollars – if only it still existed as more than a name?

And why not?

But this isn’t the way Publishers work right now, so … where does this plan all go wrong? Why hasn’t it happened already? What might prevent people from trying this?

1. Cojones

At the moment, the funding decisions that Publishers make are so distantly removed from the actual point of capitalizing on them that it’s quite easy for the people making the funding decisions to blame many other departments and personnel within their organization should the investment go poorly. Indeed, this plausibly deniability, this easy abrogation of responsibility by the decision makers – and the great distance between them and the people actually implementing the game – are root causes of a lot of the practical problems in the Publisher/Developer relationship whenever they do “external” publishing (i.e. publish a game made by an independent / external studio, as opposed to a wholly-owned internal studio).

A lot of the benefits for the New Way cited above stem from removing that indirection; that means a bunch of people making hundred-million-dollar decisions would be exposed to rather more scrutiny and responsibility than they hold right now. I’ve heard people (usually the ones who don’t really understand VC’s, have acted naively or foolishly with them, and come away poor and bitter) describe VC’s as “arrogant”, “bullies”, and “too demanding”; while I don’t agree with that, just think how you’d act if you were the named individual responsible for a handful of $10million investment decisions, and how that might come across sometimes. Could the individual people working for Publishers accomodate such a change? If they were content with that level of personal exposure to risk, would they be working in the games industry, or would they already be working in the higher-paid VC industry?

2. The Art of the Sale

Another issue is that the success of selling a studio depends on, well … your ability to sell!

Publishers do not, generally, have any experience of selling companies. A publisher might spin-out or sell off one division every decade, at most – and many of those are instigated by the division itself (management buy-outs), or are fire-sales (find a buyer at any cost, no matter how low). They don’t have staff who are experience in doing this, they don’t have any contacts suitable for doing it, and they don’t (generally) maintain the level of immersion in the marketplace of studio buyers to be able to setup great deals when selling on a studio. Look at how much time the individual staff at VC’s spend purely “networking”, both looking for things to invest in (new purchases), but also looking for, befriending, understanding, and keeping up to date with the needs and desires of potential buyers (people who might acquire some of their portfolio).

3. Organizational Change

Lastly, have a look at the typical VC organization – a handful of Partners (maybe half a dozen people who make investment decisions), a handful of Entrepreneurs in Residence (EIR – maybe one or two domain-experts who make recommendations and help in due diligence). This is enough to manage billions of dollars of investments.

Now look at the typical Publisher organization – 50 people in each of marketing, customer service, and sales, perhaps 15 handling external develoment (finding and making development/publishing deals), and another 10-50 people in internal support roles. This is enough to own 1-3 development studios.

This isn’t to say that Publishers would need to downsize. Rather, it’s to point out that if the external development side started doing sales of studios for as much as $100million a time, their revenues and profits would suddenly massively eclipse (20 to 1, perhaps even 200 to 1) the whole of the rest of the organization, despite being outnumbered more than ten to one. Sooner or later, the “rest of the organization” would become politically weak and subservient to the massively profitable “trading in ownership of development studios part”.

The VPs of the current departments may well find that the total pie they’re sharing in becomes much bigger, and much more than makes-up-for the fact that their slice has got smaller, but will they accept their slice going from being a “Vice President” sized slice to a “Operations Manager” sized slice?

A few little Notes…

1. When I wrote Publishers are from Mars, Developers are from Venus, I had *no idea* that NCsoft had just decided to shutdown its European development studio, and make a swathe of redundancies in European publishing. Sheer coincidence, and sad for a lot of people involved, but very interesting nonetheless.

NB: if you work for a publisher or developer and are interested in picking up any of the good NCsoft Europe staff, especially development, QA, localization, and customer support – and you have jobs in or within commuting distance of Brighton – let me know. Lots of people are suddenly looking for stuff to do next…

2. I said that “Developers exist to make a loss, every day”, and some people questioned that.

Yes, I really mean this: the more they spend, the greater the potential profit, and they should be maximizing their potential profit. Obviously, there is a point of diminishing returns, but generally speaking whenever you have an R&D lab, you want to pump as much money into it as you can possibly spare. Generally, R&D labs are rather good at soaking up almost infinite amounts of money.

Compare the revenues and the expenditures of, say, GTA IV with those of Bookworm Adventures. The latter may have been much much more profitable in percentage terms, but the former made a bigger amount of money overall. Often, the sheer amount of money you make is more important than your profit percentage.

3. I decided to write these blog posts after a comment I made to Steven Davis about the problems of publishers owning development studios, which he replied to with “Actually, the publishers should fund these things like a movie studio or VC. Let them be independent, get them off the books, and use your money to control distribution or via publishing rights.”

I’d been thinking along similar lines, but I also realised I saw some big problems with the approach, so I thought it would be interesting to explore in more detail. But if he hadn’t made the comment, I probably wouldn’t have got around to it :).

In my experience relatively few people in the games industry realise this, but all these things are cyclical (it’s a lot more obvious in non-niche industries, like the IT industry, where you have many more companies, and the billion-dollar companies can’t be counted on one hand). So, what’s next? What’s going to happen over the next 3-5 years?

Some (recent) history

My last job was working for a large publisher (NCsoft – http://ncsoft.com) where we were setting up a new internal development studio from scratch. When I arrived I there was only one other person (plus my manager). We were doing a lot of other things at the same time – external development, pitching new internal projects, etc – but over the course of the first year I spent a lot of time looking at what we had to do to get a studio up and running, starting from scratch.

Interesting and fun. But also … surprisingly difficult. I’ve been one of the first employees at a couple of startups, and founded some, so I’m accustomed to starting up teams and departments, and a lot of the problems we encountered with this studio were just variations on familiar themes. But then there were also some new ones, side-effects of being inside a huge, well-established, publisher – one whose head-office was on the other side of the world, where the vast majority of the staff didn’t share any languages with the vast majority of the publishing office in our country, and our staff.

To summarize: the things that should have been completed fast were incredibly slow, and the things that should have been easy often turned out to be extremely hard. My definition of “should have” here is based on “whatever plays to the strengths of large corporates”.

As that became clear, one option would have been to throw up our hands and say: “this company is crap! No other similar company works this way!”. Instead, I dug deeper, and tried to understand how it was that we seemed to be seeing a lot of the opposite of what I expected. Sure, a lot of it could be explained by some over the top internal politics, and some by issues with individuals, but … this is a billion-dollar public company, and it’s foolish to think that management could be so weak and disorganized that a few internal battles and a few individuals could cause major aims of the organization to fail. No, there were underlying problems that were natural side-effects of the way the company worked. IMHO, these same issues are almost certainly causing problems for other internal development studios already, and will probably be major contributory causes of the move away from internal studios (when that day comes).

Publishers exist to make profit, every day; Developers exist to make a loss, every day

I could stop there. In that one sentence is encapsulated a problem so powerful and subtle that it’s more than capable of causing all the secondary problems – the ones people actually notice – that lead to publisher/developer acrimony when the two are together in the same company.

A traditional publisher is a box-shifter that pays a hefty license fee for exclusive rights to import a popular, trendy product from a foreign country. The things they need to be good at are:

• Identifying the Next Big Thing, and signing an exclusive deal before anyone else gets it
• Efficiently importing that thing and distributing it out to mass-market consumers (this is where most of the opportunity for profit exists)
• Persuade as many people as possible to buy the product, as quickly as possible, for as high a price as possible (this is where ALL their revenue comes from)

Why did I mark the SECOND point as the point for profit? Because profit is extracted through the differential between the costs generated in that bullet point, and the price point that the publisher – arbitrarily – places on the product as sold to retailers (who then, typically in retail (forget the games industry – this is normal for all industries!) double the price again before selling to consumers).

The price point can be … anything you want. The volume you sell comes from the third bullet – but you have NO control over how much you sell. You *try* to sell as much as you can, but you cannot wake up tomorrow and *decide* to double sales. However … in contrast, you can wake up tomorrow and *decide* to halve costs. Or double them. So you focus on that middle bullet point: Efficiency (while making sure you assign a healthy slab of money to a sales + marketing department, and set them “targets” to try and meet).

A traditional developer is an R&D (research and development) laboratory. They try to be as scientific as possible, whilst spending every day working with masses of unknowns (and several unknowables – what is “fun” anyway?). After working for an indefinite period of time (no way of telling how long it will take) they’re trying to create (or discover) something that has never been created before, and which satisfies various criteria – many of which cannot be measured until after the project is complete.

They absorb money like a dry sponge in a puddle, with very little to show for it. The things they need to be good at are:

• Securing as large a pile of resources as they can, and spending it to the fullest
• Trying crazy stuff that they can’t explain, and waiting to see what will happen
• Sticking as close to the cutting edge as possible, and always investing in long-term improvements

Why do they have to secure a large pile of resources?

Because their success is limited only by two things: their resource, and their skill. That translates into three concrete things:

• How good is their equipment? (”equipment” means EVERY TOOL they use to do their work – including lots of indirect things that you may not think of as “tools”)
• How much reagants and raw materials do they have? (everything consumable … including “time” … that could contribute towards doing MORE experiments)
• How good are their staff?

Those three things are, in turn, only limited by “money” and “the quality of the people they hire”.

Publishers hate this. No, that’s not strong enough; Publishers REVILE, DESPISE, RESENT and LOATHE Developers for always, ever, and only going after those two things. And … they don’t understand it.

Frankly, as a box-shifter, with “efficiency” your only concern, WHO GIVES A F*** HOW “GOOD” YOUR PEOPLE ARE?

But that’s not the worst. No, the worst is this: as a box-shifter, the only thing you can directly control is your costs. Everything in your business, from the structure, to the choice of staff, to the processes, is designed to reduce costs. And what does every R&D laboratory obsessively try to do? Yep – raise costs!

If you ask a Publisher to create, fund, nurture, and partner with a Developer, you are asking the staff to encourage, to aid and abet, the one thing that you are already telling them every day to hunt down and destroy. Capiche? Does anyone see a problem here?

Developers in the Wild: R&D for profit

Well, this is clearly insane – how could a Developer ever make a profit? The answer can be found most easily by looking to the one place in the world where R&D laboratories make more money than anywhere else: Silicon Valley.

In the Valley, the Technology guys have become Entrepreneurs (or found an Entrepreneur to work with), and they’ve gone out there and applied their intelligence to a new problem: “Given this thing I’ve created, which is novel and cool and awesome, how could I use it to drive a product that people would pay for, and which (because of my NEW tech) I can sell cheaper than what is available, or (because of my NEW tech) does something people have been trying to pay for but been unable to find a working solution for?”

Despite appearances otherwise, good Developers are very similar to Valley Technology Startups: it’s all about the monetization, the capitalization – what bridge are you going to build between “what you’ve created” and “someone who has money and a problem”, and HOW are you going to build that bridge?

“Sell the exclusive publishing rights” is one bridge. It can be built many ways.

“Create an infrastructure that lets us deliver this product to the public, and take money from them” is another bridge, with just as many potential schematics.

But then there are others too, many others. Just because those two are the ones that the game-playing public tends to talk about (and are the two that Publishers are most familiar with) doesn’t – by any stretch of the imagination – mean those are the only ones that exist. Ask Blitz (an independent developer) about their Advergames for Burger King (definitely not-a-game-publisher). But, in general, just like in the Valley, the “other” bridges are tricky to invent, and tend to make someone rich just once or twice once invented and done for the first time. There are always new bridges to invent, and if the Technology person’s main role is to invent new tech, the Entrepreneur’s main role is to invent new bridges. So don’t be surprised if you find it hard to think of some.

What happens when a Publisher catches a Developer, puts it in a cage, and ships it back home? Or, more specifically, what do they do to the people that are thinking up innovative new bridges for monetizing the Developer’s assets, and trying to implement them? I’ll give you a clue: if everyone you know believes the world is flat, and has never walked more than a few hundred miles, and one day you meet a person who claims to have walked around the circumference of the planet, would do you do?

Yep. These people tend to be first in the firing line when nerves start to fray and the tensions between Developer and Publisher flare up. They’re an easy target – they make no sense to the Publisher, and their very existence is an affront to their core business model, to their box-shifter mentality (it suggests that the box-shifter is doing a simpler business, something run by simpler, less imaginative, more stupid, people).

UPDATE: I’ve just written a followup looking at one of the possible future directions coming out of this

You need ALL the attached jars, download them into the same directory. Then run jnlpcreator-bin.jar

NB: this is a work-in-progress – the Save and Test buttons don’t do anything yet, but you can get a lot of the functionality just by using the Preview button and copy/pasting out of the window that it pops up.

NB2: for some reason, teh popup window doesn’t popup on OS X. I have no idea why.

NB3: there are still some old references to the original hardcoded paths in the codebase and href at top of the JNLP; you’lll need to manually correct those for now, but next version I’ll replace them with some clever uto-generated stuff.

Files to download (you need ALL of them):

1. jnlpcreator-bin
2. log4j
3. tmachine-basics

– good game/website integration, very smooth
– very basic game design – most games seem to ahve never been playtested, or designed by very novice game-designers. Most simply aren’t fun, and need just a few basic tweaks.
– control systems mostly awful, bouncing and snapping and with unnecessary lag.
– No ability to see your achievements! Achievements rock, but they don’t seem to understand the main point of them – showing them off.
– artwork and sound are extremely polished.

Summary:
– a very highly-polished (graphics, sound) attempt to clone Pogo, but without fully understanding how and why it works so well, and ultimately coming tragically short. Go play Kongregate instead – its much more of a mixed bag, but the overall experience is much, much more satisfying.

(Notes from several weeks ago, accidentally clicked save instead of publish at the time)

Speaker: Shannon Posniewski, Cryptic

I was expecting something shockingly naive and/or stupid from the title of the session. The first thing the speaker said was that the title was completely wrong, so I ran with that. With that out of the way, the talk was fine, although small things kept coming out during the talk that were hard to believe or worrying claims.

So it was going OK, until … right at the end, just before the Q&A, and partly during the Q&A, the speaker dropped some serious shockers:

– they’ve never actually used this custom DB: everything that the speaker says RDBMS’s can’t do he doesn’t yet know for sure that his proprietary DB server can do either
– they never bothered to try alternative SQL servers, they just used MS SQL Server
– they’d never heard of Versant until after they’d developed this proprietary DB – apparently they never did any evaulating of other possible DB engines
– they’ve created their own proprietary query language based on C and C syntax, and don’t find any value in using SQL

Disclaimer: I recently started to work for the company that published CoH/CoV, and apparently forced the speaker into using SQL and an RDBMS in the first place. I didn’t join the company until a long time after this had all happened, and I work in a different division in a different continent, so everything in this talk was new to me (although I admit I’d be fascinated to hear what NCsoft’s thoughts were on these development directions)

My own commentary in [ square brackets ], any mistakes/misunderstandings my own fault :).

Title

The title is deliberately wrong and provocative – it’s really that current vendor solutions are inappropriate for the needs of MMOs

Databases 101

[Adam: spent 15 mins explaining basics of modern database usage:

ACID explanation

Relational explanation

SQL explanation

Keys explanation]

SQL DBMSs are safe, tested, used by practically everyone, lots of experience of using them exists.

Why not use it [SQL + RDBMS]?

SQL is optimized for the general case, and is generally “OK” – it will never be terrible. It will always work..

Sometimes queries take a long time to setup – creating an index automatically, etc.

It self-optimizes on the fly with caching etc, but that introduces stalls at unpredictable times. Unpredictable latency caused by this is very bad because sometimes you e.g. try to pick up something in the game world from within the game client, and nothing happens, so user is confused – because of that latency.

[Adam: if this strikes you as very strange behaviour for an RDBMS, you are not alone. It might help to know that the end of the talk, the speaker revealed that they'd only ever used Microsoft SQL Server and hadn't bothered trying out any alternative RDBMS's, and didn't really know much about the others]

Inserts are extremely slow – just for each single row that you insert. Because of all the overhead of checking for consistency etc, inserts are slow.

Once you’ve got data into the database, the per-row overhead to update anything in the row is also fairly expensive (although you can do several changes per row at once for almost no extra penalty).

Object-Relational Impedance Mismatch

Trying to map OO systems into Relational databases is painful and shoehorning it comes with lots of performance loss.

Every class I’ve ever seen in database architecture talks about normalizing all your data. Doing it that way – the “right way” – can really backfire on you.

[Adam: ... apart from all the other classes in DBA, the ones that also talk about DE-normalizing data. I went to different database classes than the speaker, apparently :) ]

On COH we had this idea that we’d store statistics on each individual player – how many times did they pick stuff up, how many times did they kill people, how long they spent online, the number of times they trash-talked in chat, etc.

[Adam: i.e. gameplay metrics, a la the work done by Bungie with Halo heatmaps, or by various 3rd party companies at the moment in the games industry. Cool stuff]

We made a table where there was one row-insert per player PER piece of data that had changed. Some of those stats change all the time, like “how long have they been online”.

So the traffic to the database was extremely high, because each thing had to be updated constantly.

[Adam: Well ... IME this isn't the *normal* way of dealing with that use-case, so I'm not surprised it bombed. If they'd approached it in a more standard way, using simple multi-server database organization, I suspect it might have worked out fine from the start]

So … we threw out all the tranditional DB concepts, and made one column per stat, with no actual names for columns, instead just have “Stat 1 value”, “stat 2 value”, etc.

[Adam: I really don't get how this ever even seemed like a sensible idea, but there was a long queue of people asking questions at the end so I didn't manage to ask before I had to go to next session]

In the interests of performance, we were twisting our data, either in code, or in the DB, or in both places. This became a huge impedance to our development process.

Solution: Caching

Problems:
– unexpected unpredictable 1-second and multi-second latencies

[Adam: I'm really suspcipious about this problme - where is it coming from?]

Solution:
– created a server process that all DB calls from other server processes were indirected through
– it had a second thread that was busy writing stuff out to the DB
– appeared that all writes were instantaneous, but they were being buffered.
– i.e. write-through cache
– all changes you made in game-server memory would show instantly as having been committed

Problems2:
– throughput was still far too slow, although at least now there was no latency

Solution:
– instead of updating every stat every time it changes, we did “snapshots” and only occasionally save them
– …but means that there was no ACID properties being used any more except for Durability

Since that was shot anyway, we went whole-hog, basically. What we decided was we’d keep all the data for the players on a given map on that game server. The database was being used just as a storage engine instead of as a transaction engine. The business logic is now all inside the game-server, and we just periodically flush it all out to the DB.

Problems3:
– lost data
– server-server comms

Solution:
– flush data faster

Conclusion: we optimized the servers, the database engine, to the point that we lost all the benefits of RDBMSs. All the ACID stuff etc had just become a hindrance to us.

Final solution: Cryptic DB

Requirements
– regain ACID benefits
– low predictable latency
– high read/write throughput
– minimize code-db impedance (the schemas changed very often, and that was very painful)
– backup and restore easily

[Adam: suggests that hiring a professional DBA, and using a commercial OODB, would be the best option to start with]

Relaxations
– offline character access is unimportant
– domain integrity became the responsibility of the programmer
– general ad-hoc queries can be slow (we don’t use them)
– SQL not needed

Background – some stuff they’d already done

In CoX we took all static data (mission data, character class template etc) and didn’t put it into a DB. We stored it on disk and loaded it into memory manually.

To help us do this, we created a “StructParser” [just a serialization tool]. This was what we used to start building Cryptic DB. We wrote a pre-parser that looked at our C structs in the source code and wrote out parserss/serializers for it.

Using this for live data as well as static gave us benefits of not having two different storage systems in parallel.

[Adam: in general, that's a pretty good thing to achieve]

Allowed us to create generic mutators, because we know the datatypes and memory layout. So, now we have a generic mutator database. Only simple operations at this point, like mathematical addition of a single counter.

Complicated things like checking whether to allow a trade (lots of rules and things to check) need business logic. We decided to sepaaret the data and logic away from the database, and put it into our game server.

A particular game server owns the data for an entity. E.g. a team server would know about teams, and anything that has to do with teams will be held on that server. It knows how to modify those entities.

[architecture diagram]

Most transactional actions are now being done locally only, so execute extremely fast.

To make it easier to develop, we created auto-transactions. These instrument and change the code so that transactions are written in C. We did field-level locking on the items we were modifying.

[C code with a bunch of special macros implementing a basic transaction, but mostly written in C]

[Adam: here he gave a simple example of distributed transaction processing (although he didn't call it that), enough to convey the general idea, although he ommitted the details that make DTP hard / interesting technically]

Cryptic DB uses basic journal-based techniques for data consistency longterm.

Conclusion

We’ve tried stored procedures etc, doesn’t work. Other MMO devs have had to resort to using solid-state disks etc to make it work fast enough. So, don’t just believe what they say in database classes.

[Adam: the mention of SSD is disingenuous: only one MMO company has decided to try that, and their from Iceland so they're a bit odd anyway (/me ducks and runs for cover) - but seriously for them it's part of an overall game-design and/or technology strategy - they aimed for and achieved gameplay that Cryptic hasn't (single-server world)]

Q: Did you look at federated data as a solution?

We looked a little, didn’t try to implement it, because we’d worked out a proprietary solution already that we reckoned would work.

Latency would still be a problem anyway with it.

Q: How do you handle corruption of data in your DTP?

There’s certain things we relax – we don’t worry about cosmic rays flipping bits randomly.

Within the database, everything is checksummed.

Q: what’s the throughput of your cross-server transactions? How many of your servers are local versus distributed?

I would say at a guess we have a very small number of distributed TP’s. We have a zoned games, so almost everything is local, probably less than 5%.

I have no idea how much slower cross-server transactions are. I don’t think they’re particuarly slower, the biggest problem is just the overhead of having to wait for the locking of data. Probably a couple of times slower, but not an order of magnitude.

Q: you’re saying that with your proprietary “Cryptic DB” you keep all data in memory. What do you do when you hit the limit of RAM in a machine?

It’s a pretty straightforward task to virtualize your memory. If you’re WoW you’ve got no chance of fitting it all into memory, so you only keep in memory the working set of characters that are online right now.

If you still run out, then we make a second shard. We’re talking tens and hundreds of thousands of simultaneous players before we get to that problem.

Q: Do you cluster machines to make the DB?

It’s one giant machine with a lot of memory in it

[Adam: which would suggest you could benefit from SSD, maybe... :P]

Q: Have you considered co-locating the data cache with the servers themselves?

No. We haven’t needed to consider it yet.

You can compress the data in memory and other tricks, so it’s not a problem yet.

Q: how do you do backups?

A set of three servers work together to make sure they can handle the load and so that they can ensure they are being back up correctly.

These are effectively zipped flat files, for simplicity. There’s one server frontend to rest of games, responding to read requests and making in-memory changes for transactions that come through. The second server is slaved to it, and every once in a while writes out an image of it’s DB in a safe way. The last server picks up the things that have been changed and merges them into the master DB. Master DB is then copied back to first and second servers.

At any one spot in the change the data is consistent within that one machine. There may be a huge journal at some point, but it will be consistent.

It turns out that MMO players care more about not being able to play than about losing ten minutes of game data. By about an order of magnitude.

When someone claims they lost a Sword of Death, don’t worry too much, just give them a new sword, because you save a lot of pain and money in customer support costs.

Q: Sounds like you’ve made an OODB, how does that compare with Objectivity or Versant? And the disadvantage of an OODB seems to be the loss of querying ability, so you can’t research behaviour or do metrics to examine your gameplay etc – how do you sovle that?

When we started working on this we didn’t know that Versant existed. We only found out when we first spoke about it publically.

Other ones we knew about but we already had what we wanted from a previous game that was fully tested by us and used in produciton. We were going to have to worry about learning the vendor specifics, and converting our data and pipeline stuff.

I’m also not convinced that an OODB would have worked for the stuff we’re trying to do. We’re writing a lot of code, and the fact you can write AUTO_TRANSACT at the top of a function, and knowing it will Just Work is worht it’s weight in gold.

Re: adhoc queries, we have created a proprietary query language that looks a lot more like C than it does like SQL.

If for some reason we lack a feature we need in our DB, it’s so easy for the DB to interact with C code that you could just write the damn thing in C and have the DB execute that.

Q: Is this in production?

No

Q: How do you test it?

We eat our own dogfood. The true test will be the next time we have a game that goes live.

We have a zillion stress tests, and run that stuff routinely, of course. Right now we just have test-harnesses that fake it.

Q: Could you have made Cryptic DB from the start, or did you only learn it by using SQL from the start? Did you get anyt benefit from SQL?

We got nothing from SQL, it was forced upon us by our publisher who forced us to use SQL. NCsoft were doing network operations and backups etc, so they said look we already know how to use it, and go ahead with it.

I have no idea if we’d have used it given free chance.

We learnt from it that variant latency is a killer, and learnt where the major bottlenecks were in MMO data writing, so we did learn a lot by working with SQL that was transferable knowledge.

Q: How do you handle verisoning on the data, like schema changes?

The actual DB if it sees a field that it doesn’t know, it just ignores it silently. If we add a field that isn’t there, it silently creates it.

We set things to zero or null to create them.

Q: Do you have a central contorl of the DB design, the schema?

Every programmer can add whatever fields they want. We haven’t run into any problems with it yet.

There’s a certain data structure which is the character definition. At the start of the project someone sat down and designed that, and that doesn’t change often. What does change is the occasional adding of a new data field every now and then [i.e. no additional tables being added]

Q: How much did it cost in money and hours to build?

Whole system from start to finish was two people for maybe a year.

Q: Did you try multiple implementations of your SQL server? Different vendors?

No, we did not. There was no indication to us why different ones would ever be faster, so it didn’t seem worth investigating it. I think it is problems endemic to all databases in general.

unless, maybe, you just spent huge amounts of time and effort on database administration. We don’t know how many players we will have, so we can’t actually define optimal schemas and things in advance.

One of our main goals was to spend more time on game coding and less on database coding.

Q: did you spend a lot of time profiling your SQL server?

I think we tried profiling everything, automatically created indexes, stored procedures, etc. We got maybe 15%, 20% improvement, and DBA’s think that’s great, but we were looking for multiple hundreds of percent improvement in performance.

Q: Does your new DB store quests?

It stores state for how you’re progressing through, but not yet the actual quest template data.

Q: Have you looked at any vertical file DB systems?

No.

It’s entirely possible that it’s much faster than using Relational. We were already used to using SQL so we kept in that direction.

Q: will you be releasing this as a third-party DB?

Maybe – but not at least until we’ve put it into production for ourselves!

Normally, this isn’t a problem, as the default firewall setup I always use prevents any remote logins except from known-good hosts. However, this server was accidentally provided with partially missing firewall code by the hosting company, and so I couldn’t run my firewall without first upgrading the kernel. And I’d been “too busy to get around to” doing that…

Oh well. Fortunately, I had a very recent backup of the blog, and I’ve now discovered a couple of major flaws in WordPress’s backup system (note: it doesn’t bother even trying to backup your uploads, embedded images, etc) that I can now change my backup procedure to accomodate.

Apologies to anyone trying to follow the links in the last 6 hours. You were probably locked-out – I firewalled off the whole machine for a few hours to investigate how they got in and what they did once they were in. You still have to wipe all harddisks and re-install, of course, but you need first to find out how they did it, or you have no way of making sure they don’t get in again.

Oh, and I kept enough incriminating evidence to give to the police / FBI if the hosting providers manage to track down the perpetrators. One of them left his IP address in, but I’m pretty sure that was just another compromised host (have informed the large, famous, north-american telco it came from. Probably one of their naive users with a compromised windows 2k/xp box).

Adam

oh, the drama!

Another First Post from Adam…