Posted on

Debugging odd problems when writing game servers

Cas (@PuppyGames) (of PuppyGames fame – Titan Attacks, Revenge of the Titans, etc) is working on an awesome new MMO RTS … thing.

He had a weird networking problem, and was looking for suggestions on possible causes. I used to do a lot of MMO dev, so we ran through the “typical” problems. This isn’t exhaustive, but as a quick-n-dirty checklist, I figured it could help some other indie gamedevs doing multiplayer/network code.

Scenario

  1. Java, Windows 7
  2. DataInputStream.readFully() from a socket… a local socket at that… taking 4.5 seconds to read the first bytes
  3. and I’ve already read a few bits and bobs out of it

Initial thoughts

First thought: in networking, everything has a realm of durations. The speed of light means that a network connection from New York to London takes a few milliseconds (unavoidably) – everything is measured in ms, unless you’re on a local machine, in which case: the OS internally works in microseconds (almost too small to measure). Since Cas’s problem is measured in SECONDS – but is on localhost – it’s probably something external to the OS, external to the networking itself.

Gut feel would be something like Nagle’s algorithm, although I’m sure everyone’s already configured that appropriately ;).

That, or a routing table that’s changing dynamically – e.g. first packet is triggering a “let me start the dialup connection for you”, causing everything to pause (note that “4 seconds” is in the realm of time it takes for modems to connect; it’s outside the realm of ethernet delays, as already noted)

My general advice for “bizarre server-networking bugs”: start doing the things you know are “wrong” from a performance view, and prove that each makes it worse. If one does not, that de facto that one is somehow not configured as intended.

Common causes / things to check

1. Contention: waht’s the CPU + C libraries doing in background? If system has any significant (but small) load, you could be contended on I/O, CPU mgiht not be interrupting to shunt data from hardware up to software (kernel), up to software (OS user), up to JVM

Classic example: MySQL DB running on a system can block I/O even with small CPU load

2. file-level locking in OS FS. Check using lsof which files your JVM is accessing, and what other processes in the machine may have same files open.

(NB: there’s a bunch of tools like “lsof” which let you see which files are in use by which processes in a unix/linux system. As a network programmer, you should learn them all – they can save a lot of time in development. As a network programmer, you need to be a competent SysAdmin!)

Classic problem: something unexpected has an outstanding read (or a pre-emptive write lock) which is causing silliness. I remember several times having a text editor open, or etc, that was inadvertently slowing access to local files (doh).

3. can you snoop the traffic? (i.e. its socket, not pipe)

Classic problem: some unrelated service is bombarding the loopback with crap. eg. Windows networking (samba on linux) going nuts

Try running Wireshark, filtered to show only 127. and see what’s going through at same time

Also: this is a good way to check where the delay is happening. Is it at point of send, or point of receive? … both?

There’s “net traffic” and there’s “net traffic”. Wireshark often shows traffic that OS normally filters out from monitoring apps…

4. Check your routing table? AND: Check it’s not changing before / aftter the attempted read?

5. Try enabling Nagle, see if it has any effect?

My point is: use this as a check: it ought to make things worse. If not … perhaps the disabling wasn’t working?

6. Have you done ANY traffic shaping (or firewalling) on this machine at any time in the past?

Linux: in particular, check the iptables output. Might be an old iptables rule still stuck in there – or even a firewall rule.

Linux + Windows: disable all firewalls, completely.

7. Similarly: do you have any Anti-Virus software?

Disconnect your test machines from the internet, and remove all AV.

AV software can even corrupt your files when they mistakenly think that the binary files used by the compiler/linker are “suspicious” (IIRC, that happened to us with early PlayStation3 devkits).

8. On a related note: security / IPS tools installed? They will often insert artificial delays silently.

CHECK YOUR SYSTEM LOG FILES! …whatever is causing the delay is quite possibly reporting at least a “warning” of some kind.

9. (in this case, Cas’s socket is using SSL): Perhaps something is looking up a certificate remotely over the web?

…checked your certificate chain? (if there’s an unknown / suspicious cert in the chain, your OS might be trying to check / resolve it before it allows the connection)

10. (in this case, Cas is using custom SSL code in Java to “hack” it): Get a real SSL cert from somewhere, see if it behaves any different

(I think it would be very reasonable for your OS to detect any odd SSL stuff and delay it, as part of anti-malware / virus protection!)

11. “Is it cos I is custom?” – Setup an off-the-shelf webserver on the same machine and check if YOUR READING CODE can read from that SSL localhost fast?

(it often helps to get an answer to “is it the writer, the reader, both … is it the hacked SSL, or all SSL?” etc)

12. (getting specific now, as the general ideas didn’t help): if you bind to the local IP address instead of 127 ?

13. Howabout reverse-DNS? (again, 4 seconds is in the realm of a failed DNS lookup)

Might be a reverse DNS issue … e.g. something in OS (kernel or virus protection), or in JVM SSL library, that’s trying to reverse-lookup the 127 address in order to resolve some aspect of SSL.

I’ve had to put fake entries into hosts file in the past to speed-up this stuff, some libs just need a quick answer

Result

Turns out we were able to stop here…

Cas: “Well, whaddya know. It was a hosts lookup after all – because I’m using 127.0.0.2 and 127.0.0.3 it was doing a hostname lookup. I added those to hosts and all is solved

Me: “(dance)” (yeah. It was skype. The dance smiley comes in handy)

Posted on

All Game Developers should read Pat Wyatt’s blog…

I noticed a few months back that Pat Wyatt has been blogging rgularly and in a lot of detail last year. This (IMHO) is big news: Pat is an awesome developer who held key positions in the teams behind many of the bestselling computer games (e.g.: Diablo 1 + 2, Starcraft, Warcraft) and went on to co-found Arena.Net (creators of Guild Wars).

I worked with him briefly in the past, and he’s friendly and full of advice and knowledge – but while he was happy to share, IIRC it was rarely in published form.

I’ve had a tough few months, but I’ve been dipping into his blog a few times, and it delivers in spades. Here’s a few hilights:

Assertions: enable them in live builds

(I’ve always felt this was the “right” way to do it for servers – where you don’t have to worry so much about frame-time, and assertions are more valuable at runtime because they help with the hardest-to-trace bugs … but it’s hard to get broad data on what the performance cost is)
http://www.codeofhonor.com/blog/whose-bug-is-this-anyway:

“The bug was easily fixed by upgrading the build server, but in the end we decided to leave assertions enabled even for live builds. The anticipated cost-savings in CPU utilization (or more correctly, the anticipated savings from being able to purchase fewer computers in the future) were lost due to the programming effort required to identify the bug, so we felt it better to avoid similar issues in future.”

…and a great rule of thumb for any Programmer:

“After my experience reporting a non-bug to the folks at Microsoft, I was notably more shy about suggesting that bugs might be caused by anything other than the code I or one of my teammates wrote.”

Some bugs are due to … user’s broken hardware

http://www.codeofhonor.com/blog/whose-bug-is-this-anyway:

“Mike O’Brien, one of the co-founders and a crack programmer, eventually came up with the idea that they were related to computer hardware failures rather than programming failures. More importantly he had the bright idea for how to test that hypothesis, which is the mark of an excellent scientist.

He wrote a module (“OsStress”) which would allocate a block of memory, perform calculations in that memory block, and then compare the results of the calculation to a table of known answers. He encoded this stress-test into the main game loop so that the computer would perform this verification step about 30-50 times per second.

On a properly functioning computer this stress test should never fail, but surprisingly we discovered that on about 1% of the computers being used to play Guild Wars it did fail! One percent might not sound like a big deal, but when one million gamers play the game on any given day that means 10,000 would have at least one crash bug. Our programming team could spend weeks researching the bugs for just one day at that rate!”

AI cheats to improve game balance in RTS’s, starting with Warcraft/Starcraft

http://www.codeofhonor.com/blog/the-making-of-warcraft-part-3:

In most Warcraft missions the enemy computer players are given entire cities and armies to start with when battling human players. Moreover, Warcraft contains several asymmetric rules which make it easier for the AI player to compete, though these rules would perhaps be called outright cheating by most players.

One rule we created to help the computer AI was to reduce the amount of gold removed from gold mines to prevent them from being mined-out. When a human player’s workers emerge from a gold mine those workers remove 100 units of ore from the mine and deliver it back to the player’s town hall on each trip, and eventually the gold mine is exhausted by these mining efforts. However, when an AI-controlled worker makes the same trip, the worker only remove 8 units of ore from the mine, while still delivering 100 units into the AI treasury.

This asymmetric rule actually makes the game more fun in two respects: it prevents humans from “turtling”, which is to say building an unassailable defense and using their superior strategic skills to overcome the computer AI. Turtling is a doomed strategy against computer AIs because the human player’s gold-mines will run dry long before those of the computer.

Secondarily, when the human player eventually does destroy the computer encampment there will still be gold left for the player to harvest, which makes the game run faster and is more fun than grinding out a victory with limited resources.”

Posted on

Concepts of “object identity” in game programming…

Hume just posted his Lessons Learned from the warmup for Ludum Dare 23 (48 hours to write a game from scratch – starts this weekend!) – and his positive experience using an Entity System.

In his epic comment (sparked by a different Adam – not me, honest), is this gem:

“Using the entity system for the first time was unreal to me. It’s like polymorphic code. I did really weird things on the fly. For example:

– In the health processor, if the enemy was just destroyed, set a flag in the lifecycle component.
– In the lifecycle processor, if the fresh kill flag is set, extract its loot component and put that into a new entity with a small randomized velocity component and a gravity component so that the loot drops; then, remove most of the other components from the entity and add an explosion component.

The “enemy” still has the same entity ID — any other components that are looking for that entity will still find it (e.g. missiles homing in on the wreckage, or score processors looking for slain entities) — but by swapping one set of data with another, its implementation has changed from an enemy to some kind of non-interactive effect object.”

(emphasis mine)

Identity. It’s important.

(Quick sidenote: for all the people asking questions like “but … which variables do I put in Component A as opposed to Component B? How do I manage Events in an Entity System? … etc” – Hume’s approach above is a good concrete example of the first-draft, easy-to-write way of doing things. Copy it.)

Identity in games

This is one of those things that newbie game programmers seem to underestimate, frequently.

And when I say “newbie” I include “experienced, skilled programmers with 10+ years of coding experience – but who haven’t yet shipped a game of their *own*”.

(e.g. I’ve seen a couple of studios that started as Digital Agencies, or as Animation Studios, etc – that then transitioned to writing their own games. This is the kind of thing that they often struggle with. Not for lack of skill or general programming experience, but for lack of the domain-specific experience of game coding)

Examples of Identity in games, off the top of my head – all of these are independent, and interact in complex ways with each other :

  1. Game-role: e.g. … “enemy”, “powerup”, “start location”
  2. Code ‘object’ (in OOP terms): e.g. … “the sprite you are drawing at position (4,5) is part of Object X. X is coloured THIS colour”
  3. Gameplay ‘object’: e.g. … “the sprite at (4,5) represents a Tank. If a Tank sprite ever touches a Glass sprite, we need to play the Broken Glass noise”
  4. Physics element: e.g. … “5 milliseconds ago, our Physics Engine thought this thing was THERE. Now it’s over HERE. Don’t confuse the Physics Engine! Make sure it ‘knows’ they are the same object – not two separate objects”
  5. Network “master/clone”: e.g. … in multiplayer, there are N copies of my avatar: one per computer in the game. One of those N is the original – and changes to the original are constantly used to overwrite the clones; changes to clones are LOCAL ONLY and are discarded. Which is original? What do we do with incoming “changes” – which local Code Object do we apply them to? (your Code Object will be different from my Code Object – but they’ll both be the same identical Network Object, save mine is flagged “clone”)
  6. Proper Noun object: e.g. … “The Player’s Tank” is a very specific tank out of all the Tanks in the game. Many lines of game code don’t care about anything except finding and operating on that specific tank.
  7. Game-Over representation: e.g. … after the player has killed all the enemies, and they see a Game Over (won/lost/etc) screen, and you want to list all the enemies they killed … how do you do that? The enemies – by definition – no longer exist. They got killed, removed from the screen, removed from memory. You could store just the absolute numbers – but what if you want to draw them, or replay the death animations?
  8. …etc

Identity in Entity Systems

ES’s traditionally give you a SINGLE concept of Identity: the Entity (usually implemented as a single Integer). Hmm. That sounds worryingly bad, given what I wrote above. One identity cannot – by definition – encompass multiple, independent, interrelated identities.

But we’re being a bit too literal here. ES’s give you one PRIMARY identity, but they also give you a bunch of SECONDARY identities. So, in practice…

Secondary Identities in an ES

In OOP, the Object is atomic, and the Class is atomic. You cannot “split” an Object, nor a Class, without re-defining it (usually: re-compile).

In ES, the Entity is atomic, and the Component is atomic. But the equivalent of an OOP Object – i.e. “an Entity plus zero or more Components” – is *not* atomic. It can be split.

And from there comes the secondary identities…

A Primary Identity: e.g. “The Player’s Tank” (specific)
A Primary Identity: e.g. “a Gun Component” (generic)

A Secondary Identity: e.g. “The Gun component … of the Player’s Tank Entity” (specific)

Revisiting my ad-hoc list of Game Identities above, I hope it’s clear that you can easily re-write most of those in terms of secondary identity.

And – bonus! – suddenly the relationships between them start to become (a little) clearer and cleaner. Easier for humans to reason about. Easier *for you to debug*. Easier *for you to design new features*.

Global Identity vs. Local Identity

Noticeably, the network-related Identities are still hard to deal with.

On *my* computer, I can’t reference entities on *your* computer. I cannot store: “The Gun component … of YOUR player’s tank”, because your “Player’s Tank” only exists in the memory of your computer – not mine.

There are (trivially) obvious solutions / approaches here, not least: make your Entity integers global. e.g. split the 64bit Integer into 2 32bit Integers: first Integer is the computer that an Entity lives on, the second is the local Entity Integer. Combined, they are a “global Entity ID”.

(I’m grossly over-simplifying there – if you’re interested in this, google for “globally unique identifiers” – the problems and solutions have been around for decades. Don’t re-invent the wheel)

But … at this point, they also offer you the chance to consider your game’s network architecture. Are you peer-to-peer, or client-server?

For instance, P2P architectures practically beg for unique Global entity numbers. But C/S architectures can happily live off non-global. For instance:

  • On each client, there are ONLY local Entity numbers
  • When the client receives data from the server, it generates new, local, Entities
  • …and adds a “ServerGenerated” component to each one, so it’s easy to see that they are “special” in some ways. That component could hold info like “the time in milliseconds that we last received an update on this object” – which is very useful for doing dead-reckoning, to make your remote objects appear to move smoothly on the local screen
  • The server *does* partition all entities from all machines. But none of the clients need to know that

Or, to take it further, if your network arch is any good at all for high-paced gaming:

  • The server differentiates between:
    1. The entity that the game-rules are operating on
    2. The entity that client 1 *believes* is current
    3. …ditto for client 2, client 3 … etc (each has their own one)
    4. The entity that the game-rules accept (e.g. if a hacked client has injected false info, the game-rules may override / rewrite some data in the local object)
  • The server also tags all the entities for a single in-game object as being “perspectives on the same thing”, so that it can keep them in synch with each other
  • The server does Clever Stuff, e.g.:
    • Every 2 milliseconds, it looks at the “current entity”, and compares it to the “client’s belief of that entity”. If it finds any differences, it sends a network message to the client, telling it that “you’re wrong, my friend: that entity’s components have changed their data. They are now X, Y and Z”

… or something like that. Again, I’m grossly over-simplifying – if you want to write decent network code, Google is your friend. But the fastest / lowest latency multiplayer code tends to work something like that.

How?

Ah, well.

What do you think?

(hint: you can do wonders using Reflection/Introspection on your entity / components. By their nature, they’re easy to write generic code for.

But you WILL need some extra metadata – to the extent that you may wish to ‘upgrade’ your Entity System into a SuperEntity System – something with a bit more expressive power, to handle the concept of multiple simultaneous *different* versions of the same Entity. Ouch)

Yeah, I’m bailing on you here. Too little time to write much right now – and it’s been a *long* time since I’ve implemented this level of network code for an ES. So, I’m going to have to think hard about it, refresh my memory, re-think what I think I knew. Will take some time…

Posted on

Realm of the Mad God – a great game, interesting monetization

I’ve recently been playing the excellent Realm of the Mad God – a very fast-paced 2d co-operative shooter. My feeling is that it’s going to be one of the most important games of 2011/2012, as it continues to grow in popularity. Typical experience of this game is that within 30 seconds of being dumped into the main level, you’re surrounded by monsters, and then surrounded by other players, all on the same screen as you, blasting away in a rainbow of colours.

Sounds good. As if that’s not enough … it’s the guys who’ve been working with AmitP (Amit J Patel) (Amit maintains one of the best up-to-date collections of links and algorithms for indie game-developers). If you haven’t seen Amit’s pages, I recommend browsing through the blog – his links collection is OK, but his blog posts on algorithm design are excellent. If you get as far as the posts on “how I auto-generated a realistic 3D-world”, you may notice a striking similarity to the 2D worlds used in RotMG …

Anyway, it parallels an idea for an MMO shooter I’ve had kicking around for a long, long time. For me, it’s been a delight to see what works (and doesn’t) about the core ideas. The RotMG authors have done a great job of making a fast, simple, quick, easy-to-grasp game.

The Good

Fast. No barriers to play

This is how MMO-shooters should be: fast, furious, permadeath – but very quick to get back into the fray. You should expect to die tens of times every hour.

Permadeath – but paralled with some perma-advancement

Your progress is split evenly between items (which can be banked) and avatar stats (which are lost forever upon death).

Perma-advancement increases variety, unlocks new features

With 10 classes, there’s plenty of variety – and each class can only be unlocked by achieving a minimum level of progress with one or more other classes.

NB: this part could be improved and expanded IMHO. In particular, the classes are wonderfully varied – but merely unlocking classes isn’t enough these days. Plenty of games have shown that permanent-unlocks work best when there’s a variety of game-features in there. Also, the classes themselves would work better if there were some cross-effects (c.f. Diablo 2’s Lord of Destruction expansion, which had abilities in one class improve abilities in your older classes, re-vitalizing them for re-play)

Free game, paying is optional – payment kicks in when you’re most bought-in to the design

Free players get a tiny storage for permanent items (1/50th of what paying players get – it’s not enough! … so pay!), and are only allowed 1 class “alive” at once.

In a delicious twist, if you don’t pay for the game, the only way to take advantage of a newly-unlocked class is … to commit suicide … since you’re only allowed a single character per account (unless you pay)

You can ONLY benefit from other players, never suffer

(there’s actually a case where you can suffer, sadly – Thief’s get killed as a secondary-effect of other players teleporting to them, since the game doesn’t have a “prevent people from teleporting to me” flag)

This is the one that should have most wannabe-MMO-designers sitting up and paying attention. If you group-up with other players:

  1. You all get the same experience-points as if you’d single-handedly killed every monster
  2. You get the points just for being nearby – no need to score hits just to “tag” it for yourself
  3. Mob strength is constant, but player damage is multiplicative on number of players present

Net effect:

Every player is willing and eager (*) to collaborate with every other player, without words being exchanged, without fear of being ripped-off.

In a game that’s fast paced and frantic, you don’t have to keep pausing to negotiate. Other players can ONLY benefit you, so … run with them.

(*) – or just ignores the other players. Their presence doesn’t provide negative impact on you. It’s only their absence that is negative (in game-design terms).

Interesting design choices for lag

As a real-time game with dozens of players on screen at once, lag is guaranteed to effect gameplay. We’re always saying “try to work around lag through game-design changes”, so here’s the decisions they made:

  1. When packets are lost, everything moves in exactly the same direction it was going, at exactly the same speed, forever
  2. “Speed” used above is the “on-screen speed, including any rubber-banding effect”
    • FAIL: this means monsters and players often move MANY times faster than they are allowed to – so that when the lag stops, the side-effects are magnified
  3. Your avatar can’t be hit NOR damaged while it’s missing packets from the server
    • For the early parts of the game, this *almost* completely fixes lag problems
  4. Projectiles (bullets etc) that your client didn’t receive are queued up and sent to you all in one go once the packet-dropping stops
    • FAIL: this multiplies the damage output of enemies (NB: not players!), breaking all the designed-in balance in the game
    • In mid to late game, this ruins the gameplay – players end up running around never seeing a single enemy, because if you’re close enough to see it, a single flicker of lag will cause you to receive ZERO damage initially, followed by MORE damage than the monster is capable of – delivered instantaneously
  5. The client is authoritative on player liveness/death
    • MILD FAIL: in effect, coupled with the other features of the game, and the lack of lag compensation … this means you CAN and SHOULD (and, for some cases, effectively: MUST) cheat. You can run a bot on your machine – and if the network is less than perfect, you have to, in order to play the game properly.

Overall, apart from the massive security flaw (where anyone could write a bot to be invulnerable – and the developers are encouraging them to do so), it seems very close to a good solution for an MMO shooter.

I’m surprised by the way they approached the “save up the enemy bullets, then unleash them all at once”. My guess is that it wasn’t designed, it was just an accident: maybe they took a slightly lazy approach to compensating for lag (they don’t), and the net effect is this. It looks very much like what you get when using TCP for game-data packets (I really hope they’re not using TCP; if so, most of the lag is the developers’ own fault)

The Bad

I’ve unlocked half the classes, and looked at what classes other people play (and which classes rise high on the leaderboards). There’s good variety, and almost all the classes get used – even the beginning class, the one you get for free, works well.

Except one.

Unfortunately, at around 50% through unlocking the special classes, one of the classes is horribly unbalanced. The Assassin (which is supposedly an upgraded Thief – but is a massive downgrade) is almost impossible to play. The special ability fails completely when there’s lag (which is frequent in this game), and the class is the weakest, lowest-range of the lot. Looking around, you rarely see Assassins (I suspect: you only see them when people are desperately trying to upgrade them, to get past this dull and frustrating point in the upgrade tree).

Worse, because the *only* way to unlock the higher level classes is to reach the level cap with this class … you’re forced to play it. Over and over again. Watching the bad game-design … over and over … and over … again.

Every time the assassin dies, it’s like another twist of the knife:

We know you don’t enjoy this

We know that a mistake in our game design has you stuck here

(and our overall game design makes that “mistake” into “a disaster”)

We know that this whole process is turning “a class that wasn’t much fun” into “a class you hate”

And there’s nothing you can do about it!

So, single-handedly, it’s driven me to *not* purchase any game credit. I’d enjoyed the game enough to that point that I’d already decided to buy it – and if this had been on iTunes, I’d have paid already. But since it’s not an iPhone/iPad game, and paying for it is a bit more difficult, I hadn’t made the payment yet.

As it stands, I’m still playing occasionally, but now it’s for research rather than for enjoyment, which is a great pity.

Monetization: money thrown away

I think the developers are missing an ENORMOUSLY successful way to make money from this game. In fact, it’s so big, I suspect they could increase their revenues by a substantial multiplier.

With a permadeath game, there really is no need to actually delete the dead character. If the player isn’t paying, they are forced to kill their character sooner or later to change characters.

Taking a leaf from Flickr’s book, why not keep ONE single character in storage, with a tempting “buy now, for goodies *and to have this one returned to you, ready to play*”?

I’d set it so that when you change class (if you’re a free player), only the last character of the PREVIOUS class is retained. If you switch from Warrior to Knight, you can die many times as a Knight, and your Warrior remains on ice. But if you then grow tired of the Knight and switch to a Rogue … the Warrior is tipped out, and replaced with the last Knight you had.

i.e. you setup the exact flow of decision-making and options and “safety” that the player would have had, if only they’d purchased sooner, and allow them to benefit from it retroactively – if only they make the decision to pay.

Of course, it’s a very limited “retroactively” – it’s a sampler, to let you see the benefits of paying.

(*) – Flickr’s early promise was “upload your photos in highest resolution, you can view them for free – but only low-res versions. HOWEVER, we keep the high-res versions for you – forever, for free – until you decide to purchase a subscription. At which point, not just your new photos, but ALL your photos, become magically available at highest res”. It was a great way of simultaneously offering a high-value to paying customers, while making non-paying customers feel they weren’t committing themselves to loss. It reassured a lot of potential customers at a time when Flickr wasn’t yet famous, and most people weren’t yet “hooked” – it bridged that gap.

Analyse this

So, the interesting question is: how common is this problem?

Are the dev team correlating “players who pay” and “the point at which they pay”?

More importantly, are they correlating “players who DON’T pay” and “how their experience differed from the average”?

The last time I saw an MMO with a level-based kick in the teeth this bad … was in Tabula Rasa. We had a point where poor signposting by the quest designers meant many players were given quests that were many levels too hard for them, and effectively impossible to do. Those players died over and over and over again in a short time – and they hated it.

The dev team knew “you’re not supposed to do that quest”, but often they (randomly) gave it to new players as the first quest. I wasn’t privy to the arguments over whether this needed to be changed (and there were definitely arguments), but I did see the analytics that eventually got produced. They showed an almost perfectly smooth, averaged, graph of player behaviour – bar a big notch at this particular location. It stuck out like Rudolph’s nose on a snowy day.

I wonder if there’s a similar notch in RotMG? For a game that’s almost *designed* to drive people to rage-quit … what stats do they see on “what the last thing was before a player stopped playing forever?” … and what stats do they see on “…stopped playing for a long time, but eventually came back”?

Posted on

Mongo DB is WebScale. MySQL is not WebScale.

There’s good reasons for adopting Mongo, I’m unconvinced (but open-minded) that performance is one of them. Here’s a ROFLMAO viewpoint on it:

“If your write fails, you’re ****ed”

Obviously, MySQL’s not perfect, but in most cases I’ve seen, it’s been lack of competence on the developer side, and the lack of basic DBA skills – not problems with MySQL itself – that’s broken scalability. In which case, I’m a little suspicious that a company that fails to scale MySQL will equally fail to write their code correctly on Mongo. In many ways, throwing away SQL makes it much easier to prevent scalability…

Posted on

The nature of a Tech Director in games … and the evils of DevOps

Spotted this (the notion “DevOps”) courtesy of Matthew Weigel, a term I’d fortunately missed-out on.

It seems to come down to: Software Developers (programmers who write apps that a company sells) and Ops people (sysadmins who manage servers) don’t talk enough and don’t respect each other; this cause problems when they need to work together. Good start.

But I was feeling a gut feel of “you’ve spotted a problem, but this is a real ugly way to solve it”, and feeling guilty for thinking that, when I got down to this line in Wikipedia’s article:

“Developers apply configuration changes manually to their workstations and do not document each necessary step”

WTF? What kind of amateur morons are you hiring as “developers”? Your problem here is *nothing* to do with “DevOps” – it’s that you have a hiring manager (maybe your CTO / Tech Director?) who’s been promoted way above their competency and is allowing people to do the kind of practices that would get them fired from many of the good programming teams.

Fix the right problem, guys :).

Incidentally – and this will be a long tangent about the nature of a TD / Tech Director – … my “gut feel” negativity about the whole thing came from my experience that any TD working in large-scale “online” games *must be* a qualified SysAdmin. If they’re not, they’re not a TD – they’re a technical developer who hasn’t (yet) enough experience to be elevated to a TD role; they are incapable (through no fault of their own – simply lack of training / experience) of fulfilling the essential needs of a TD. They cannot provide the over-arching technical caretaking, because they don’t understand one enormous chunk of the problem.

I say this from personal experience in MMO dev, where people with no sysadmin experience stuck out like a sore thumb. Many network programmers on game-teams had no sysadmin experience (which in the long term is unforgivable – any network coder should be urgently scrambling to learn + practice sysadmin as fast as they can, since it’s essential to so much of the code they write) – and it showed, every time. In the short term, of course, a network coder may be 4 months away from having practiced enough sysadmin. In the medium term, maybe they’ve done “some” but not enough to be an expert on it – normally they’re fine, but sometimes they make a stupid mistake (e.g. being unaware of just how much memcached can do for you).

And that’s where the TD-who-knows-sysadmin is needed. Just like the TD is supposed to do in all situations – be the shallow expert of many trades, able to hilight problems no-one else has noticed, or use their usually out-dated yet still useful experience to suggest old ways of solving new problems that current methods fail to fix. And at least be able to point people in the right direction.

…but, of course, I was once (long ago) trained in this at IBM, and later spent many years in hardcore sysadmin both paid and unpaid (at the most extreme, tracking and logging bugs against the linux kernel) so I’m biased. But I’ve found it enormously helpful in MMO development that I know exactly how these servers will *actually* run – and the many tricks available to shortcut weeks or months of code that you don’t have to write.

Posted on

Speaker Evaluations – GDC Austin 2009

Conferences don’t make these public.

But they should.

So … here are the evaluations (from the audience) for our panel session at AGDC 09.

Judge for yourself whether you want to attend any future sessions featuring us again (Adam Martin, Bill Dalton, Rick Lambright, Joe Ludwig, Marty Poulin).

Head Count: 74; Evaluations: 32 (43% response rate)

  • Overall rating of the presentation – 88% (AVG: 86%)
  • How relevant was the topic to you? – 86% (AVG: 84%)
  • How well did this class meet your expectations? – 94% (AVG: 84%)
  • Would you recommend this session to a colleague? – 90% (AVG: 84%)
  • Evaluate the speakers’ ability to communicate – 94% (AVG: 86%)
  • If there were visual aids (slides) how were they? – 74% (AVG: 60%)

All of those are above average, and I’m glad that a particularly high number would recommend the session to their colleagues.

It seems that we did particularly well on fulfilling the remit (very high number for “met expectations”), and that our speakers had an awesome ability to communicate (almost 10% higher than average for the other speakers at the conference).

Audience Comments

  1. The most entertaining session I attended, but didn’t sacrifice information value.
  2. Interesting format, I wouldn’t mind seeing more of this, but it is time consuming
  3. Good stuff
  4. Slow, confused start lost valuable time for Q&A
  5. Should have done middleware
  6. Only 3 topics covered. Expected others

Comment 4 – yeah, something I’m unhappy about too, (it wasn’t our fault, it was the people running the conference), but there was nothing for it but to grin and carry on. Someone screwed-up the radio microphones, and we lost a lot of time at the start waiting for them to fix it. There was nothing we could do – they had connected the mics from a different room to *our* speakers. We didn’t find out until the person in the other room started talking, and it all came out through our speakers :(.

Comment 6 – we covered 4 topics (oops, audience can’t count :P). We all wanted to do more, but at GDC conferences, the organizers only give us 1-hour slots. With 4 speakers + moderator, I think that was pretty good, especially considering the time we lost at the start.

Perhaps someone will clone this format for a future conference (seems a good idea), and try to get a 2-hour slot for it?

Posted on

Can OnLive work, technically? If so, how?

This week, a game service was announced that would stream games to your home TV without you needing to own a console or PC. A lot of people are wondering: are these guys smoking crack?

EDIT: Richard Leadbetter at Eurogamer has an article with some great “side by side compare” video to show theoretical quality you can achieve with off the shelf compressors right now. He comes to a similar conclusion on the issue of latency, although having seen the video I disagree on the quality (he feels it’s unacceptable) – it’s clearly inferior, but it still provides a very nice experience, especially if you’re sitting 6-12 feet away from the screen.
Continue reading “Can OnLive work, technically? If so, how?”

Posted on

Tabula Rasa: Going down in a burst of glory

http://www.gamasutra.com/php-bin/news_index.php?story=22528

“It is probably safe to say that, despite decades of ever more spectacular Hollywood visions of extra-terrestial domination, humanity in its worst nightmares never imagined it would have to contend with spawn-camping aliens.”

(also … If that article is accurate, sad but unsurprising to hear that (apparently) the underpowered server tech for TR yet again managed to make a misery of gameplay, even at the very end. If that article is accurate, then well done to the ops for managing to get some instancing sorted out, but note to self: never let this happen with future twitch-based / FPS MMOs)

Posted on

Installing and using PHP Eclipse IDE on OS X

I wanted to knock something up for Facebook, and I thought I’d try out PHP development on my shiny macbook air laptop (I usually develop on a much more powerful windows/linux PC).

This is tortuous, painful, and mostly undocumented on the official site. Sigh. After some experimentation, here’s how to make it work (IMHO you shouldn’t need to do this – although the Eclipse project allows plugin developers to package their plugins in really stupid ways, and doesn’t make it easy for anyone (users, developers, etc) – it’s still the fault of the plugin developer if they ALSO do not make it easy for the users to install).

First problem: get Eclipse

IME, most programmers who would use Eclipse already have it. The PHP plugin website won’t really help you, mostly taking the attitude of “install an extra copy of eclipse, just for doing PHP development; if you already have eclipse … work it out yourself”. I kind of understand why, but still feel that the first duty of every developer is to make their stuff easy to install!

If you don’t have Eclipse, I highly recommend you do NOT follow the PHP instructions (by downloading their pre-made “PHP + Eclipse all in one”) because then you are doomed to having multiple parallel installs if you ever need to use any other programming language; learn how to do it properly instead.

Download eclipse here (you want the latest stable version, currently called “Ganymede” (no, I don’t know why they stopped using Version numbers either – yes, it does make life more difficult for all normal users who haven’t memorized the funny names. Sigh)). You can get “Eclipse with Java” or “Eclipse with C/C++” or whatever you want – they are all identical, some just have extra plugins pre-installed.

NB: the people that run the Eclipse website have some pretty icon-artists, but a cruel sense of humour (or just suck at website design, maybe? I shouldn’t complain – it used to be a LOT worse than this) and don’t provide bookmarkable links to the OS X version (that I could find, at least); you’ll just have to go to that page and scroll till you find it. Right now, the current “standard” Eclipse version is at the VERY BOTTOM OF THE SCREEN (c.f. my previous comment re: sense of humour), and at the right hand edge of that box is a tiny link saying “Mac OS X”)

If you’re new to Eclipse, you will probably find that downloading eclipse is one of the most confusing downloads you’ve ever done; if so, this is part of that same problem mentioned above where the Eclipse project makes plugin installs ridiculously difficult: all those many confusing different Eclipse versions that you cannot tell the difference between are actually the same, but differ only in which plugins are pre-installed.

Yes, this is stupid. Yes, it’s badly documented. Sorry. You’ll learn to live with it – they’ve been doing this for almost ten years now so don’t expect it to be fixed any time soon.

I suggest you run Eclipse once, now, before going on to the next step – if Eclipse doesn’t start at this point, don’t waste time confusing yourself with the PHP plugins until you can get Eclipse working on its own!

Second problem: get PHP IDE (now renamed to “PDT”)

You can try to do the automated-install; Eclipse is bad at handling automated installs, has very poor error-handling if anything goes wrong (it just crashes and doesn’t explain), and plugin developers usually screw-up the auto-install API in ways that can actully render your copy of Eclipse unusable (this happens *a lot*). I would advise never using it if you can avoid it.

This one’s pretty messed-up. According to the website and the list of requirements, if you want to use the latest version (vesion 2.0, not yet “packaged”), then you have to download approx 6 massive files.

I found that if you download the “Eclipse for Java” pacakge then it has some of those built-in already, and even if you don’t, several of those have EACH OTHER built-in (WTF?). I suggest you don’t take any risks, and that you do this the long way (download EVERYTHING).

First, go to this download page.

Decide which version you want; right now you want “2.0.0 Stable Builds”, but soon that will be what you get from “Latest Releases”, so check there too.

Then download ALL the zip files listed under “PDT” AND everything listed under “Build Dependencies”. Right now, there are 3 files for “PDT” (SDK, Runtime, and Tests), and 5 files under “Build Dependencies” (Eclipse gtk, EMF, DTP, GEF, and WTP).

Third problem: OS X can’t unzip the files

If, like most OS X users, you’re using Stuffit Expander to unzip the files, by default it won’t do it, because they all overwrite the same directory name (and StuffIt is designed to “protect” you from that, which is nice).

That’s only slightly annoying to get around, but you are still screwed, because OS X itself is (apparently – I couldn’t find a way around this using Finder) hard-coded to prevent you from copying the contents of the directories into the Eclipse directory. When you try to it says “delete the target directory first, or cancel?” (unlike windows, which says “only overwrite files which are the same, otherwise copy all the missing files … or cancel?” which is 99% of the time what you wanted. I have no idea why Apple uses a “destructive” copy – and gives you no alternative!)

Here’s how to get around both problems: The solution – Manual install via Terminal

Fortunately, if you switch to using the Terminal, and run “unzip” by typing it in manually, by default it’s setup as a unix variant that acts in the same way that Windows works by default.

First, make sure you are in the directory where you saved the ZIP files, e.g. by typing:

cd ~/Downloads

(assuming you saved them to your personal Downloads folder)

Then you just type:

unzip [name of zipfile1]
unzip [name of zipfile2]
…
unzip [name of zipfileN]

…which resolves the incompatibilities in the distro files, and then to install the plugin you type:

cp -Ri eclipse [location of your eclipse folder – usually: /Applications/eclipse]

Note that the “-R” is *required*, and that there is NO trailing slash after “eclipse”. The “i” after the “-R” is optional, it might be good to know if you have problems, but it allows you to get confirmation before overwriting any files. Thankfully, you can just hold down the enter key and it will do the defautl (do not overwrite) as it goes through each file; there are many hundreds of files to copy, and you may already have hundreds of them, so this is handy.

Installation complete

Now start Eclipse – this took 5 minutes for me, where normally it takes 30 seconds, don’t ask me why) – but eventually it worked.

Test it works – go “File -> New Project” and scroll down to the PHP folder, and select “PHP Project”. If there is no PHP folder in the list, then the install has failed. Start again. Good luck.

Otherwise, it should let you create a project, in which case: You now have PDT / PHP-IDE for Eclipse installed and working.

Writing a PHP file

I’m assuming you know how to do this, or can find a tutorial (any PDT tutorials should be fine – it works the same way as the mainstream language plugins for Eclipse, so *any* tutorial on creating a source file and building it ought to work).

Fourth problem: Running your PHP

Oh crap. If you go ahead and create a project, give it a name, and hit OK, you’ll find that the PHP IDE seems designed to not allow you to develop or test PHP on a server; it only supports developing and testing on a local (inside Eclipse) private PHP interpreter. If you’re new to programming, this is fine to get started and learn some basic PHP.

If you’re an experienced programmer, you’ll probably hat that: unless you enjoy tracking down unreproducable bugs and tearing your hair out, you need to develop on the same software + version that runs your production server (in most cases, this will be an Apache2 server running the PHP5 module). Since OS X comes with Apache2 *and* PHP5 built-in, you *already* have a server on your machine that is probably 98% the same as the live server you would use (so far Apache2 + PHP5 on OS X seems to act almost identically to the same versions on Linux, FreeBSD, etc – as you would expect).

(98% is annoyingly short of 100%, but it’s a lot closer than using the bulit-in interpreter)

I can’t find any options in the Run Dialogs to control how it invokes the running of the code from a remote server (or even a local one!) – if you go digging through all the config options, they’re just missing.

NB: there IS something that looks like it might do the trick, where it has a list of “Server” and lets you choose a “PHP Server” – but THIS IS A LIE (there is no cake), do not believe it, this is for something else entirely; it’s just that someone made a poor choice of name for those labels).

Instead, what you have to do is be a lot more careful when creating new projects. Do this:

  • File -> New… -> PHP Project
  • Fill in project name as per normal with Eclipse
  • UNcheck the “default” option for “Project Contents”
  • Click the Browse button under “Project Contents” and navigate to wherever you keep your source DIRECTORIES for all your projects (see more on this below – and you may end up crying when you see what has to be done)
  • Click Finish
  • …NB: BE CAREFUL: it asks you a sneaky extra question, and your answer depends on how you manage source control (see below)

Special Note: where do you keep your PHP?

If:

  • you are just developing locally for now on OS X
  • AND you want to use the Sites directory to save your PHP files in
  • AND you want to use your Apache2/PHP server instead of the “fake” one that comes with eclipse

Then:

  • select your personal Sites folder as the Project Contents above
  • AND answer “Create project in /Users/[your name]/Sites/[project name]” when you press Finish above

This will:

  • automatically create a new sub-directory in Sites with the same name as your project
  • mean that the address of your project is “http://localhost/[projectname]/”
  • mean that if you delete the project in Eclipse, and select “delete from disk as well” when you do, Eclipse will delete ONLY the “[projectname]” subdirectory from your Sites folder, and leave everything else intact

…ALTERNATIVELY…

If:

  • you use proper source control which has a unique root directory for each source project

Then:
EITHER:

  • select the folder for that project as the Project Contents above
  • AND answer “Create project in . (Deleting the project will delete the entire )

OR:

  • select the PARENT folder that contains the project-specific source folder as the Project Contents above
  • AND answer “Create project in [PARENT folder of source root folder]” when you press Finish above

I actually strongly recommend the first option, since this will ensure that Eclipse doesn’t mess with your source control’s PARENT folder (which in most source control systems will either screwup the system (happens with crappy source control like CVS) or just be ignored because you won’t have write-access (happens with the more high quality source control) – but this can upset Eclipse if you do some other things wrong in the future.

FINALLY!

After all that, I finally had a working PHP IDE on OS X. Yes!

I haven’t tried debugging yet, but I found the following links that look pretty sound for setting it up. Bear in mind that the first one tells you to setup your Project Contents differently – just adapt what it tells you depending upon what you did above – the author doesn’t seem to fully understand Eclipse’s arcane approach to projects (given the name he uses for his Project!), which is fine, but IMHO not recommended.

How To Setup a Free PHP Debugger using Eclipse PDT + XDebug

Posted on

MMO Blogger Round-up

On this site I have a rather subtly-hidden Blog Roll. When I started blogging, the site had less on it, and the roll was easy to find – and short. Now it’s not. And it’s long. And each link on there has been carefully considered. There’s some gems in there (although a lot of them are updated so infrequently few people track them).

So it’s time to call-out some of the interesting things to be found in the blogging world of MMO people.

By the way … you can tell who’s working on uber-secret or personally exciting projects these days because they’ve suspiciously stopped blogging for months at a time. Lazy slackers, the lot of them. The more you do, the more you should blog! :P

There are some that should be on the blogroll but aren’t (yet), and some other bloggers I should mention (but I’m sticking to the blogroll only for this post – I’ll go through others next time). Feel free to add your own recommended reading in the comments.

Blogs to read:
Brinking (Nabeel Hyatt)
* Who? serial entrepreneur, raised funding and sold companies
* What? currently running a funk-tastic social / music / games company with a bunch of Harmonix guys
* Why? big commentator on the games/apps/making money/predictions parts of All Things Facebook

Broken Toys (Scott Jennings / LTM)
* Who? became infamous in the early days of MMOs as a player of Ultima Online who ranted publically, amusingly, and sometimes even insightfully
* What? ex-NCsoft, now doing intriguing web games at John Galt Games
* Why? In his heart Scott’s still a player, and more than anyone else I’ve seen he interprets the world of MMO design, development, and playing through the players’ eyes. Interesting point: he’s mostly concerned with life-after-launch. Funny that. Players kind of find that bit the most interesting. Also keeps a close eye on community-management screw-ups, and WoW generally

Bruce Everiss
* Who? ex-head of marketing for Codemasters
* What? um, I’m not sure what he’s doing these days, apart from becoming a “professional blogger”
* Why? he aims to comment on every single interesting piece of news in the mainstream games industry. That’s a lot of commentary. Always something to read! IMHO he is often completely wrong about anything online-games, and a lot of business and “future of industry” stuff – Bruce is from an older age of the industry. But … he says a lot of interesting things and sparks a lot of interesting debates in the process. Worth reading. Just remember he is extremely (deliberately, I’m sure) provocative, and don’t take it too seriously.

Coke and Code (Kevin Glass)
* Who? A programmer working in mainstream IT
* What? An insanely prolific author of casual games “in his free time, as a hobby”
* Why? Because he’s better at making games than many professionals I’ve met, and he is very very prolific, making new libraries, toolsets, editors, games, game engines – and commenting on it all as he goes, and throwing up new games for you to play all the time

Erik Bethke
* Who? ex-Producer for Interplay
* What? CEO of GoPets, an online casual virtual world that’s especially big in Asia (and based in South Korea)
* Why? A hardcore WoW player who analyses the game-design as he goes, and relates very honestly a stream of both emotional experiences and seminal events in the game that should give you lots of things to be thinking about, especially if you’re a designer, business person, or product manager.

Extenuating Circumstances (Dan Hon)
* Who? ex-MindCandy, current CEO of SixToStart
* What? one of the first Bloggers (on the whole of the internet!) in the UK, and an awe-inspiring assimilator of “everything happening on the internet, with technology, with media, with entertainment and the future of the world” for all of the ten years I’ve known him.
* Why? He’s still an excellent tracker of all those things, and finds memes very quickly. Nowadays he just auto-posts links (lots of them, every day) with a few words of commentary scattered here and there (del.icio.us descriptions) – making his blog a ready-made news filter for you :)

Fishpool (Osma Ahvenlampi)
* Who? CTO of Sulake (makers of Habbo Hotel)
* What? a very technical commentator, often in great detail, on the issues of running a 100-million user virtual world, with observations about Habbo’s community, business, and culture thrown in
* Why? He posts very rarely, but when he does, they’re usually full of yummy detail

Futuristic Play (Andrew Chen)
* Who? ex-VC (Mohr-Davidow Ventures)
* What? entrepreneur with a web-background who’s come into the games industry and bringing lots of useful stuff with him
* Why? blogs a LOT on advertising (and how to make money out of it in games and web and casual), and on metrics, and how you can use them to run you games or web business better. Also has a long fascination with what are the best parts of the games industry, and the best of the web industry, and how we can each put those best bits together to be even better

Off the Record – Scott Hartsman
* Who? ex-Everquest, ex-Simutronics
* What? Senior Producer for MMOs – but previously an MMO lead developer, and once (apparently) a Game Designer.
* Why? he’s funny, he knows his stuff, and he’s worked on some of the most important MMO projects outside Asia, so he’s got an interesting perspective going there.

Orbus Gameworks (Darius Kazemi)
* Who? ex-Turbine, now CEO of Orbus (a games-metrics middleware company)
* What? Likes the colour orange *a lot*, infamous for networking his ass off at games conferences (*everyone* knows Darius), very friendly, generous – and mildly obssessed with the use of metrics and stats to improve the creativity and success of game design (in a good way)
* Why? If you liked the Halo heatmaps when they came out, you’ll love some of the stuff they post on the Orbus company blog. A year ago they were posting heatmaps-on-steroids. If you thought “metrics” equalled “spreadsheets of data” then prepare to have your view changed pretty thoroughly.

Prospect Magazine/First Drafts (Tom Chatfield)
* Who? section-editor of the highly respected socio-political print magaine Prospect
* What? a highly-accomplished English Literature post-grad (bear with me here) … who also happens to have been a lifelong hardcore game player, I think the only person I know who got a hardcore character to level 99 on Diablo2, and now plays WoW a lot.
* Why? although Prospect only very rarely (like, only a few times ever) covers games, it’s very interesting to see what the rest of the world – especially the highly educated and highly intelligent but non-technical, older generations – thinks of us. And a bit of culture in your blog reading is probably good for you, too.

Psychochild (Brian Green)
* Who? ex-3DO/M-59, now the owner and designer of the revamped, relaunched, more modern Meridian-59
* What? an MMO game designer who disingenuously describes himself as an indie MMO designer but like most of the others has probably spent too long doing this and knows too much (compared to many of the modern “mainstream” MMO designers) for that to be true any more
* Why? lots and lots of great design ideas and commentary here for anyone wanting to do MMO design

Scott Bilas
* Who? programmer on Duneon Siege
* What? …in particular, responsible for the Entity System (one of my main areas of interest)
* Why? Scott’s phased in and out of blogging, but when he does blog he tends to do good meaty programming posts that contain lots of source code and some useful lesson or algorithm.

Sulka’s Game (Sulka Haro)
* Who? lead designer for Sulake (Habbo Hotel)
* What? more of a Creative Director than game designer, more of a web background than games, but above all a community/product/creative person who knows his stuff. Also a big player of MMORPGs
* Why? are you cloning Club Penguin or Habbo Hotel and want some pointers about revenue models, community management, and how to be successful with virtual-item sales? You might want to read his posts ;)

The Creation Engine No.2 (Jim Purbrick)
* Who? ex-Codemasters, ex-Climax (both times working on MMO projects)
* What? originally a network / MMO academic researcher, then a network coder, and now the person who runs Linden Lab (Second Life) in the UK. Very big proponent of all things open-source, always doing interesting and innovative things with technology
* Why? Keep an eye on the more innovative technology things that are done with Second Life (stuff you don’t tend to read about in the news but – to a tech or games person – is a heck of a lot more interesting by a long long way), and get some insight into the life of serious open-source programmers who succeed in living and breathing this stuff inside commercial environments

The Forge (Matt Mihaly)
* Who? developer of one of the earliest commercially successful text MUDs, now CEO of Sparkplay Media
* What? spent many years running Achaea, a text-only MUD that made a healthy profit from pioneering the use of itemsales (virtual goods) – and the things weren’t even graphical – and has now finally (finally!) moved into graphical games with the MMO he’s developing
* Why? one of the few MMO professionals who talks a lot about his experiences playing on consoles (especially Xbox), which makes for a refreshing alternate view – especially from the perspective of an MMO person talking about social and community issues in those games. Just like Brian Green, claims to be an indie MMO designer, but probably knows far far too much for that to be even vaguely justifiable

Vex Appeal (Guy Parsons)
* Who? ex-MindCandy
* What? Guy is an extremely creative … guy … who had a small job title but a big part in inventing and rolling out a lot of the viral marketing stuff we did for Perplex City (online game / ARG from a couple of years ago)
* Why? Awesome place to go for ideas and info on the cutting edge of doing games stuff with social networks. Usually. Also … just makes for a fun blog to read

We Can Fix That with Data (Sara Jensen Schubert)
* Who? ex-Spacetime, currently SOE
* What? MMO designer, but like Lum / Scott Jennings, comes from a long background as player and commentator, and shorter background as actually in the industry. Like Darius Kazemi, spent a lot of time in doing metrics / data-mining for MMOs
* Why? Take Darius’s insight into metrics for MMOs, and Scott’s knowledge of what players like, don’t like, and ARE like, and you get a whole bunch of interesting posts wandering around the world of metrics-supported-game-design-and-community-management. Good stuff.

Zen of Design (Damion Schubert)
* Who? ex-EA (Ultima Online), currently at Bioware (MMO)
* What? MMO designer who’s been around for a long time (c.f. UO)
* Why? Damion writes long detailed posts about MMO design, what works, what doesn’t, practicalities of geting MMO development teams to work together, how the playerbase will react to things, etc. He also rather likes raiding in MMORPGs – which is fascinating to see (given his heavy background as a pro MMO *designer*)

[NC] Anson (Matthew Wiegel)
* Who? ex-NCsoft
* What? Dungeon Runners team
* Why? was doing lots of interesting and exciting things with data-mining/metrics in the free-to-play low-budget NCsoft casual MMO. Watch this space…

People with nothing to do with games, but you might want to watch just because they’re interesting:
Bard’s World (Joshua Slack)
* ex-NCsoft
* Josh is one of the key people behind Java’s free, hardware-accelearted, game engine (JME)
Janus Anderson
* Who? ex-NCsoft
* What? um, he’s been taking a lot of photos recently
* Why? watch this space
Mark Grant
* Who? non-Games industry
* What? an entrepreneur, web-developer, and Cambridge Engineer
* Why? very smart guy, and interesting posts on web development (no games tie-in)

Posted on

How to hack an MMO (Raph Koster)

“The first thing to realize is that encryption of the data stream isn’t going to stop anyone serious.” – this page lists half a dozen conceptual ways that people will try to hack your MMO. It’s nothing like as exhaustive as the title suggests, but given it’s on Raph’s blog, the comments section is likely to pile up with many different additional concepts and information from world + dog.