Day: April 4, 2010

From the IGN walkthrough:

“If you have trouble grabbing the beam, just keep trying—we promise it works, but lots of readers have told us it’s not always easy.”

I’m a pretty good AC player, but after 10 minutes of trying to do that one standing jump, I gave up and stopped playing for a long time in frustration.

When game developers talk about “games should be so easy that all players can complete them; no-one should ever have to give up / fail to complete a game because something is too hard”, I usually disagree.

But in this instance, where the game is extremely, excessively difficult on something that the designer obviously intended to be extremely simple – and where the player has spent hours being taught that this will be easy – you have something different going on. It’s a failure of the control scheme; in fact, it’s a bug.

It’s a side-effect of the heuristics that AC uses to decide “what the player is trying to do” – heuristics that are far from perfect, while being very good.

In the first game, it took me a long time to get past the intro – no, really – because if you *try* to jump over gaps, then you fail. The heuristics were so heavily weighted towards “allowing” you to jump off buildings that running over a small gap became very difficult – until you learnt that the character “automatically” jumps small distances.

On the whole, I’m very impressed by the AC2 heuristics – compare it to Mirror’s Edge (a beautiful game, but feels a lot less fluid). I find them a bit too simplistic – I would love another 25% or so of user-control, and another 50% of precision on directional control – but (as ME shows) they got closer to perfect than any other game so far.

BUT … what do you do about a bug like this, one severe enough to make me stop playing the game entirely?

They had a huge QA team already (this is Ubisoft, after all), and such a vast amount of content in this game (multiple entire cities, modelled in fine detail), that there’s no way they could be sure to catch this bug.

Or is there?

This is the raison d’etre for a whole segment of in-game analytics / metrics: data-mining to discover undiscovered bugs.

Good metrics for game designers are VERY hard to describe, and IME the vast majority of the industry doesn’t know how to carefully hand-pick the few numbers they really need out of the millions of stats availalbe. Here’s a good example of how to pick well.

If the game reported

“the quest-point at which people stopped playing”

…then you *might* discover this bug. But it’s too coarse-grained.

If the game reported either/both:

“the segment on the map where people stopped playing”
“the segment on the map where people spent most-time during a mission”

…then you’d quickly and easily discover this bug. By “segment” I mean, essentially, a small patch of polygons approximately 6’x6′. This is relatively easy to measure algorithmically using greedy-polygon grabbing and hashing – although it would take a little care to make sure the measurement of the value didn’t take much CPU time (it could easily be pre-compiled for any given map, of course).

I’m not 100% of the “stopped playing” part – this is a console game, and while that info would be useful, it would mostly stop evenly distributed over quest-end points. Where it was more / less likely, it would be obvious just from knowledge of the story. ALTHOUGH: still well worth doing *in case* there were anomalies there – that should set off alarm bells.

However, the “spent most time during a mission” is more cut-and-dried.

This probe gives you a set of local maxima. It’s categoriesed by mission, making it one level finer than doing it over the entire world-map (which is too much, too uncategorised info), and it’s also coarse enough to correlate closely with user-behaviour (it merges results mission-by-mission; recurring bugs are very likely to show up by people doing the same mission and getting stuck at the same point).

The mission-based merge of results also has a nice side-effect: it tends to iron-out any anomalous results due to people wandering around the open-world game.

So. With a little bit of probing, using probes that you could/should have invented at the start of development (i.e. without knowledge of exact bugs that would occur) this bug could be ironed out. The three remaining questions are:

1. does Ubisoft do this level of automated-bug-detection,
2. do their designers bother to look at the anomaly-date,
3. and if so … why hasn’t the game been patched?