“If you have trouble grabbing the beam, just keep trying—we promise it works, but lots of readers have told us it’s not always easy.”
I’m a pretty good AC player, but after 10 minutes of trying to do that one standing jump, I gave up and stopped playing for a long time in frustration.
When game developers talk about “games should be so easy that all players can complete them; no-one should ever have to give up / fail to complete a game because something is too hard”, I usually disagree.
But in this instance, where the game is extremely, excessively difficult on something that the designer obviously intended to be extremely simple – and where the player has spent hours being taught that this will be easy – you have something different going on. It’s a failure of the control scheme; in fact, it’s a bug.
It’s a side-effect of the heuristics that AC uses to decide “what the player is trying to do” – heuristics that are far from perfect, while being very good.
In the first game, it took me a long time to get past the intro – no, really – because if you *try* to jump over gaps, then you fail. The heuristics were so heavily weighted towards “allowing” you to jump off buildings that running over a small gap became very difficult – until you learnt that the character “automatically” jumps small distances.
On the whole, I’m very impressed by the AC2 heuristics – compare it to Mirror’s Edge (a beautiful game, but feels a lot less fluid). I find them a bit too simplistic – I would love another 25% or so of user-control, and another 50% of precision on directional control – but (as ME shows) they got closer to perfect than any other game so far.
BUT … what do you do about a bug like this, one severe enough to make me stop playing the game entirely?
They had a huge QA team already (this is Ubisoft, after all), and such a vast amount of content in this game (multiple entire cities, modelled in fine detail), that there’s no way they could be sure to catch this bug.
Or is there?
This is the raison d’etre for a whole segment of in-game analytics / metrics: data-mining to discover undiscovered bugs.
Good metrics for game designers are VERY hard to describe, and IME the vast majority of the industry doesn’t know how to carefully hand-pick the few numbers they really need out of the millions of stats availalbe. Here’s a good example of how to pick well.
If the game reported
“the quest-point at which people stopped playing”
…then you *might* discover this bug. But it’s too coarse-grained.
If the game reported either/both:
“the segment on the map where people stopped playing”
“the segment on the map where people spent most-time during a mission”
…then you’d quickly and easily discover this bug. By “segment” I mean, essentially, a small patch of polygons approximately 6’x6′. This is relatively easy to measure algorithmically using greedy-polygon grabbing and hashing – although it would take a little care to make sure the measurement of the value didn’t take much CPU time (it could easily be pre-compiled for any given map, of course).
I’m not 100% of the “stopped playing” part – this is a console game, and while that info would be useful, it would mostly stop evenly distributed over quest-end points. Where it was more / less likely, it would be obvious just from knowledge of the story. ALTHOUGH: still well worth doing *in case* there were anomalies there – that should set off alarm bells.
However, the “spent most time during a mission” is more cut-and-dried.
This probe gives you a set of local maxima. It’s categoriesed by mission, making it one level finer than doing it over the entire world-map (which is too much, too uncategorised info), and it’s also coarse enough to correlate closely with user-behaviour (it merges results mission-by-mission; recurring bugs are very likely to show up by people doing the same mission and getting stuck at the same point).
The mission-based merge of results also has a nice side-effect: it tends to iron-out any anomalous results due to people wandering around the open-world game.
So. With a little bit of probing, using probes that you could/should have invented at the start of development (i.e. without knowledge of exact bugs that would occur) this bug could be ironed out. The three remaining questions are:
- does Ubisoft do this level of automated-bug-detection,
- do their designers bother to look at the anomaly-date,
- and if so … why hasn’t the game been patched?
3 replies on “Assasin’s Creed 2: Understatement of the Century”
Wow, and I thought it was just me complaining about this kind of problem (and the solution) :)
I never got to that jump in AC2 – I was too frustrated by the race against your brother right at the beginning. One of the first things you do in the game, almost no explanation, super hard, THE AI CHEATS, and you HAVE to finish it to continue the game. Who *does* that these days in a mass-market game?
But I agree Mirror’s Edge was much, much worse.
And yeah, metrics are the way to go.
Ideally, during testing at least, every little thing the player does should be written to an OLAP cube. Ideally, on non-MMO games, you’d also do this with the live game, perhaps with periodic “phone home” collections, where the anonymized play pattern data from each player (who does not opt out of the data collection) gets added to the central cube. On MMOs, since you are processing everything centrally anyway, this is easy.
The designer can then take one of the Business Intelligence analysis tools available on the market and go rooting around in this data, exploring for patterns. E.g. to use your example a designer might ask the question, “which quests have problems?”
1 – Check the completion rates of the different quests.
2 – Taking a quest with a relatively low completion rate. If it has multiple completion conditions or steps, what is the typical completion rate of those conditions, when the quest as a whole is not completed. E.g. a kill ten rats quest has a “Killed_Rats” dimension in the OLAP cube, with the number of rats being the measure. Here, we see that most players are giving up after the second or third rat.
3 – We check the player deaths dimensions against the mob killer measure and see that players rarely, if ever die at the hands of a rat. So we now know that players are not giving up because the rats are killing them.
4 – Let’s check the average life expectancy of rats and we see that the average rat lives just a few seconds, double checking, we see that they are nearly always killed by players. We now know that players are camping rat spawns, but seem to be giving up after killing just a few rats.
5 – Now we can surmise that the players are camping rats and killing them easily, but seem to get frustrated for some reason. Before we simply turn the spawn rate up, let’s see who among the players is killing them. By examining the rats killers, we see certain player names come up at the top of the list. By adding the player guild to the crosstab, we see that these players are all part of the same guild and by adding level, we see that these are all high level players. Therefore we can surmise that a griefer guild is camping the newbie areas and blocking the newbie from completing their kill ten rats quest.
6 – The GMs can investigate further and the dev team can devise methods to prevent this in the future.
@David
As someone once said: “Who has the time for this?”
In practice, no dev team is going to spend the time to record *everything*.
Equally, no design team will bother to drill-down *every* possible combination.
IMHO, OLAP/pivot-tables are rarely useful in game-design for non-trivial games. They are too specific, and are appallingly bad at the kind of queries that designers actually need to make.
IME, tables are also far too difficult for humans to understand and interpret. Compare them to graphs (OLAP outputting graphs in real time is a lot better than the OLAP tables I’ve normally seen – but still only partially successful).
The real challenge IME on actual game teams is working out the small subset of data that you should actually bother collecting and reading on a regular basis. Expensive datamining systems can help by doing some automatic anomaly detection, but I’ve never had the chance to run something really clever, like Autonomy, on an MMO dataset.