Category Archives: reputation systems

#Gamification, #StackOverflow: How they create new sites from the community while blocking spam

If you’re not a StackOverflow user …

  • It’s rapidly become the go-to place for answers to precise technical questions
  • It has a bold, “points-based reputation controls everything”, moderation system (e.g. get upvoted enough and you become a moderator – there is no human intervention!)
  • It worked so well they expanded to infinitely many clone sites, for any topic you can think of, collectively termed “StackExchange”
  • …and the process for creating a new clone is itself points-based (not human moderated)

Historically this kind of setup has been a recipe for disaster – too easily gamed (taken advantage of), both by selfish users and purely malicious griefers. SO has had, and still has, many problems – some of the design choices that worked early on caused more problems than they solved as it scaled up in size. But overall it worked, and continues to work, very well.

The main site is easy to understand – by the time you’ve earned 10,000 reputation you are probably so enmeshed in the community it’s probably safe to give you moderator tools. (Note “probably” – it causes serious damage in a large minority of cases, but overall it works well, and it’s very cheap. This generally means no corporate sponsors/advertisers/subscriptions are needed)

But the process for creating new sites is less obvious, more convoluted, and – for the brand – potentially a lot more dangerous. I’m involved in two sites going through the process right now, and it’s interesting to compare them.

Site creation overview

  1. A special site – Area51 – allows you to create new sites, with this FAQ
  2. Your new site goes into phase 1: “Definition”
  3. If it passes, it goes to phase 2: “Commitment”
  4. Once it’s live, it’s monitored for a while, and if the site was a bad idea / disaster / failure, it gets shutdown
  5. If it passes the probationary period, the site is live

SO is worried about a bunch of things. From game-design and community-management perspective, I’d expect them to focus on e.g.

  • Will anyone use the site? (ask questions)
  • Are there any experts around to answer the questions?
  • Is the community large enough to be self-sustaining?


This checks the first worry – will anyone ask any questions?

To prove this programmatically, they force you to ask 40 “Good” questions.

To determine if a question is good, it has to receive 10 more upvotes than downvotes. This is arbitrary, but it means e.g. 10 different people felt it was a good question, and no-one thought it bad (or more for both sides, so long as there’s 10 more up than down votes).

Riiight … so you write 40 questions, get 9 friends to upvote all of them, and away you go … right? Easily gamed/abused.

Wrong. Area51 uses a modified version of SO’s voting. Each user is limited to asking 5 questions – which is fair and harsh. Fair because: if there’s really a community ready to go, it will have questions from many people (at least 8 people will have to pose good questions to get the 40 needed). Harsh because: most people will struggle to think up more than 1 or 2 good questions.

Still easily gameable, but now enough to dissuade idle / bored people.

Similarly each user can only upvote 5 questions. Choosing to up/downvote is very easy, so this isn’t “harsh” at all. It’s (almost) equally gameable: other users (anyone, anywhere) can register and counter-game by downvoting bad questions, forcing the collaborators to work harder.

Sadly, this scheme has a non-obvious element – the need to get 40 questions to 10 upvotes – that MOST users fail to understand, and SO has done nothing to fix. On, more upvotes is always better; on Area51 the 11th upvote (and all afterwards) are not only worthless, but actively delay the proposal because they squander upvotes that the user could have used on other questions.

So, for instance, the Computer Science Educators proposal was popular but spent many months failing to pass this phase because people arrived, upvoted the top questions, felt they’d helped … and left. Because of the design-flaw, not only were they “not helping” but the surge of high voted questions above the fold encouraged the next wave of newcomers to do the same. #facepalm.

By comparison, the IMHO less valuable Microbit proposal, for a politically driven educational tool that distracts from CS education, appears to have been supported by people with better understanding of the rules, and got through much more quickly. I don’t mind a microbit SE site, but … not if something so niche and political gets through at the cost of the CS Educators site (because it diverts attention away).

Solution: SO should change Area51’s visual design so that any question with 11 or more upvotes is displayed as “accepted” instead of a number, and the total number for each question is shown in smaller type somewhere else on the page.


Guarding against the second concern is phase 2. With 80 people signed up as “committed” to microbit, and only 65 “committed” to CS Educators, microbit has now overtaken the older proposal. Right? Wrong.

Microbit proposal CS Educators proposal
Screen Shot 2016-10-05 at 12.33.01 Screen Shot 2016-10-05 at 12.32.53

Clicking the link at bottom right of each info panel shows that SO has a different approach at this stage. They use not one but three measurements to pass – it’s got harder.

Microbit would be winning here, except … SO judges you on the weakest link in the chain. And two of the three criteria are biased against people who don’t use SO much:

Microbit proposal CS Educators proposal

Screen Shot 2016-10-05 at 12.06.03

Screen Shot 2016-10-05 at 12.05.59

Measure 1: Total number of committers

Raw score of “how many people have clicked a button to say they believe this is a good site worth adding to the web”. Very much in the vein of SO.

EXCEPT: I see no “anti-commitment”; true SO ideals would suggest that we have a way to say “no, I don’t think this deserves a site” – and have it cost you some of your positive influence on committing to sites you do like. This is the essence of what made SO successful, and it’s intriguing that they’ve dropped it here.

I suspect (guess) that the relative infrequency of people proposing/committing to new sites (a few a year) vs voting/asking/answering SO questions (hundreds a year) means that the danger of people putting in negative votes at no effective personal cost was considered too likely. Or the cost to individuals of gaining enough positive commitments to “earn” the right to anti-commit was too high.

Measure 2: Require committers with > 200 reputation elsewhere

This is the classic gating strategy SO doubled-down on when they became too successful / too big: a requirement for minimum amount of positive reputation before enabling basic features that spammers tried to (Ab)use.

Earning 200 rep on “any” site is much too easy, and I think they’ve made a mistake here. It ought to be something like “earn 200 rep on 2 different sites, or 500 rep on one site”. I say this because over time SO rep has continual inflation problems, just like real-world currencies in live economies. A 200 rep barrier on one site is no barrier to people gaming the system – there are now so many obscure SE sites that it’s easy to find a gameable one. But finding multiple gameable ones would be substantially harder.

Measure 3: fudge factor

…this is the solution to the problems with Measure 2. I’d prefer a better Measure 2, but I can see value in having Measure 2 be very simple to describe, and then to fix the problems later.

Official stance on measure 3

The one piece of data we have that tells us a lot and is hard to game is a user’s reputation on the existing sites.

If you have a lot of reputation, you’re much more likely to actively use the site, because you’ve shown that you actively use similar sites

If you have a significant amount of reputation across multiple sites, you’re even more likely to actively use the site, because you’ve shown that you actively use many such sites

On the other hand, if you’re some random person off the internet with no reputation, you’re very hard to quantify but there’s a good chance that you won’t contribute very much

Here’s the formula we have right now. It’s almost certainly wrong and we’ll be tweaking it as we go:

“by running a spy network I am griefing”

If you’re an MMO designer, and you *still* don’t grok the griefer-mindset, or you somehow hope/believe that “one day, there will be no griefers”, then maybe this RPS interview with the always-fun-to-watch Goonswarm will help you:

MT: We are griefers. If nothing is going to happen then we’re going to try to find something that screams and bleeds and poke at it.

RPS: Griefing is something goons are known for doing, but now I’m talking to you it’s not something I can imagine you personally doing.

MT: Technically speaking, by running a spy network I am griefing.

RPS: But would you go out and aggravate other players for the Hell of it if you were a lower ranking member of Goonswarm?

MT: Well, most lower ranked Goons make their money by doing that. Scamming people is a very quick way of making money in Eve. Rather than making an honest buck, you take that buck from somebody else.

and, much further down, maybe this will help you see how griefers often serve just as positive and valuable a role as all your “preferred” player-types:

RPS: For my money, Eve might be the most fascinating game in existence today. But that doesn’t stop it from being interminably boring as well.

MT: Right. I mean most Eve players are stuck in high security space mining, and a lot of the core PvE in Eve has you sitting there are watching three grey bars slowly turn red.

Goonfleet is a socialist alliance. We give people ships so that rather than being forced to rat [fight low-powered AI NPCs] they can take part in PvP, we teach them how to scam so that they don’t have to mine, we teach them how to make ISK most effectively, we give them a lot of ISK and we reimburse their losses. This way they can focus on the fun aspects of the game, like griefing and warfare, so they’re not forced to endure derp-derp-ing around high sec.

If they play your game, you should be glad; if they grief, you should be asking yourself why – and if you’re a commercial operation, you should probably be asking:

“are they fixing a problem for us?

can we afford to leave them to it, part of our unpaid workforce?


is it worth our time trying to fix the problem itself, or should we accept their help and move on down our never-ending list of pending fixes?”

GDC09: Meaningful Social Reality Games

Austin Hill, Akoha


Conference organizer introduced this as “during this first talk, think about the platform they’ve made, as much as you do the game; that could be especially interesting for this audience”.

I totally support the principles and the ideals. The game looks fun and interesting, and at the same time taking a very “Don’t worry, be crappy” approach to core game design: lots of classic mistakes made, obvious stuff. Is this a case of being brave enough to deliberately make the mistakes they understand (because they’re easy to fix later when you’re more successful – and it leaves you more spare time to focus on fixing/avoiding the mistakes you don’t understand yet) – or just naivety?

Interesting to hear the philosophy that fed into the creation of the game, the speaker’s personal journey and how it informed the design. On the other hand, I was a bit disappointed how little actual content there was in this talk. It was perhaps 50% or more made up of a few long video clips. They were long and very little was pulled-out / emphasised from them. Most had very little information content per minute. Worst example was a mildly entertaining video of one of their players giving an intro to the product – but, frankly, so what? This was “new” and “interesting” 4 or 5 years ago, but by now it’s happened thousands of times over, and we’ve all seen it for many games. I didn’t understand why we were watching it.

I have a sneaking suspicion that – given he’s a VC – the speaker was pitching that video stuff to show “look, we have players who love our game”. That’s interesting and exciting to investors who have little or no immersion in the online world, but IMHO for game developers that’s just par for the course these days. No?
Continue reading