#Gamification, #StackOverflow: How they create new sites from the community while blocking spam

If you’re not a StackOverflow user …

  • It’s rapidly become the go-to place for answers to precise technical questions
  • It has a bold, “points-based reputation controls everything”, moderation system (e.g. get upvoted enough and you become a moderator – there is no human intervention!)
  • It worked so well they expanded to infinitely many clone sites, for any topic you can think of, collectively termed “StackExchange”
  • …and the process for creating a new clone is itself points-based (not human moderated)

Historically this kind of setup has been a recipe for disaster – too easily gamed (taken advantage of), both by selfish users and purely malicious griefers. SO has had, and still has, many problems – some of the design choices that worked early on caused more problems than they solved as it scaled up in size. But overall it worked, and continues to work, very well.

The main site is easy to understand – by the time you’ve earned 10,000 reputation you are probably so enmeshed in the community it’s probably safe to give you moderator tools. (Note “probably” – it causes serious damage in a large minority of cases, but overall it works well, and it’s very cheap. This generally means no corporate sponsors/advertisers/subscriptions are needed)

But the process for creating new sites is less obvious, more convoluted, and – for the brand – potentially a lot more dangerous. I’m involved in two sites going through the process right now, and it’s interesting to compare them.

Site creation overview

  1. A special site – Area51 – allows you to create new sites, with this FAQ
  2. Your new site goes into phase 1: “Definition”
  3. If it passes, it goes to phase 2: “Commitment”
  4. Once it’s live, it’s monitored for a while, and if the site was a bad idea / disaster / failure, it gets shutdown
  5. If it passes the probationary period, the site is live

SO is worried about a bunch of things. From game-design and community-management perspective, I’d expect them to focus on e.g.

  • Will anyone use the site? (ask questions)
  • Are there any experts around to answer the questions?
  • Is the community large enough to be self-sustaining?

“Definition”

This checks the first worry – will anyone ask any questions?

To prove this programmatically, they force you to ask 40 “Good” questions.

To determine if a question is good, it has to receive 10 more upvotes than downvotes. This is arbitrary, but it means e.g. 10 different people felt it was a good question, and no-one thought it bad (or more for both sides, so long as there’s 10 more up than down votes).

Riiight … so you write 40 questions, get 9 friends to upvote all of them, and away you go … right? Easily gamed/abused.

Wrong. Area51 uses a modified version of SO’s voting. Each user is limited to asking 5 questions – which is fair and harsh. Fair because: if there’s really a community ready to go, it will have questions from many people (at least 8 people will have to pose good questions to get the 40 needed). Harsh because: most people will struggle to think up more than 1 or 2 good questions.

Still easily gameable, but now enough to dissuade idle / bored people.

Similarly each user can only upvote 5 questions. Choosing to up/downvote is very easy, so this isn’t “harsh” at all. It’s (almost) equally gameable: other users (anyone, anywhere) can register and counter-game by downvoting bad questions, forcing the collaborators to work harder.

Sadly, this scheme has a non-obvious element – the need to get 40 questions to 10 upvotes – that MOST users fail to understand, and SO has done nothing to fix. On SO.com, more upvotes is always better; on Area51 the 11th upvote (and all afterwards) are not only worthless, but actively delay the proposal because they squander upvotes that the user could have used on other questions.

So, for instance, the Computer Science Educators proposal was popular but spent many months failing to pass this phase because people arrived, upvoted the top questions, felt they’d helped … and left. Because of the design-flaw, not only were they “not helping” but the surge of high voted questions above the fold encouraged the next wave of newcomers to do the same. #facepalm.

By comparison, the IMHO less valuable Microbit proposal, for a politically driven educational tool that distracts from CS education, appears to have been supported by people with better understanding of the rules, and got through much more quickly. I don’t mind a microbit SE site, but … not if something so niche and political gets through at the cost of the CS Educators site (because it diverts attention away).

Solution: SO should change Area51’s visual design so that any question with 11 or more upvotes is displayed as “accepted” instead of a number, and the total number for each question is shown in smaller type somewhere else on the page.

“Commitment”

Guarding against the second concern is phase 2. With 80 people signed up as “committed” to microbit, and only 65 “committed” to CS Educators, microbit has now overtaken the older proposal. Right? Wrong.

Microbit proposal CS Educators proposal
Screen Shot 2016-10-05 at 12.33.01 Screen Shot 2016-10-05 at 12.32.53

Clicking the link at bottom right of each info panel shows that SO has a different approach at this stage. They use not one but three measurements to pass – it’s got harder.

Microbit would be winning here, except … SO judges you on the weakest link in the chain. And two of the three criteria are biased against people who don’t use SO much:

Microbit proposal CS Educators proposal

Screen Shot 2016-10-05 at 12.06.03

Screen Shot 2016-10-05 at 12.05.59

Measure 1: Total number of committers

Raw score of “how many people have clicked a button to say they believe this is a good site worth adding to the web”. Very much in the vein of SO.

EXCEPT: I see no “anti-commitment”; true SO ideals would suggest that we have a way to say “no, I don’t think this deserves a site” – and have it cost you some of your positive influence on committing to sites you do like. This is the essence of what made SO successful, and it’s intriguing that they’ve dropped it here.

I suspect (guess) that the relative infrequency of people proposing/committing to new sites (a few a year) vs voting/asking/answering SO questions (hundreds a year) means that the danger of people putting in negative votes at no effective personal cost was considered too likely. Or the cost to individuals of gaining enough positive commitments to “earn” the right to anti-commit was too high.

Measure 2: Require committers with > 200 reputation elsewhere

This is the classic gating strategy SO doubled-down on when they became too successful / too big: a requirement for minimum amount of positive reputation before enabling basic features that spammers tried to (Ab)use.

Earning 200 rep on “any” site is much too easy, and I think they’ve made a mistake here. It ought to be something like “earn 200 rep on 2 different sites, or 500 rep on one site”. I say this because over time SO rep has continual inflation problems, just like real-world currencies in live economies. A 200 rep barrier on one site is no barrier to people gaming the system – there are now so many obscure SE sites that it’s easy to find a gameable one. But finding multiple gameable ones would be substantially harder.

Measure 3: fudge factor

…this is the solution to the problems with Measure 2. I’d prefer a better Measure 2, but I can see value in having Measure 2 be very simple to describe, and then to fix the problems later.

Official stance on measure 3

The one piece of data we have that tells us a lot and is hard to game is a user’s reputation on the existing sites.

If you have a lot of reputation, you’re much more likely to actively use the site, because you’ve shown that you actively use similar sites

If you have a significant amount of reputation across multiple sites, you’re even more likely to actively use the site, because you’ve shown that you actively use many such sites

On the other hand, if you’re some random person off the internet with no reputation, you’re very hard to quantify but there’s a good chance that you won’t contribute very much

Here’s the formula we have right now. It’s almost certainly wrong and we’ll be tweaking it as we go:

Leave a Reply

Your email address will not be published. Required fields are marked *