Metacritic Refuses To Pull Negative Review That GameSpot Admits Was Factually Inaccurate

If you head on over to the Metacritic page for Natural Selection 2, you might notice something strange. Of the seven reviews listed, only one is scored below an 80.

Also, it doesn't actually exist.

Last week, GameSpot reviews editor Kevin VanOrd pulled his site's review of Natural Selection 2. The review, scored 60/100 and written by a freelancer named Eric Neigher, had been eviscerated by readers and commenters who pointed out a number of mistakes -- for example, the review said the recently-released indie game was $US30, when it actually costs $US25. Other inaccuracies involved the game's engine and load times.

So VanOrd removed the review, citing "several inaccuracies" and apologizing to his readers. Yesterday, GameSpot ran a re-review by a new writer, Ashton Raze, who scored the game an 80.

But the original review is still on Metacritic. A big yellow 60 sits right on Natural Selection 2's front page, warning readers that "disappointing execution holds it back." Clicking "read full review" will take you to a broken link.

So why hasn't the review been replaced? I asked Metacritic head Marc Doyle, who told me that they have a one-shot policy for all reviews and all gaming outlets. (Kotaku is not listed on Metacritic, as we do not use review scores.)

"Yes, the critics we track know - and I spoke to the GameSpot team about this this week - that we only accept the first review and first score published for a given game," Doyle told me in an e-mail. "I'm explicit about this policy with every new publication we agree to track. It's a critic-protection measure, instituted in 2003 after I found that many publications had been pressured to raise review scores (or de-publish reviews) to satisfy outside influences. Our policy acted as a disincentive for these outside forces to apply that type of inappropriate pressure."

"We only accept the first review and first score published for a given game... It's a critic-protection measure."

It's a policy that seems to be enforced with the best of intentions -- just about every reviewer in the gaming industry has heard shady stories about publishers trying to get scores changed -- but it doesn't account for situations like this, where a writer's mistakes could seriously impact the fate of a game. I asked GameSpot's VanOrd how he felt about this whole situation, but he wouldn't comment on the specifics. "The review had multiple factual inaccuracies that cast a shadow over the entire piece, so we chose to pull it and reassign, which is a rarity, of course," he told me.

"So what?" you might be asking. "Why does it matter what kind of review scores this game has on Metacritic? Aren't review scores just arbitrary anyway? You guys don't use them!"

For better or worse, Metacritic has become an important tool for people in the gaming industry; some publishers even use it to decide whether to give out bonuses, a practice I've criticised in the past. And studies have shown that a game's Metacritic score can have significant impact on its sales.

For an indie game like Natural Selection 2 -- a game made over the course of six years by a team of just seven people -- one mediocre score could be catastrophic. (Although they seem to be doing okay, and Kotaku's Kate Cox had a ton of praise for the competitive game in our review.)

I asked Hugh Jeremy, community manager and jack-of-all-trades over at Natural Selection 2 developer Unknown Worlds, what the team thought of this whole situation.

"We fully respect Gamespot's journalistic processes and are very thankful that they took the time to review NS2. Many critics will not touch smaller indie games with a ten-foot barge pole," Jeremy told me. He said they've reached out to Metacritic for an explanation, but they haven't yet heard back.

"In general, it is extremely important that score aggregators reflect accurate and timely information about the reviews they are aggregating," he said. "Their scores are used by players, publishers, investors, other critics and even developers themselves to judge the success or failure of a product. Those scores can make or break dreams."

While Eric Neigher, GameSpot's original reviewer, was certainly entitled to give the game whatever score he felt was appropriate, that score was pulled. GameSpot has re-reviewed the game. Should the most important game score aggregator on the planet not update their listings to reflect that?

This isn't the first time that Neigher's reviews have come under fire. In 2010, the developers behind Monday Night Combat publicly questioned if he had played the game before writing a review for 1up. (I reached out to Neigher earlier today to see if he'd like to give his thoughts for this story, but I haven't yet heard back.)

And a glance through some of Neigher's old work shows some rather bizarre choices.

"And if you're all butt-hurt about not having the same stuff as the kids across the Pacific, I hear you, but please believe me when I say that if you allow that butt-hurtedness to prevent you from buying and playing this game, you fail at life and will never have sex. Yeah, I said it. Because it's true. Don't say I didn't warn you," he wrote in his review of Yakuza 3.

We'll continue to follow and update this story as we hear more.


    Well this is embarassing... it shows a big design flaw for Metacritic in that they won't remove flawed reviews? Not suggested flawed reviews but factually flawed ones. Damn...


      But let's be honest here, the entire system is broken. Subjectively scoring anything out of 100 is always going to be in essence, flawed. The only thing that makes Metacritic valuable as a resource is that everybody has to work by the same rules without exception. If you go and tinker with the score then you leave the whole thing open to a never-ending review process...

      God just thinking about having to constantly review and update scores based on how many nerds take the time to comb through reviews searching for factual inaccuracies is actually hurting my brain.

      I reckon it's more of a flaw in how Gamespot go about doing their reviews.

        Absolutely nailed it. Metacritic are not to blame here.

        I worked on a game that had good reviews all round - except from one prominent site who gave the task of reviewing our game to someone who began with "I hate this genre of game".
        Killed the average.

        What you say is true, but it's Metacritic policy that is hurting the devs now.
        Gamespot retracted the invalid review. Although - Gamespot should really make the old link go to the new review - that should begin with an explanation for the current situation.

          But metacritic have a fair point, too. They have to ensure that there's no way for publishers to pressure sites into changing reviews, even if they do it under the guise of "factual errors". It's up to the site to ensure their review is accurate before they publish it. If they open that door even a crack then some publishers are going to try to kick it wide open.

            Also I would add that it isn't Metacritics fault that publishers insist on using them to decide payout bonuses. Metacritic does what it set out to do, and it bears no responsibility on what others choose to do with the information they provide.

            Last edited 16/11/12 12:51 am

            Then again Metacritic have now essentially admitted that their score for Natural Selection 2 is inaccurate.

            The bottom line is that Metacritic should only be concerned with providing their readership with accurate meta scores. If their own policies prevent them from doing that then why should any of us trust their scores?

            I mean, hell, how many times has something similar to this happened and we simply didn't read about it on Kotaku? And how is a score being lower than it should be due to poor reviewing any better than a score being higher than it should be due to publisher interference? Both are untrustworthy scores.

        That goes without saying, but by Metacritic not putting the new review in instead of the old, it's polluting the score knowingly, this is a flaw in the system.

        The way the Metascore is "calculated" is flawed, basically GameSpot has more influence than other review sites.

        NS2's Metascore currently sits at 79 when 7/8 of the reviewers gave it 80 or over.

    I have nothing to base this on except what was written in this article, but the only review-score effecting error looks like the mention of load times and I can't imagine that would make a 20 point difference.

      Someone on Reddit posted the original review as a comment.

      It was apparently based on a single attempt at playing on a low-spec laptop and (as someone who has not played NS) seems to show a lack of understanding about the game.

    Oh but it DOES account for situations like this.. this is exactly the point.. GameSpot dropped the ball, they didn't check the freelancer's work. They just assumed that everything they had written was factual and accurate (or more importantly, not how GameSpot would have wanted the game's review to be painted). Now that GameSpot have identified the error, through external sources, they want to change the score... the score was given, GameSpot acknowledged and published the score... now they want to change it... maybe they should only employ people who will review games in the way they want in future. Lesson learned.

    Last edited 15/11/12 3:22 pm

      That's all good and proper except Metacritic gives GameSpot more weight over Metascores than other review sites. So they're actually both responsible, and both owned by CBS funnily enough.

    I guess gaemz jernalism is broken again... tch, it only lasted, what, 2 or 3 days?

    Just going to point out that when this same situation happened with Neverwinter Nights 2 (1up posted a review that gave the game a 5/10 and later took it down and replaced it with a review that gave the game a 6/10), Metacritic replaced the original score with the revised one. Seems like Metacritic's policy on revised review scores only applies to games released by small developers rather than major publishers.

    "Kotaku is not listed on Metacritic, as we do not use review scores"
    For this I thank you.
    The only reason I ever look at Metacritic is to read funny whining reviews.

    Everyone seems to be overlooking the fact that GameSpot has more influence on the Metascore than the other review sites.

    The original review that GameSpot pulled scored 60, while the other 7 reviews scored 80 or over, yet the Metascore is 79. It doesn't take a mathematician to figure out how much weight GameSpot has on Metacritic and it's likely because both companies are owned by CBS.

    If that 60 score was treated as equally as the other 7 scores the Metascore would obviously be 80 or higher and this wouldn't be such a big issue.

      The average of 91+90+85+80+80+80+80+60 is 80.75, whereas Metacritic gave it 79. It's not 'obvious' that the score would be 80 or higher if you knew anything about maths, and rather than Gamespot being given more weight, it's actually the top scorers, and AusGamers, who as Metacritic says average around 2.5 points higher than other critics, have been slightly adjusted down in weighting.

      You're right, it doesn't take a mathematician to figure out how Metacritic generates its scores, and sir, you sure ain't no mathematician. The real question is, why do you - as an individual consumer - care about Metacritic scores in the first place?

    As much as I understand why they do it, the nature of games these days is that the game you have several months down the track may be completely different to the game reviewed at release. Especially with MMOs. What happens then with the one shot policy? Do you forever have a review written for a game that no longer exists in that form?

Join the discussion!

Trending Stories Right Now