Metacritic Treats Some Outlets Differently From Others. Here's How. [Update]

Industry favourite Metacritic, a site which aggregates review scores, does not simply add up every score it can find and come up with an average. It weights scores, giving some outlets more of a say in the final aggregated tally than others.

While the outlet has never publicly disclosed the criteria by which it does this (on the site's FAQ page it says it will "absolutely not" reveal how each outlet is weighted), Adams Greenwood-Ericksen from Full Sail University, along with some of his students, set about cracking Metacritic's code for us. After months of research, they've finished their work, and according to a report on Gamasutra "their findings were almost entirely accurate".

There are six "classes" of score, with publications ranked "lowest" all the way up to "highest". You'd think that those at the top, those with the biggest influence on a game's score - and as a result a game's critical performance and, in some cases, a developer's pay (or even future existence) - would be the biggest and most influential outlets. The IGNs, Gamespots and Eurogamers of the world.

Nope.

The 29 websites in the "highest" category contain several that I literally have never heard of. One of them is a volunteer fansite. Eurogamer and PC Gamer are only ranked "medium". Giant Bomb is "lower". Edge Magazine, perhaps the most respected review outlet in the world, is ranked beneath Yahoo Games.

Note that the list, perhaps due to the time taken to research it, is a little old. Some outlets on there are no longer in business, while other newer sites like Polygon do not feature.

Metacritic's very existence, or at least its importance in the eyes of many industry types, is bad enough. But if this research is accurate - or even mostly accurate - it only opens the site up to further criticism. Who is determining which sites carry more weight? And how? Does the site keep the weighting a secret because, for a site that's all about hard numbers, the process is completely arbitrary?

When people are losing their jobs over Metacritic scores, those are questions that need answering. We've contacted the site for comment and clarification on the list's accuracy, as well as Metacritic's weighting methodology, and will update if we hear back.

UPDATE - Here's Metacritic's response to the study:

Today, the website Gamasutra "revealed" the weights that we assign to each gaming publication (for the purpose of calculating our Metascores), based on a presentation given at the Game Developers Conference this morning. There's just one major problem with that: neither that site, nor the person giving the presentation, got those weights from us; rather, they are simply their best guesses based on research (the Gamasutra headline is misleading in this respect).

And here's the most important thing: their guesses are wildly, wholly inaccurate. Among other things:

* We use far fewer tiers than listed in the article.

* The disparity between tiers listed in the article is far more extreme than what we actually use on Metacritic. For example, they suggest that the highest-weighted publications have their scores counted six times as much as the lowest-weighted publications in our Metascore formula. That isn't anywhere close to reality; our publication weights are much closer together and have much less of an impact on the score calculation.

* Last but definitely not least: Our placement of publications in each tier differs from what is displayed in the article. The article overvalues some publications and undervalues others (while ignoring others altogether), sometimes comically so. (In addition, our weights are periodically adjusted as needed if, over time, a publication demonstrates an increase or decrease in overall quality.)

Metacritic's weighting system revealed [Gamasutra]


Comments

    That is essentially the point of a good meta analysis (akin to what meta critic is doing). You probably dont want them to weigh no name publications' review more than well respected and reputable ones. That is why scores are weighted. However, it would be better if they gave a weighted and an unweighted score to give more transperancy

      What would be better is if we could stop attaching arbitrary scores to media based on peoples personal views/bias.

      A review should contain 3 things: the reviewers opinion (liked/disliked), a summation of the things they liked/disliked, an unbiased pros/cons of the product (eg. You will probably like it if you like: X,Y,Z, You probably won't is you dislike: A,B,C etc.).

      I don't see how a bias score can help more than the above. I know good reviewers set out to try and give an unbias opinion on what they review, but that is ultimately impossible, if you simply do not like a particular genre of something it would be impossible to give a review that would reflect what someone who loves that genre would think of the product.

      Last edited 28/03/13 11:10 am

        This exactly!

        The whole concept is messed up. You can't assign a quantitative value to something which is simply based on a persons subjective perception of it.

        The fact that a game will get two different scores from different people, this itself makes the entire process totally redundant in itself. If scores were appropriate, then a game would get the same number regardless of who gives it, thus making reviews unnecessary .

        Agreed. People are much better at making binary decisions. The 'scale' is basically useless when your talking about subjective experiences.

          Well that is it isn't it, it IS a binary decision, do you see the movie or dont you, do you play the game or dont you. it isn't do you watch half the movie because it only got average reviews, or play the first levels because it got average reviews. Once you have paid your entry fee the reviewers part is done with.

        I really don't think reviewers set out to remove bias from their reviews, at least not entirely. Many reviewers accept that we can never, ever give an unbiased account of something. Good or bad, every decision is influenced by our tastes and beliefs. The problem with the internet, ironically, is that people make simplified accounts of everything based on their own bias without a hint of self-awareness. Another problem with the gaming community in general is that there is essentially no room for interpretation. For a gamer, there must be an answer or they will hate you. Put these together and you get the ridiculous opinions we always see about reviews. People are paid off, they don't understand the game and reviews should be a structured report like you listed above. Reviews are very much opinion pieces as apposed to the government-sanctioned buyers's guides like people seem to make them out to be.

        I mean reviews are meant to be read: correct? We agree.
        Reviews should be interesting and insightful: i think this is where we disagree.

        The best reviews are written by people who know the genre, you'll get a guy or girl who loves shooters to write a piece on CoD and someone who likes platformers to do Mario. Sure, it adds huge bias considering you could have chosen someone who hates the genra and ended with a different result but you'd get very little insight or reward from your reading experience. I'll always believe it's up to the creator to decide what they want to say with their work, not the masses. In that vein, metacritic is cheapening creative work by essentially telling people what they mean by their own review and assigning them what is essentially an arbitrary number.

          I am not saying reviews shouldn't be lengthy or enjoyable, I appreciate the work a lot of reviewers do, going to length about games.

          But suggesting someone who likes this genre does this review, someone who likes that does that one is extremely rare, unless it is dedicated to that medium most (like here at kotaku), such as newspapers, generalist magazines, or magazines about different media (such as rolling stone doing game reviews) hire a single person to review all games.

          This still doesn't defend the need for scores, it isn't metacritic forcing people to give scores, you have to give scores to be on metacritic, kotaku doesn't get aggregated because it just gives yes/no on games. All metacritic does is assign a weight to your score, then they are telling you how valuable your opinion is, it is wrong, but could be easily fixed by just not giving scores.

        Thats the thing though, opinions are bias (to the extent that they are subjective). People's opinions on a game will be different. Even two people who like the same genre or series. It isnt just a binary decision per se. That is an oversimplification. It IS binary in that you either buy it or you don't. BUT the extent to which you like it is not necessarily binary. The scores are an attempt to reflect *how much or the degree to which* someone might like a game, not necessarily whether they should buy it or not. The scores are an attempt to show you will like the game about 'this much' if you like pros A, B and C and if dont mind the cons of X, Y and Z.

        A meta analysis attempts to dilute some of it by pooling all the scores together. Assigning different weights to the publications may appear to be making the whole process redundant (as weightings themselves are subjective), but that's why I said it'd be important to give a weighted and an unweighted score. That way, we get the transparent unbiased unweighted pooling of scores (as well as the weighted score) and they dont reveal their secret formula. We get more infomation and everyone's happy.

      I think Rotten Tomato's meta-analysis of critic's responses to films is a far more accurate way and seems to remove the requirement of weighting some reviews more than others though it is perhaps a little too binary for the industry to use as a benchmark of success.

    " if, over time, a publication demonstrates an increase or decrease in overall quality."

    Who the fuck determines "quality"? So in reality, Metacritic is not a review aggregator. It is reviewing reviews.

    This means that it is just as open to being bought as any other reviewer. For shame!

      There are some ways you can objectively measure the quality of a publication's scores. If you take any pair of titles, you will expect most publications to rank them in the same order. While you'd expect a publication to disagree occasionally, if it does too often then it isn't a good predictor for the general opinion. It would be legitimate to say the publication's scores are low quality for that purpose.

        So essentially you are using volatility as a predictor. I still disagree with the concept that Metacritic should be used as an aggregator by the industry when the process involves manipulating the weighting given to publications according to formulae that aren't made public. Metacritic is just saying 'trust us'.

          I'm not a fan of the algorithms being secret either. But I can understand why Metacritic keeps theirs secret: if it were public, then they could be replaced.

          I do find it a bit weird that they manually split publications into fixed tiers though. It would seem fairer to either treat everyone the same, or have everyone in their own tier selected algorithmically (e.g. based on how good a predictor they are).

    I think Edge is probably weighted low because its a consistent outlier. This is because they use a rating system where 6 actually is better than average.

      Yeah I'm assuming most of the weighing is based on standard deviations from previous averages. I am just assuming though.

        In our defence, we kind of have to because they won't tell us this stuff at all. Kind of pisses me off that companies would tie their bonuses to a system that isn't transparent.

Join the discussion!