Something Is Wrong With Video Game Reviews

A month or so ago I was interviewed by ABC2's Good Game on the topic of video game reviews and, more specifically, review scores. You can watch their feature on the issue tonight at 8.30, but I thought I'd take the time to outline some of my own thoughts on the matter. Something seems to be wrong with video game reviews and, personally, I'd like to see them change.

By most measures Super 8 is a good movie. I thoroughly enjoyed it. My wife cried at the end, while I thought the ending was a teensy bit underwhelming. My friend, who watched with us at the cinema, complained afterwards about the plot. My brother in-law wondered why every single shot in the entire movie had lens-flare - a legitimate complaint.

At the moment, on Metacritic, Super 8 is sitting at around 72. A resounding success - an apt reflection of a well made movie with a handful of flaws.

Compare that to L.A. Noire – a game I did not enjoy. My wife thought it had an interesting concept. My friend loved it and is currently on his second playthough. My brother said it was the worst game he has ever played and has since traded the game in.

At the moment L.A. Noire is sitting at 89 on Metacritic. This is not an apt reflection of opinion. Not in my experience, at least.

There is something wrong with video game reviews.

Roughly a month ago I was interviewed by the TV show Good Game about MetaCritic and the issue of reviews – why do video game reviews skew so highly when compared to music or cinema. Are game journalists pressured by publishers to give high scores to mediocre games, have I ever been excessively pestered for giving a game a low score, what have the consequences been... stuff like that.

Good Game airs tonight at 8.30pm on ABC2, so I’ll leave it to the show to answer those questions – but it occurred to me during the interview that the act of reviewing a video game, based on a number of factors - some within our control, some not – is a broken process that needs to be reworked from the ground up if they are to retain any real value to consumers. Because, in their current state, video game reviews are practically worthless.

I could spend hours going into specifics – the publisher/editorial relationship, the price of games, the expectations of readers, the language we use, the curious hivemind thinking that plagues MetaCritic – but mainly, ultimately, I think the issue with reviews is one of perspective.

We have to start treating video games as an experience. Not a product.

Games are inherently more complicated than any other medium. There are far more parts holding the experience together compared with any other medium you could name – mechanics, visuals, design. With that in mind I think reviewers, and gamers in general, have been more likely to discuss video games in the same way they would discuss a TV or a brand new car. A game is ‘polished’. Mechanics are ‘fundamentally sound’. Reviews are scientific almost – the discrete parts of the product are taken apart and discussed using the same language used to review a brand new toaster.

But a game is not a toaster. It doesn’t perform a function in that cold clinical way, it won’t help you make a delicious sandwich – it is a video game, it is an experience.

If L.A. Noire was a toaster – I would recommend that appliance. Thoroughly. That toaster has had plenty of money invested in its development. It is fundamentally sound. If I were to take that toaster apart it would be made up of the finest components money can buy. What an incredible toaster L.A. Noire would make.

L.A. Noire is not a toaster, it is an experience. It’s entertainment - it’s a video game. But when we discuss video games there is very little attempt to engage with it as such. Why aren’t we asking these questions: does it make sense? Is this engaging? Is anyone having any fun here?

Because at the end of the day, L.A. Noire is not a toaster - Super 8 is not a toaster. They’re both subjective experiences. But the difference is this: I could walk out of the cinema, hop onto Rotten Tomatoes and find a million different reviews that reflect that experience. I’d have a far more difficult time doing that with L.A. Noire.

I didn’t enjoy L.A. Noire. It’s probably unfair to use that game as an example, but look at it from this perspective: ex-Gizmodo Editor Nick Broughall thinks it’s the game of the year so far. So do plenty of my friends. My brother? He hates it with the fury of a thousand angry suns. You probably have your own perspective.

But that’s the point - the video game reviews that you and I read do not reflect that variety. They skew towards an almost clinically similar conclusion, almost universally. That makes them practically worthless to you and I as consumers.

There is something wrong with video game reviews.

And I’d like to see them change.


Comments

    Out of curiosity Serrels, the LA Noire user review on metacritic is 7.4 - is this more in line with your expectations?

    http://www.metacritic.com/game/playstation-3/la-noire

    I've been giving a lot of thought to the user ratings in metacritic as of late - they seem to fit more in line with my experience of the games in question.

    How do you view the user ratings system? Do you think this 'crowdrating' method has more potential than game journo reviews (which often curiously have the ability to be skewed or biased)?

      I find user reviews to be far too filled with outrage to be fair, really. They're often either giving 10s to something because it got a a, trying to compensate, or they give 1s for the same reason.

      Now, there should be very few 1s, and probably even fewer 10s. I find the information in most critic reviews to be more helpful than the final score.

        I've found some reviews are hit and miss with the information they portray. Some reviews have missed glaring problems or issues or even concepts, and I've found myself playing the game going 'how did they miss this?'.

        While I agree that crowdsourced reviews are mostly a) nerd rage or b) fanboys, the end score from this amalgamation of geekery more closely reflects how I feel about a game than the combined official reviews. It's also a handy source for finding any glaring issues, flaws or so forth with the game that may have been 'skimmed over' by the reviewers (for whatever reason).

        Therefore; go to official reviews for information about the experience, then go to the gamer reviews for the value placed on the experience and anything glaring(just take a tablespoon of salt with you).

        As a species we enjoy classifying things. I don't think we'll ever get rid of a ratings system but we need a more meaningful and broad way to demonstrate a subjective experience to people of varying tastes and interests.

        The question is how we go about this?

          For me it depends how you view the data on metacritic. If you read the reviews, yeah, your only going to get hyperbole and opinion from the individuals writing them.

          But metacritic isnt about individual reviews, its about the 'meta' of those reviews - you know thus the name.... Its about getting a quantifiable value for the quality of a product from the qualitative user reviews.

          Long story short, the user scores are actually quite accurate - once you get enough user reviews. The maths works something along the lines of, skewing the averages between 1s and 10s to find a mid point where the score can 'settle'. So you will have all these people giving different data in different forms, but once you have a broad enough sample (see enough reviews) you begin to simply get the broad trend of these reviews.

          The one falling down, is that over a longer time period (years, not months) these scores tend to decay, as newer reviews are less positive. This is mainly a factor of how well the game 'ages' as people buying it after release will tend to compare it to the new, rather than its 'peers' (stuff released at the same time).

          On the topic of differences vs film review, the obvious answer stems from GearBox's comments Re: Duke Nukem Forever.

          Film reviewers are respected for their opinions, and present opinions as such - a qualitative perspective on something given a quantifiable value, with full understanding that it is 100% subjective. Moreover the power dynamic is different - reviewers are viewed as peers of the cinema industry and getting the respect of a reviewer with no bias is basically a requirement to be taken seriously.

          As evidence of this have a look at http://www.metacritic.com/movie/cars-2

          The user score and reviewer scores are almost identical!

          Now look at games reviewers. Games are always given higher marks than the user reviews tend to lean towards. Why the higher scores?

          Two choices, one is the review method is off, two is that reviewers are bought. Answer would be a combination.

          There is more than a bit of evidence that reviews are bought, GameSpot and Jeff Gerstmann, the whole GearBox telling reviewers to play nice go elsewhere, but that can happen in movies.

          As Serrels said, its about a review of a product where your attempting to quantify the whole way down, examine the pieces.

          Using the toaster analogy, all the parts that make up the toaster are exquisite. But Mark missed the ONE part that is the MOST important. DOES IT MAKE GOOD TOAST!

          You can get a platinum coated toaster with diamond encrusted knobs that gently serenades you as it cooks. But if it makes bad toast, whats the point?

          Mark said that he would buy the La Noir toaster, because the parts were great. But according to his opinion the toast wasnt worth it, that right there is the crux of the matter.

          So what can be done? Its all down to the reviewers, looking at how they review, what they say. Then ignoring this and giving the whole 'gut' feeling to the review. XD

        This is spot-on. User reviews are far too often emotionally motivated - people either want to tell everyone how much they absolutely love or absolutely loathe it. There's very little room for mid-range reviews. At least a critic has some kind of integrity and responsibility to a larger publishing entity.

        Yeah, I agree.

        I generally find them to be unreliable and unrepresentative. They attract the extremes of opinion and often don't at all reflect mainstream attitudes to the game or movie.

          Does anybody know how metacritic calculates that user review average? Is it just a straight out average of all the scores submitted? Or do they try to filter out the worst excesses at either end e.g. exclude the top and bottom few percent and take an average of what's left?

    You make a fine point there, Mark. I have often thought the same of video game reviews being reviewed with greater reference to the technical aspects of a game rather than the experience of playing it. Sure, I can understand that if the technical makeup of a game hinders the experience, then it should rightly be scrutinised, but there is far too much emphasis on it at the moment.

    The opinion of the experience of playing a game, as a whole, is highly subjective. I think we can all agree on that. Therefore, if reviews did start focusing more on these aspects of a game, rather than simply the technical, it could create an even larger divide between the highest and lowest scores we see on Metacritic. Though, to be honest, I can't help but feel that this would be a much more accurate reflection on the enjoyment of a game. Judging a game as a whole, rather than the sum of its parts.

    I give this article a 6.7 out of 10.

      Thanks man. On Metacritic that means 'generably favourable'. I can live with that.

        Awesome :).

        only in the movie department mark.

        in the game department it's merely mixed or average since they generally only use the upper 25%

    Isn't that why kotaku invented the likes/hates system, to review games based on experience rather than mechanics/sound/design.

    Maybe video game reviews need to change. But as experiences and impressions vary with every user, the best a subjective reviewer can do is score based on measurable features and their own experiences and let the community do the rest.

      I've always liked the way Kotaku does game reviews, at least in the way it dispenses with a pointless numerical score.

      There's a risk that disassembling a game into 'Likes/Dislikes' still weighs it up clinically without really trying to appreciate the whole experience.

    I use review scores as a very loose guide only. I'll play, watch, listen to and experience things I want to (and can afford to) and come to my own conclusion.

    Some things which are now considered great were once considered rubbish. Some things that are now considered rubbish might one day be considered wonderful.

    You can talk about what you like and don't like in a game, but you can't really put a game to a definite scoring system.

    I really like the setup of TotalBiscuit's first impressions of games "WTF series", it let's you judge whether you think you'll like a game or not by showing you the early sections of a game then him giving his opinions on what he likes and does not like.

    Hi Mark - I had to stop and re-read your sentence:

    "We have to start treating video games as an experience. Not a product."

    At that point in the article I had concluded the exact opposite - that the subjective experience of a video game varies so much for everyone, which causes problems with ratings (the rest of the article goes on the explain what you really mean though regarding technical specs != good game).

    Given that games are designed to produce different experiences, how could the average punter believe ratings based on the reviewer's subjective experience?

    I wonder if it's the composite. aggregate score that's the problem. What about a score for visuals, a score for glitchiness per gameplay hour, a score for story, a score for...

      That is how a video game review should work...

      Interesting, because yoyogames.com, a amatuer game development site, actually has that system.

      The only problem is that people will put 1 star in the story a lot, even if it's a game that doesn't need a story, like Proun.

    So basically... a Rotten Tomatoes, but for video games.

    I think the system is alright as it is, but a rottentomatoes-ish site would be a great additional resource to make choices as to what games to play.

    I agree so much with you Mark. I've been thinking about this for a fair while and you pretty much said everything I was thinking. A game that has a 90 rating could be a game I seriously hate. Case in point for me is Duke Nukem Forever, copping a lot of flak but it's something I'm really enjoying despite it's flaws. Who's to tell me a game is crap and I shouldn't play it? That's my choice and my choice only.

    Horses for courses!!!

    @Mark

    1.How does your vision differ from 'New Games Journalism' as espoused by Kieron Gillen (and others) around 2005? It seems to be the pretty much the same thing. I suppose I'll have to watch to find out.

    "New Games Journalism rejects this, and argues that the worth of a videogame lies not in the game, but in the gamer. What a gamer feels and thinks as this alien construct takes over all their sensory inputs is what’s interesting here, not just the mechanics of how it got there. Games have always been digital hallucinogens – but games journalism has been like chemistry, discussing the binding reactions to brain sites. What I’m suggesting says what it feels like as the chemical kicks in and reality is remixed around you."

    2. Your proposal retains the major problem that reviews aren't a good buying guide, as you still have to know if you and the reviewer have similar taste to know how to weigh their review. How do you propose to solve this?

    Skim read and kept seeing the word "toaster". Article made me hungry.

      Toasted Waffles anyone?

    Something the Extra Credits crew over at Escapist Magazine brought up recently is that more often than not game reviewers will describe the experience, then apply a score to it. Not why they enjoyed or didn't enjoy it, not what it's similar or dissimilar to, or how it did/didn't affect them - just what it is.

    "You kill vampires - 7/10"

    Whereas is film reviews the reviewer is more comfortable comparing a film to other, similar films in the same medium with the assumption that the person reading/viewing the review will have some familiarity with these examples and say "Ok, well I liked A, so maybe B will appeal to me!".

    While I see the merit is such a system, I also immediately spotted the big flaw: fanboys. The main issue with video games is that it's still an immature industry largely populated by an immature audience. Sure, the average gamer is 30, but that's a whole lot of under 18s with a few 70 year olds dragging the average up. Us (allegedly mature) late 20s to early 30s gamers are still outnumbered by screaming 10 year olds, arrogant 16 year olds and 12 year olds who just discovered how to swear and ask for boob pics. I KNOW that doesn't describe all 10, 12 and 16 year olds, so if you're 10, 12 or 16 and think I am being unfair, be assured I am probably not referring to you.

    The point I am getting at is, if you were to say, put forward the hypothesis that someone who enjoys Modern Warfare 2 might enjoy Battlefield Bad Company 2 because they look and play similarly, you would have a handful of people who'd say "you know what? That makes so much sense I'm going to have sex with it follow that advice and buy the game I have not tried yet.

    You would also have a shitload of other people who would say "NO WAY, BAD COMPANY 2 IS WAY BETTER, IT HAS DOOBLEWHATSITS AND GONZJIBBERS. MODERN WARFARE 2 IS JUST A HERPY PILE OF DERPLE DOOP."

    And vice versa.

    This is why game reviews end with some arbitrary score. The reviewers whether deliberately or unwittingly disconnects him/herself from his/her words of derision/praise and says "this is the value I place on the experience".

    Now you still get a backlash from someone who is on his second pristine in Modo Waffle 2 on day 1 because he thinks Module Whacker 2 is a 15/10 and some jerpy tortlebort gave it a mere 8/10, but we've become a lot better at writing this off as a difference of opinion. Also, we still seem to believe 7/10 isn't a bad score so we don't rage too hard - only when a game is subject to universal vitriol does a reviewer dare to venture down towards a 6 or even a 5, lest he get fired. Only the very worst games like Duke Nukem Forever will be subjected to the misery of a 3.5, which is a step up from Big Rigs Over the Road Racing, but a step down from Pimp My Ride.

    Realistically though, we're beginning to realise that when a reviewer things something is crap, they slap a 7/10 on it. So if we actually like a game that fairly earns a 7 because it's not an 8 or 9, we assume the reviewer thought it was crap anyway.

    The way games are reviewed has to change, frankly I don't think there's a system that will actually work.

    You can watch the good game feature, it's live on their site

    http://www.abc.net.au/tv/goodgame/

    I actually prefer a review system that lists the highs and lows rather than the review score so my expectations aren't skewed by the magic number. I'm not sure we should be so precious with games to review them differently to any other medium though. Yes games are an experience, so shouldn't we rate them on that as well as the underlying game mechanics? I agree with the point about the clinical dissection of games, they're not toasters as you say, and they should be reviewed for their ability to be a game. The reviewer's experiences with the game should be distanced from the critique and the focus should be on "is this enjoyable regardless of my own preferences?"

    Sorry but I have to disagree with you here. Reviews are a guide that should help you decide whether you might enjoy a game or not, not a method to tell yo whether a game is good or not, which is almost impossible as experiences are subjective as you have shown with your L.A. Noire example.

    This is true across all media, a good review will talk first technically using the language of the medium and then further expand on this as to whether the reviewer enjoyed it or not stating the reasons why.

    The power of this is that over time people will tend to form affinities with particular reviewers as they tend to find a similar fit to their tastes. This is not limited to traditional reviews as I find the same thing happens amongst my friends as some of enjoy particular types of games.

    For this reason I completely avoid aggregator site sites Metacritic as they basically dilute the views of the reviewers whose opinions I care about. While are agree that there are issues with the game reviews in general (publisher/editorial relationship is a huge issue) I do not think that the solution is to start from the ground up with a new way to review games but to provide an alternative to the aggregator sites that better connects people with reviews that will matter to them more.

    OK.

    So you've delivered your verdict on videogame reviews.

    They suck, and that's cool.

    So, what are you going to do about it?

    You, out of all the people who read Kotaku, are in the best position to change things, being the editor of a videogames website which frequently and obsessively refers to reviews and scores.

    So, what are you going to change things?

      Fair call. On the other hand, we don't do reviews here.

      Would you guys like to see Kotaku Australia reviews? I mean proper ones?

        I have a fuzzy memory of Wildgoose doing reviews, I think.

        Or at least, he gave his opinions on new releases, sometimes through a "first hour" log of what happened in a game and what he thought about it.

        I'd love it, personally. Even the comments section would be a great place to crowd-source opinions on what worked and what didn't, given the calibre of people who comment on this site :P

        Assuming you mean without a score attached - Yes. But not at the expense of the investigative / quality news articles you've brought to the site.

        YES.

        I would also like to see some comment about the "Australian" experience. I mean specifically when a game has a multiplayer component - how's the experience for an Australian gamer, taking into account latency and the obnoxiousness of hearing "rocket" pronounced "racket".

        However, you do frequently refer to other reviews and review scores, posting links from the Kotaku parent website which mention game scores.

        The Critical Reception posts with graphs showing the scores from the bigger sites comes to mind.

        And sure, the Kotaku reviews don't have scores, but are still rather traditional, otherwise.

        It's hard to consider change though. I know this comes topic pops up for us at xboxworld.com.au every few months. My last review gave up all pretence of being a review, and just analysed why movie tie-ins are so bad, considering I've been involved in making many of them in recent years :D

        But ultimately it comes down to the audience. I write completely different reviews for XBW than I do for TimeOut Sydney. Feedback suggests the audience is appreciative of both approaches, and if they become less so, I'll have to change my style.

        But having said all this, I would like to see Kotaku AU reviews :)

    Problem is that unlike anything else.

    Video game producers put to much stock in the review score and as a result the metacritic average.

    They seem to view the number as all thats important from the review even though in most cases the review score is detached from what is actually mentioned in the article.

    With some games reviews mentioning no flaws and getting a 80% then another article by the same writer with game flaws pointed out in the review getting 90% with no clear distinction as to why there was any real difference.

    This wouldn't really be an issue if it wasnt for the fact that those 10 points apparently determine the success or failure of a game.

    We have a system that for some reason sits on the cusp of a maximum score so that when something truely groundbreaking or deserving of the distinction a 5star rating would show it in the film industry instead it's lost among the rest of the games that scored in the 90+ range either due to fandom or bought reviews

    I find that I ignore the score and read what the reviewer actually says about the game. If they sound like they have similar tastes to mine, I can usually trust their judgement. That is, when I read reviews at all; I prefer to make up my own mind, play the game and then read the reviews. Same with movies, really.
    As for the trend of extreme ratings, maybe its because with most games, the experience lasts much longer than with a movie, so your opinion becomes more extreme one way or the other due to the extended exposure to the game. Ie. If you don't like a game, playing it longer will likely make you like it less; if you are enjoying a game, playing more of it will make you like it more. Although, come to think of it, you rarely, if ever, see reviews in GameInformer (as an example) with a score of less than 50%, so maybe 75 is the new 50?

    I always liked old school Hyper scoring.. A percentage for graphics, gameplay, sound and then an overall.. *shrugs* I don't even really pay attention to scores anyway, I usually come on here and see what everyone's been saying, then trawl googlge for some other opinions. Unless of course it's a game i've been hanging out for, which i'll buy no matter what..

    This might be ever so slightly OT, so in advance, I'll say that I thoroughly enjoyed the article, and I have to agree-- VG reviews need a complete working over. The system just doesn't seem to work right...
    In any case, I just have to say that most of the time, when I buy a gaming magazine expecting solid, researched reviews, I end up with a magazine full of blathering praise, rating every game an 8.5/10 at the very LEAST. It makes me wonder if they get paid to do so, and it really cheeses me off.

    I agree - to an extent. There are games like Shadow of the Colossus where the experience is everything and by not commenting adequately on that then you are missing the best part of the game. But experience, as you told us, is subjective from user to user. Sometimes to make sure everyone understands your review or your opinion you have to throw in generic descriptors like "great combat" or "awesome soundtrack" (note the lack of jargon). While you want to convey your personal experience you also have to touch on a games broader appeal.

    Essentially I agree with you that reviewers are often more clinical (something I try to avoid on my site and I'm guessing Kotaku does too) but there is still a place for some of that in a review. It just can't be the ENTIRE review.

    I think your article makes several great points.

    Some reviews do make the mistake (well, a mistake in my opinion) of comparing games to a toaster precisely because they're both commercial purchases and a certain type of consumer wants value for money.

    People buying a toaster want feature x, y and z - some people buying a game want to know that its graphics are this good, or that its multiplayer can do that amount of things.

    Video games are, indeed, an experience - the sum of all of the separate parts - and shouldn't be judged based on the combination of how good their sound, graphics, gameplay, story, longevity, etc are.

    I guess what I'd take from this article, and forgive me if I'm wrong here, is a shift in the way games are reported and reviewed to reflect a lot more subjectivity and, I suppose, a much more nuanced approach to how people found the game. Similarly to how a novel or a film is reviewed, with a much greater emphasis on the opinion of the reviewer and their view of the game in aggregate rather than as a sum of a bunch of technical scores.

    It's definitely an attractive goal, but would developers - and especially publishers - want to discard the Metacritic system, whereby they're assured of a sort of herd mentality and can proudly claim to have a "90+ game", or a "85+" game to their investors?

    The way I see it, sites like IGN in particular are skewing people's perceptions on review scores.

    You see a review from them and its always up there, between 8 and 9.5 most of the time, and when you look at people's comments on reviews, they seem to think any game that gets under an 8 in any review is complete crap and worst game of the year.

    This is somewhat relevant: http://www.hejibits.com/comics/cash-cap/

      exactly and it's why metacritic went out of its way to give video games a different scoring system than any other media.

      a 60 for a movies is considered generally favourable

      for a game to be considered generally favourable it has to get a 75 or higher

Join the discussion!

Trending Stories Right Now