game design
The Expert Bias: Reviewing for a New Culture
Posted by Maggie Greene at 3:30 AM on July 20, 2008
![]()
Danc at Lost Garden has another take on the utility (or lack thereof) of game reviews in today's gaming landscape, this one looking at the 'expertise bias.' He points out the disparity between reviewers and players when it comes to looking at new games — especially ones that have a gentle difficulty curve. His basic operating premise is that because game reviewers have plowed through so many titles and mechanics, they're looking at 'difficulty' in an entirely different light than vast portions of the audience. What will the future look like? He posits observation of other players is going to become increasingly important to developers, and if reviews can't keep up with that, they will really fall by the wayside:
If you are serious about providing objective insight into a game, either a title you are building or one your are reviewing, your expertise is not enough. In fact, your vast mastery of game related skills is mostly likely causing a giant bias in your judgments. You need to fight this bias by observing other players over and over again. They will do things with the game that are a source of wondrous insight. Your expertise becomes a tool for making great changes based off these insights, not one for predicting a priori exactly how all users will react to the game.
As for the current review industry, it is built on the unstable foundation of expert opinion in the absence of actual player observation. As games evolve and become ever more about first time learning experiences, the traditional game review will become increasingly irrelevant. It is arguable that they've already stopped informing most buying decisions and now serve as little more than entertainment for the hardcore niche. As the value proposition of reviews falter, the vast, churning, capitalist forces of creative destruction will replace them with a much richer set of game criticism that offers real value to its readers.
We've heard a lot about why the reviewing structure is broken, but this is an especially thoughtful take on the problem. I'm not sure it's one that can really be gotten around (critics — of the game, film, food, or book variety — tend to get those positions by being 'experts'), but it makes for interesting reading to be sure.
Soul Bubbles: A classic game ill treated by expert reviewers [Lost Garden]

Comments (AU Comments · US Comments)
Jerrod
Posted September 18, 2008 7:59 AM
I wholeheartedly agree with this post. Actually, for the past few months, I've been trying to develop a website that's based around trying to get "reviews" (if you could even call them that) away from the "expertise bias". It's called GameMagi. You might wanna check it out if you're curious. I can't really sufficiently describe what it does here.
Raian1
Posted 3:59 AM 20/7/08
At last! Someone took notice of Soul Bubbles, a game that deserves much more attention than it has so far received. And it's a perfect choice for a topic about difficulty, considering that to most expert players, Soul Bubbles will be easy.
But that said, not every game is meant to provide a challenge. As Miyamoto once said, the playing of a game can be fun regardless of challenge and reward, and Soul Bubbles certainly does that. The bubble physics, the artistic style, the sheer variety of puzzles and obstacles, it's all so much fun just to play.
Raian1
Triple8
Posted 3:58 AM 20/7/08
A good reviewer tries to be objective but there's an elephant in the room that this article is ignoring: it is impossible for a reviewer to be completely objective. No matter how hard you try, your biases will influence your opinion.
Every review is and will always be an subjective opinion. They should try to get rid of any biases they have, because the ones who don't fail to recognize how egotistical it is to think people want hear about their biased opinion.
The best thing reviewers can do is make their tastes very clear to the reader. That way readers can know if their tastes are similar to that of the reviewer.
Triple8
omnibot2000XL
Posted 3:57 AM 20/7/08
Reviews are worthless. They are written by people who a) get the game for free, instantly devaluing it; b) are not usually self-selected to be primed to like that kind of game (eg if I buy a wrestling game, chances are, I like wrestling games. When a review starts out "if you like seeing lots of sweaty men..." chances are the reviewer has an inherent bias against the sport and the game; c) necessarily need to power through the game as fast as possible, thus playing it in a way that few players will.
Thus, the reviewer's experience is almost always totally different than the actual gamer's. Today, demos, screenshots, movies, and ESPECIALLY word of mouth are what matter, not if the GamePro guy gives it a "Smiley Thumbs Up."
omnibot2000XL
Al3xandr0s
Posted 3:51 AM 20/7/08
Also, they can most certainly affect gamers' views. No, nobody (sane) just buys something because a review rated it good or doesn't because a review rated it bad, but they still check the overall vibe all the reviews together give, though they're more likely to believe a friend's opinion of course. They also have just as much experience (if not more) as any professional reviewer, and wouldn't consider tough something they find easy or vice versa. Since when is the difficulty curve something that can make a grand game bad or a shitty game good anyway? Sure it can be a negative/positive aspect depending on how it is, but it's certainly not a major factor of the game if the actual game is good/bad and worth/not worth the trouble of the initial difficulty... Unless it's just so tough it's a broken game of course.
Al3xandr0s
Ichidou
Posted 3:51 AM 20/7/08
You know, they really bring about a good point. Considering that the core audience for things like the Wii and DS are far greater than that of the hardcore gamer, it's never dawned on me that reviews for games for these people would be skewered by the judgement of game reviewers.
Now that the point has been brought up, I would personally love to see the editor's take on the game, plus give the game to someone else (Not another editor) and let them go hands-on for about an hour or two and see how they feel. Chances are that the casual gamer won't invest nearly as much time and energy into a videogame as someone that frequents this site, for example.
Ichidou
mikes_stalker
Posted 3:48 AM 20/7/08
Quite an interesting article. I really do believe that game reviewing has turned to ignoring the true base of reviewing and more about trying to appeal to the base.
I remember back in the days when game reviewing actually recommended renting games or even borrowing from games. Also there is some passion about games they are reviewing, both good and bad. Feels like game reviewing is reviewing for the sake of reviewing. Awesome article you brought up Greene. Made me think hard about the future of game reviews.
mikes_stalker
Al3xandr0s
Posted 3:47 AM 20/7/08
Well, from checking out various magazines from time to time, I can see a lot of those so called professionals often could barely be called gamers... I often spot many mistakes on the history of reviewed games, and most reviews sport the same ol public relations screenshots showing they weren't dedicated enough to take a few of their own. They also mostly go by hype and/or paychecks from publishers and the fanboyism of the current magazine they work for. Most of them are not gamers, just general journalists who couldn't get a job on the field they studied it seems.
Al3xandr0s
Lezard
Posted 3:41 AM 20/7/08
What he says is pretty true. On the one hand, you want to be knowledgeable enough that you know what you're talking about. On the other, if you only take your particularly level of expertise into account, you're not really doing favors for those more unfamiliar or less experienced in the genre than you are.
There are people that consider Ninja Gaiden Black an excellent. These are generally the people that are able to overcome its insane difficulty curve. How do they express their appreciation and admiration of the game to someone that has trouble struggling through it on Ninja Dog difficulty, which contains elements that essentially mock the player for sucking so much?
Lezard
Drake Lake: L'Heure Exquise
Posted 3:40 AM 20/7/08
Are they going to fall by the wayside, though? Most of the 'core' gamers (who obviously have more experience, and can share that connection with reviewers) are the ones who look up these reviews, more so than so-called casual gamers that for the most part are willing to pick up a game based on its premise rather than quality experience.
Drake Lake: L'Heure Exquise
JohnnyLA
Posted 4:13 AM 20/7/08
Movies, food, and literature have the same problem but I don't see any of them "going by the wayside".
Their are going to be two types of reviewers: hardcore and mainstream (a.k.a Ebert/Good Morning America flavors).
JohnnyLA
El-Suave
Posted 4:51 AM 20/7/08
@Raian1:
You are so right - Soul Bubbles really is a great game picture to accompany the article, which is also insightful. Even though it still gets good to great reviews it's always sad when they're dragged down by complaints about difficulty.
I would consider myself very much a hardcore gamer but I totally can appreciate the zen qualities this title posesses and since there is more than enough gameplay in there complaints about difficulty aren't very appropriate.
The problem is that review websites only speak to the hardcore player, which makes reviews not matter much because experienced gamers have their own means of knowing if a game is good with all the media that is provided these days, also in this day and age most major releases are pretty much guaranteed not to be total failures. Therefore reviews often are not very helpful and downright fail in some areas like some Wii and DS reviews. When journalists are always complaining about minigame collections like Carnival Games they are neglecting to remember that we (at least people of my age) all got started with simpler games that were very meaningful back then. I.e. Summer-, Winter- or World games from Epyx were cornerstones of my gaming childhood but when you play them today, they aren't really fulfillg anymore - but they (or their updated versions like Mario & Sonic at the Olympics) might be very relevant and enjoyable to a new gamer, and that perspective is lacking from reviews.
El-Suave
Zerozaki Ishiki
Posted 5:09 AM 20/7/08
Soul Bubbles is, quite honestly, one of the best games the DS has. There have been some downright shocking reviews of it, reviews that make you wonder if they even played the same game.
I can't help but wonder if it hits back to the Penny Arcade argument; Soul Bubbles is not a game designed to be played straight through. Played one level at a time over a few weeks, it works perfectly, but I could see it all grinding together a bit if played in one sitting.
I'm still only about halfway through, but the last couple of levels have been pretty tricky; never enough to frustrate me, but harder than I thought the game was going to get. I honestly liked that it wasn't hard, since it was so relaxing and fun.
Most of the other complaints are nonsense - replayability? Who gives a fuck? It always tells you what to do? Not if you don't click on the hints. Repetitive, lack of variety? Bullshit. The game is doing an amazing job of introducing a mechanic and then adding twists to it. It always has another trick up its sleeve. Again, I have to put this down to play the game in one sitting and getting bored with it.
Play the game a level at a time, and savor it. Because it's well worth a trip to Toys R Us, where Eidos buried it.
Zerozaki Ishiki
lachinay
Posted 5:01 AM 20/7/08
Having been a professional reviewer for a decade (although I am no more), I really don't get the point. It's absolutely acceptable for a title to get great reviews and no sales and vice versa, hell, it's what _usually_ happens in movies, books, anything!
If anything, videogames are slowly moving towards a "normal" situation in which experts review titles, you know, expertly (ie on the basis of their expertise), and the great mass of customers doesn't care. Or, this would be if gaming journalism was made of true professionals, which isn't, for a lot of reasons (one of the most important being pay, which is ridiculous, because evidently the ones who should pay think that reviewing games isn't a real job).
lachinay
rateoforange
Posted 4:59 AM 20/7/08
Whoops. I swear, I should start composing longer comments in Word or something.
rateoforange
rateoforange
Posted 4:58 AM 20/7/08
Separated even from the effects of hardcore backlash, most casual games are shallow, trivial, and poorly executed. Games as a medium may not have literary quality plots, but the gameplay itself is refined to a high degree. Casual games with rudimentary gameplay may have a place in the popular
Because of this, our current crop of reviewers exists to inform people who are into gaming and they do it in forums frequented by those same people, like enthusiast mags and websites. If gaming ever goes as mainstream as movies, newspapers and other mainstream type organizations will get their own crop of reviewers and have different standards.
Jeff Gerstmann should never feel he has to give a pass to umpteenth Wii minigame collection to overcome his 'expertise bias', just like A.O. Scott of the NYT doesn't have to sign off on every piece of popular excrement that gets extruded out of the Hollywood anus. Just because a kids CG movie is popular with a certain audience doesn't make it good, and experts should be allowed to point that out.
More charitably: our current crop of reviewers exists to inform people who are into gaming and they do it in forums frequented by those same people, like enthusiast mags and websites. If gaming ever goes as mainstream as movies, newspapers and other mainstream type organizations will get their own crop of reviewers and have different standards.
rateoforange
akruble
Posted 4:55 AM 20/7/08
Difficulty is always subjective. I note the reviewer's take on the game's difficulty but it has no influence on whether or not I would be interested in the game. Maybe that's because I don't care if a game is "easy" or "hard," but rather that it's fun.
akruble
Nightshift Nurse: Mile High Colonic
Posted 5:40 AM 20/7/08
@EditorinChief: Probably because the games industry as a whole can't keep up with the network of geeks that produce/populate these blogs and message boards who constantly scan the net for any tidbit of news.
/End snarky reply ;)
Nightshift Nurse: Mile High Colonic
EditorinChief
Posted 5:33 AM 20/7/08
It's trite topics like this one that make me wonder why the game industry accepts glorified message board posts as "editorials."
/End snarky gamer comment
Not really true in my opinion. Most game reviews are not done locked up in a hotel some where. That is only some games. Most games go through many people at an office as opposed to just one single person. Also, a reviewer, a decent one anyway will be able to realize things like this one.
EditorinChief
Nightshift Nurse: Mile High Colonic
Posted 5:25 AM 20/7/08
@pandafresh: I would imagine because starting a publication (print or online) of your own takes large investments of both time and money from multiple people. And that the potential (high) risk of failure is not worth everyone's collective involvement.
Furthermore, gamers are frequently so damn fickle and bitchy that getting them to care is an uphill battle unto itself.
Truly, complaining about games seems to have upstaged the games themselves in this last half-decade. Genuine enthusiasm has become a commodity in all corners of the industry - from player to professional.
Nightshift Nurse: Mile High Colonic
ExistentialEgg
Posted 5:25 AM 20/7/08
@Zerozaki Ishiki: I agree. I've been playing games for over 25 yrs. Soul Bubbles is "easy" to a hardcore gamer yet it's my favorite game for the DS by FAR. Beautiful art direction and whimsical design, just a true treat to play.
ExistentialEgg
Nightshift Nurse: Mile High Colonic
Posted 5:21 AM 20/7/08
@Nightshift Nurse: Mile High Colonic: It should read "...And I use the term..."
Apologies for any other typos...I find I really have to hammer on my keyboard to get every single letter to appear.
Nightshift Nurse: Mile High Colonic
pandafresh
Posted 5:19 AM 20/7/08
im just confused as to why so many people talk about how game reviews are wrong, but yet i rarely see anyone try to do anything different.
personally? i think theyre fine to base some idea of what kinda game yer gunna get yourself into.
pandafresh
Nightshift Nurse: Mile High Colonic
Posted 5:19 AM 20/7/08
I wholeheartedly agree with the notion that games as they exist now are little more than entertainment for the hardcore demographic. An I use te term "entertainment" loosely...as I find it has less to do with genuine interest and more to do with stoking various fanboy fires to generate hits/advertising revenue/"street cred".
There's no denying that the structure as it currently stands is broken (primarily for the reason above), but I fear little can be done about it until games wholly filter out into the mainstream and every publication that normally features literary, film, performance, or music reviews offers opinions on gaming as well.
Related to the issue of reviews catering to the hardcores fetishes is the idea of "fair weather" reviewing. We've all seen it happen...a less popular console plays host to an excellent game which loses a few points because of its platform. Though I feel it's become more pronounced in recent years (Uncharted, anyone?), the practice is still as old as game reviews themselves.
Movies aren't faulted for the studio that produces them, likewise games shouldn't be judged based on their intended destination. But until games have moved entirely into the sphere of mainstream consumption this type of pandering (or exercising of personal bias) will continue.
Nightshift Nurse: Mile High Colonic
Servant_of_Jashin
Posted 6:10 AM 20/7/08
what the says does make sense. i for one only read reviews to see if a game that i might buy has a serious flaw like controls issues and or cameral problem. other than that i usually already made the decision to buy the game before reading the review.
Servant_of_Jashin
hagridore
Posted 7:21 AM 20/7/08
I just picked this up yesterday and I'm having a blast. Shame its only available in ToysRUs; it will probably disappear very soon.
hagridore
Nipz
Posted 7:34 AM 20/7/08
Some games i've recently played have been extremely well crafted that recieved really poor reviews. I think a lot of the reviewers find it hard with the casual games, either getting to picky over controls which they go easier on in hardcore games or just missing the point of the casual game. So casual games end up with lower overall reviews. That been said it sickens me the amount of rushed casual games its as if they think these gamers dont give a **** about quality... comon EA and Ubi. Everyone wants quality but not everyone knows how to find it.
Nipz
stevesan
Posted 7:33 AM 20/7/08
Reviews are subjective...in any media. What we need is a diversity of reviewers. The problem these days is that most reviewers are of the "hardcore" crowd. We need people who review games for other possible audiences, like casual and mid-core.
stevesan
Nawara_Ven
Posted 8:10 AM 20/7/08
It works the other way too.
Godhand, for example, is extremely difficult, and I've read reviews that pan it for its limited gameplay... but the reviewer makes no mention of any event beyond stage 1-1.
The reviewer is worthless to the hardcore audience when his lack of skills do not allow him to even experience a respectable portion of the game!
Nawara_Ven
bialia
Posted 8:36 AM 20/7/08
@JohnnyLA: good point. it seems to me that as many people just like to read/watch ebert because movies are something they are really into. the same thing with core gamers reading and writing game reviews.
compared to good morning america's movie reviews, game reviews in newspapers, they're apples and oranges.
@Triple8: relative objectivity is important, but personal bias is inherent in criticism of anything. of course it is subjective. that's not an "elephant in the room." it's assumed to be part of parcel of the whole concept. i don't think it's being ignored. criticism is opinion. it is why people have favourite sources for reviews.
bialia
whalleywhat
Posted 8:39 AM 20/7/08
I actually see a lot of the opposite, where a lot of reviewers seem to really enjoy a game but force themselves (or are forced by their editors) to talk down to some imagined audience of incapable morons.
Every game can't be everything to everyone, just like every games journalist can't. MSM reviewers should reflect their audience's wants and needs and hardcore enthusiast reviewers should reflect theirs.
whalleywhat
Evil Tortie's Mom
Posted 9:31 AM 20/7/08
+1 for both this article and Maggie's bringing it to our attention.
Evil Tortie's Mom
EditorinChief
Posted 2:56 PM 20/7/08
@Nightshift Nurse: Mile High Colonic:
You're right.
None the less, I disagree with the article.
EditorinChief
KRIGBERT
Posted 6:50 PM 20/7/08
I remember picking up Gears of War, my first console FPS since Goldeneye 64, and really struggling to get through it on the easiest difficulty setting - later I read a review of the game complaining that that particular setting was too easy and that it'd ruin the game. The multiplayer is also highly praised - sufficed to say, there was no way I could enjoy anything but a little bit of co-op. As the more immature elements of LIVE repeatedly pointed out, their moms were better at this game! :p
If a game is likely to give that kind of an experience to a non-hardcore gamer, I think a good review should point that out.
KRIGBERT
Gene-ius
Posted 1:42 AM 21/7/08
"It is arguable that they've already stopped informing most buying decisions and now serve as little more than entertainment for the hardcore niche."
As a general rule, this stands. People either go to Metacritic or have a game recommended to them by a friend if they are actually planning to shell out. People read reviews for entertainment. Even the most famous games reviewers haven't reached the level of, say, restaurant or theatre critics: a bad review from one of the top critics in those fields is enough to close a business or shut down a production. However, I doubt even Yahtzee could sway people into buying or not buying a game, and for games reviews and reviewers to have an economic impact they have to come out in force. I honestly can't remember the last time, if indeed there was one, that even a major site like IGN managed to songle-handedly harm a game's sales to the point where the developers really felt it.
For many people, myself included, we go to reviews in order to have confirmation. Generally I have my prejudices towards a game and unless it's alarmingly good or a real disappointment reviews fail to raise an eyebrow, even if they do regularly raise a smile (thank you, PC Gamer). It is a very rare occasion where good reviews will cause me to buy a game which I'm not already considering, though reviews did draw my eyes to both Okami and SSX Blur, both of which I thoroughly enjoyed.
Reviews aren't obsolete because gaming opinion is still valued (and everyone seems to have an opinion on gaming, so we're not stopping that anytime soon), but really, people's recommendations are starting to come from elsewhere. Reviews simply can't compete with marketing or the myriad other ways publishers can get games into people's hands.
Gene-ius
glamnesia
Posted 7:39 AM 21/7/08
This debate about the one true and right way to review video games is a load of old fluff. If there's a demand or an insight into a new way to review games then it will naturally come about.
There is endless room on the internet for different formats and points of view. The idea that one way is somehow superior to another inflates the importance of journalistic criticism.
glamnesia
enix2093
Posted 2:28 PM 21/7/08
This is the exact reason I am creating/heading this: [www.sudr.org]
enix2093
fuchikoma
Posted 1:29 AM 22/7/08
Reviews have been skewed that way from the days of NES game reviews in magazines. Those were not for casual players who fire up SMB/Duck Hunt every month or two.
They become increasingly irrelevant in that form - not for number of readers, which should remain constant or grow - but because more "gamers" are new and casual so on average the reviews mean less.
I don't think this means they're going to be replaced by any means. Maybe paralleled with a more mass market counterpart?
fuchikoma
GingerDadless
Posted 8:08 AM 20/7/08
http://gamebrainspew.com Reviewing games to find the fun.
GingerDadless
FaceF18
Posted 6:58 AM 20/7/08
I think if this is put into perspective with other industries facing similar concerns it seems like less of a big deal. In movies, more artistic movies with better acting and scripts are usually praised by the "hardcore" review group, but may make less money than the big budget summer action movie that doesn't leverage the advantages of the medium to the same degree.
Games are an interactive medium, so the gameplay, and its ability to satisfy the end user, is kind of like the equivalent of the acting or the script in a movie, and it makes sense for reviewers to judge the content from the perspective of someone with experience in the genre and the capacity to play it the way the designer intended it to be played. No one criticizes movies if a grade schooler can't understand the plot, but the movie probably won't generate a lot of sales to grade schoolers. The review industry isn't broken; for any medium, the critical community will inevitably develop in the same way the game review community has.
FaceF18
raiseplease
Posted 9:45 AM 20/7/08
I disagree. I think the point of a critic is to be expert. For example, I rarely play sports games. I could pick up the latest FIFA game and it may enchant me, and I give it rave reviews. But I don't have the history to realize EA crippled the franchise mode from previous games, or that they released a virtually identical game to last year.
When a game reviewer says a game is too short, it's not just him, but his experience of other games in the same genre that gives him objectivity. Otherwise there would be no too short or too long, and we couldn't have reviews at all, because then it would just be bias. It may be very well a good game if you play it for 15 minutes at a time rarely, but so will a similar-themed one that lasts for three times the total duration.
raiseplease
CoreWolf
Posted 4:46 AM 20/7/08
Sure there's problems with reviews, but for the most part they still work. You know if a game is above 70 in metacritic then it's worth looking into, if it's above 80 or 90 you'll most likely love the game if it's of a genre you like. On the other hand you know that pretty much all of the 69 and below games are worthless and you can filter them out entirely. There are of course exceptions but they're few and far between.
And about difficulty - i get the feeling that it's generally only seasoned gamers that bother to check out reviews, and they've usually played plenty of games so there's virtually no disparity between the gamer and reviewer. That point is almost entirely invalid. I also don't believe that many non 'core' gamers read reviews, or if they do they obviously ignore them based on the quantity of horrifically bad titles that still manage to fly off the shelves.
Aside from things like metacritic scores the best way to find out whether you'll enjoy a game is to find a specific reviewer that seems to be like minded and look to him/her. That's what i've done with films for ages and it works great most of the time.
Also: APPROVE MY ACCOUNT GOD DAMNIT!
CoreWolf