Why User Reviews Are The Best Tool We Have

Why User Reviews Are The Best Tool We Have
Image: Steam

It’s an increasingly common refrain: user reviews, particularly on Steam and Metacritic, can be unhelpful. We’ve covered all the facets of review bombing for years, but what’s interesting is that some developers themselves are backing the idea of scrapping user reviews altogether. But one prominent indie developer and publisher has struck out at the discourse, highlighting why user reviews are absolutely essential.

The debate fired up earlier this week, although the discussion about the weaponisation of Steam reviews and its effect on developers has always been bubbling under the surface. Earlier this year, the three-team studio behind Kunai complained about how a single reviewer had trashed their user rating from 8.1 to 1.7. The devs behind AI: The Somnium Files made a public plea for help after their rating plummeted years after release. Various games have been targeted by communities for review bombing in the past, with the campaign against Taiwanese horror game Devotion so successful it was pulled from Steam.

So user reviews, evidently, aren’t perfect. But despite all the flaws, David Szymanski, creator of the outstanding retro shooter DUSK, argued that most of the criticism against user reviews, Steam’s in particular, was misdirected. “I want to again reiterate how damaging [removing user reviews] would be for a lot of developers, Symanski wrote in a thread that went viral.

“The biggest problem facing indie developers is the fact that it’s almost fucking impossible to get people to care about your game among thousands of others,” Szymanski wrote.

A big point of contention for the DUSK developer was, as they saw it, the perception of review bombing. “[Review bombing is] rarer than people think for indies,” they argued, “and are almost always the result of a cataclysmic failure elsewhere”. For more context, I reached out to Szymanski and his New Zealand publisher Dave Oshry, and asked why the narrative against user reviews had gained so much traction among other developers.

“I think the complaints are rooted in genuine concern, but not necessarily rooted in what’s actually best for everyone vs what would be best for their personal feelings,” Szymanski said over email. “It’s almost always someone who’s angry — about a review they got, the reception their game received etc. — or someone that’s scared because they’ve absorbed this narrative that there’s a mob of slobbering feral capital ‘G’ Gamers out there just waiting for an indie game to review bomb into obscurity.”

Szymanski’s view, and one backed up by Oshry, was that user reviews get unfairly scapegoated when there are deeper problems, like a lack of marketing, the natural internalisation of criticism that all creatives face, and the inherent challenge of standing out in an environment when hundreds of competitors are released every week. And even clever marketing, as seen by Devolver Digital and even New Blood’s shtick of selling body pillows, isn’t a silver bullet.

There’s also the real fear that all indie developers and creatives of any stripe fear: nothing. “Getting cheered is great, getting booed isn’t, but getting silence? That’s the worst,” Oshry said.

The DUSK creator added that there’s ultimately no one tool that’ll fix all issues for smaller developers. “There are certainly things about the review system that could be improved, but I’m not sure any of that would be a huge boon for [developers]. You could argue that it should have a “neutral” rating in addition to thumbs up and thumbs down, but I don’t know whether it’s more common for people to default to negative or positive, and if the latter than obviously devs benefit from the lack of that feature. Meme reviews tend to trickle upwards currently, and those can be a bit of a wild card in addition to making it harder for players to find reviews of substance.”

A good example in DUSK‘s case was a negative review which simply read: “Why are you reading the negative reviews?

Image: Steam

Another instance is cases where user reviews run counter to critical opinion. Off the back of Szymanski’s thread, the maker of Primordia shared screenshots showing how his game received a 55/100 review on PC Gamer — only to spend 7 years in the top 250 rated games on Steam.

“What good is a 100/100 review if the game is bad and nobody plays it,” they wrote.

I asked the pair whether a bigger underlying issue was discoverability, something PlayStation has been heavily criticised for. “The most important area of improvement is absolutely discoverability, and tweaking the algorithm and tools so that eyes can get on games that don’t have eyes on them yet, and may not have the ability to do that for themselves,” Szymanski said.

Oshry also noted how other platforms, like TripAdvisor or AirBnB, force users to post detailed reviews covering multiple categories to ensure reviews have more utility for everyone. That said, there’s an underlying assumption there that short reviews aren’t genuine, or that they’re not useful to the developers or the broader public. Many of the best “funny” Steam reviews, particularly positive ones, capture the experience of a game within a few lines. That’s often enough to make for some powerful word of mouth marketing on a game’s store page, although there’s no direct data to actually show how much impact a popular positive review has on sales.

The New Blood founder, who also works as a senior director on the survival game Icarus, argued that something Steam could target is outdated user reviews. A lot of indie games typically launch in early access, and as games improve post-launch, Steam (but also platforms like Metacritic and, to an extent, OpenCritic) doesn’t do enough to ensure that new users are reading experiences that are reflective of a game’s current state.

“I definitely believe Steam should prompt players to update reviews of games that have left early access or have been updated frequently,” Oshry said. “Some of the most popular reviews for games are ones that are YEARS old and quite out of touch with the current state of a game – and those reviews are certainly not helpful. Perhaps they shouldn’t be weighted as highly?”

But Oshry also argued that filtering through feedback is part of the creative and development process. “It’s very easy to look past the memes and get to the heart of the matter. ‘Bad performance’, ‘Broken servers!’ ‘A buggy mess’, the usual culprits — and usually all things developers can take action to fix,” Oshry said.

user reviews
To combat review bombing, Steam introduced a chart system in 2017, highlighting volumes of positive and negative reviews historically and over a shorter period of time.

“If you release a product devoid of the most common issues for which people leave negative reviews, then the only reason left for them to leave a negative review… is if they just plain DON’T LIKE IT. And that’s fine! But what game developers (indies especially) fail to do to combat that … is proper messaging. Often times I’ll see negative reviews that say a game ‘wasn’t what they expected’ or ‘wasn’t what was advertised’. Well, whose fault is that? It’s a messaging issue and a communication breakdown with how your game is presented to players.”

Oshry also questioned why more smaller developers don’t put time into engaging with negative reviews, something he’s seen a lot of success with in New Blood’s various games. “I have personally “flipped” countless negative user reviews into positive ones simply by engaging with the player who left it and addressing the issues they had. More often than not, they don’t hate your game, they just got frustrated! We’ve all been there.”

“I think developers need to put themselves in the mindset of their players more often. After all, if not for them – who are you making games for?”

user reviews
A shot of the store page for NBA 2K21 on the Epic Games Store.

None of this detracts away from the fact that Steam’s user reviews aren’t perfect, and those problems get exposed more readily because of Steam’s scale and importance in the industry. But it also highlights how uneven the criticism is. Some storefronts have no review functionality at all, particularly on consoles, although there is some hidden metrics applied in some instances. (One indie developer who will remain unnamed, for instance, told Kotaku Australia that Nintendo’s eShop will not highlight discounted games in the store’s specialist deals page unless that game has a Metacritic rating of at least 70.)

The Epic Games Store, as another example, only features reviews pulled from OpenCritic. That adds more layers of gatekeeping: only the voices of major mainstream outlets are typically shown, since the site’s layout can only display three reviews on the page. Any new voices have to go through OpenCritic’s approval process to be considered, a barrier that user reviews don’t face. (A review from Kotaku Australia, for instance, does not appear on OpenCritic listings.)

And even with all of those qualifiers, that’s still a vast improvement on what a regular user might see when scrolling the PlayStation Store, the Nintendo eShop, or the four people who accidentally browse the Microsoft Store. Is it right that Steam cops so much invective when other platforms are so lacking?

It’s not, according to Oshry.

“Steam is the biggest PC storefront and it does have the most in depth and visible review system,” the New Blood Interactive founder said.

And that visibility — for those fortunate enough to get it — can be powerful. Users can and have often remarked about how their purchasing decisions have been gently or directly influenced by the right quip, the perfect explanation that encapsulates exactly what it was they were looking for out of a game.

It’s natural to think, then, that outdated reviews can be just as effective at turning customers away.

Image: Conan Exiles

The former CEO of Funcom talked about this when Conan Exiles, the studio’s survival MMO, was review bombed from Chinese users. In Rui Casais eyes, Steam should have applied a region-locking system, especially in cases of multiplayer games where the experience can vary wildly due to player counts, dedicated servers and how a game’s state differs from one region to the next.

“Let’s say that I’m in New Zealand, and I have a terrible connection to the servers because they are just too far away. I would give the game a thumbs down, because it just doesn’t work for me. And that’s fair,” Casais told Games Industry. “But if I live in Germany, and I’m sitting next to the servers, I’ll have a great experience and I’ll give the game a thumbs up. [In Germany] I shouldn’t be influenced by the negative opinion of the person who happens to be far away from the servers.”

That’s a weird grey area: the user reviews are fair and accurate, both from the developer’s perspective and the player’s perspective. Is it a fair image to portray to a potential consumer? That really depends on where that consumer lives, because the situation in New Zealand or another small region may not apply if they’re connecting from New York, London, France, and so on.

Another thorny problem not easily solved by reviews, and one that’s perhaps a bit harder to resolve than what Oshry or Szymanski indicated, is where games don’t neatly fit into a simple category. Browse the indie titles on Steam — or even some major AAA games — and you’ll quickly see a lot of comments about users bouncing off a title, leaving a negative review because it wasn’t for them. Perhaps they hit a game breaking bug that genuinely ruined the experience — but the bug was down to a rare configuration of hardware and drivers that clashed in an unusual way.

While writing this article, I went down a Steam rabbit hole just for extra context, hoping to better understand the gravity of what developers have to face. One survival game received a negative rating because players couldn’t chop down trees. An indie point-and-click adventure received negative ratings because it was a sci-fi adventure, rather than a cyberpunk story, even though it was advertised as sci-fi.

I searched through reviews of games I was more familiar with. Project Wingman, the Perth-led love letter to Ace Combat, has a string of negative reviews for different reasons. One starts by saying the game is “a great accomplishment”, with good controls, familiar design and spectacular views. The player wrote the review after 7.1 hours playtime; Steam logged their total playtime to date at 30.2 hours.

“As much as I like the appearance of the game, I’ve already grown very tired of the gameplay – and I’m not even halfway through the campaign,” the review said.

What about Hades, one of the most critically acclaimed games of the last 12 months? One negative review — written after 19 hours — refuses to recommend the game because it has no colourblind settings, despite saying “the game is a masterpiece”.

According to Steam, that reviewer went on to play almost 150 hours of Hades.

And what are developers supposed to do when actors, say, negatively target their game for the inclusion or perception of certain characters, beliefs, or politics? How do you discourage malevolent acts while allowing users to lobby for improvements, as seen recently with the PC port of Nier Automataegregious localisation errors, atrocious server performance, excessive cheaters, or sheer bewilderment at the state of a release?

Perhaps, as Oshry notes, there is no answer.

“A more detailed review system would lead to better reviews, but also less user reviews overall. That’s less consumer voices being heard, which I’m not so sure is a good thing,” he said.

For developers with the financial might to break through the firewall of platforms — or to be forcibly hosted by them — that might not be a bad thing. But for indies, not having user reviews at all could be catastrophic.

Comments

  • I don’t think I’ve ever read a Steam review. I’m curious – is it something you other commenting typed people do?

    • Typically will read a review if I’m interested in a game, and depending on what is commented on (and how accurate it is), will buy the game or put it on wishlist and wait for discounts. I typically go through the daily discovery queue only to ignore a lot of stuff. (both of not suitable game/quality and way over-priced games).

    • I did for a while until it was clear that a lot of indie game developers (my preferred genre) on Steam would get 1,000 of their friends to upvote their game regardless of quality, or they’d employ some dodgy Russian service to get their members to all upvote the game. Either that or a game would be so terrible that it would be hilariously funny and cool to upvote a game because it’s “ironic” and that’s what everyone else is doing. But I’ve been accused of being more then a little jaded before.

      • The ‘lolz’ reviews annoy me since they really don’t push me towards purchasing a game or not. Especially since there are so many joke ones on *ahem* adult titles which don’t address whether the game is fun or not.

        Having seen some clear review bombs happening, it comes across as a head in the sand mentality from the Dev ‘It doesn’t happen to me, so it musn’t really happen much at all’.

        • It’s not that black and white, and they’re not that black and white about it either. Their view here is that not enough indie developers engage with reviews to turn things around when they have them.

          Of course, that’s if the reviews are a problem in the first place. The bigger hurdle is discoverability. But it’s also a bit weird for Steam to be the vector of so much criticism, when every other platform (hi Microsoft Store, PSN) is so much worse.

    • Oh, I don’t make a single purchase without at the very least a skim through user reviews.
      Even stuff I’ve been looking forward to.
      They’re incredibly useful in aggregate: The cited example in the article is good. If it looks mediocre but has thousands of overwhelmingly positive reviews, maybe take a closer look. If it’s mixed, start paying attention. If the reviews are complaining that the dev has abandoned the game in an unfinished state, that’s vital information. If the reviews are slamming launch bugs, but the discussion threads show a constant patch schedule as recent as last week, that’s good to know, too.

      User reviews are even useful when they’re shit.

      What I mean by that: If the bulk of the reviews are negative, but they’re complaining about the game being too SJW/woke, you filter them out and look for the positives. Is it actually a good game? Have the negative reviews prefaced their criticism with praise for the game itself? Or, let’s say the game is mixed, but all the negatives are whining that the devs didn’t meet a kickstarter promise or commited the unforgivable crime of alpha/beta character wipes before launch. Those can be ignored, too.

      Sometimes if the only thing anyone has to complain about is trivial bullshit, that’s a kind of endorsement for every other aspect of the game.

      Whereas if the thing corrupts your hard drive, kills your CPU, has performance that makes your eyes bleed, or feeds your bank details directly to Russia/the CCP, you can bet your ass it’s going to be the dominant theme in user reviews.

      • The other thing I forgot to mention is that what user reviewers focus on can tell you a lot about the game and whether it’s up your alley. Also, if developers respond. I still haven’t purchased Edge of Eternity – despite having had my eye on it for a while – because of the detailed user reviews that have listed an uncommon pain point that I know would drive me right up the fucking wall. Something I haven’t seen mentioned in the few ‘legit’ publication reviews I’ve seen.

        It’s still on my wishlist, because in response to those call-outs, the devs have actually responded that these are things they’re working on. That one game is a perfect example of what’s so useful to me about user reviews.

  • I tend to find user reviews more informative than any that has come from some sort of media (no dig at kotaku), this is the same for movie/TV show reviews.
    The thing that makes user reviews better is that it’s easy to spot reviews from like minded people who will point out games (or movies) that you are likely to enjoy.
    There are way too many variables that add or subtract from the overall gaming experience and even then it’s up to the player to decide what they like.

    You just can’t get an accurate overall review of a game without a large variety of reviewers

  • I cant say I have ever read a review and based my purchase on one, personally I do not not see Steam reviews of Metacritic as even rating on an opinion level. Especially when they are clearly being manipulated by players not to rate the game, but to serve up outrage for whatever the gaming crime they think the devs are guilty of. Most times when that happens I wonder if those gamers picture themselves as truly heroic, you know, going full Enjolas and the rest of the Les Mis gang, in full red-flag-waving revolution mode.

    Cos I always think they are truly hilarious.

  • There are far more negatives that would come with the removal of reviews than exist with reviews as it stands.

    If you don’t like reviews don’t read them. If a dev does not want reviews don’t buy their game. They are not honest developers.

    • Again, that’s too black and white. The problem faced here isn’t that devs don’t want reviews of their game – people don’t want reviews to be weaponised by bad faith groups, or to mislead those who might potentially like the game (but click away because a review is 2 years out of date and doesn’t reflect the actual state of a product). It’s a bit more complex.

      • I get the point your making. But i still hold the opinion that any dev that wants to remove user reviews entirely is very suspect and wants to control the narrative so they can push out subpar products.

  • I’ve said before that, although I see where people are coming from when they criticise user reviews (especially in the age of review bombing, no matter how infrequent), they present a perspective that is wholly impossible to replicate. By their very nature, no critical or journalistic piece can replicate the perspective of the end user. Nor should they – the fact that they present refined alternative perspectives is what makes them valuable and useful.

    But they’re still written by people paid to evaluate a huge number of games in a relatively short period of time, which is a different use case to the majority of people buy the games themselves. For that reason, they’re still a vital part of the gaming landscape.

    • This is especially noticeable in the case of games like Assassin’s Creed: Odyssey, where reviewers complained that in order to complete in what I can only assume must’ve been one fucking sitting, they were ‘forced’ to buy the exp booster to shorten the ‘grind’ to boost their level requirements that was gating story progression. The non-reviewer experience by comparison is played over weeks, not days, and the booster was never necessary if you even bothered to complete the objectives in the camps and hideouts the game sent you to as part of the story.

Show more comments

Log in to comment on this story!