It’s an increasingly common refrain: user reviews, particularly on Steam and Metacritic, can be unhelpful. We’ve covered all the facets of review bombing for years, but what’s interesting is that some developers themselves are backing the idea of scrapping user reviews altogether. But one prominent indie developer and publisher has struck out at the discourse, highlighting why user reviews are absolutely essential.
The debate fired up earlier this week, although the discussion about the weaponisation of Steam reviews and its effect on developers has always been bubbling under the surface. Earlier this year, the three-team studio behind Kunai complained about how a single reviewer had trashed their user rating from 8.1 to 1.7. The devs behind AI: The Somnium Files made a public plea for help after their rating plummeted years after release. Various games have been targeted by communities for review bombing in the past, with the campaign against Taiwanese horror game Devotion so successful it was pulled from Steam.
So user reviews, evidently, aren’t perfect. But despite all the flaws, David Szymanski, creator of the outstanding retro shooter DUSK, argued that most of the criticism against user reviews, Steam’s in particular, was misdirected. “I want to again reiterate how damaging [removing user reviews] would be for a lot of developers, Symanski wrote in a thread that went viral.
The important thing that is relevant for a lot of us and that for some reason gets ignored in the discussion is the effect positive reviews have toward tipping someone into a potential sale, or heading off concerns a player might have about quality.
— David Szymanski (@DUSKdev) June 29, 2021
If you’re, say, Cruelty Squad, and you have the unfair appearance of being a low effort shitpost when in reality that’s not the case, reviews are what can save you. As they did for Cruelty Squad.
— David Szymanski (@DUSKdev) June 29, 2021
“The biggest problem facing indie developers is the fact that it’s almost fucking impossible to get people to care about your game among thousands of others,” Szymanski wrote.
A big point of contention for the DUSK developer was, as they saw it, the perception of review bombing. “[Review bombing is] rarer than people think for indies,” they argued, “and are almost always the result of a cataclysmic failure elsewhere”. For more context, I reached out to Szymanski and his New Zealand publisher Dave Oshry, and asked why the narrative against user reviews had gained so much traction among other developers.
“I think the complaints are rooted in genuine concern, but not necessarily rooted in what’s actually best for everyone vs what would be best for their personal feelings,” Szymanski said over email. “It’s almost always someone who’s angry — about a review they got, the reception their game received etc. — or someone that’s scared because they’ve absorbed this narrative that there’s a mob of slobbering feral capital ‘G’ Gamers out there just waiting for an indie game to review bomb into obscurity.”
Szymanski’s view, and one backed up by Oshry, was that user reviews get unfairly scapegoated when there are deeper problems, like a lack of marketing, the natural internalisation of criticism that all creatives face, and the inherent challenge of standing out in an environment when hundreds of competitors are released every week. And even clever marketing, as seen by Devolver Digital and even New Blood’s shtick of selling body pillows, isn’t a silver bullet.
There’s also the real fear that all indie developers and creatives of any stripe fear: nothing. “Getting cheered is great, getting booed isn’t, but getting silence? That’s the worst,” Oshry said.
The DUSK creator added that there’s ultimately no one tool that’ll fix all issues for smaller developers. “There are certainly things about the review system that could be improved, but I’m not sure any of that would be a huge boon for [developers]. You could argue that it should have a “neutral” rating in addition to thumbs up and thumbs down, but I don’t know whether it’s more common for people to default to negative or positive, and if the latter than obviously devs benefit from the lack of that feature. Meme reviews tend to trickle upwards currently, and those can be a bit of a wild card in addition to making it harder for players to find reviews of substance.”
A good example in DUSK‘s case was a negative review which simply read: “Why are you reading the negative reviews?”
Another instance is cases where user reviews run counter to critical opinion. Off the back of Szymanski’s thread, the maker of Primordia shared screenshots showing how his game received a 55/100 review on PC Gamer — only to spend 7 years in the top 250 rated games on Steam.
“What good is a 100/100 review if the game is bad and nobody plays it,” they wrote.
Alright, here’s why you should never remove user reviews. @DaveOshry @DUSKdev
Primordia was 7 years in the top 250. We owe everything to user reviews.
Thank you from the bottom of my heart. ????????????
[Look at other games too, Fran Bow, Deponia, etc] pic.twitter.com/Sc1ka5cVgI— James Spanos (@Dualnames) June 30, 2021
I asked the pair whether a bigger underlying issue was discoverability, something PlayStation has been heavily criticised for. “The most important area of improvement is absolutely discoverability, and tweaking the algorithm and tools so that eyes can get on games that don’t have eyes on them yet, and may not have the ability to do that for themselves,” Szymanski said.
Oshry also noted how other platforms, like TripAdvisor or AirBnB, force users to post detailed reviews covering multiple categories to ensure reviews have more utility for everyone. That said, there’s an underlying assumption there that short reviews aren’t genuine, or that they’re not useful to the developers or the broader public. Many of the best “funny” Steam reviews, particularly positive ones, capture the experience of a game within a few lines. That’s often enough to make for some powerful word of mouth marketing on a game’s store page, although there’s no direct data to actually show how much impact a popular positive review has on sales.
The New Blood founder, who also works as a senior director on the survival game Icarus, argued that something Steam could target is outdated user reviews. A lot of indie games typically launch in early access, and as games improve post-launch, Steam (but also platforms like Metacritic and, to an extent, OpenCritic) doesn’t do enough to ensure that new users are reading experiences that are reflective of a game’s current state.
“I definitely believe Steam should prompt players to update reviews of games that have left early access or have been updated frequently,” Oshry said. “Some of the most popular reviews for games are ones that are YEARS old and quite out of touch with the current state of a game – and those reviews are certainly not helpful. Perhaps they shouldn’t be weighted as highly?”
But Oshry also argued that filtering through feedback is part of the creative and development process. “It’s very easy to look past the memes and get to the heart of the matter. ‘Bad performance’, ‘Broken servers!’ ‘A buggy mess’, the usual culprits — and usually all things developers can take action to fix,” Oshry said.
“If you release a product devoid of the most common issues for which people leave negative reviews, then the only reason left for them to leave a negative review… is if they just plain DON’T LIKE IT. And that’s fine! But what game developers (indies especially) fail to do to combat that … is proper messaging. Often times I’ll see negative reviews that say a game ‘wasn’t what they expected’ or ‘wasn’t what was advertised’. Well, whose fault is that? It’s a messaging issue and a communication breakdown with how your game is presented to players.”
Oshry also questioned why more smaller developers don’t put time into engaging with negative reviews, something he’s seen a lot of success with in New Blood’s various games. “I have personally “flipped” countless negative user reviews into positive ones simply by engaging with the player who left it and addressing the issues they had. More often than not, they don’t hate your game, they just got frustrated! We’ve all been there.”
“I think developers need to put themselves in the mindset of their players more often. After all, if not for them – who are you making games for?”
None of this detracts away from the fact that Steam’s user reviews aren’t perfect, and those problems get exposed more readily because of Steam’s scale and importance in the industry. But it also highlights how uneven the criticism is. Some storefronts have no review functionality at all, particularly on consoles, although there is some hidden metrics applied in some instances. (One indie developer who will remain unnamed, for instance, told Kotaku Australia that Nintendo’s eShop will not highlight discounted games in the store’s specialist deals page unless that game has a Metacritic rating of at least 70.)
The Epic Games Store, as another example, only features reviews pulled from OpenCritic. That adds more layers of gatekeeping: only the voices of major mainstream outlets are typically shown, since the site’s layout can only display three reviews on the page. Any new voices have to go through OpenCritic’s approval process to be considered, a barrier that user reviews don’t face. (A review from Kotaku Australia, for instance, does not appear on OpenCritic listings.)
And even with all of those qualifiers, that’s still a vast improvement on what a regular user might see when scrolling the PlayStation Store, the Nintendo eShop, or the four people who accidentally browse the Microsoft Store. Is it right that Steam cops so much invective when other platforms are so lacking?
It’s not, according to Oshry.
“Steam is the biggest PC storefront and it does have the most in depth and visible review system,” the New Blood Interactive founder said.
And that visibility — for those fortunate enough to get it — can be powerful. Users can and have often remarked about how their purchasing decisions have been gently or directly influenced by the right quip, the perfect explanation that encapsulates exactly what it was they were looking for out of a game.
It’s natural to think, then, that outdated reviews can be just as effective at turning customers away.
The former CEO of Funcom talked about this when Conan Exiles, the studio’s survival MMO, was review bombed from Chinese users. In Rui Casais eyes, Steam should have applied a region-locking system, especially in cases of multiplayer games where the experience can vary wildly due to player counts, dedicated servers and how a game’s state differs from one region to the next.
“Let’s say that I’m in New Zealand, and I have a terrible connection to the servers because they are just too far away. I would give the game a thumbs down, because it just doesn’t work for me. And that’s fair,” Casais told Games Industry. “But if I live in Germany, and I’m sitting next to the servers, I’ll have a great experience and I’ll give the game a thumbs up. [In Germany] I shouldn’t be influenced by the negative opinion of the person who happens to be far away from the servers.”
That’s a weird grey area: the user reviews are fair and accurate, both from the developer’s perspective and the player’s perspective. Is it a fair image to portray to a potential consumer? That really depends on where that consumer lives, because the situation in New Zealand or another small region may not apply if they’re connecting from New York, London, France, and so on.
Another thorny problem not easily solved by reviews, and one that’s perhaps a bit harder to resolve than what Oshry or Szymanski indicated, is where games don’t neatly fit into a simple category. Browse the indie titles on Steam — or even some major AAA games — and you’ll quickly see a lot of comments about users bouncing off a title, leaving a negative review because it wasn’t for them. Perhaps they hit a game breaking bug that genuinely ruined the experience — but the bug was down to a rare configuration of hardware and drivers that clashed in an unusual way.
While writing this article, I went down a Steam rabbit hole just for extra context, hoping to better understand the gravity of what developers have to face. One survival game received a negative rating because players couldn’t chop down trees. An indie point-and-click adventure received negative ratings because it was a sci-fi adventure, rather than a cyberpunk story, even though it was advertised as sci-fi.
I searched through reviews of games I was more familiar with. Project Wingman, the Perth-led love letter to Ace Combat, has a string of negative reviews for different reasons. One starts by saying the game is “a great accomplishment”, with good controls, familiar design and spectacular views. The player wrote the review after 7.1 hours playtime; Steam logged their total playtime to date at 30.2 hours.
“As much as I like the appearance of the game, I’ve already grown very tired of the gameplay – and I’m not even halfway through the campaign,” the review said.
What about Hades, one of the most critically acclaimed games of the last 12 months? One negative review — written after 19 hours — refuses to recommend the game because it has no colourblind settings, despite saying “the game is a masterpiece”.
According to Steam, that reviewer went on to play almost 150 hours of Hades.
And what are developers supposed to do when actors, say, negatively target their game for the inclusion or perception of certain characters, beliefs, or politics? How do you discourage malevolent acts while allowing users to lobby for improvements, as seen recently with the PC port of Nier Automata, egregious localisation errors, atrocious server performance, excessive cheaters, or sheer bewilderment at the state of a release?
Perhaps, as Oshry notes, there is no answer.
“A more detailed review system would lead to better reviews, but also less user reviews overall. That’s less consumer voices being heard, which I’m not so sure is a good thing,” he said.
For developers with the financial might to break through the firewall of platforms — or to be forcibly hosted by them — that might not be a bad thing. But for indies, not having user reviews at all could be catastrophic.
If we remove Steam reviews, I’ll likely never buy an indie title again unless I’ve already bought from the developer. https://t.co/15e6G6KNiK
— WuYixiang | 吳羿翔 (@WuYixiang4084) June 29, 2021
Leave a Reply