Cyberpunk 2077 Has Revived A Debate Over How Games Are Reviewed

Cyberpunk 2077 Has Revived A Debate Over How Games Are Reviewed
Image: Cyberpunk 2077

With the first round of Cyberpunk 2077 reviews — or impressions, in the majority of cases — officially out, the internet is starting to process the state of 2020’s biggest game. Along with concerns over the amount of bugs, division is emerging over the lack of clarity around the game’s console performance, the developer’s approach to transparency, and exactly what is being kept from the public before release.

A large running theme among the coverage earlier this week was that, right now, Cyberpunk 2077 is pretty buggy. Exactly how buggy varies from person to person. I ran into countless visual glitches and a quirk early on that stopped progress, with some occasional hard crashes to desktop along the way.

Others had a different experience. PCGamesN, as just one example, had more severe problems, with corrupted save games and bugged quests:

And that’s just the stuff that’s intentional. Even after a 49GB day-zero patch – the product of CD Projekt Red’s unfortunate last-minute crunch – Cyberpunk is still plagued with bugs. These include quest progression issues and save-file corruption – though fortunately it autosaves so often that I never lose too much progress – and an amazing variety of visual glitches of which floating objects are only the most noticeable. On one memorable occasion, I give a peddler of illegal snuff braindances two buckshot shells to the face, point-blank. Rather than his head exploding, he sinks gently into a squat against the wall, his mouth open in surprise. Though he’s dead, I can still talk to him and browse his wares – when I ask about business, the corpse’s head rotates to face me, jaw slack, then snaps back to staring straight ahead. It’s extraordinarily creepy.

Games of this size and scope are always going to be buggy. But there’s a difference between not being able to grab loot because they’re carrying too many bobby pins, versus not being able to move forward because the game hard crashes or randomly resets your PC (which happened to me a few times).

So Cyberpunk 2077 isn’t alone here. But what’s fuelled the fire is a decision that no console codes would be shipped out prior to release, which has brought a bigger spotlight on how games are reviewed, and precisely what can and can’t be said about a video game before its official launch.

Ghost of Tsushima
Ghost of Tsushima (Screenshot: Gabriel Esteves – Email)

Here’s the score. Whenever there is a review period for any game, there’s always some kind of agreement between the publisher, website, YouTuber, influencer, whoever the content creator on the other side happens to be. This generally covers some basics: don’t leak the damn game, don’t publish footage until day X and hour Y.

In other circumstances, publishers and developers may ask for additional conditions. Most of these are generally based around spoilers, and are generally pretty reasonable. A good example: reviewers playing Death Stranding were asked to only use official screenshots during the first major embargo, which lifted a week before launch. Once the game officially launched on November 8, press were asked to only post edited footage to the end of Chapter 3 at a maximum of 20 minutes. But before that? No direct screenshots, no direct video, and no B-roll was provided.

Another example: Ghost of Tsushima. Unlike other games, there were no preview or early embargoes for Sucker Punch’s epic. Reviewers were allowed to capture their own footage of any open world location, activity or collectible, but any quests, tales or the main story was limited to the “A New Horizon” mission, near the beginning of the game’s second act. But if there were bugs or thoughts or anything that happened after that point, reviewers were free to mention it, provided it didn’t include any spoilers. Similarly, capture was restricted to a total of 30 minutes of video, 20 screens, with a maximum of 10 GIFs — that could only be 10 seconds long.

It’s worth noting that you never hear about all of this, because the boilerplate language on these things often say you can’t even mention the restrictions.

If you’re getting early access to any big game, there’s always going to be some give and take. The whole process operates on a degree of trust. It’s also a commercial arrangement: the developer and publisher are relying on the audience of whoever follows that content creator/website/outlet/newspaper/whatever to help drive sales of the game. But it also exists as a level playing field. Embargoes keep things as even as possible, ensuring that a small Australian website or YouTube channel gets the same access as, say, Polygon or IGN.

Of course, there are instances where this doesn’t happen at all. There’s been scenarios where our US partners have been playing something not available to outlets in Australia, or instances where everyone gets access to the game, but only major US/European channels were included on an earlier round of previews, with others tied to a later, second embargo. It was prominent during the review cycle for the next-gen consoles: this website, for instance, was included in the first round of PS5 previews and review units, but missed out on the first impressions of the Xbox Series X hardware. This doesn’t include industry blacklists either, which can be global, sometimes regional and, in certain cases I’ve heard, restricted to individuals. (As in, an outlet won’t be specifically blacklisted, but a publisher may ask that a particular individual not be assigned a review of a certain game.)

I mention all of this because, for the most part, Cyberpunk 2077‘s review restrictions were actually pretty reasonable. Unlike every other game from a major publisher this year, CD Projekt didn’t impose any restrictions on user screenshots ahead of the initial review embargo. That’s usually a massive no-go for many studios: even the PC re-release of Death Stranding this year wouldn’t let users take their own screenshots, forcing everyone to publish hideously ugly letterboxed images with watermarks. (In case you’re wondering: indie games, and many BB games, typically have little to no restrictions at all, although many will often kindly ask that spoilers are avoided as a courtesy.)

There’s always debate on how long a review embargo should be. I’ve been pretty vocal about how six days was not even close to enough for Cyberpunk, but it’s not the first time that’s happened. There’s even weirder scenarios around certain publishers where a review embargo doesn’t lift until the day after its available for release locally. (This happened, for instance, with Jedi: Fallen Order.)

But still, all of this usually works in balance. Reviewers accept it not necessarily because of an implicit fear of losing future access, but because what’s asked is generally pretty fair. You don’t want to spoil things for your own audience, after all, and it’s very, very rare where an embargo actually stops you from doing your job as a reviewer.

Image
Cyberpunk’s review terms meant images like this, captured on a GTX 1080 at a lower resolution, could be published. Normally, however, all you’d be allowed to see would be …
cyberpunk 2077 sex scenes
… this official shot. It’s clearly at a higher resolution and it’s from the same scene — when you meet Evelyn Parker at the beginning of the game — but it also might have no relation to the conditions, or hardware, people are actually playing on. A key difference here too: look at the background behind Evelyn, particularly to her left.

Where Cyberpunk has copped heavy criticism is for two things, one understandable, and the other not so much.

The first is around the lack of review coverage for console codes. This would have risen a red flag under any normal circumstances, but it was always going to be a problem given that CD Projekt specifically delayed Cyberpunk 2077 to December to allow for more performance optimisations on all platforms. Understandably, that does two things: it suggests that the older consoles were having a harder time running the game to an acceptable level, so much so that management saw fit to delay the game’s entire release.

So, obviously, people want to know how that shook out. For anyone weighing up whether to get the game now or later — especially if the experience will be tied into your access on a next-gen console or new PC upgrades — that’s a key factor. But it was also further highlighted when CD Projekt published footage of the game on Xbox One X and PS4 Pro, but not the original Xbox One or PS4.

Because of this, a couple of major reviewers publicly said they wouldn’t be posting their reviews, or said they’d declined access to the review code entirely. That’s a big deal: even though the review period was relatively short, every hour with a game as big as Cyberpunk counts. That’s especially true if making content is your daily job, because it means you miss out on the algorithm-driven views that naturally come from the entire gaming industry all talking about the one thing, all at the same time.

Jeremy Penter, creator of the ACG YouTube channel, said on Reddit that he’d struggled to get a code, despite the publisher initially reaching out. “I could not figure out why I was having so many issues with getting code when they had asked me to cover it originally. Then was discussing it and realised, I cover all the platforms and had discussed console versus PC in depth and what I wanted to do in later videos,” Penter wrote.

Australian reviewer Ralph “SkillUp” Panebianco added that he wouldn’t be publishing his review until he could show “the reality of the game with my own footage”.

The release of B-roll footage is completely standard, and again, happens with preview and review footage. Another major example: reviewers could only use official B-roll footage for their The Last of Us 2 reviews, and user-taken screenshots were forbidden as well. But none of that even came up in discussion, because a much bigger talking point was a restriction around the second half of the game, which prevented any public discussion around the switch from Ellie to Abby.

Even amongst the hardcore PC crowd, concerns have been raised over the game’s performance. A few outlets have noted that the game was supplied with Denuvo DRM. One outlet, Tom’s Hardware, published initial benchmark figures but later removed them “at the request of the publisher”. Some parts of the internet have picked up their pitchforks over that alone, questioning why DRM is being used given CD Projekt Red’s ownership of the DRM-free storefront GOG, with some even saying the official PC requirements were misleading.

cyberpunk 2077

But these complaints are lacking context. For one, it’s bizarre that Cyberpunk 2077 is being criticised for using DRM before launch when they’re not the only publisher to do so. You didn’t see, for instance, any complaints against Bethesda for using Denuvo DRM during the review period. DOOM Eternal‘s Denuvo was patched out months after release, but nobody criticised the studio for locking things down before the game was released. Devil May Cry 5 used Deunvo as well, and while nobody likes it, it also didn’t receive any of the flak that CD Projekt Red are getting now.

Given that Cyberpunk 2077 will exist on a platform that will inevitably end up being copied, torrented and patched a little more easily courtesy of how GOG operates, it seems deeply unfair that they’re copping criticism for this one. Similarly, Denuvo DRM having a performance impact isn’t new to Cyberpunk. It’s not always a big deal, but the performance hit is there. But because CD Projekt Red — and Nvidia, as well, who supplied reviewers with guides about the game’s ray tracing features and performance prior to launch — were open about Deunvo’s potential impact, it’s become a talking point. They’re effectively being criticised for their transparency here. That’s deeply regrettable given Denuvo’s being patched out for launch anyway, something other games often take weeks, even months, to do.

And by the way, there’s the other side of the console coin here too. Cyberpunk 2077 isn’t the first game to withhold code from a certain platform until launch. Consoles were excluded this time, but over the last few years — at least in Australia — local distributors and publishers have sometimes excluded access to the PC version of a game at launch, whereas PS4 or Xbox One codes would be distributed days, sometimes a week or more, beforehand.

But it also puts a spotlight on the reviews process, and what information should and shouldn’t be in the public eye. Is it fair to reveal benchmarks considering the Deunvo overhead, but also considering the game has been patched over the review period, and that multiple Nvidia drivers have been made available prior to launch — and none of this includes any potential optimisations from the day-one patch?

I’d argue that it’s not. It makes sense to mention what the experience was like, of course — if the performance wasn’t enough to reasonably play with ray tracing enabled, that’s definitely worth mentioning. And the Cyberpunk restrictions were generous enough that people could actually show the difference in still shots, something many other publishers don’t and haven’t allowed.

None of this, of course, excuses that customers have every right to see and know whether a game runs satisfactorily well on their hardware before making a purchase. But the whole episode just highlights how little we talk about how games are reviewed.

Cyberpunk 2077 has found itself in the spotlight, primarily because it’s permanently hogged it for the better part of the last few years. So it’s going to receive an extra degree of analysis and criticism, not because a blockbuster production of this size doesn’t deserve it but because other major titles have skated under the radar in a way that Cyberpunk isn’t being allowed to.

There’s always the hope that the extra focus on Cyberpunk creates an environment where publishers going forward are more open — having seen the criticism Cyberpunk received — but if my experience of the industry is any indication, I doubt much will change. That said, sometimes there’s progress: the public discourse and backlash meant publishers stopped organising “review trips” for games, resulting in fairer, less clouded coverage of those titles.

Ultimately, many of the launch issues surrounding Cyberpunk 2077 will be rectified in due course. Anyone who played the launch day, or pre-release versions of The Witcher 3 can attest to the amount of performance improvements and optimisations that game received, not to mention massive quality of life fixes with the UI and animation system. And the experience of games like No Man’s Sky, Star Wars: Battlefront 2 and even Fallout 76 are worth remembering: everything is, and always has been, a work in progress. (That said, some fixes are way more important than others.)

Cyberpunk 2077 will continue being the centre of debate, discussion and division for weeks and months to come. That’s a good thing — it opens the floor to a better, more open dialogue what about games are, what they could and should be, the way they’re marketed, and how we as a community deal with it all. But we also need to talk about the wider industry, how the review process works and how different publishers and games try to put their own stamp on what you, the consumer, ultimately end up seeing.

Comments

  • An interesting article. How did the reviewers feel about not being allowed to cover the Abbie/Ellie swap given it seemed to change a core gameplay mechanic just as much as it was a story issue?

      • God, that’s terrible. And the moment reviewers break any agreements they get blacklisted as well. It’s a scummy thing to since you guys then get blamed for BS agreement terms, which you have to follow to even get the review copies. I feel like NG should get even more flak for what they did after reading this article.

        • Well, it’s not necessarily NG’s direct call either. It’s a joint thing between publisher and developer, but also they’re in a tricky spot too where that game had already been leaked (and then you add the weirdness of COVID on top of that).

          On the whole, the way The Last of Us 2 was handled was actually done really well. Everyone got access super early, and a special set of restrictions was outlined so people could do a preview of the start of the game, and then follow on with their review with the exact same code. Since real-life preview events weren’t available, it was a really good setup that actually worked very, very well.

          But, of course, you don’t hear much about that either because, again, the industry doesn’t talk about it.

          • It’s definitely not fair to anyone on the receiving end (though I think from what you’ve said Sony might be more to blame) since those sorts of things are consumer issues that would definitely influence whether or not people would buy the product. It reminds me of the Sword and Shield thing in Japan where Game Freak didn’t tell their Japanese customers that Pokemon were being left out of the games and it triggered outcry plus refund demands. Yeah, there’s still Pokemon in the game, but is it the product as advertised by the developer? Arguably not and we get returns for less.

            At least the rollout of the actual review copies were competent and I’ll give them credit for that. God knows I’m definitely seeing CDPR in a new light with them being half arsed issuing review copies. After seeing that article of the reviewer who had a seizure I can’t help but think that might’ve been missed so easily yet damaged so many people just as Porygon’s episode did in Japan. People like me would’ve been absolutely screwed if she hadn’t posted her experience with the game and that’s why they should be flogging out reviews to as many people as possible.

        • ND already had nearly everything leak, that might be why they went “you cant confirm the leaks” because more people might have cancelled their pre orders. Maybe?

      • Thankyou for being so candid. I realise we don’t always agree on a lot of things. some bigger or smaller issues. still this was an excellent article and your honesty and thoughts here are appreciated despite any differences we may have.

  • Everyone has a different definition of what constitutes ‘fair’ and I’m just glad/lucky that KotakuAU tends to land about where my own does.

    • Agreed. For the longest time I have not really paid attention to which articles are Australian written and what has been imported. On the whole, I have seen Kotaku decline in quality over the last few years.

      But finally I have taken notice and wow, it has really shown me the massive quality divide that exists. I don’t think I have given Alex and Leah nearly enough credit for their work. As m2d2 said below, back to back awesome articles indeed. I really hope the US Kotaku take notice and up their game because at the moment, they look like amateurs. Whiny ones at that.

    • Ive never seen a shit article, let alone a “not good” article produced by the Au Team and im been here since the days of David Wildgoose. Normally id never bother with the Local version of a website, but when it comes to kotaku, its Kotaku.au All the way

    • Agreed, Alex, a stellar series of articles. Keep up the good work. *synchronised claps*

      In terms of Cyberpunk, for me some games are like operating systems, don’t be an early adopter. Or at least make sure your hardware is up to the task first. Personally I very rarely buy games on release. I like post launch reviews, and these days, its digital which removes the element of scarcity. And of course there’s the pile of shame, lets not dwell on that. Fairly sure that this will/has sold well and most organisations will want to ensure that I won’t see their products at a third of the price in 6 months and therefore will have plans in place post launch to address bugs and enhancements, like in any sensible business plan and agile software release.

  • I for one dont see the point of reviews, and I especially don’t see the point of places like meta critic. At the very most I have some favourite YouTubers whose style and rational opinion is of interest to me, like ACG, and while rarely do I actually fully agree with them, still I watch mainly for entertainment value, not so much for educational purposes.

    Cyberpunk was always going to have issues, everyone (including myself) have set our expectations so high and given how crap 2020 has been, the game has almost become this irrational beacon of hope. So either people are going to be disappointed, that a mere computer game doesn’t fill that void we so desperately need it to, or other people are simple doing to downvote and troll the hell out of it for their own amusement. Honestly, there is only opinion I care about, and that is mine.

  • I think reasons like this are why some gamers are turning away from games reviewers or youtubers and instead wait for public opinion before making a plunge on a game. You don’t know what agreements are made with a reviewer and a publisher/developer or what restrictions are placed on a reveiwer. Basically every reviewer of TLOU2 said up front that they weren’t allowed to cover the second half of the game. Some gamers said that stunk because how are they to be expected to make a decision on whether to buy a game based on a review of half a game?

    And on the other side, pre-orders are becoming more of a dirty word with games. Where’s the incentive to pre-order a game if it’s incomplete or buggy as hell on launch and you don’t get much from pre-ordering anyway?

    It definitely feels like developers and publishers have lost track of ‘make the best game you can, let reviewers and gamers love it’ by placing all these restrictions. Fair enough if they don’t want their story spoiled in a review they might expect for major spoilers to not be discussed, but I think any reviewer who puts major spoilers in a review with no warning will find themselves quickly looking at a 0 read count on their articles so they shouldn’t really worry about that.

    But if they want to hide the fact their game doesn’t run as well on some platforms, well that’s gonna come back to bite them at some point. People are intelligent, more intelligent than publishers releasing games give them credit for. If they come out straight and say ‘Cyberpunk is buggy right now and runs like crap on PS4 & XBox One systems’, people will take that on board and maybe be ok with it. But they won’t be ok with trying to hide stuff like that, and I’m happy to see some reviewers saying they won’t put out a review straight away because of that even if it means they might lose out on those precious Day 1 clicks, views, likes and subcribes

  • I must admit, I came into this article thinking this would be a critique about features versus bugs, and in a world of every AAA title getting 8,9’s and 10’s, where does one draw the line between the vision of the world and the impact of the bugs on playability.

    Still, an interesting article about something we the consumer very rarely get insight into.

  • My only gripe with review shenanigans is when a game sends out console codes for reviews and then dont release the PC review copies… cause the game is still being ported under Crunch by a contracted third party, which is a sign its going to be a disaster. (For example Batman Arkhem Knight)

  • One thing I have seen across all of the reviews so far is there is some kind of expectation of a ‘next-gen’ gameplay experience.

    What is that exactly, no reviewer will state what they believe is next gen gameplay, just that is hasn’t met some kind of wierd expectation. Not even yourself, Alex, who pinned for that very thing.

    It feels like the expectation is that somehow these guys would create a new genre, and that is the expectation.

    I think thats kinda crazy.

    • There’s absolutely some next-gen tech in here — the AI-supported facial recognition (which the Polish government helped fund with a grant) works real well, although you can tell the upper half of the facial models aren’t on the same level as what, say, ND did in The Last of Us 2. And the way the city is to built to support everything that it has, and everything loading in as smoothly as it does (even with the inevitable pop-in) is super impressive.

      CDP are gonna give one hell of a GDC talk in a year or two’s time, let me tell you.

      But design wise it still adheres to a lot of traditional last gen principles, inventory management and such is still Every Ubisoft Game You’ve Ever Played, and the edginess of the writing a lot of the time (although I’m starting to hit some quests now that have a finer touch) doesn’t have that same mature undercurrent that really made The Witcher 3 stick.

      So to answer your question. Next-gen gameplay. I want to see things I haven’t seen before, designs that weren’t physically possible, worlds not built with bullshit tricks and cut-off corners because of some 2012 hardware limitation, and a ton of cool stuff with AI. We’re already seeing how neat things like DLSS can work — now it’s time to really put AI generation to work in other fields too. What’s the next step beyond procedural generation? What can ML do for, say, the AI in a sports game? Animation systems that can learn and grow over time just by taking in new data from watching live matches?

      But also, I really want that cat game.

  • Really good article. The fact that the stipulations of embargoes often restrict talking about the details of embargoes is a very frustrating thing for readers.
    It is a crappy practice which obviously only serves huge corporations interests and keeps consumers in the dark.
    I applaud the transparency in the article.

Show more comments

Log in to comment on this story!