Why Are Game Developer Bonuses Based On Review Scores?

Last night, Obsidian's Chris Avellone tweeted an interesting detail about his roleplaying game Fallout: New Vegas.

"[Fallout: New Vegas] was a straight payment, no royalties," he said in response to a fan question about the game's financial success. "Only a bonus if we got an 85+ on Metacritic, which we didn't."

Metacritic, an aggregation website that collects scores from selected review sites and compiles them as a weighted average, currently lists the Xbox 360 version of Fallout: New Vegas at 84 (out of 100). The PC version is also listed at 84. The PlayStation 3 version of the game is listed at 82.

In other words, Obsidian may have missed its bonus and lost out on a significant amount of money because of a single point.

We've reached out to New Vegas publisher Bethesda, the company that financed the game, to try to confirm Avellone's statement, but they would not comment. If the New Vegas designer's tweet is accurate, then Bethesda put a portion of Obsidian's financial fate in the hands of a select group of game reviewers.

Finances have been an issue for Obsidian — earlier this week, the independent studio had to let go of 30 staff because a game it had been developing for the next Xbox was cancelled. So a potential Metacritic bonus may have been no small matter.

I understand the logic used by publishers like Bethesda when they dole out bonuses based on Metacritic numbers. As an aggregation of critic review scores, a Metacritic average can be an important benchmark for the perceived quality of a game. And it certainly makes sense that a boss would want to reward its employees based on the quality of their work.

Except Metacritic scores are not objective measures of quality. The Xbox 360 Metacritic page for Fallout: New Vegas consists of 81 reviews. If Obsidian's bonuses were determined by this aggregator, they were not based on the game's quality — they were based on 81 peoples' opinions of the game's quality.

Metacritic scores are not objective measures of quality.

Look through Metacritic's list of critic reviews. The list of selected websites is comprised of both professional and volunteer reviewers. Some write for the web. Others write for print. Some scores are weighted more heavily than others (Metacritic does not publicly discuss the formula it uses to create its averages). Some scores are even treated differently than others — a 7 at Game Informer does not mean the same thing as a 7 at Edge, for example.

Many of the reviews attacked the game for its bugs and glitches, many of which were fixed in subsequent patches and downloadable content packs. While reviewers may have been justified in marking down scores for the buggy product, those scores may not have been relevant after a month, or even after a week. Most review outlets don't change their scores once patches have been released. Is that something Bethesda took into consideration?

There is no such thing as an objectively good game. Nor is there such thing as an objectively bad game. We all secretly hate some games that are beloved by the rest of the world, and everyone has their favourite black sheep. I've strongly disliked some highly-rated games, like Dragon Age 2, and fallen deeply in love with some poorly-rated games, like Suikoden V. Should my personal opinion really be condensed into a mathematical formula and used to decide somebody else's bonus?

At Kotaku, we don't use review scores. Metacritic doesn't count our reviews. What if that made the difference? What if an outlet's choice of reviewer changed everything? What if a developer's bonus was determined by a single person's arbitrary distinction between a 7.8 and a 7.9? What if a game studio faced financial trouble after it missed its bonus by a single point?

This isn't healthy for anybody involved. It's not healthy for a reviewer to have to worry whether his criticism will directly affect peoples' jobs. It's not healthy for developers to focus on pleasing reviewers, rather than pleasing consumers. It's not healthy for individual opinions to impact bonuses and salaries.

Publishers need a better tool for measuring a game's quality. I don't know what that tool is. I don't know that it exists. But using Metacritic to hand out bonuses is dangerous — for developers, reviewers, and, quite frankly, you.

(Disclosure: While working at Wired.com, I gave Fallout: New Vegas a 9/10. My review appears on the game's Metacritic page.)


Comments

    This is depressing. It's not surprising... but it is a downer...

    Companies keeping money from the people that make them money. I loathe review scores.

    Metacritic is probably the least bad option. The only available objective metric is sales data and that can take a year to collect. It does suck that Obsidian missed out by one point. However, they also released one of the buggiest games in recent memory. The fact it was "patched out" a month later is basically irrelevant because a majority of the full price sales are made in the first couple of weeks. The game should not have been released in the state it was, though I wouldn't expect any better from Obsidian.

      Metacritic should never be an option, the problem with that though is that even buggy games get high scores, just look at Skyrim, the PS3 version is arguably more buggy than Fallout:NV and that still sits at 92%! Sales data can be collated each and every month it doesn't need to be after a year.

      It's very disappointing for Bethesda to make such a clause in a contract, suggesting that 85% is what they define as a successful product, at the same time they released Rogue Warrior, something that should never of been released.

      Sadly, the final problem with these contracts is that it takes two to tango and sadly Obsidian signed it! With Obsidian now having a cancelled project with MS, I fear they will have trouble over the next year and they deserve a lot better.

        Yeah, it really should be related to sales and not reviews. It certainly always used to be, a lot of contracts actually included sections about receiving a bonus for exceeding the predicted sales.

        Unfortunately, in my experience, the average employee was never told what the predicted sales figures were or how the game was selling (well, nothing specific beyond "really well" or "not as well as we had hoped"). So if a bonus showed up you were just pleasantly surprised.

        Basically, never rely on a bonus.

        I would argue that they don't.

        Their games are always, always buggy... from KoTOR2 to New Vegas to Alpha Protocol. They are consistently reviewed as having the same issues, and yet they never change them.

    Shouldn't their bonus be based on something that is actually tangible - like sales. I'm sure New Vegas has sold enough despite the missing 1 point. This is like having subjective KPIs in your job - they don't work and someone always ends up with the short straw. Of course it was a little silly for obsidian to agree to such an arbitrary benchmark. But of course you find this same sort of thing in the office - the boss has all the power.

      But how many times have we seen high quality games that don't get the sales they deserve? Particularly ones that get zero support from their publishers in terms of advertising and such, which usually effectively nails their coffin shut.

    I think a lot of people are missing a valuable point here. If everyone purchases a game because of "hype" but the game turns out to be rubbish, how are we then to send a message to the developer that we don't want more of this type of game? They already have our money! This sort of bonus encourages developers to actually make a good game. Metacritic may not be the best answer but at th moment I can't think of another.

      No. Metacrtic is 100% about marketing and not game quality.

      Look at the Modern Warfare 3 Metacritic review from Critics as opposed to the public on PC. This tells us three things:

      1. The general public have a polar perception of game reviewing. If you like it it's a 9 or 10, and if you don't it's a 1 or 2. In actual fact, the game was decent but unimaginative, good for people who liked the previous games and absolutely noone else. Not a 1, but sure as hell not a 10 either.

      2. Profiessional Critics are either playing different games, or sell their review scores to publishers/developers.

      3. Metacritic is not a tool to gauge the quality of a game, at least precisely. And should therefore not be used as a basis for bonuses.

      The reality of the matter is that there is no way to dole out bonuses other than waiting until after the game is released to see the reaction of the general public. Once the hype dies then the game's true degree of quality will be ascertained. That doesn't help with bonuses, but it's the best way to see if a game is worth buying.

    Colour me surprised... I can see where the publishers are coming from, but yeah straight payment for developers aren't out of the norm in the industry. This is why the system is broken, because unless a company can get another project quickly (or concurrently) well, they're screwed.

    This is why I want to see a return to royalties.

    Alec Guinness didn't ask for as much a fee for being in Star Wars, instead he asked for a small royalty on all Star Wars products that featured him and well, he didn't end up out of pocket huh?

    6 Million sales and no bonus. Seems fair.

    Sure, meta-critic isn't an objective measure of game quality, it's just a culmination of many peoples subjective opinions of a games quality. I still think it qualifies as the most scientific approach around to measuring game quality, which is itself such a nebulous vague concept. It's the best measure of quality around that most approximates objectivity, I cant think of a better alternative.

    BUT, it still shouldn't be used as any kind of benchmark for financial reward for developers. Surely that should be based purely on sales.

    I'm curoius to know if they made back development costs on that game.. If they did, posthumous bonuses should be paid, regardless of scores

    if they had an issue about it in the first place, they should not have gone into the contract with it.

    no use crying over spilt milk.

      They don't seem to be the ones crying. It sounds like something they regret, but they aren't complaining to the world, gamers are doing that for them.

    Good, maybe next time they'll actually take on board the constant criticism that their games are game-breakingly buggy.

    I could never finish F:NV for this exact reason.

Join the discussion!

Trending Stories Right Now