On October 1, President Obama took the stage at a press conference and did something unusual for the leader of the free world: He explained why a website wasn’t working. Specifically, Obamacare’s overloaded online service healthcare.gov, which hadn’t had a particularly smooth launch. For a moment, the president sounded indistinguishable from a PR rep for a video game developer like Electronic Arts, Blizzard or Rockstar.
“Like every new product rollout, there are going to be some glitches in the signup process along the way that we will fix. I’ve been saying this from the start,” Obama explained, sounding for all the world like a Blizzard rep, explaining away Diablo III‘s crippled online launch.
“We’re going to be speeding things up in the next few hours to handle all this demand that exceeds anything that we had expected,” he continued. Somewhere across the country, former Maxis head Lucy Bradshaw likely nodded in solidarity, recalling when earlier this year she had had offered a very similar explanation in relation to her company’s disastrous launch of SimCity.
“It is true that what’s happened is the website got overwhelmed by the volume,” the President told the Associated Press five days later on Monday the 6th, “and folks are working around the clock and have been systematically reducing the wait times.” In an office outside Seattle, ArenaNet president Mike O’Brien may have felt the hairs on the back of his neck stand up, recalling the moment when he halted sales of Guild Wars 2 to keep the game running smoothly.
That same day, top White House technology advisor Todd Park elaborated on the ongoing problems millions of people were having signing in to healthcare.gov. “Right now, we’ve got what we think we need. The contractors have sent in reinforcements. They are working 24-7. We just wish there was more time in the day.” Most gamers have heard that refrain, too.
On October 1, the same day that the Obamacare website went live, Rockstar Games launched GTA Online, the ambitious multiplayer counterpart to their already-ambitious game Grand Theft Auto V. Everyone who’d bought GTA V got GTA Online for free, meaning that the moment Rockstar’s servers went live, millions of people attempted to access them and begin playing together at once. As predicted, it was a fiasco. The service was essentially broken; almost no one could log in.
Is anyone going to get this stuff right?
There are a few constants in life: The sun will rise in the morning, and set in the evening. The traffic lane you’re in will be moving slower than the one next to you. Movie theatre popcorn will always be way too expensive. And if a popular game or online service launches, it will always be broken for at least a week.
At this point, those of us who play video games have come to expect it. If millions of people are excited about an online game, we gird ourselves ahead of time. “I’ll have fun playing this… in two weeks, when it actually works,” we joke, just a little bit bitterly. And so often, we’re proven right. Will that ever not be the case? Or is this simply something that we have to learn to live with? And even if that is the case, shouldn’t there be some things we can take for granted? I think there are, and I’ve listed them here.
First, some background.
Thanks, Diablo III
Launch-day server overload is old hat for players of massively multiplayer online games like World of Warcraft and Guild Wars. But when Diablo III tripped out of the gate, meeting frustrated players with nothing but endless error messages, it raised our collective awareness of the stability problems that plague large-scale online games.
That’s probably because the Diablo series has long been thought of as a single-player game with an online component. Developer Blizzard’s decision to give the third game an online requirement was a controversial one. The game’s launch was their chance to prove that an always-connected gaming experience was actually good for customers in some unforeseen way.
Suffice to say, it wasn’t. Diablo III fell on its face and became the first, best example of why always-online just isn’t good for people who play games.
Since then, we’ve seen it happen plenty more times. Guild Wars 2 had incredibly jammed servers at launch, with ArenaNet’s president eventually being true to his word and putting a hold on new sales until they stabilised servers. Final Fantasy XIV couldn’t get it right on the second time around, and was a disaster for a good chunk of time after launch. And of course, SimCity was such a colossal, flaming wreck that the SimCity Disaster-Watch tag on our site is still being updated months after the game came out. (In fairness, that tag is being updated mostly with ways the game itself doesn’t work; the servers are, for the most part, fine.)
A week after launch, GTA Online appears to be mostly stable, though only yesterday did they manage to get out a patch to address the widespread bug that was deleting players’ characters.
However, that does raise the question: Is this the new status quo? Can we expect, from here on out, that every major online launch will be plagued by functionality-killing server overload?
It seems that, at least for now, the answer is a resounding “yes.” Rockstar should’ve been the best candidate to have a smooth launch — they have a huge reservoir of money, and should have theoretically been able to buy enough servers and pay enough technicians to get everything working. Furthermore, since GTA Online launched two weeks after GTA V, they had a pretty good idea of just how many millions of players would attempt to log in on day one. But while they did fix the majority of GTA Online‘s problems relatively quickly, the fact remains that the game didn’t work very well for the better part of a week. If these guys can’t get it right, who can?
Well, actually…
Some People ARE Getting It Right
There is reason for hope: Some online games have managed to pull off relatively failure-free launches. Star Wars: The Old Republic‘s launch went, as Fahey puts it, “so smoothly that people forget about it.” Funcom’s two recent games Age of Conan and The Secret World both launched with some minor hitches, but players were able to get in and play from the first day. (That’s likely because Funcom learned more than a few hard lessons from the prolonged failure of the Anarchy Online launch.)
The Secret World worked somewhat differently than some other online games, using what Funcom calls a “single server” approach, which puts all players together but separates them out into “dimensions” that they can travel between. While I can’t say for sure that that approach would work for every online game — it’s entirely possible that other factors contributed to TSW‘s relatively smooth launch, including the fact that it never really attracted players on the scale of a WoW or a Guild Wars 2 — it does appear at least possible to take a different approach to online games that makes it less likely to encounter the sorts of traffic jams that we so often see.
(Side note: I’m not a server expert. For a good breakdown of the gist of how some MMOs function, check out this article from Carbine Studios’ lead network engineer David Ray about how the servers work for their still-in-beta MMO Wildstar.)
Those games aren’t the only ones to get it right. Activision’s massively popular Call of Duty games normally launch to a huge influx of players, and while their launches haven’t always gone entirely smoothly — there are usually intermittent crashes and bugs for the first few days of play, though some versions of the game run better than others — their servers don’t seem to get hellaciously outmatched as some of the other games in this article. It’s a bit apples and oranges — Call of Duty functions very differently online than, say, Guild Wars 2, but still good to see that these games can launch without complete failure.
Activision has launched an annual Call of Duty every year since 2006, so they’ve probably got this down to something of a science. If other companies had that much practice, would they do a better job, too? And by that reasoning, is there any really excuse for a company to get this wrong more than once?
Valve’s Steam service has also come a long way — it wasn’t that long ago that it was a complete wreck, marring the 2004 launch of Half-Life 2 with the very sorts of delays and registration woes that today’s gamers have become so accustomed to. And yet with practice, Valve too has worked out most of the kinks in their system. Granted, the occasional launch will still encounter some hiccups — I have memories of Portal 2 being harder to play on launch night than it should’ve been — but they’ve come a long way since the early days. Practice, it would seem, does make perfect.
What’s Causing The Failure? Lots Of Things.
The causes behind these failures are as varied as the services and games themselves. Unfortunately but understandably, the moment things go wrong, most video game companies hold technical details close and don’t release specifics beyond “We’re working hard to increase network stability.” When SimCity fell short, its makers got specific enough to say they were increasing both the number and stability of servers.
The nature of most failures, however, is usually evident to anyone who suffers through them. In SimCity‘s case, the servers were constantly full, and even when they weren’t, they were too overworked to provide a stable experience. That’s in part because games like SimCity and World of Warcraft all use multiple separate servers, each of which has a population cap. As a result, the individual servers become slammed, and players wind up locked out of various servers, despite having had access to them earlier. As you can imagine, sinking a few hours into a game on a given server, only to return the next day and find that server full, can be pretty annoying.
Back to Obamacare for a second. Given that the Affordable Care Act is a national initiative with, let’s be real, a lot more riding on it than some inconvenienced video game fans, there’s been more transparency surrounding what went wrong. Tech advisor Park blamed a software problem, not a server or hardware issue — the bulk of the healthcare.gov functions, he said, are ready to handle a huge influx of users, but the software that handles login creation crumbled under the weight of all the new usernames.
That underlines one of the reasons these sorts of failure-at-launch problems can feel so intractable: Massive online programs require a lot of moving parts, and if any one of them fails, the entire system can go down. To hear Park tell it, it’s as though healthcare.gov is a well-constructed house with a broken front door. From what Maxis said about SimCity, their front door worked fine, but the house itself didn’t have enough rooms. Each house, then, has its own unique structural problems.
We Should Be Able To Take Some Things For Granted
In a perfect world, companies would have the online infrastructure to make any online service work as advertised on day one. Clearly, we do not yet live in that perfect world. All the same, there are a few things we should be able to take for granted. Four things, specifically. They’re the least we should expect.
1. Let us play offline, at least a little bit.
Sure, GTA Online didn’t work all that well. But Rockstar hasn’t taken all that much heat for it — certainly not to the extent that Blizzard and EA/Maxis did for Diablo III and SimCity. The reason for that is pretty simple: Grand Theft Auto V is a massive, fun game that players can enjoy offline while waiting for GTA Online to get up and running.
The crucial failing of both Diablo III and SimCity was that they took series that were previously known as offline experiences, made them online-only, and then promptly failed to work when the games launched.
If Diablo III had offered a single-player mode and the optional — though doubtless cool! — multiplayer features hadn’t worked for a while after launch, people would have been far less upset. If SimCity had launched with its nifty-sounding online economy and trading functionality broken but a playable offline version working, fewer players would have been demanding refunds, and EA/Maxis would’ve had to eat far less crow to try to get back into players’ good graces. It’s a lesson they seem to have finally learned: The latest word is that they’re looking into an offline option for SimCity, and they’ve gone out of their way to confirm that The Sims 4 will be a single-player, offline game.
Massively multiplayer games like Guild Wars 2 and Final Fantasy XIV have a somewhat different row to hoe, given that the entire point of an MMOs is to play online with friends. This next bit may be pie-in-the-sky, but dammit, I’m a dreamer. So: Even those games should theoretically launch with some sort of stripped-down offline option. It could be small: A little area, a character creator, the opportunity to earn some XP and make sure you like your character while waiting to get in and play. It might not be compatible with the way those games currently work — and it could be wide open to exploits and cheaters — but future games could take that kind of thing into account and give players something to do offline.
However they accomplish it, some sort of offline component doesn’t seem like too much to ask. It needn’t be as grand and complete as the standalone version of GTA V, but players who bought a product deserve to use it, even if just a limited version.
It’s heartening to see companies backing off and giving people offline options: To see Diablo III‘s console version be playable offline, to see SimCity‘s makers talking about a possible offline version, and an offline Sims 4. And it’s particularly nice to see Microsoft do the right thing and relax its stance on the Xbox One’s previously announced internet requirement. To this day, I imagine what would’ve happened if the Xbox One had launched with an internet requirement and hit the same sorts of problems as recent online games. And oh, how I shudder.
2. Do not lose our progress.
Developers, I beseech you: Do not lose our progress. Do not delete our characters. Do not make us use the character creator more than once. Do not allow this to happen. Do not.
Once SimCity got up and running fairly stably, I sank a good five or six hours into it. I built a lovely little city, and was really happy with what I’d accomplished. And then, thanks to some combination of tech and server issues, I was permanently locked out of that city. All those hours, gone.
It should really go without saying, but losing progress due to server failure is unacceptable. Not only does it inconvenience players in an obvious way, it can cause them to lose faith in a game altogether. After losing that first city, I’d begin a new SimCity town only to imagine what would happen if, in a couple of days, I lost it again. My faith in SimCity was fractured beyond repair, and I’ve never really gotten it back.
Last week, upon getting into GTA Online, I made my first character. Despite the fact that the GTA Online character-creator is almost comically lousy — I can’t even tell what my grandparents look like, let alone figure out how many options I have! — I was pretty happy with what I came up with. I named her “Felina.” She was a sorta badass biker chick who liked committing crime more than she liked hanging out with friends. I got into the game, watched Felina arrive at Los Santos International Airport, and then the servers failed and I got booted. Upon restarting, I found that Felina hadn’t been saved, and I had to start over again.
Other players reported worse failures — losing characters after sinking in dozens of hours and having to start over. (Again, worth noting that yesterday Rockstar said it’s finally fixed the problem, though lost characters will stay lost.) If it’s too much to demand that all online games work perfectly on day-one — and even if it’s too much to ask that they all have some offline component — surely it’s not too much to ask that they save our progress locally in some way. As annoying as it is to lose progress online, the worst thing about it (for me, anyway) is how it undermines my faith in the game in general. Why should I invest time in this thing that may just erase all my progress?
3. Level with us. Early.
Even if everything has gone wrong — even if the server farm is on fire, if a family of poltergeists has moved into the engineering offices and a zombie redneck torture family is currently filleting the IT team, a developer or publisher can still come out ahead by following one simple guideline: Tell us what’s going on. Level with us. And do it early.
Rockstar’s GTA Online launch might have left a lot to be desired in the execution department, but they succeeded in one crucial area: communication. Prior to launch, Rockstar was remarkably forthright about the likelihood of server problems. “There will be the typical growing pains for an online game,” they wrote, “including but not limited to crashes, glitches, crazy bugs, gameplay modes and mechanics that need re-balancing and other surprises!” So it was written, and so it came to pass.
In the midst of the rocky launch, Rockstar continued to post an ongoing series of comprehensive, relatively transparent official updates, sharing details on coming fixes and clear timetables for when those fixes would go into effect. If they didn’t know precisely when an update would be pushed out, they said they didn’t know and gave an estimate.
That kind of clear communication shouldn’t be the exception. Compare that to EA/Maxis’ weirdly robotic communiqués in the midst of the SimCity fiasco. They referred to SimCity as a success, mentioning fans around the world who were loving the game, and acknowledging the game’s catastrophic failures in a smiling “oh, it’s not such a big deal” manner that smacked of corporate double-speak. They admitted error in a tone and manner that didn’t match up with their customers’ feelings of anger and frustration.
“The good news is that SimCity is a solid hit in all major markets,” Maxis’ Lucy Bradshaw wrote mere sentences after apologizing for the disastrous launch, as if the fact that EA was making money on the game would somehow make us all feel better. “The consensus among critics and players is that this is fundamentally a great game,” she continued. Why not just own it? Cut the smiling corporate happy-talk and say, “Hey, we screwed up. We know you’re mad, but we’re going to make it right. Sorry!” As Rockstar has demonstrated, you don’t have to be all gloom and doom about it. You can keep it light. After all, we want the game to work, too. So just be clear and honest with us. We’ll thank you for it.
4. Don’t make us pay for your services while they’re busted.
Players shouldn’t be paying for an online service as long as it’s not working. If a service is rocky for a week or two, those two weeks should be free, or the game-makers should extend the free trial period to make up for it. If a game is free but allows players to buy in-game items, suspend microtransactions until there’s a level playing field (as Rockstar did with GTA Online). In short: If the game is broken, players shouldn’t be paying for it.
When it comes to online games and services, consumers are becoming conditioned to expect launch failure. Yet our future will be increasingly connected and online. As our ambitions continue to outpace our technical bandwidth, it’s tough not to just throw our hands up in the air and ask, is this stuff ever just going to work?
There’s certainly an argument to be made that people only pay attention when things go wrong. Some online games have had relatively smooth launches, and there have been successful technical approaches that indicate that, in fact, some methods work better than others. But given the fact that just last week, Rockstar Games and the Obama White House — two massive organisations with seemingly endless financial and technical resources — both debuted services that were broken at launch, it would appear we still have a ways to go.
Just because things don’t always go according to plan doesn’t mean it all has to be such a series of fiascoes. If game developers can give us some things we can rely on — something to do offline while we wait for servers to start working, an assurance that we won’t lose any progress once we start playing, and clear communication about what’s really going on — it’ll only help ease our transition to this apparently inevitable brave new online world.
To contact the author of this post, write to kirk@kotaku.com or find him on Twitter @kirkhamilton.
Comments
24 responses to “In An Era Of Online Launch Fiascoes, Gamers Deserve Some Rights”
All I ask from developers, is a “Local Only” search restriction option, or server browser for server based games. If there are still problems even with these options, THEN we know it might be a server problem. Without these options, how do I know it’s a server problem, or because I’m connecting to US?
I gave up asking developers for it, now I’m asking Sony, Microsoft and Nintendo to make optional “local only match making” support mandatory. It’s a quality of service feature and if it isn’t there major parts of the game don’t work. Selling a multiplayer heavy game without it is the equivalent of selling a single player game that runs so slow you can barely play it, but you can’t even get a US centric company to acknowledge there’s an issue.
I would say it’s more like selling a single player game without the ability to save.
But you know, new games like Halo 4 can still survive with 20,000 daily players and ignore comments from Swedan players that their connection speed of 1,000 Mbps is still lagging on US hosts :/
I will eternally agree with you about this. If a game lacks this feature I will not fully engage with it. If Halo 5 lacks it, I’ll probably just hire it and trade it in after I finish the campaign
I was on the fence about picking up Splinter Cell: Blacklist. Guess what pushed it over to the “not buy” side!? 😀
I’ll at least give games a try first. GTA Online for example runs like a dream about 95% of the time. Granted we’re mainly playing co-op against AI, but I’ve very rarely had bullet lag or anything like that, which is impressive considering all the NPCs and vehicles moving around the world.
Well, in the open world, I still notice people from US, UK, Mexico, etc. and bullets having the ability to “catch up” to someone running behind cover.
Being flat out told who the host is inside the gametypes is nice, especially since for some reason developers have been removing ping notifications from their games over the last few years, so you can at least control who’s in the game.
Been thinking about creating a new “AUS” crew that will only ever accept people in Australia, New Zealand and Japan…
Actually, come to think of it, that’s how we play anyway. Maybe that’s why our games always feel so smooth.
Also, I didn’t think there was a host advantage in GTA because I assumed the world was hosted by a server instead of the P2P connection in games ike Halo/COD. I wouldn’t think a single person’s internet connection could host all of the movements and reactions and physics of the open world. I thought ‘host’ in GTA V just referred to the person who was setting up the game and in control of the options. I could be wrong though.
I always assumed that free roam was on servers (without knowing where the servers are located, most likely all in US) and that when going into a gametype you’re disconnected from the server (reserve your spot) and host the (limited) world on your console. Since pretty much all gametypes are restricted and confined to specific areas that one person can run the whole game.
That’s the frustrating part. It’s such an achievable goal. GTA Online is doing all this stuff and our connections can handle it, the tech is top notch, but they fail to do one tiny little thing during match making and it breaks the game. It’s like forgetting to put a handle on the door of a hover car. =P
I remember MW2 launching a day earlier here than the US. That first night was awesome. 24 hours later even though all the Australian players were still on I couldn’t get a match where I had any chance at all of hitting a moving target.
I’m sure everybody remembers that. Day 1 all I remember was GREEN. Then, one whole day later, an American joins the game and the match pauses to migrate host to him. Don’t just ask me, I’ve heard plenty of stories, even from MW2 day 2 of a game full of Aussies, one leaving for one reason or another, having some 9 year old CoDKid joining and getting host. From that moment onwards it was terrible for all non-US players.
Sad thing is, despite MW2 being the downfall of the gaming industry and offending me on many levels as a gamer, they eventually did fix that issue and still remain the only developer I can think of that has. Although, what they did is slow the search system to search at 50 ping matches and increase that search by 10 ping every 5 seconds. Meaning once I reach 100 ping, I just cancel and start all over again.
A bit lame, I would have prefer the option, but it was a bandaid that sticked.
Well it was always intended to do local match making, the other guys did it for World at War where it worked perfectly and all the options were in there, it just came out of the box broken. I think what went wrong is it just grabbed a list of 50 matches and used the lowest ping one, but it didn’t factor it into determining hosts so matches that were under 250ms latency for Australians were never created to begin with, and on the rare occasion where they were the odds of the match making actually grabbing it in it’s initial list was low.
I seem to remember we could usually get a match playing a less played mode, but like you said if a single American joined they’d get host and sink it all (not that I blame them, it’s not their fault match making was dodgy).
I use them as the go to example because it was so extreme, playable to unplayable in the space of a day. Also their handling of it was pretty offensively bad. They just completely ignored the issue for like two or three months, maybe longer (I know I played for a few months then ended up quitting before the patch came out). It’s technically better than games that don’t ever patch in local match making, but it’s sort of more frustrating because you know they know it’s important but they’re still leaving it as low priority. At least the other games can plead ignorance.
You know because if it’s a server problem, the Americans will whinge harder than us, if it’s just rubbish matchmaking the whinging will be coming from remote populations like ours.
Please dont name drop things in your article you did not experience first hand or have a any clue about.
The Age of Conan Launch was one of the worst in recent history. It sold million boxes on launch day yet retained less then 150k after the free month due to server and client performance/bugs. It only makes you look incompetent and unsuited for your role and your writing style in general relying heavily on copy paste 30 second journalism.
Something else to keep in mind. The army of loyalists and apologists online who will shoot down any suggestion that you have a right to be upset over poor service. They are the ones who will tell you all online launches are shit, be patient, get a life etc…
That’s the worst. Yes, online launches are almost always turbulent, I get that, but they’re still falling victim to extremely predictable issues that break the game. You can’t just say ‘of course SIm City doesn’t work, it’s only just launched’ and expect paying customers to brush off their complaints.
Nobody should know the stress a launch puts on an authentication server better than Blizzard, yet they still released Diablo III with online authentication requirements and nowhere near the hardware to cover it. They’re happy taking the money for the game and they insist on Battle.net integration, but because it’s inconvenient/costly to support it properly during the first three months we’re expected to be understanding when they choose to just ignore their responsibilities to their customers.
I’m not buying Sim City to play it 2-6 months later. Some people don’t get into end-game and want to just play an MMO from 1-60 with their first ‘free’ 30 days they paid $80 for. I don’t get why that’s so hard for some fans to understand.
Agreed, I wish apologists would go away. They’re just enabling companies. Every time someone defends a buggy product they’re setting us back a little.
I think someone should write up a code of conduct for both consumers and companies. Most companies don’t respond to these issues properly and most consumers have forgotten how to properly behave as well.
The funny part about those apologists?
You would NEVER EVER put up with this crap with *ANY OTHER PRODUCT*! Anybody watched a film that was grainy and static for the first 15 minutes and went “oh but it’s ok! it’s only just been released!” or a car that can’t start for 20 mins brand new? Oh it’s a brand new MS console! RROD is to be expected anyway!
To put it in perspective I mean it’s understandable to have hiccups on a *new* concept of a product… but come on folks we’re supposed to be living in the “brand new age of cloud powah!” and whatnot. While we expect online launches to be buggy/slow/terrible that shouldn’t be the norm! We expect it but we should also be expecting better w/ every iteration. Not be lulled in a sense of complacency at every problem that crops up!
All online launches ARE shit, and you SHOULD be patient. Doesn’t mean you can’t be upset about it, but don’t be those idiots that vent indignant rage over what is in reality a very small problem that inconveniences you in a very small way. Comic Book Guy is not someone to aspire to be like.
What you are describing has nothing to do with what the person you are replying to. He wasn’t acting like nor was there any insinuation that he aspires to be (the hilarious) comic book guy.
I don’t see anything in my reply that is directed at dogcat specifically. Are you perhaps confusing general advice with personalised advice?
I was totally about to write this. It seems like there’s this epidemic of people telling you why it’s YOUR RESPONSIBILITY to fix problems that weren’t created by you and ones that are restricting you from using the service you paid for. I’m not talking about not having the foresight to check your phone lines before having internet installed; i’ve literally been criticised by the posters of an entire thread for insinuating that when my internet drops for 4 days and my ip literally not picking up the phone (it just rings out), it’s the user’s responsibility to run diagnostics, flush dns and constantly re-change my password over and over again which will inevitably fix the problem and miraculously supply my ip with fingers fit for picking up phones. It’s enabling.
I’m really good at building pcs but in no way do i think that if i build a computer for someone and it doesn’t work, they should learn everything about how one works, re-assemble it and localise the problem. It’s not their damn responsibility, they paid for a working pc and they didn’t break it.
yeah, if D3 had an offline component, theres no way in hell we would of even seen Error 37 because there would not of been 6million people all trying to log in the moment the game went live.
Now GTA was definately going to have launch problems for its online mode and everybody expected it to happen what with it making 1billion in 3 days and you really cant blame rockstar for that.
Its like the Launch of WoW back in 2004, it was terrible due the seer number of people wanting to play it even, more so when you remember that blizzard actually ran out of copies of the game
*ctrl-F* “Dedicated Servers”
Nothing.
Seriously? Dedicated servers help, local dedicated help more. Add options for Player-run dedicated servers – they’re laggy but better than NO servers. Add LAN for crying out loud.
Stop removing features that work, then shoving your customers onto fantastic ground-breaking new ones that can’t deal with it!