The Horror: Behind The Scenes Of Launching H1Z1

Launching a game is hard. So what happens when you release a high-profile title with the world watching and things break? Lots of things. Worse, it takes at least a day to fix stuff when a retweet can happen in seconds. Let’s talk to H1Z1‘s senior game designer Jimmy Whisenhunt.

From a panic stations point of view H1Z1 started strong. The biggest issue? People couldn’t get into the game when it launched. “From our experience launching games there’s always something that breaks,” admits Jimmy. In this case the problem when everything went live was that the login servers couldn’t handle the traffic and the entire development team sat helplessly watching streams of people unable to play their new baby. “That was horrifying, because Liric alone [a Twitch streamer with nearly 700K followers] had 74,000 people watching him not be able to get in.”

Not an ideal start but also not without a certain amount of humour: “There was one French guy, I think his name was Aiekillu. He got in. There’s only one of us that speaks French.”

A fix was eventually found but nothing that could be implemented that day (“We couldn’t figure out why at the time, but eventually we found some hardware stuff that we ended up solving,” explains Jimmy), so for the time being the only solution was to “just let people trickle in slowly and start watching the streams”. The decision was made to start over: to shut everything down and start again. Slowly.

Of course, while the team are watching streams, taking notes and trying to see how the game is working in the wild, not everyone was happy. Being able to login to an online game is pretty much the point and the internet gets angry fast. “I see it every day. We’re incredibly transparent I, maybe to a fault, talk to the community who come at us in a very negative way: ‘You guys are f-ing morons’, ‘How did you not know that?’”

So how did H1Z1 not know what would happen at launch?

Well, the game had been tested as much as the studio could manage internally: “We play within the company. As many people as we can possibly get on. We hit player cap from the company which is cool because then we get a realistic expectation. We were like, OK, well it’s not exploding. The player cap works [200 per server]. Our key systems were working, everything seemed to be OK from the login side,” but with one unknown: “obviously we don’t have thousands of players trying login at once, so we couldn’t fully test it until we went live.”

“It’s the same when you’re doing a scientific experiment,” Jimmy continues. “You have a lab and a closed environment and you can glean a lot of assumptions and data from testing. But the moment that this virus goes into the jungle it can mutate and do all these different things. In a game world we can only test the mechanics to a certain degree, especially in an MMO. We only know what happens in a world with a little over 200 players within the company. Not with multiplayer servers that work together [or] multiple servers per box. We don’t know. If we need to switch host what will happen? If we have thousands of players trying to login to the same server will it hurt its performance? Or will the servers do their job?”

Eventually, of course, people managed to get in, but Early Access is, by definition, unfinished and the fun of making it all work kept coming. Soon, another issue raised its head. Or rather didn’t: “Zombies didn’t move,” says Jimmy. “That was really scary, because one of our big dreams for this game is that you want people to feel like a survivor in the zombie apocalypse. That you are afraid for your life because of the world, not just because of players. And then, ‘oh God, they don’t move. What happened? They worked fine yesterday!’” It was a server issue again, which Jimmy describes as “good news” because it was something the team were already on top of before people started reporting it.

With the initial few hours out of the way, Jimmy and the rest of the team settled in to see what the world was finally making of their game. Now it was out of the lab it was important to see if everything was doing what it was meant to be doing. Were the mechanics and gameplay, so successfully tested internally, translating to the real world? “I was watching streams, I didn’t touch data for three days,” he explains. “I did nothing but sit there and watch streams, take notes, do my own testing like, ‘am I having the same issue he is? What can I do to change that?’”

One issue that soon became apparent was that of loot. My own early experiences consisted of hours running around with a makeshift bow, pockets oozing with blackberries and little else. One evening saw my entire gear haul consist of a shirt, a knife, a tin of food and as many sticks and berries as I could carry. It’s not an issue now (there’s stuff now. Inside things!) but it took a while to get there.

The early lack of loot was not only one of the more serious gameplay affecting issues, it quickly became a sticking point given the game’s free-to-play status. “I got a tweet,” explains Jimmy, reading it out: “‘So Sony Online Entertainment [now Daybreak Games] nerfed the loot to where there’s none, so we’ll buy more air-drops.’” It was taken as sarcasm until the tweeter started calling Jimmy a “scumbag.” (The issue of the air-drops I’ll get to later.)

Again, it was another issue that only happened ‘live’ with a large number of players in the game. “We couldn’t figure out why because we didn’t have thousands of players logging in internally for us to test what happens over a couple of days,” admits Jimmy. The obvious answer was to increase the amount of loot in the world but something still wasn’t right: “we’d been upping loot, just going up and up and up. This is the first time we’d tested it with this many players obviously and we’re like ‘OK, it’s not working.’”

The final solution was to flip the loot-free game in completely the opposite direction. “We screwed ourselves because we didn’t know about a code bug,” says Jimmy. “We’re like, OK, let’s fix the code bug, don’t touch the data because you don’t want to do two things at once and, when we did that, there’s an infinite loop everywhere and you could find 17 guns and 500 rounds in an hour. It was crazy and we were like, OK, well that’s obviously that’s not going to work, so we backed it way up and then there’s nothing so we had to keep going up from there.”

It’s an issue that seems to still be seeking a full solution. My most recent games have seen me finding just enough to to be worth fighting for, but not enough to feel safe. You might find a gun in the first half hour or so, but barely any bullets. Rattling around a small deserted town with four bullets and terrified of other players (‘Friendly’ is the universal cry of someone about to kill you), makes the game feel almost like a The Last Of Us MMO. And while some have tried to embrace a more pacifist approach
almost every human contact I’ve had has ended in death or a retreating Mexican stand off.

Talk of guns, however, brings us on to what was one of H1Z1‘s defining features at launch, the so-called pay-crates. The in-game economy lets you ‘buy’ air-drops, where boxes of supplies are dropped into the map somewhere. The idea was that while you could purchase these things, you had no say where they would land and it would create a “world event” as everyone who saw it could fight for the content. It didn’t work that way. Within hours of the game going live people were buying crates that landed right next to them, and that contained a weapons, giving them an immediate paid-for advantage. Across Reddit and forums the cry of ‘pay to win’ went up.

“There were a lot of things that happened there really quickly and there was nothing any of us could do but sit and watch,” says Jimmy. Again it came down to not being able to test the game with thousands of players until it went live. “There was a bug, something that we hadn’t seen,” he explains. “When an air-drop came in for the first time on a stream that we saw, [it] landed right on top of a guy. It went against completely what we intended. We wanted it to be a world event, not a self-loot spawner. We knew immediately we had to fix it but it was already too late to say anything because everyone was already on top of it.”

While the team started working on a fix, the internet went wild. “The volume was incredible,” confesses Jimmy. “I was impressed. The first thing was they said we lied. And we said, ‘Whoa, no we never lied’. In August we announced air-drops and what would be in them. We said it could be weapons, there could be a lot of things in them. We pitched it, we explained it to the community and everyone was like ‘cool’ at the time. And we didn’t say anything since then. Because why not? And then it launched everyone said we never said there were weapons in it and we didn’t know what to say because there were articles — I think PC Gamer, Gamezone and several — had reported that we would be doing that.”

Trying to defuse the situation didn’t help, either: “I tried to talk on Reddit a couple of times and I was downvoted to oblivion and people were insulting. I started to realise that a lot of these people didn’t own the game. Someone saw something, they assumed something was happening that wasn’t without trying it themselves and it blew out of proportion. It’s horrifying because the last thing anyone wants to run into in their professional career is to be called a liar.

“I think the worst for me was not being able to react faster” says Jimmy of those fraught few weeks post-launch. Fortunately, most of the big teething-issues seem to be behind the team. Much of the forum and Reddit discussion currently focuses on minor bugs, balancing and ideas for the future (one issue I’ve encountered recently has been a weapon/animation lock up leaving me defenceless). Patches and balance measures continue to fine-tune the experience and players seem to have largely forgotten the shaky start (although they’re still murderously callous bastards).

I still think it’s a little too early to really judge the game but recent improvements have made a world of difference. When I first started playing I wandered forests for hours with nothing but a bow and pockets full of blackberries. Now there’s far more potential to feel like a survivor as you dodge the undead and the living alike, trying to scrape together enough to endure. It might have taken Daybreak a few hammer whacks to get it all working, but the fun is starting to emerge.

Update: Since publication Daybreak has asked I clarify two issues marked in the feature:

1. Daybreak made the following statement regarding the game’s ongoing development: “We understand people get frustrated and will continue to engage to solve the issues that bring this frustration.”

2. Concerning the game’s early loot issues: “[These] were resolved; they are still being fine-tuned, but the patch that was released last week fixed the issues mentioned in the article.”