Sim City: Why The Aussie Launch Was Broken, And Why Oceanic Servers Were Fixed First

When Sim City initially launched, EA and Maxis believed it would have its server issues under control for launches in other territories, but that didn't quite work out as planned. We spoke to Lucy Bradshaw, the General Manager at Maxis to find out precisely what went wrong, and to ask why Australia's Oceanic server was among the first to be patched.

According to Bradshaw, initially the expectation was that Maxis could get servers stabilised before Sim City launched internationally, but there were a few unexpected issues.

"We started to experience problems in the early hours of the morning on March 5 when we launched in North America," she said. "Our servers were being overloaded and our players were having difficulties getting into the game. At that moment we gathered our server team to triage the situation and our primary goal was to figure out how to get the rest of our fans into the game."

The initial priority, as you'd expect, was simple: increase server capacity, free up space, all customers to play the game they'd purchased.

"We deployed additional servers to address load but we also found that our servers were behaving differently under load than our experience in closed betas and load testing. We had to address the underlying issues, update server elements and work through the deployment of new servers. Once we had those underway and moving into the ecosystem we needed to go back to the original servers to update them so that they would benefit from all of the enhancements we did."

These additional unforeseen issues delayed Maxis' attempt to get things stabilised before the International launch date.

When Sim City initially launched in Australia, Maxis and EA were still in the midst of attempting to fix the game's server issues. At that point, the team was in the thick up server upgrades. We noticed that Oceanic server 1 was one of the first to be upgraded. Why? Pure coincidence.

"[P]art of our solution was to add higher capacity servers to allow players to get more into the game and play," explains Lucy. "Then, the next step was to upgrade the original servers and it just happened that we picked Oceanic 1 to be first."

It's been a difficult learning process for the folks at Maxis, but Lucy Bradshaw believes that valuable lessons were learned. The team thought they were fully prepared for the onslaught, but they were wrong.

"We learned a valuable lesson with this launch: that no matter how prepared we thought we were, we weren’t prepared enough," she says. "We thought we had all of the data we needed, but that didn’t help us anticipate the behavior of our servers during the first days of launch. We need to do more real-world testing and more thorough load testing, better telemetry mining, and more importantly, build up a larger infrastructure, so that this does not happen again."

But regardless, Maxis stands by its initial vision for the game, despite all the problems.

"When people can get into the game they’re telling us about how much fun they’re having, and how much they enjoy this new approach to SimCity," says Lucy. "The problem is that not everyone can get in, and that is our deepest issue.

"Adversity brings out the best in Maxis and it’s all hands on deck over here. It’s not just the server teams who are working around the clock, it’s the gameplay teams culling through pages of feedback and reading message boards for comments; it’s our design team thinking about how to improve some of the concepts in the game. The team spirit is high because we are close to bringing the game up to full stability and we’re excited to get everyone into the game."


Comments

    Pity the lesson learnt wasn't "this always on drm crap is a really really dumb idea"

      It's not DRM that was the issue. It was an online game with not enough servers to handle demand at launch. Much like World of Warcraft for a year!

      You can argue that it did not need to be an online game, that's fine and a different matter, but please stop throwing the catch-all "blame DRM" around. I don't like DRM either, but it is not at fault here.

        Lol if it can be done offline (it can) but it's forced to be done online it is drm. Don't kid yourself

          Except it CAN'T be done offline. If you notice the amount of space you get to build your city is drastically smaller than it was in other games. This is because they want you to specialise (you can be the tourist city or the industry city or whatever) but want you to rely on traffic from other cities in your area to fulfill other needs, you might be getting tourists from another city, or raw materials or power. No city can be all things.

          This means you have to be connected to other players, or at least, that is the goal, you need to have an array of cities around you and the goal is to have these populated by players, all working together.

          So the core concept is "flawed" in that it isn't actually a single-player game at all, its simply an MMO where you HAVE to be connected for it to function, and they failed to provide proper server stability and load capacity. This is why i don't want anything to do with it because i don't want to be forced to play online, with other people.

          I'll stick to the Anno series, among others.

          Last edited 13/03/13 11:34 pm

            funny I'm pretty sure CitiesXL provided the exact same type of game play with an offline component just fine.

            You know the last Sim City allowed cities to be connected to other cities? Your cities, you could keep the same pissy city sizes and still be single player using multiple of your own cities, or... or... you could just bring back the big cities that everyone loved in the first place, or like in 2000 where you connected to other AI cities, sure they didn't provide much function in that game, but it was still possible and could be done today, on a single machine without the online only crap.

            The reasoning for always online not being DRM is extremely poor at best.

        I disagree, DRM IS to blame. It's diablo 3 all over again. It's not a matter of whether or not this was built to be an online game, if it has single player components, there should be no requirement for it to be always connected. And guess what? The only reason why the game is always online is because they want DRM in there, not because they want the game to be a multiplayer experience.

    I'd have thought they'd have gone for that one first because less players would have made it slightly more easy to fix and if they can fix the easy one first, they can get more people on sooner and try to look better.

      And if they stuff it up, there's less people to piss off :) hehe

    Im sorry but this game launched with around 6 or so servers? How did they think it would end up?

    The game is great fun don't get me wrong im a huge fan of it... But the launch was doomed to fail when I the initial server list loaded up

    Last edited 13/03/13 1:37 pm

      Without knowing how many players each server was designed to cater for, I'm not sure we can judge based on the number of servers at launch alone. There are a few games out there (EVE comes to mind) that only have one server for their whole population, so it really comes down to capacity rather than quantity. That said, their capacity was obviously too low at launch too.

      These are probably not six _boxes_ but more like six _clusters_, and their capacity estimates for each _cluster_ is insufficient. It's hard to connote the notion of a "cluster" to play from, so, you just say "server".

      You misuderstand the concepts. In the same way, when you play WOW, you are on a "server" you are actualy on a bunch of them at the same time. In one zone you will be on one server, when you move to another zone you will be on a different server. You don't notice the change, because of the way the game is programed, but it is there.

        I understand how the backend of the system 'could' work but the logic is each 'server' or 'region' was not able to hold up its own game.

        Each 'server' has the ability to support x players, they had y number of 'servers' available. When these values were calculated it didn't marry up in the end which was obvious.

        Regardless of how the 'servers' are arranged (Considering they are all cloud based anyway which in turn runs over many many 'box's' EA/Maxis obviously didn't think the game would sell well or didn't assume how much each server could hold.

        Without going too technical the 'in a nut shell' EA/Maxis launch was doomed to fail with only around 6 'servers' accessible to the public on day 1.

        So try and spin ur 'this is how games actually work' fact of the matter is, however the game actually does work, it was severely underestimated.

    The big question here... What happens when all these really expensive server upgrades and infrastructure expansions start to show themselves in the Cost v Benefit ledger?

    Always-On DRM = "Thanks for playing when you could, but we are no longer going to pay for these servers, so you cant play anymore, ever. Have a nice day now!"

      By then they'd have expected everyone to have already moved onto SimCity 2014 or whatever the next one is going to be.

      They're all Amazon AWS servers... so its all virtualized anyway...

        V or P, it still costs money, money that EA are, eventually, going to stop paying.
        There goes the game you paid for.
        With Always-On DRM, you never "buy" a game, you "rent" it

        Last edited 13/03/13 3:54 pm

      Google EA xbox server shutdown

      That's how EA rolls

    no matter how prepared we thought we were, we weren’t prepared enough

    Yep... Blizzard learned that, too. As did the rest of the internet. Why did it take EA/Maxis so long? And this, added to the alleged lies EA has been spouting about offloading simulation to the server? Yea... Thanks, EA. The lesson you learned from Diablo 3 was "have a good scapegoat feature to rationalise your DRM". Everyone else learned a slightly different lesson...

      What part of "EA had no part in making the game always online" are you having dificulty understanding? EA never comes into the picture. Also lots of games have "always online DRM". being online is required to be able to do the interaction between cities in the zone. Why not have the DRM be online too?

        s/EA/Maxis/g, if you want to be pedantic. I was using "EA" as shorthand to refer to the both of them, as I find it difficult to believe that EA had no part in the project beyond throwing money at Maxis. The point was that what they "learned" here has been known for quite some time by everyone else. If EA had learned this as well, would they have told Maxis to find another way? Or quietly pocketed the extra sales and rode out the storm? Does it make a difference if they were ignorant or greedy, in this case?

        Being online is only necessary for inter-city interaction if you play it multiplayer. If you play one city at a time, building each up by feeding on each other, why do you need the online component? This is why the DRM needn't be always-online (server issues aside). Going from RegionA/CityA to RegionA/CityB is a hell of a lot faster without the "route through external servers on the internet" part of the journey.

        And, yes, lots of games have always-online DRM... for a while. The only one I can think of off the top of my head that hasn't backpedalled after a year or two is Starcraft 2.

          If you play one city at a time, building each up by feeding on each other, why do you need the online component?

          Because ALL the inter city interactions are done in the cloud. This is a good thing. If your in a reigon with other people, having the coding for the intercity interactions on the client side would lead to cheating. For this reason, the code for intercity things are ONLY online. If there was an offline mode, that code would have to be in the client allowing hackers to determine more easily, how the interactions work and find ways to cheating.

          If EA had learned this as well, would they have told Maxis to find another way?

          And if Blizzard had learned their lesson after launch they would have done things different for BC. And if Blizzard had learned their lesson after launch of BC they would have done things different for WoLK
          And if Blizzard had learned their lesson after launch or WoLK they would have done things differently for Cata
          And if Blizzard had learned their lesson after launch of Cata they would have done things differently for MoP.

            Because ALL the inter city interactions are done in the cloud.

            "This pen is blue, because no-one could possibly conceive a red pen". Alternatives exist - that's why it's called "system design" and not "jigsaw puzzle". Want to make the region multiplayer? Then offload to the cloud under those circumstances. Prevent single-player regions from being opened up to public access - allow private access by invitation only. Again, alternatives exist.

            And cheating? Frankly, the interactions can be pretty easily figured out through simple experimentation; system probing and network monitoring can get further than you might think, even on black boxes.

            And if Blizzard... etc
            BC. WoLK. Cata. MoP. WoW expansions, one and all. I would think they've learnt their lesson regarding releasing an always-online game that isn't an MMO. MMO subscribers accept server downtime as a matter of course. They rant, they rave, and they go back once hte servers come back up. Sim City and D3 aren't MMOs, they don't have audiences that are as understanding regarding the impact of server downtime on their recreation.

            Yes, I would have expected Blizzard to have learned a thing or two from WoW when they handled the Diablo 3 release, but it was rebooting a sixteen-year-old franchise. How can you forecast the adoption rate for a game where your last launch statistics are over a decade old, and no recent parallels exist? Ditto for Sim City. But all of this misses the point. The lesson wasn't "make sure you have enough servers on launch day" - that's a great way to lose all the revenue that you get from sales by wasting it on hardware, overtime and upkeep. The lesson was "don't do always-online unless you absolutely have to, because your servers will fall over on launch day, guaranteed, and will result in a PR nightmare".

            I'm tired of arguing this. At the risk of sounding somewhat hypocritical at this point, please consult xkcd #386 if you feel compelled to argue further, and reconsider whether pushing your views in an anonymous forum (nay, comment section) to persuade one person you will likely never meet is a valuable use of your time. I wish you luck in your future endeavours, but I have better things to do. Goodbye.

      Even then its been proven that their "scapegoat" doesnt even hold up:
      http://www.rockpapershotgun.com/2013/03/12/simcity-server-not-necessary/

        This is exactly what the Devs have stated publicly the entire time. This isn't new news, it just demonstrates RPS's inability to understand what people say, or their choice to use selective hearing to only hear everyother word.

        Something along the lines of "servers handle calculations ................. cities and can't be done in the client." where the .................. is the word between.

    SimCity 2014, we even moved the Reticulating Splines to our servers

    I would disagree they "picked oceanic server 1 first". For me when I manage my servers I patch, maintain or upgrade the least used servers first. Once that live test bed has passed all the checks and balances you then roll out to the rest of the servers.

    Standard operating procedure if you ask me, not the PR spin given by some random picking of a server to be the first patched. It's simple economics if things went from bad to worse then you want that to happen on the least busiest server first.

    I'm far more worried by the incredibly bad 'agent' programming that results in stupid, unrealistic swarming behaviours for all services, and the ridiculously tiny 2x2km city size.

      Work with the engine, not against it. Learn how the agents opperate and design your city arround that.

      A few people have figured it out and have built massive cities. You just need to figure out how the engine works and develop strategies to cope with it.

      Could you imagine the memory requirements if a city of even 10k people had to have assigned work and home locations for each and every singel one? Imagine the size of the database!

      I admit, it would be cool to be able to go into the DB and look at all the data, but the ram requirements would be masive.

    “When people can get into the game they’re telling us about how much fun they’re having, and how much they enjoy this new approach to SimCity,” says Lucy.

    Well, that's a lie since the frequency of interacting with other players is low, boiled down to things like sharing resources and other slight variations. Everyone would have enjoyed the game just as much if they didn't have to interact with another person.

      You are the problem with modern society.

        Because I don't hide behind lies? I'm okay with that.

        She, on the other hand is pissing in my mouth and telling me it's just as good as water. If you want to drink it up, be my guest.

          I was reffering to your statement that you would enjoy things just as much if you never had to interact with another person.

            Well, I referred to this game specifically so you've misinterpreted my OP.

    Antarctica 1 REPRESENT!

    I would say only one thing : Origin Sucks!
    Anytime I see big titles coming out on this platform it's a disaster and somehow I am not even suprise that simcity does not even work after one week being out. BF3 was an absolute nightmare and now the same dinner is served again by EA Origin with the new Simcity. that platform need either to go or EA need to employ people that knows what they are doing.

    Last edited 13/03/13 6:39 pm

      Sim city has been working fine since last friday. Haven't had a problem at all.

      WTF would you know. You don't even play the game. All you do is look at the forums with all the other winers like yourself who refuse to play the game because its EA and Origin and talk shit about things you have no clue about.

      SimCity is great. Had problems the first 2 days like everyone else but since Friday the biggest problem I have had is getting off it to go to bed.

    Fascinating read here:
    http://answers.ea.com/t5/Miscellaneous-Issues/Traffic-quot-AI-quot-This-is-why-services-and-traffic-are-broken/m-p/737060

      Server issues aside, that seems pretty unforgivable in game design

    The oceanic servers were fixed? I haven't been able to get on them since I bought the game 3 days ago. They are always full. I've been playing all my games on the Antarctic server!

    I've also had multiple problems with not being able to save the game locally. The servers always bugger up and don't save the last 10 or so minutes of the game when it boots me out. I'm mostly over the game now. It took two days of errors and bugs to make me tire of it.

      Funny, I'v been on both oceanic servers since Friday 8th without a problem.

Join the discussion!