When EB Games took pre-orders for the Nintendo Mini NES Classic earlier this week, the website crashed under the sheer volume of traffic from eager buyers. This happened two days in a row, leaving a mob of angry customers in its wake. With all the hype around the classic console, you’d think EB Games would have expected the level of traffic to its online store and worked to ensure its website was reliable. We take a look at where EB Games might have gone wrong and what businesses can learn from this debacle.
People who were trying to buy the Nintendo Classic Mini NES from EB Games on Monday were pulling their hair out after the website crashed because the servers in the back-end couldn’t support the amount of traffic that was coming through. Many customers who had their hopes up after adding the Mini NES to their cart ended up missing out on pre-ordering the console because they were stuck in checkout hell – the website wouldn’t let them complete the transaction. Other customers didn’t even get that far as the page failed to load at all.
So the next day, EB Games beefed up its servers. But that wasn’t enough and the website crashed again on Tuesday. The company released a statement to appease angry customers who missed out on the pre-order due to the failures of the website:
“Despite juicing up our servers, our website just couldn’t cope with the record traffic of tens of thousands of enthusiastic gamers. We were running 45 servers, each with 32 CPUs, for a total of 1440 CPUs handling the website.
“On a normal Tuesday, we have about 500,000 page views. Yesterday, we hit over 7,500,000.
EB Games hasn’t given us anymore detail about the hardware and software supporting its website but we can make a few assumptions based on what happened during the website crashes. Some users who visited EB Games online were greeted by the Microsoft Information Internet Services (IIS) welcome page so it’s safe to assume that is the web server that is being used. There were some customers who saw a message on their browser that pointed to EB Games hosting at least a portion of the website or web app on the Microsoft Azure public cloud platform.
First off, lets address the hardware. The 45 servers should have been enough to handle 7.5 million requests in one day, given that they’re not concurrent. There are online retailer websites out there that handle more requests with fewer servers than that.
Now, assuming that the website is hosted in a public cloud environment, you’d think that you could just spin up capacity to meet the demands on the fly. After all, aren’t public cloud services all about scaling up on-demand? Not always.
If you want a web app to be scalable, you need to build elasticity into it which should be factored in during the development process. That may not have happened with EB Games online.
If the web app itself isn’t that scalable, there are other measures that EB Games could have considered to handle the heavy traffic coming through. On the web server front, perhaps EB Games should have used NGINX instead of IIS. The NGINX open source web server is good at handling concurrent web requests. This might be a controversial suggestion given there have been benchmark reports that have been published that claim IIS outperforms NGINX, but there is also a lot of internet chatter among IT professionals that question those results. It’s a nuanced topic and I’d love to hear from IT professionals out there who have used both IIS and NGINX about their real-world experiences in the comments section below.
So let’s assume the EB Games web app isn’t built for scaling and it could have used a better web server. It would take a long time to overhaul all of this and the company needed to find temporary fixes to try and ensure the website doesn’t crash again the next day. EB Games has already said they had ramped up its servers in preparation of day-two of the traffic onslaught, but that still wasn’t enough.
One thing the company could have done was implement more intelligent load balancing, spreading the pre-order and checkout specific traffic across multiple back-end servers dedicated to this task alone. Each of these back-end servers should also have session limits in place to ensure the server can respond successfully to users whose translations are already in progress. They could also set specific rules for session limiting using a web application firewall.
EB Games might retort that there was session limiting but it seems the company based it on the total amount of concurrent users. This is a big problem when you consider how many customers that got in early to add the Mini NES to their carts only to get screwed over at the checkout process. One way EB Games could have prevented this was to prioritise session IDs of users who have already started the buying process so that they could complete the checkout. That way, it frees up a space for another user to complete the checkout process.
We may never know exactly what happened behind-the-scenes on the EB Games online store and I doubt the company will be releasing anymore comments about the Mini NES pre-order debacle. Please bear in mind that we’ve made suggestions based on what we think might have been happening in the back-end. But the incident has served to highlight the importance of good solution architecture for web apps, especially for businesses that deal with online sales.
As customers continue to favour online shopping, your retail website serves as the face of the business. It might be the first thing that your customers interact with when they consider doing business with you and first impressions matter.
We’d love to hear from the IT community about other ways EB Games could have handled the pre-order situation better. It’d be great to get some healthy discussion going on about good solution architecture for websites.
What are your guesses as to what EB Games’ architecture is? What strategy do you think the company used to handle Tuesday’s traffic? Let us know.
This story originally appeared on Lifehacker
Comments
35 responses to “What EB Games Can Learn From The Nintendo Classic Mini NES Debacle”
lmao: leaving a mob of angry customers in its wake!!
This was awesome, so many tilted gamers were TRIGGERED!
http://lolworthy.com/wp-content/uploads/2016/02/triggered-h3h3-gif.gif
What EB games learned from this issue: Holy crap! Look how many people want this thing, get on the phone to Nintendo and order every bit of stock they’ve got. Don’t worry about the website, just keep rebooting the servers every day and people will still keep coming back for more!
Maybe it’s Nintendo that needs to learn not to have ridiculously limited supplies of a product with such high demand
After the wiiu they were probably shocked to have a product in high demand.
The big w website crashed aswell, but Im not seeing anywhere near as much hysteria about it. I wonder why.
Because that wasn’t their last possible chance to secure one 😛
Also they shut down their site because of a security breach I think.
Their site was taken down days before because of a security issue exposing customer details.
or maybe its the general public who need to learn that they don’t actually “OMGZ I NEEDZ THIS” piece of plastic with old games.
This hasn’t been an exercise in how Nintendo or EB failed – this has been an exercise in how immature and pathetic the consumerism movement has become
you realise you are on a site devoted to video games right? some people like old games, that is somehow less acceptable or more immature then current gen gaming?
Yeah, but it’s not unreasonable to characterize the reaction as fucking childish.
There’s a pretty damn big difference between, “Holy shit, that was some surprising demand,” and the hand-wringing and tears over this.
Wow man, did you completely misunderstand what I wrote
Gamers, the collectors and nostalgia chasers, are a passionate people. You can judge us for it, but at the end of the day opinion on entitlement and consumerism is just… not clever or insightful. We made society here a place where you accrue stuff you like. It’s an identity thing now. It why Pop Vinyl figures exist.
And ultimately it hurts nothing and no one and your attitude is better used on subjects more deserving of representation. People like you enjoy seeing people get pissed off. And that means everyone ultimately picksa person with your feedback criticising others for minor rage like Mini NES Gate as a sad, lonely, troll who envies passion others present.
I don’t enjoy seeing people getting pissed off – you don’t know me, so you cant pass judgment on me based on a handful of words you see on a screen and have clearly skewed towards your point of view
What I find amazing, that as Video game and Technology lovers, we’ve read these stories on this page (and others) over the last few days and instead of stopping to think “Wow, some people are completely over-reacting over this” we decided to take the pitchforks to EB and Nintendo instead. Really.. is that the group you want to lumped into? You want to be known as being a part of a group that gets so vicious and vitriolic over missing the purchase of an item that will be restocked in 1/2 months time?
There’s a difference between ‘passion’ and outright embarrassing human behaviour. Bombarding a retailer’s social media website with comments about how pathetic they are (and the rest…) for not being able to service their needs at a time of extreme demand is not what I would call passion. And I cant believe that a rational person could think that is passion.
If I throw my controller in rage over something in a game, and then I cite “oh I am just passionate”, 99 out of 100 people will tell me I have anger issues and took it too far. “Oh I cant believe people would throw their controller over a game, its just a game – relax” – and I would agree with that, its not passionate, its immature behaviour at its worst. In my opinion, this is one step away from that.
Nah. You’re a tear drinker, dude. No escaping it. Look how defensive you got. So many wordz.
Fair enough, your opinion
Enjoy your life crying over a missed retro console and blaming everyone else but yourself with the defence of “passion”!
Dude. Seriously. Why does crying give you such a rager?
Prolly should point out EB customer service staff were receiving death threats over the phone from the “passionate people”.
Yeah. it must be like how you have Muslims and Fundamentalust Muslims right? Being passionate is bad! And everything is as black and white as you say!
>Comes to a gamer/technology page
>Insults gamers/technology buffs for their passion
>Argument is nonsensical
Seriously though champ, step off on the judgement 🙂 There’s alot of relevant information from this debacle that could be put to a better purpose by EB on managing events of this nature, though personally I dont think Nintendo will even care – limited releases drive demand up, justifying the production/development of more of them, resulting in popular products which they will make bank on.
read my comment/reply to superdeadlyninjabees above
On a side note. EBNZ now has it available for order….. importing shouldn’t be too hard
Or mass amounts of people could have pre-ordered it from their local brick and mortar store avoiding mass hysteria and a river of tears
EB apparently were not accepting on store pre-orders.
My local one was. They only had about 6 people pre-order one though.
All the servers in the world won’t help if your software and database queries are inefficient. It’s not uncommon for poor design to mean that you take down servers with a single request.
Really, I think it’s just a case of the software and architecture never being designed for such a high volume of traffic. They would have had set targets in mind about their expected daily traffic and concurrent user load then made sure that the systems were optimised to handle it, not doing any unnecessary work for possible future situations that might never happen. Even now, the choice is going to be whether they spend the time and money on being able to handle this kind of load in future or recognising it as a blip and doing enough so they can respond quickly next time if necessary.
Spot on. A chain is only as strong as its weakest link. If your web servers are all referring back to one database, that becomes the bottleneck. The only solutions to that are to beef up your database server, rely less on database calls (cache! cache! cache!), or design the system to split calls to multiple database servers .
In any case, you’re completely right in that the EB site is just not designed for such high traffic. I imagine they’ll be doing a redesign of the base code very soon as a response to this.
If they usually get 0.5 million requests a day, and got 7.5 million that day, it is probably safe to assume that most of the extra 7 million requests were concentrated around the release time. So it was most likely a lot more than just a 15 fold increase in traffic.
Why do EB need to learn anything from this debacle? As much as commenters on here like to (justifiably) vent their rages at EB Games, we are not their target market. Most people on here are the people that will take their time either studying every market place to find the best deal (rarely EB), or make the bulk of their purchases for the PC (mostly non-physical). EB is the impulse item store, or for when a family member wants to buy a present.
EB probably didn’t want to throw a bunch of cash fixing their servers for a once off spike in traffic, especially when they know that there was no way that these crashes would adversely affect their sales (it still sold out, and I’m sure if the mini SNES appears, that too will sell out in minutes once the hype train is at full speed)
I don’t think there’s anything they CAN learn from this.
I mean… millions of people were trying to buy something that only existed in the tens of thousands. I don’t think there IS a winning strategy for a scenario where almost everyone involved is going to be walking away empty-handed.
This was always a losing proposition, it doesn’t matter what the fuck they could’ve done, the point is that it was always going to result in millions of complaints because people weren’t ever going to get what they wanted and were always going to shoot the messenger.
People can complain that there might’ve been various better ways of letting people pretend they stood a shot, or reinforcing that it was a fucking crapshoot, or giving some nicer, more placatory kind of error message, but it’s all bullshit. Those people were always going to complain. Always.
Maybe they can warn people up-front on their store page that Nintendo is stingy with supply and no matter how many tabs you open or how rapidly you mash the checkout button, simple arithmetic is that you are far from fucking unique and factors outside of control are going to give a million other people a shot ahead of you in the queue, so be aware that this is a dice roll, not a human right.
The only thing I can think of for these times when supply is dramatically lower than demand is to do what the brits did with the Olympics & make it a lottery. You register for the shiny thing you want, the people are randomly selected & offers are sent out a week before release then they pay for it. If it’s not paid for in a day then they miss out & the offer goes to the next person down the list.
Run that for 2-3 weeks before the release and it’d spread out the server load and give everyone a fair chance of getting it rather than people feeling like they’ll miss out unless they’re on the site hammering refresh at 23:59 in the hopes they get in before everyone else.
I’d rather see an error page which says, “There is no stock available. This is the kind of situation that pre-orders are actually useful for. But since you didn’t pre-order, I guess you won’t be too upset by this shortage.”
In this instance, hammering refresh at 23:59 would mean that you’d be at it for 12 hours before the NES Classic got listed on the page.
7.5 million requests. A lot of people tried several times to buy it. Plus requested what? How many pages did you have to go through to get to the check out. Each one is a separate request. At 7.5 million I sincerely doubt half the population of Australia was trying to buy one that morning.
I don’t care what anyone says, customers have no right to be salty and abuse EB Staff when this was available SINCE AUGUST. I can tell you there was a good 3 weeks these were available to order in stores back then.
When do we talk about customers that tried loading the browsers on multiple devices and tabs? Is that not a contributing factor? It’s about high time consumers take responsibility for not having their shit together months ago rather than placing the blame on retailers.
Lies, man. Fat fucking lies. I TRIED to order a unit at a store back then. They would not take my order. They were sending us away. You should probably read up on the whole thing before weighing in at the eleventh hour, friendo.
Also, retail is a role. One the requires you to negotiate people and their parting with their money. If you can’t provide customers with the information properly, your customer is probably going to get frustrated and make you feel things. That’s your incompetence making you feel that. Not the customer.
Funny you mention incompetence, it’s literally the reason you don’t have a NES Mini right now. I happen to know a lot about the stock levels and advertisement and dispersal of this console, I work in a store that sold them.
3 weeks they were available to preorder in store before the stock was out (at least in my store anyways), if your EB didn’t have them, did you think to try JB? EB Games and JB Hi-Fi gave you everything you needed to know, price point, description of the product, release date, the fact it was a limited edition, they did no wrong. The rest was up to you and you let yourself down. I hear there’s a few going round on eBay for a hefty amount, but I’m sure you’re really good at asking “is that the best price?”, so good luck “friendo”.
Now if you don’t mind, I have two fantastic retro consoles to enjoy, because I did the smart thing and preordered day one. 🙂
There are plenty on eBay for 2-4 times the RRP. I wonder how many people trying to buy were genuine fans vs eBay scalper scum?