Trying to buy the Nintendo Classic Mini NES from EB Games this week has, to put it mildly, a nightmare. And earlier this afternoon, EB Games published a statement outlining precisely why.
After their website crashed on Monday, EB announced they were increasing server capacity to handle up to 60,000 requests at once. They thought, at the time, that would be sufficient to handle the excessive demand for Nintendo’s mini console.
Plainly, it wasn’t. And in a statement published on Twitter, EB explained that they had over 7.5 million page views yesterday – a day when they would ordinarily have around 500,000.
Please see our statement regarding the Mini NES. pic.twitter.com/Re8evKKGMI
— EB Games Australia (@EBGamesAus) November 16, 2016
The full statement is reprinted below:
Despite juciing up our servers, our website just couldn’t cope with the record traffic of tens of thousands of enthusiastic gamers. We were running 45 servers, each with 32 CPUs, for atotal of 1440 CPUs handling the website.
On a normal Teusday, we have about 500,000 page views. Yesterday we hit over 7,500,000.
Thank you for proving like us that you love the 80’s, unfortunately Nintendo is unable to supply additional stock this year. We pride ourselves on responding to all our customers as quickly as we can. Right now, we have over 40,000 comments and messages so we may take a little while to respond.
This does not meet our usual high standard, however the demand for the Mini NES has been unprecedented.
On the bright side, I hear Target and Big W might have some more stock of the Mini NES for December. Failing that, there’s always Ebay.
Comments
71 responses to “EB Games Explains Why The Nintendo Mini NES Broke Its Website”
yeah but what about RAM and bandwidth for these servers?
They also seem to have absolved themselves of all blame for any inefficiencies in their own code.
Yup and also their DB
No good having all that grunt and poorly configured load balancing
Maybe they should use MongoDB. I hear it is web scale:
But, but, but ….. the site works so well with Microsoft Access….
😛
lmao I like how people in the comment section go ape shit for the NES
All that hardware is good but the major failing was the workflow; once the item was in a user’s cart it should have been temporarily ear-marked for that user and the stock count decremented.
That way even if the servers buckled those who got the item into their carts didn’t lose out.
EB, Amazon, you name it and the most common complaint was the item being added to their cart only to have it disappear on before they get to the confirmation page.
This is a simple solution but from experience ive seen people get around such restrictions by opening multiple tabs and hitting Add to cart all at once causing SOH to go into negatives when the server normally wouldn’t allow it.
Thanks, I’m actually a Web developer these days and I keep forgetting those simple things.
Even if the cart is cookie based that will alleviate the multiple tab solution but it won’t solve the problem if the user is using multiple browsers and even multiple machines.
The only way around this is to track the cart server side and we are back to square one again, 🙁
Yeah its a incredibly annoying issue, but hey if you can work out a way to fix it patent the shit out of it and make some dollarydoos 🙂
I haven’t done coding in a long time, but I feel if its done server side, the only way to cleanly do it is to have a lot of checks done on the data, and move to a queue based system.
Something like:
Check SOH, If SOH is positive then assign IP address a unique queue number and minus SOH by requested amount (you could limit it to 1 per IP addresss) then Check SOH is not negative and if that’s fine proceed to payment.
Then if SOH on the 2nd check is negative the system can stop the payment and inform the customer they are out of stock and hopefully update the webpage as well.
This is the thing, are solutions already out there. They may not be a cure all but they are out there.
So far when I’ve booked airline tickets, tickets to ACMI or just plain old passes from TickeTek, I am advised on the site the tickets have a hold and if I wish to keep them I need to complete the transaction within minutes.
If I don’t, the hold is released and I have to start again.
Given the limited stock, something like this should have been in place. Again, it doesn’t get around the tab issue (or multiple browsers for that matter) but it is far better than the free for all we saw the resulted in carts becoming empty before payment confirmation.
Heck, some even got multiple payment authorisations but no order confirmations so it shows the transactions are completely non-atomic.
yeah unfortunately not all web devs are created equal 😛
Yeah, this is pretty much the entire issue. I can look past sudden spikes in traffic disrupting the website but EB Games is a very pre-order focused company. They’re not new to this. At this point they’ve been through enough website revamps and huge pre-order rushes that they can no longer justify their standard inventory item with place-holder price tag system.
Given how eagerly they pursue pre-orders I would really expect the EB website to be the top tier of pre-order systems but I can’t even do basic things like electing to pay off my entire pre-order once the price has been finalised. I have to wait for them to randomly deduct the remainder from my account.
Given that their problem was with the site getting overloaded, this sounds like it’d make problems worse.
With a regular web store shopping cart, you should be able to browse the site and update your cart using a read-only replica of the stock database, with the cart data being stored in some ephemeral session storage linked to the user’s browser cookie (you could use memcached or equivalent for this). You only need to hit the master database stock when completing the checkout process. And if the user leaves the site with things in the shopping cart, you can simply expire their session storage without having to touch anything else.
If adding things to the cart reserves stock, then you’ve suddenly got a lot more web requests that will need access to the master database (or some other central reservations database). And when expiring the data associated with abandoned shopping carts, you’ll again need write access to the master database to release the reservations.
Given that most products effectively have unlimited stock (the retailer has it on hand, or will order it in response to your purchase), this added complexity isn’t going to benefit the user. It’s only these limited stock/high demand situations where it matters, and you could argue that a more scalable site is going to be more valuable here.
LOL, the only thing their post explained is that their network engineer has no idea what they’re doing (or at least the person who wrote the statement for them).
As @pointzeroone mentioned, the metric basically means nothing and is almost certainly them just quoting the maximum potential of their cloud provider. I suspect they turned on unlimited scaling for the website itself, but still had a single db instance.
1440 CPUS, one Ethernet cable….
Given Australia’s Internet, would it be mean of me to also imply the cable is only 10 Mb/s?
probably also powered by a potato battery 🙂
Might explain the outage.
Worker #1: “Frank, have you seen the potato powering the server?”
Worker #2: “No, but I just made fries. Want some?”
Worker #1: “Why not?”
It used to be a hamster dieing on its hamster wheel, but since Portal 2, its been a potato battery. Aint progress grand? Not sure I’d want hamster fries…
Aren’t emulators a thing?
Ebay…. HAHAHAHAHAHAHAHHA. Fuck no I’m not paying some asshole scalper.
Considering I would guess most of the ‘pissed off” people who missed out were scalpers who planned to flip them on Ebay, there’s really no great loss.
** PUTS ON FLAME HAT **
People who truly wanted them had the opportunity to pre-order, especially since most places who offer preorder don’t require full payment, so its pretty risk free. And its not like the retailers would have had a hard time reselling a missed preorder. Also, as a society, are we that impatient for things we cant wait another month or two for a restock? Why can’t people be this passionate over endangered animals, destroyed forests and the killing of the planet they rely on to actually live…?
This. Totally. And I know it’s not a popular view.
Look – I can understand being cautious about preordering an unfinished game. There’s risks. The game might not be all you expected it to be.
But the NES library is pretty well documented. Those games are 25-30 years old. When the MiniNES was announced it was pretty clear what you were getting. 30 emulated games, HDMI connectivity and a replica controller. I didn’t even decide I wanted one til about a week after announcement – and my partner had no trouble preordering one for my birthday. And at the cost of one AAA game ($100) it was hardly a risky venture.
I can understand peoples disappointment at missing out – even if they decided late that they wanted one. But I can’t understand the white hot rage some are displaying (not many on Taku anyway). If you’ve been waiting 30 years – what’s an extra month or two?
My NES Mini is collecting dust….
Some users here have been suggesting emulation.
First and foremost, those things are illegal. Owning the cartridge, etc., does not make your ROM image legal; the ROM itself is an illegal copy thus having it in one’s possession means one is holding an illegal copy. Period.
In terms of penalties, they are harsh for individual but for corporations they are in the tens of thousands (32,500+ per offence from memory).
Truth be told, I’m no lawyer so take what I say with a grain of salt but I myself avoid such things as I’m self-employed and don’t want to find out the hard way that being self-employed is enough to get the corporate scale penalty.
One last thing; making a backup of your own PlayStation disk, etc., is also illegal as to do so you have to circumvent the copy protection which is also illegal.
Not saying this to make anyone stop; just getting the facts out there so others can weigh up their little protest against the (Japanese) Man for not manufacturing enough stock against the legal ramifications if they are caught.
I guess making a backup copy via breach of copy protection is like the old ‘tree falling in the woods’ aphorism.
It’s a copyright mater, that’s a civil mater. And the imposed penalties have to be based on real world losses incurred from the copyright infringement.
Call me skeptical but since when has the real world mattered in the court room?
Justice Nye Perram thought it mattered, when he bounced DBC LLC out on their ass for pathological inability to claim fair costs.
That’s the exception, it is not the norm.
Think about it; one easily gets harsher penalties for downloading a movie than for rape and murder.
Really is a stick time we live in when violating copyright is deemed more damning than taking another human beings life and/or dignity.
Well, I haven’t seen any cases of the studios successfully doing that in Australia, so I think it might actually be the norm here? In the US though… sure. It’s insane over there.
And yes. Treating ‘corporations as people’ for the purposes of the law and damages is insanely immoral.
Name one case in this country where someone has gotten that harsher penalty for copyright infringement. That isn’t distribution mind you, we’re are talking about downloading.
Since that’s how copyright law works in this country.
Ripping a ROM is legal, its what is behind format shifting. There is no DRM breach, as its a physical rights management, not digital. Basically, the same theory behind why its OK to rip wav files from a CD to mp3’s.
Thats all based around the Telecommunications Act which allows for a copy, or reasonable facsimile, of any digital data you purchase. Intended for software purchases, it extended to entertainment once things went disc shaped, and the key part is ‘reasonable facsimile’.
It allows from a copy that only has to be functionally the same to the user, not exactly the same.
Its something thats never been tested as far as it could be though, so I asked a lawyer mate. In short, they told me to go look at the Hansard 2nd readings on the Act’s, which are basically where they formally state the intent of an Act and what its trying to achieve (a kind of overall summary). Under both the Telco Act and Copyright Act it was clear they were trying to protect the consumer when they were just trying to do the right thing.
Which to me says format shifting negates DRM.
If you upload that copy, its a different situation, but if you are just downloading a copy of something you already own, which law takes precedence?
Isn’t the software in the cartridge is protected by copyright law, though? That means even if the circumvention of the 10NES lockout is ignored, an unauthorised duplicate is being made.
Furthermore, while pre-dating the laws doesn’t the 10NES lockout chip count as DRM?
Copyright laws are there, and yes, they offer some protection, but the backup laws under the Telco Act are specifically there to work alongside copyright. They arent meant to be exclusive to each other, but work to allow legal breaches of copyright.
Format shifting has been around since before consoles (was an argument when VHS recording became a thing), and thats one of the important parts here. Basically, if you’re format shifting, you’re fine. If you arent, your not.
Its been tested with VHS, mp3’s, PS1’s, and now movies and TV shows, and in almost every case (I’m thinking that mini antennae business in the US might fall under this. they lost), format shifting (or a variation) has been the winner, no matter what other DRM questions have come up.
Hm.
Is ripping a ROM format shifting though? The software starts off digital and ends up digital. For want of a better term, only the container changes; the binary format is identical before and after the process.
With tapes, one is going analog to digital and CDs while not so much their binary representation changes in the process.
Yes, it is. The last entertainment medium that was analog was vinyl. Everything relevant since has been bits and bytes, VHS and cassette’s included, which are magnetic tape set on or off to provide the data.
CD to mp3 is the easiest example, where there is a significant technical change from one format to the next, but no practical change to the user. For the most part (personal preference aside), it still sounds the same. Some might prefer one after the other (the wav format of a CD is definitely a superior format), but the core song at the middle is the same in both formats.
The format shift with a ROM isnt so much in the software, but in the use of that software. It shifts it from a cartridge based OS to a software based OS. Like with MAME switching things from a hardware to a software base, with software emulating the hardware programming on the motherboard.
@grunt
After work I might look further into that. As far as I know, VHS and audio cassettes (unless otherwise indicated) were in fact analog; there is no such thing as bits and bytes with them, only wave signals.
Either way, I think I’ve let this tangent run long enough. Thanks for the alternative view.
You could be right, not going to bet the house on it 🙂
That aside, good discussion, hopefully you can see that its not as simple as “breaching DRM can only be bad”, and that there are other things muddying the water.
For me it simplifies down to the issue that if you own it, you shouldnt get in trouble. Thats not the intent of either law, and the Hansard readings support that.
If you dont own it, different story, but it is the uploader thats the first one to break the laws, go after them.
@grunt: There is still one element that concerns me though. If I may propose this thought experiment.
I download, say, the new Pokemon Sun and Moon games right now and get caught.
I am caught and found guilty of infringement (obtaining a copyright protected work from an unauthorised source, for example in Legalese), which scale of penalty do I fall under?
For individuals (from memory) it is around $1,600 per offence (two in this scenario). But, given then I am a sole-trader, am I charged as an individual or as a corporation/business where the penalty is upwards of 32,000 per offence?
I see this when I have my taxes done; I am expected to document and pay corporate scale tax but benefit wise I only get individual scalings.
VHS tapes and cassettes that were generally consumer targeted, ie music and films are analogue.
There are digital variants of these technologies but were mainly used in professional audio and film. DAT tapes were not much different from regular cassette tapes but a DAT recorder wrote digital information on the tape. ADAT tapes were pretty much the same, but with VHS sized tapes.
Yeah, being self-employed forces you to take these things way more seriously. It doesn’t make us any more likely to get busted but on the off chance it actually happened the stakes are ridiculously high. My friends always crack the shits because I won’t let them use my WiFi, but they don’t get that what I’d be risking by letting them download crap they’re probably not going to watch/play anyway.
Maybe we should email ausops@ebgames.com and ask them why the site failed?
Somethingsomething IBM, somethingsomething didnt test rebooting a router?
Alex’s second paragraph says to me – a layperson – that the server capacity being upgraded, and attaching a number value to it, could possibly be interpreted that they would be able to adequately serve 60k people. Who want to buy the same thing.
That’s not Alex’s wording, it’s EB’s Monday announcement (post-crash). I know better, of course. But that could lead people to believe (word-of-mouth, social media, etc) that EB was drowning in the things, so it’s open slather.
The ‘big deal’ statement in the tweet is more bragging, if we’re being really critical here. Turning an apology into an ad for similar products down the road.
Not only do they neatly excuse themselves from any responsibility (ie they never actually state how much stock they had, but of course they are under no obligation to do this) after Tuesday’s debacle, they effectively erase anything they may have said or promised they could do on Monday.
In doing so, they move the blame onto Nintendo as if it was Nintendo’s fault that one of their retail partners kept flogging a dead horse since Thursday and they had already allocated Mini NES units to all other relevant parties for sale.
I say again: if Nintendo is to be criticised, it should be criticised for having to be at EB’s whim and not service the other retailers in this country. And for not utilising its own official Ebay page.
What’s the bet that EB Australia absolutely dwarfs Nintendo Australia?
This isn’t defending Nintendo, nor is it attacking EB, all I am saying is it’s another example of how games retail in this country is severely disadvantaged because it’s such a niche market.
Can EB tell us if its site was attracting hits from international visitors? That would pour even more salt into the wound.
Part of me wonders whether some amount of DOSing by someone who has one already was involved. Kind of a “Haha I have one and you can’t have one” type affair.
Nah, I doubt it. But that would make a good comedy novel so get writing! 😛
I bet all the EB Games staff members got their hands on a console if they wanted it. Am I right?
Wrong franchise, mate. You’re thinking of Dick Smith, 😛
Can vouch: nope, no chance, no sir. We had the exact same amount of opportunity as the rest of the general public. Most of us have no Mini NES and a shitload of abusive phone calls in our direction instead.
Am I, righto.
EB Employee here, can confirm that the only staff who got one were those who preordered, like everyone else (I didn’t want one so never bothered with it, still don’t want one for that matter). They were up for order since August and we were letting people know about it since then, and hardly anyone card about it until about a month or so ago, so to all the people abusing us and the social media team because it’s apparently our fault you missed out, no, it was your own fault you didn’t preorder months ago when there were plenty left.
Depends on the circumstances. If they came too late then it is the fault of the customer.
But if the item was in the cart and the user couldn’t confirm payment or (as some have reported) they got payment confirmation but not order confirmation then yes it is the fault of EB for not having the IT up to scratch for the rush.
Some cases maybe the fault of the consumer but at the same time the handling of the scarce stock is still the fault of EB.
At least some of us are being constructive; like how I for one am proposing putting extra measures in the event there is high demand of scarce stock.
Ball is in EB’s court now. They can either let history repeat or learn from it.
Shhh careful with the tone around here.. people don’t want to/like to take responsibility for their failed actions.. its everyone else’s fault remember?
The joys of working retail; it’s always our fault and never the customers’, even when it’s beyond our (the store workers) control. That being said, if anything Nintendo should really be held most responsible, they did this with the wii and the first wave of amiibo, under stocked their retailers to create a false hype, and I can see this happening again in the future with Nintendo products unfortunately, it’s really just a vicious cycle with them and time again consumers miss out and retailers have to take the flack
I know for true that uncollected Mini NES preorders have gone to staff on preference. Don’t pull that shit. You guys (not all) take games home and play them. You sell us used discs as new. You fail to stock shelves and send us away because you can’t be stuffed unpacking gear.
I was refused preordering the Mini NES in store at the time. Now why would that be?
Don’t take the high ground. You can’t. Your employer’s standards are low.
Look at what happened. EB gets business because they lived longer than the competition. You are video game Catholicism.
Can we expect a statement from Nintendo itself?
Apologies if I’ve missed the “We reached out for comment but had not heard back at the time of writing” post.
Note I’m not asking, just if Kotaku/any local gaming media has sought anything from AU HQ.
EB’s big fail here was promoting the thing so much. They only have themselves to blame for the huge influx of traffic over this. An influx in traffic that, might I add, was not experienced on other sites.
When you announce to the internet on your website and all forms of social media that an extremely limited item that a lot of people want will be going on sale on a certain day at a certain time, and the only way they can obtain this item is through your website, you don’t really have the right to excuses when your website can’t handle the load.
Hopefully Nintendo has learned their lesson over this and if/when they release the mini SNES, they will make sure there are more than enough units to go around.
Oh poo, I meant to make that a reply. Sorry.
But is it promoting or informing? They have to put the message out there to try stem the tide of enquiries that they would be flooded with if they said nothing at all.
That doesn’t make sense, though. It’s not exactly a different SKU like the Slim version of a console that Nintendo is already currently manufacturing.
Nintendo’s at great pains telling us that it is just putting out a singular device now – the Switch is the be and end all. It would be contradictory to revert back to manufacturing two or more different consoles and actively support them as demand dictates.
There’s conservative business practises, but then there’s saying you will do something and then going and doing the complete opposite.
They should have had a ballot system. Give people a few days to register with a credit card so they don’t all pound the website at once and then notify the winners via email. Their infrastructure would have held up and everyone gets a chance at getting one.
I was about to suggest the exact same thing. Knowing full well that the demand for the thing was enough to destroy the web-store, why not just get people to register their interest, and pull names out of a hat? Sure, you’d still have a lot of customers missing out, but at least you could say it was fair, and we wouldn’t be at the mercy of the technology.
They’ve had these things available for pre-order since August, what more do people want? Based on the interest through pre-ordering, I expect everyone thought there was enough on hand, but given the timeframe to Christmas, it seems to me that a perfect storm of several things all happened at once that changed that – mostly timing and price.
$100 is an ideal gift price for this sort of thing, and being 6 weeks to Christmas a lot of parents are either finishing their shopping, or just starting.
Nintendo is also notorious for being conservative with day one stock.
This a global thing as well, dont think its exclusive to Australia and EB.
We hyped the shit out of something we knew we would have very little stock of.
This stopped you going to other retailers, got you to our site and kept you trying to use it instead of going down to Target.
We also generated a ton of press. Bad press about our website, sure, but a shitload of free advertising with the EB Games logo and that we sell the Mini NES.
All in all a win for us.
We’ve done it before, and will do it again soon.
All the people losing their minds over this is hilarious… It’ll be available again after christmas.. and you know what? It’ll likely be a lot cheaper too in post-christmas sales.
Those people that also bought for $500+ on ebay… what in the world were they thinking?!?!
buying the mini nes has a nightmare!
Websites crash, life goes on. Being able to buy something is not a right its a privledge. Some people wish they could buy food and they dont flood comments sections or call people and give them abuse because they want to eat. Calm down, it’ll be alright.