Foxtel Now died in prime time last night, just as tens of thousands of viewers were logging on to watch the debut episode of season seven of Game of Thrones. Now, Foxtel has explained what happened, and what it’s doing to make things right for customers.
Here’s Foxtel’s official statement:
Game of Thrones is a global television phenomenon and the launch of Season 7 was the most anticipated television event of the year.
The number of Australians subscribing to Foxtel’s streaming service, foxtel now leapt by 40% in the 48 hours prior to last night’s screening of Episode 1. We expect a record audience of 1.5 million Australians to watch this episode via an iQ box, Foxtel Now or the Foxtel App.
The combination of new foxtel now customers signing up and existing customers upgrading to get the Drama Pack so they could watch the show, put unprecedented pressure on our technical operations. Foxtel’s Identity Management System (IDM), which verifies customers’ entitlement to view content, which usually handles around 5,000 processes a day was hit with 70,000 transactions in just a few hours.
Unfortunately, due to this massive surge in demand, a significant number of customers experienced difficulty logging in to foxtel now and the Foxtel App. The system was unable to verify some customers’ entitlements. We are unable to say exactly how many customers were affected because the system is unable to separately identify customers who have a problem logging in but later successfully manage to do so.
We had anticipated heavy usage for last night’s premiere; however, the traffic that eventuated far exceeded expectations. Foxtel’s engineers are examining what steps can be taken to mitigate any recurrence.
Given that the problems were driven by the volumes of people who sought to sign up on the day of broadcast Foxtel advises others who wish to sign up before next week’s episode to do so a few days in advance.
We note that problems were experienced in the USA, India and Latin America, also driven by the extraordinary surge in demand for the program.
The system is now working and customers are able to watch the episode either on demand or at an encore screening. Encores are scheduled for today Tuesday, at Midday, and 9.15pm, on Wednesday at 1.20pm, Thursday at 10.10pm, Friday at 11.25am and Saturday at 8.30pm. Foxtel will also add multiple repeat screenings of the episode on the showcase channel.
Foxtel unreservedly apologises to customers who were affected by this issue.
We truly appreciate our customers’ patience during last night’s outage and will do everything possible to ensure that this does not occur again.
Comments
25 responses to “Here’s Why Foxtel Went Down Last Night”
I didnt use foxtel to watch this so Im curious how much people are actually paying them for what is likely a single show they couldnt watch with the rest of us.
the Drama pack actually has a heaps of good shows in it. pity it’s 720p only.
You cant have 1080p?
Nope the steaming doesn’t support 1080p yet.
lol ‘yet‘
I know right, they still make customers pay a premium for HD channels on normal foxtel cable accounts. what is this? the year 2005?
The company is SO behind the times and SO overpriced. Ill literally never give them a cent of my money until they can prove they want to actually offer a good service.
Good for you!
You get free ads though. Yes, even on streaming.
70,000 is not massive by any measure in this day and age.
Really does show how out of touch they are in terms of both technology and scale.
I don’t think you can blame them for not foreseeing a 40% growth in users in 48 hours…
Actually, I can. What we are seeing is despite their being high demand Foxtel clearly have not updated their infrastructure to cope. If at all for that matter.
Why not? Game of Thrones is one of the most popular shows on the planet, together with its HBO stablemates it’s one of the most valuable exclusive properties that Foxtel has access to, and it featured heavily in launch and subsequent promotions for Foxtel Now. How could they not expect a big increase in users?
everyone would want to get the most out their 2 weeks trial so they would all sign up at the latest point
its not difficult to understand foxtel (not you vanit)
OK that makes more sense, perhaps
My thought was why would you wait until the very last moment to sign up for a service that you ‘had to watch’ as soon as it was available. Signing up, making sure there was no issues and testing it before the event would make sense.
I could imagine this kind of thing happening if they were using a database that blocked reads when writes were occurring. If they were used to a fairly low rate of sign-ups, the writes might not noticeably affect the identity server. But if you had a bunch of people trying to subscribe right when everyone else was trying to log in to play the show, things could break.
Now this doesn’t really excuse what happened: it is a core part of their business, and there are obvious well known ways to avoid the problems (sharding, relaxing atomicity guarantees, etc). But it wouldn’t be the first time a company has failed to get their product to scale.
Glad to see a reply from someone else that understands software architecture. Applications can very easily fail to scale in non-obvious ways in trivial parts of non-trivial components, and that’s very hard to even test ahead of time. I’m all for shitting on Foxtel, but this will continue to be a problem for software engineers until the end of time.
of course people were going to sign up in the last 48 hours they’re offering two weeks free
I’m happy to report that TPB had no such issues.
Yup, Downloaded the torrent to my seedbox in europe. Then streamed it from there. Took all of 5 seconds.
News is putlocker was able to pump it out at 1080P with no probs.
Seems more convenient that seeing it on Foxtel, by appointment…
“Tuesday, at Midday, and 9.15pm, on Wednesday at 1.20pm, Thursday at 10.10pm, Friday at 11.25am and Saturday at 8.30pm.”
B-But TPB is blocked….
8.8.8.8
After spending about 20 minutes unsuccessfully trying to become a Foxtel customer on my computer, tablet and Xbox in order to get a 720p version, I spent 3 minutes torrenting a beautiful 1080p copy for free.
But congrats to Foxtel on the success, the first 2 paragraphs of their statement where they humble brag about how they’re just SO in demand we’re promising. I’m sure their paying customers who were unable to watch the episode were suitably consoled by the fact that Foxtel broke records or something …
hey maybe they should embrace torrents and distributed networks so this never happens again
they wont tho, they think torrent is an enemy… however it is highly efficient at getting what needs to be done and can also now watch as you download…. a “live” download