'We Do Apologise': Life At Sony Customer Service During The PSN Attack

'We Do Apologise': Life At Sony Customer Service During The PSN Attack

A few days ago, I got an email from someone who said he was a customer service agent for PlayStation. He asked me -- and, really, all of Kotaku's readers -- to cut Sony a little slack in the wake of a DDoS attack that took the PlayStation Network down for several days. Have some sympathy for the people who work there, he begged.

"You would not believe the amount of abuse we have taken from old and young alike," he wrote.

Let's call him Dan. Dan, whose real name is not in fact Dan, asked not to be identified in this story. But he was willing to get on Skype and chat with me about what it was like to be a Sony customer service agent during one of the biggest PSN outages in recent history. He told me about the abuse they took. About how they tried to maintain civility while dealing with customers both new and old. About the threats from kids and parents and people of all ages. About how people like him -- the ones who weren't getting paid six- and seven-figure salaries to run this monolithic corporation -- were taking an unfair amount of abuse.

It started on Christmas morning. Dan was stoked to go to work -- sure, it was a holiday, but he was getting triple pay, and he had volunteered thinking he'd be part of the giddiness of the day. "I had hopes of going in, being part of someone's Christmas, making people happy and just helping them out," he told me.

Dan says he was one of the first to get to his office, which handles customer service for various countries across Europe. For an hour or so, everything seemed fine. Then, at around 10am GMT, Dan says he started getting calls from people who wanted to know how to enable their PlayStation Network accounts. They had followed the instructions, they told him, but the website wouldn't verify their email addresses. It wasn't loading for them.

Soon, it became clear that something was seriously off.

"I put them on hold and asked a couple of colleagues, 'Hey, have you guys had any issues like this?'" Dan told me. "Slowly but surely, everybody in the office was getting [the same thing]. I made jokes, like, 'Yeah, this would be the shittiest day for a DDoS-type thing.' And then it was reported to me that, yeah, we were having a DDoS attack."

As the PlayStation Network remained down, the calls just kept coming in. By the middle of the day, Dan says, they had a queue of 178 people, with wait times of 65 minutes per customer. During this flood of calls, the higher-ups didn't explain much to Dan and crew, other than the fact that it was "an attack." At this point, the job of Sony customer service was to A) keep people calm, and B) promise that the company was working on the problem. When customers pressed, they'd have to explain that, according to Sony's terms of service, the company had the right to bring down the network for maintenance at any time.

Still, the outage was brutal -- PSN was pretty much entirely down for 48 hours, and connectivity remained intermittent for days after that. That meant that people couldn't get their PlayStation consoles online, couldn't download patches, and couldn't get software updates. Games that required online connections even if played solo -- Destiny, for example -- weren't playable at all. So it's understandable that PlayStation owners were upset. Less understandable: PlayStation owners getting abrasive toward Sony customer service representatives.

"One of my colleagues, she had a guy that was just yelling and wouldn't give her a chance to respond," Dan said. "Every time she tried saying the terms of service, he was saying she can take the TOS and, you know, sit on it pretty much. And she broke down. I watched one of my friends hang up the phone and just start crying."

And then there was the guy who threatened to use his media contacts against them. "I had a guy who was like, 'I have connections to The Sun,'" Dan said. "I was like, 'And?'"

There was the one who said he would find out where they were and track them down. "My response was, 'OK, great, then I'd be able to show you in person where the terms of service says that we can do this.'"

One person even threatened to kill himself if Sony didn't fix PSN, Dan says. "We had to escalate that to the [Sony Computer Entertainment] department. I don't know what they did with it. We do take things like that seriously."

Throughout the outage, Sony's customer service agents were in the dark as much as the rest of us, Dan says. They didn't know what Sony was doing, when the servers would be restored, or even how Sony planned to avoid this sort of situation in the future.

"We don't get memos or anything from system engineers," Dan said. "Our team leaders who are in touch with the [Sony Network Entertainment] and the [Sony Computer Entertainment] people tell us, [the] agents. Like when PS3 resources came back up, my team leader stood up with a notepad that said 'PS3 resources still back online, PS4 still being worked on.' That was on the fly. I was in the middle of a call when I look over and see my team leader stand up and show me the notepad. All I could do was give him a thumbs up then turn back to the customer."

Notepads! Imagine trying to deal with a deluge of angry customers under these sort of circumstances. People would ask Dan what was going on, and he just didn't have specific answers to any of their questions.

"There were people saying, 'I just got this for Christmas, and we can't get on the network, is there anything that's gonna be done?'" Dan said. "All I could say is, look, we do apologise."

Eventually, Sony did wind up offering a bit of compensation to PlayStation Plus subscribers who were affected by the outage. And today, PSN seems to be working just fine. (Dan says he's heard that Sony has just started an initiative to bolster their servers just in case this happens again.) But it's worth considering, every time we're ready to rail on a customer service agent, that he or she might have just been through hell.

"Just talk to us like we're normal people," Dan said. "We're not there to ruin your day; we're there to try and help you, and we will help you if we can."

Illustration: Tara Jacoby


Comments

    I get people being upset about the hacks and all, but at the end of the day no matter how much security Sony or MS use if someone wants to bring it down they will.

    So getting angry at Sony/MS over downtime is such a waste of time.

      Not only that, but getting angry at the person on the support line is utterly pointless - that person isn't involved in either preventing or resolving such incidents. If you're going to get that upset and abusive about being unable to connect to an online gaming service then you probably need to take a good look at yourself.

      My problem is that both companies have been working to integrate their respective services more deeply into the new consoles such that they become a single point of failure for a large portion of the console's functionality.

      I don't necessarily have a problem with the deep integration since it offers obvious benefits, but they do need to make the services more resilient to attack. Why were the attackers able to break authentication for everyone all over the world in one go, for instance?

    The thing that confuses me is that both Sony and MS were warned by the group that pulled this crap all month that they would be targeting PSN and XBL on xmas.

    And both were somehow still surprised.

      Warned or no, stopping a DDoS isn't as easy as putting a chair up against a door.

        And that's not always effective anyway :P

        https://www.youtube.com/watch?v=C6HZQ7wBhg8

        According to LS all they needed was some specific new hardware. Now I don't know crap, but I'd like to know if Sony and MS actually made any sort of moves/changes in order to prevent these attacks or if they just ignored them.

          "Specific new hardware" was all they said, which translates to "we have no idea what we're talking about because we're idiotic douche bags". I vaguely recall someone saying that Sony/Microsoft's best chance at defending against these attacks is to upgrade their server capacity or something, which is a waste of money (if PSN/Microsoft aren't going down with their current amount of players right now, then there is no point in raising the capacity. When a DDoS happens, we just have to deal with it and blame the people doing the DDoS. NOT the company).

          Could have completely misquoted what the person said, but I think that's the gist of it.

          To add, Lizard Squad were supposedly flooding PSN with 1.2TB per second. Even if Sony and Microsoft had the money, they still would not be able to stop the attack :/

          Last edited 06/01/15 7:02 pm

            Fair enough. I'd prefer a little more communication from the big wigs about what and why though. Also I'd love to see these kids get hunted down and arrested too.

            I get that there might be repercussions on the stock market or for the very richest of the rich, but I'd value a little humanity/honesty from these people I literally throw money at constantly.

            Last edited 09/02/15 7:42 pm

    I have worked help desk and tech support in the past, and it just eats your soul. Especially in the case of a DDoS attack where there is nothing they can do to make the issue better. At that point they are just paid whipping boys/girls only there to take a verbal beating. Yes the person calling is the customer, but that doesn't mean the person on the other end is the enemy. Treat them well and they will do every thing they can. Also accept the fact that they don't have a magic wand and can't fix every issue.

    I can sympathise with this. I work for an energy company, and I started off in the call centre organising connections for customers to new properties. When something went wrong on a Friday, people were without power for the weekend. The pawns (us, me) had to calmly tell the customers there was nothing to be done until Monday.

    It's unbelievable what people will say to personally insult and belittle you in those situations. Even so, I understand where they're coming from. After all, if a customer can't express dissatisfaction to call centre staff, apart from writing a letter (that won't be seriously looked at) what else can they do?

    Still, customers need to realise that call centre staff are pawns and are doing their job according to company policies and procedures. I've seen many people break down and cry after a particularly brutal customer. Believe it or not, it takes a special, thick skinned person to be a good performer in a call centre role.

    Last edited 06/01/15 11:19 am

      I did a fair bit of call centre customer service back in the day, too. Call centres, man... the real enemy is pretty much always management, not the customer.
      Management are the ones who are monitoring your pee breaks and demanding that you to sell shit people don't need which is unrelated to their call and follow impersonal, robotic scripts and spend less time on the phone per customer even if that means their problem doesn't get solved and they have to call again, because stats-wise, two calls are better than one. Fuck call centres and fuck call centre management. Your force people to deal with abusive customers and who do they have at their back? Overbearing, critical, watchful management who are resentful of having to pay you the pittance they do.

      But the customers... They're just, y'know. Scenery? Usually really stupid scenery that you vent over as a group with anyone in the breakroom. "Man, I just had this one customer who..." Because there are idiots out there.

      Eg:
      "Why are you idiots sending me a reminder notice?! I just paid my bill yesterday!"
      "You paid yesterday, huh... what's the date on the reminder notice?"
      "It says... two days ago. Oh."

      Every day. Without fail. Every. Damn. Day.
      And that's just for one operator in a centre of hundreds. 'The customer is always right' hasn't been true for ages. The real trick is making them not feel like the dumbass they are.

      ...Of course, the other side to that coin is when you get your coworkers who roll their eyes, mocking, "Oh my god, that guy wanted to use a 44FF-D form when it's OBVIOUS they need a 44FF-E if you read that fine print thing at the bottom and have intimate knowledge of the product. It's almost as if they don't deal with these forms all day every day for years! Those idiots."

      After all. No matter who you are, where you are or what the situation, it's always everyone else who's the problem.

        Hahaha oh man, agree with every. Single. Word.

        my favorite call ever was one guy called up because his mouse wasn't working. Keyboard worked just fine but he couldn't control the mouse. Asked him if he had an optical mouse and I got the response of "Uhh..." So I asked him to pick up his mouse and let me know if there is a red light at the bottom. He told me there was, and I asked him to check and see If something was obstructing the light, like a piece of tape.
        silence.
        silence.
        silence.
        "...Yeah, it's working now."
        I could hear some of his co-workers laughing in the back ground, and I had a hard time not joining them. He didn't even give me a chance to request he take our customer service eval before he hung up.

    I'm usually so polite to customer service people, the only exception being the once where I yelled at an EB Games customer service manager for over-selling Zelda 3DS XL pre-orders and cancelling mine the day before it was meant to arrive and forgetting to tell me. They gave me $50 and I didn't really feel bad, especially considering that EB Games is just a fucking terrible company.

      Yep, that's a pretty shitty thing to do. Shame on you EB manager!

      I was an employee and my BOSS sold my ORDERED AND PAID FOR Zelda 3DS (that was for my partner's birthday) to a customer when there were none left. EB totally encourages shit like this too. Luckily, I was friends with the guy who bought it and he was nice enough to just sell it back.

    I read the Not Always Right site regularly and it shows how utterly repulsive and horrid people can be without any justification on a regular basis. There are many stories of people being absolutely wonderful to counterbalance it, but it's never quite enough to remove the feeling that hell is other people. It's why I just hide in the corner at work and avoid anything that requires me to talk to clients.

    It would seem that they had the wrong script. Customer Service reps saying Sony is allowed to do this in the TOS is a ridiculous response and would anger anyone. Saying "we're under attack" would have put all the anger to the attackers and put the callers on side.
    You can't stop DDOS easily.
    You CAN manage customers and call centres better.

    Last edited 06/01/15 11:25 am

      Any script is bad. I work in IT and it's pretty infuriating when I have to to through all these basic steps when I know that isn't the answer. Let's get to the problem!

        oh man, those days... I still have flash backs... I have been in war zones, and my time working help desk and dealing with scripts has scared me more than incoming mortar rounds.

      I agree with this, but there's probably some stock-price-related reason that they can't up and admit through customer service channels that they're under attack.

    Working and managing customer service (in insurance - please forgiver me) I do feel sorry for these guys. However they need some good customer service counselling and training as well to help them. Quoting TOS, legal wording or anything along that nature is definitely the worst thing you can possibly do. You need to explain the situation clearly and calm the caller down. As soon as you throw TOS or T&Cs at people they get very defensive, then aggressive quite quickly.

    It's not easy and some people are beyond assistance but it can be done. How the Sony employee appears to present that day shows that they do not have the proper CS skills provided to them. Not their fault either.

    The only thing I'm thankful for working in a call centre in Australia is that I have the same accent as most of the people that call up; I mean the way people treat call centre staff from here is bad enough, but can you imagine the amount of abuse centre workers in India and the Philippines must get?

    We've even had customers refuse to talk to some of my workmates because they wouldn't believe they were speaking to someone in Australia, just because they're people who've immigrated to Aus and still have a bit of their accent.

      My girlfriend used to get that a lot because she still has a British accent and also said that the Indians and Philippinos who worked in Australian call centres got the worst of it.

      Basically, customer service work is something I would never put myself forward for, but I know enough not to get angry at the person on the other end of the phone when I'm ringing them up. It gets you nowhere, doesn't help the situation in any way whatsoever and the person on the other end is more likely to think I'm a tosser and will be less helpful than normal. Some people don't understand that though :(

      I remember around 10yrs ago, Telstra's big boast was that none of their inbound call-centres were off-shore, and for the most part, that was true... there was a bit of a media fluff about off-shoring a call-centre, but that was actually IBM who were doing the IT for Telstra, not the same thing.

      That policy has since gone away, but at the time, no-one believed it anyway because a lot of the high-turn-over centres (outbound sales, 1st-contact tech support) were untrained backpackers and immigrants, so there was always a mix of accents in those areas, which made people THINK it was off-shore.

      I wonder sometimes if the reason Telstra abandoned the 'no off-shoring' policy is partly because everyone already assumed they'd done it, which made oz-only centres an expensive waste of money on goodwill they couldn't keep.

    How can anyone be calm and collected and rationally respond to someone who is threatening to kill themselves over not being able to get online and play with other people is just beyond me.

      I'll go out on a limb here and say those other people probably wouldn't want to play with such a person anyway, so it's actually a win if that guy can't connect :P

    I've got the upmost sympathy for the customer service guys having to deal with these terrible people. But I'll still get mad at the companies because it's them who've encouraged this ideology that gamers just can't go without their online for any length of time.

      It's hard, right? Because sometimes you ARE getting fucked by the company you're calling and you ARE rightfully angry, but where do you get to express that to? Not the people responsible - they're hiding behind a shield of customer service innocents, like terrorists hiding their weapons caches in churches. It's bullshit. What, maybe you can send a letter and that'll get read? Ahahahaha, MAYBE by their PA, but even that's unlikely as hell.

      Which is why it's so hard to actually BE the customer who's getting fucked. You can soak it up yourself and repress your rage, bend over and take it with a forced smile.
      Or you lash out at someone who isn't personally responsible.

      Great choices, there. Real satisfying. Real fair.

      I know I've had more than one occasion of a customer service rep asking, "Hello? Are you still there?" as I've had to grit my teeth and control my anger against a really rough corporate fucking. Because the answer is basically, 'Yeah, I'm repressing my rage because you don't deserve it, but you wouldn't believe how mad I am'. (Which I usually cut down to, 'Yeah. This is less than ideal. What're my options, here?')

    Sounds like they had bad management and inadequate training on dealing with unhappy customers (not really surprising). I can't think of any thing worse to say to an upset customer in that situation than quoting the TOS. I would've refused to do it. I understand that they probably couldn't tell the truth of the situation for various reasons, but saying its ok, we're allowed to do this by our own rules, is a ridiculous thing to say.

    I've worked in call centres and have total sympathy for the people answering the calls, the idiots calling need to realise its not the person on the phones fault, there is nothing they can do about it, abusing them isnt going to change anything and calm the f down.

    But there are ways to manage these situations even when you aren't allowed to actually tell them anything, or don't know anything. Don't say things that are obviously going to enrage the customer further, even if it is dictated from above, in these situations no one is monitoring calls anyway. Use phrases like, yes we know its a problem, yes i understand this is ruining xmas for your child, we apologise, we are working on it as quickly as we can, we hope to get it working as quickly as possible, sorry I don't have an eta, sorry there is nothing more I can do... and repeat until the customer gives up. Sometimes you need to let them vent for awhile, let them say whatever they have to say, interrupting them isn't going to help either. Most importantly, don't take it personally and don't stoop to their level. Chances are they are still not going to be happy, but in this situation that is not the goal, the goal is to simply get them off the phone, so you can get to the next one, because the longer the wait times the worse it'll be. Also most call centres have an abuse policy where you are allowed to terminate the call under certain circumstances.

    Also I assume they atleast put an automated message on the phones? There is nothing worse for a customer than waiting for an hour on hold only to be told its a known issue and there's nothing they can do (oh and by the way we're totally allowed to do this according to our TOS)

    People should take their vitriol out on Lizard Squad, not the poor sods working the helpdesk who can't do anything about it anyway.

    What I am curious about is; is it Sony/MS choice or a game developers choice to force games to be 'online' even though the game can be, in essence, a single player (offline) game?

    I generally play single player games and mostly choose to ignore or actively avoid online play.

    Many games were still unplayable as they would simply not load, as my account could not be verified. I have discs, ipso facto, I have a purchased game. The system should not 'care' whether I have an account or not. Just my opinion...

    Welcome to life in a call centre bro.

    @transientmind really nails it.

    Though, contact centre workers (usually) have some lee-way in what they can say to rude people. Which really just exacerbates things, even though some things are completely justifiable.

    This kind of stuff is why I am extremely wary of "always online" games or "only streaming" content. Things that you paid for can so easily become unavailable at the whim of some idiots, or even just by chance if a cable somewhere breaks or there's a power outage or whatever.

Join the discussion!