How Heroes Of Newerth Suffered A "Catastrophic Hardware Failure"

How Heroes Of Newerth Suffered A "Catastrophic Hardware Failure"

PC game Heroes of Newerth has suffered a “a catastrophic hardware failure”, perhaps the worst we’ve ever seen for an online title, resulting in the game being completely unplayable. Here’s how it happened.

Developers S2 Games have posted a full report on the circumstances leading up to the failure, and it sounds like everything that could have gone wrong, no matter how astronomical the chances, went wrong.

Following the DDoS attacks in May, we retained a world renowned internet security firm to secure our network (this required significant software and architecture changes). After our network was secured, our then current hosting provider began experiencing connectivity issues. On June 9th, we began a transition to one of the premiere hosting companies in the world. Though the transition period was rocky, we had the system in a near 99% functional state. It was at this time that CPU upgrades to the database boxes were needed to allow for additional capacity and consistency of service. On Tuesday, June 21st, the replacement of the CPUs was started. Upon bringing the boxes back online, the engineers noticed anomalies in the raid setup, causing a total loss of data on the master and slave boxes. Following these events, they immediately attempted a restoration process on the data, it is unknown at this time if any of the data is recoverable. Concurrent to this happening, we began work to bring up our previous provider’s network that was still in place with the data from June 9th, which brings us to today. We are currently running on a temporary system while we construct our new environment at another hosting firm. Due to this system’s temporary nature, not all services may be operating at 100%.

We are planning for another maintenance window at which point we will switch to the new infrastructure. During this transition, we intend on having the game playable and will inform the playerbase of this maintenance at a later time.

Continuing their brutal honesty from earlier in the week, S2 “hold ourselves completely accountable for the events that have transpired”, and have promised every player silver and items as compensation.

Considering shit happens, even to the best of us, the mark of a man (or a company!) isn’t that mistakes are made, but how people react once they’ve made one. And S2’s transparency has been welcome in an industry that all too often is anything but.


Compensation & Apology [S2]


  • Their transparency is worse than Sony’s
    It took them 1 month of attacks and downtime to actually explain what was going on.

    Prior to this message, there would be zero warning of downtime, zero communication of the downtime, zero usage of the in game message of the day function, and a forum response buried in 20 pages of complaints 6 hours later.

    • Sorry phaded, but they’ve been keeping their player-base quite informed through the forums, twitter, in-game messaging,admin IRC etc.
      It wasn’t quite spoonfeeding but , considering that their main two forms of communication (chat server and forums) were down, information was still available and accessible.

  • So basically, in trying to do the right thing and upgrading their security, network etc they broke the game so as to be unplayable. That’s terrible. For the company as well as their players.

  • Poor guys… on unrelated news season one of League of Legends is nearly over so now is the time to start playing.

    Did I mention it is free 😛

Show more comments

Log in to comment on this story!