Power-loss post-mortem

[bradfitz]

The post you've all been waiting for! Why we lost all our power, and why it took us so long to come back up afterwards....

(Warning: it's late and I'm tired/rambly, so this post might be incoherent... go ahead and ask questions...)

Why we lost all our power...Another customer in the facility accidentally pressed the EPO button, then depressed

Comments

[morti]

Excellent, thanks.

[tsutton]

Thanks for posting it. Does Internap have CCTV which they can catch whoever did it and get him/her to pay for the loss? (i.e. hardware replacement, loss of profit, etc)

[kloostec]

You can't really charge someone that kind of money for tripping and falling and hitting the switch accidentally... If, however, they pressed the button labelled "Do not press this button" for the sole purpose of seeing what it would do... then I'd be all for that ;)

[eddy_]

You can't really charge someone that kind of money for tripping and falling and hitting the switch accidentally

That's fair enough if they'd owned up straight away, but how much time was lots because they covered it up?

[tsutton]

Perhaps. But the fact that he/she covered it up and left the building without informing anyone.

Now that's bad and that person should be punished.

[phonetic]

ya done good, brad & company. kudos.

[imdaewen]

ICON LOVE! Fantastic.

[phonetic]

thank you. :)

[kimothy]

Oh my. I LOVE the icon!! lmao

[tefkas]

Thanks for the update.

[ex_phelyan]

Ooooh, the bane of the EPO buttons. I've been witness to a whole room full of servers suddenly becoming very quiet because someone had tried to walk out of the door with an A3 graphics tablet on his shoulder and had managed to bump the switch. Shit happens...

[galbinus_caeli]

It's better than hitting the fire suppression button. I remember my first time in a raised floor and being told that if I hit that button I would be fired, my family would be fired, and all my decendents for seven generations would be fired.

One button to cause tens of millions of dollars worth of equipment damage. Plus data loss. Plus lost business.

[ex_phelyan]

Ouch... surely there's inert gases that are used in equipment rooms. Mind you, at my last company our co-lo used Halon in their fire suppression system. Wouldn't want to be in the server rooms when a fire breaks out.

[galbinus_caeli]

This was sixteen years ago. Some of the systems were halon, but some were powder. I think halon or something similar is state of the art now. People can hold their breath for a few seconds to get out of the room. Fire can't.