02282011 Release from Captivity
One call at 0405 that lasted till 0730. Depsite going round-and-round, waking up people, it came down to power cycling a card.
At this time of the morning, everyone's running on half-speed and I imagine if this happened at 1600, things would've resolved quicker.
Violation of the 3am Rule
Documentation needs to be plainly named, easy to find and read -accurate info helps too. When someone is woken up at 0300
and has to figure out a problem -especially if they're not familiar with the environment or configuration- this speeds things up.
Not the case last night but it didn't factor into the resolution; it only wasted time. I've wanted to go through this scenario for a
while so I did learn a few things.
Now the machines call someone else to say they’re broken.
At this time of the morning, everyone's running on half-speed and I imagine if this happened at 1600, things would've resolved quicker.
Violation of the 3am Rule
Documentation needs to be plainly named, easy to find and read -accurate info helps too. When someone is woken up at 0300
and has to figure out a problem -especially if they're not familiar with the environment or configuration- this speeds things up.
Not the case last night but it didn't factor into the resolution; it only wasted time. I've wanted to go through this scenario for a
while so I did learn a few things.
Now the machines call someone else to say they’re broken.