lightning
in the late afternoon yesterday we had a gorgeous rain shower. it started while i was in a meeting, and the raindrops were falling so hard on the skylight it was sometimes hard hear people.
at one point during the rainstorm, lightning struck very near to the building, bouncing the electricity. alarm bells immediately started ringing in the server room, so i gladly excused myself from the meeting to go check on my systems.
this outage was odd, and we likely lost some equipment. some offices had lights but no power to the walls, others the opposite, but most were unaffected. unfortunately, our server room was unpowered, so batteries were squealing for attention from every rack.
at first we thought we had a breaker trip, so we headed outside and sat in the pouring rain fiddling with the keys we needed to get into the mains. it wasn't going to be that simple--all the breakers were fine, so we either had a problem originating off-site or something burned out on-site.
soaking wet, we headed back in to reevaluate the situation from inside the building, and i headed off to coordinate with our office manager as well as to collect information from other offices in our building. by the time i had to leave, the power company was on the way down and my computers still weren't back up.
i have no idea whether we still have problems this morning, but i'm expecting that our jobs didn't run last night. this will also be a good test of our disaster recovery system. i don't think it passed. :)
i did get the rare opportunity to yell out "if you work in my group, you can go home!"
[update: now that i'm here, i see that things are definitely all kinds of not working.]
at one point during the rainstorm, lightning struck very near to the building, bouncing the electricity. alarm bells immediately started ringing in the server room, so i gladly excused myself from the meeting to go check on my systems.
this outage was odd, and we likely lost some equipment. some offices had lights but no power to the walls, others the opposite, but most were unaffected. unfortunately, our server room was unpowered, so batteries were squealing for attention from every rack.
at first we thought we had a breaker trip, so we headed outside and sat in the pouring rain fiddling with the keys we needed to get into the mains. it wasn't going to be that simple--all the breakers were fine, so we either had a problem originating off-site or something burned out on-site.
soaking wet, we headed back in to reevaluate the situation from inside the building, and i headed off to coordinate with our office manager as well as to collect information from other offices in our building. by the time i had to leave, the power company was on the way down and my computers still weren't back up.
i have no idea whether we still have problems this morning, but i'm expecting that our jobs didn't run last night. this will also be a good test of our disaster recovery system. i don't think it passed. :)
i did get the rare opportunity to yell out "if you work in my group, you can go home!"
[update: now that i'm here, i see that things are definitely all kinds of not working.]