What a week...
(Hopefully, I'm not jinxing things by posting this now.)
This has easily been one of the most hectic weeks I've had in a long while. On Sunday, the air conditioner in the UPS room started leaking... right onto the UPS itself. (That's "Uninterruptible Power Supply"; a battery (or, in this case, set of 64 batteries) that keep an even power flow, blocking power surges or maintaining power in the case of a black- or brown-out condition.) This, naturally, had the effect of shutting down the UPS, and thus shutting off power to the server room. So Monday, our network was down completely. Joy. We were able to put the power into bypass mode (bypassing the UPS, meaning the room was getting power, but without the UPS' protection. But since it wasn't working anyway, I suppose we didn't lose much there). But, as it turns out, a fuse must have blown, as we weren't getting power on all the lines. Some hasty rerunning of cables, and we were able to get the most important servers onto the active power leads quickly enough, and the rest of the servers got powered back on once that fuse was replaced. Although the SAN (Storage Area Network, essentially a big file server) lost its network configuration AGAIN, which is quite annoying. (According to NetApp, it's a known issue, and we (hopefully) have a fix for it now.) Still, that was quite a busy Monday.
Tuesday... well, not much actually happened on Tuesday, but there was one thing that I'll mention later, which was a bit worrisome.
Wednesday, the file server in the Newtown PA office had a hardware failure. Although we have a support contract, the turnaround time is 4 hours; I realized it would be quicker to grab a server from the office (in Secaucus NJ) and drive it over to Newtown (about 70 miles). Annoying, but we got it back up and running by noon, so it was worth it, I suppose. (Unfortunately, as we were done by noon, and I had nothing really to do at that office, and couldn't justify going home that early, I wound up driving back to Secaucus. So that whole trip ate up about 3 1/2 hours of the day.)
Thursday, when I got in, the phone system was down. Which, given that I don't really deal with the phones, wasn't a situation I wanted to have to deal with. But, being the only one there, I really had no choice. Checked the equipment in the phone room, but everything appeared to be powered on. So, I sent a quick e-mail to the guy who usually handles the phones. Who then calls me a couple minutes later. On my desk phone. The one that wasn't working five minutes ago. The system magically fixed itself, I guess, because I certainly didn't do anything, and no one else was in the office yet who couldn't done anything. Weird. No idea what happened there.
Today... actually, nothing happened today, really. Had to replace a power supply in the Exchange server, but they have redundant hot-swappable supplies, so no downtime there, at least.
All in all, not a bad week per se, but a rather active one. Everything was recoverable, at least, and we didn't have nearly as much down time as we could've had.
Still, the last time I remember having a week quite that hectic, the universe decided to cap it off by having the building catch fire over the weekend. Which brings me back to Tuesday, when, apparently, someone in another department came in to find a toaster oven had been left on overnight.
Seriously, if the building catches fire again, I think I'm quitting outright. I can't go through that again. (But here's hoping it won't be necessary.) (Please, universe?) (Like I said, I hope I'm not jinxing it by mentioning it...)
This has easily been one of the most hectic weeks I've had in a long while. On Sunday, the air conditioner in the UPS room started leaking... right onto the UPS itself. (That's "Uninterruptible Power Supply"; a battery (or, in this case, set of 64 batteries) that keep an even power flow, blocking power surges or maintaining power in the case of a black- or brown-out condition.) This, naturally, had the effect of shutting down the UPS, and thus shutting off power to the server room. So Monday, our network was down completely. Joy. We were able to put the power into bypass mode (bypassing the UPS, meaning the room was getting power, but without the UPS' protection. But since it wasn't working anyway, I suppose we didn't lose much there). But, as it turns out, a fuse must have blown, as we weren't getting power on all the lines. Some hasty rerunning of cables, and we were able to get the most important servers onto the active power leads quickly enough, and the rest of the servers got powered back on once that fuse was replaced. Although the SAN (Storage Area Network, essentially a big file server) lost its network configuration AGAIN, which is quite annoying. (According to NetApp, it's a known issue, and we (hopefully) have a fix for it now.) Still, that was quite a busy Monday.
Tuesday... well, not much actually happened on Tuesday, but there was one thing that I'll mention later, which was a bit worrisome.
Wednesday, the file server in the Newtown PA office had a hardware failure. Although we have a support contract, the turnaround time is 4 hours; I realized it would be quicker to grab a server from the office (in Secaucus NJ) and drive it over to Newtown (about 70 miles). Annoying, but we got it back up and running by noon, so it was worth it, I suppose. (Unfortunately, as we were done by noon, and I had nothing really to do at that office, and couldn't justify going home that early, I wound up driving back to Secaucus. So that whole trip ate up about 3 1/2 hours of the day.)
Thursday, when I got in, the phone system was down. Which, given that I don't really deal with the phones, wasn't a situation I wanted to have to deal with. But, being the only one there, I really had no choice. Checked the equipment in the phone room, but everything appeared to be powered on. So, I sent a quick e-mail to the guy who usually handles the phones. Who then calls me a couple minutes later. On my desk phone. The one that wasn't working five minutes ago. The system magically fixed itself, I guess, because I certainly didn't do anything, and no one else was in the office yet who couldn't done anything. Weird. No idea what happened there.
Today... actually, nothing happened today, really. Had to replace a power supply in the Exchange server, but they have redundant hot-swappable supplies, so no downtime there, at least.
All in all, not a bad week per se, but a rather active one. Everything was recoverable, at least, and we didn't have nearly as much down time as we could've had.
Still, the last time I remember having a week quite that hectic, the universe decided to cap it off by having the building catch fire over the weekend. Which brings me back to Tuesday, when, apparently, someone in another department came in to find a toaster oven had been left on overnight.
Seriously, if the building catches fire again, I think I'm quitting outright. I can't go through that again. (But here's hoping it won't be necessary.) (Please, universe?) (Like I said, I hope I'm not jinxing it by mentioning it...)