This Should Have Been a House Episode...
...for geeks.
Well, I spent the last week fighting Cornerstone's file server. Apparently, the RAID controller went south taking all the data on the RAID array with it. What is more, its timing was perfect because instead of dying in an instant, it appeared to die over several hours (think Titanic sinking), and during those hours the full backup was corrupted and the previous backup was overwritten with the corrupt file. Oh, things looked bad. *Then* the off-site backup storage device died. *Then* things looked bad.
So it was an unpleasant week filled with long hours of failing to pull the data from the RAID array, running test after test (each which assured us the hard drives and data were fine). The new card from Dell was seeming to have as many (if different) issues as the old one (can you say "Kernel Panic?"). So Monday afternoon (after the requisite House realization), I called Dell again, and they agreed to swap the motherboard and the card, and they had the new hardware on site in about an hour (I didn't know UPS had a courier service). The tech swapped the motherboard and card, and we rebuilt the system, calling it a night about 3AM Tuesday morning.
"We" constituted Austin and me. He did great work. And now everything is up and running again. At least it was when I left Cornerstone at 2PM yesterday.
God was gracious. Despite everything that happened, we lost no data. The old array had been fine. The failed motherboard just could not communicate with it.
Anyway, I learned from this that working on a server is *not* more fun than going sailing. I figured that was the case, but it's good to have empirical evidence.
Well, I spent the last week fighting Cornerstone's file server. Apparently, the RAID controller went south taking all the data on the RAID array with it. What is more, its timing was perfect because instead of dying in an instant, it appeared to die over several hours (think Titanic sinking), and during those hours the full backup was corrupted and the previous backup was overwritten with the corrupt file. Oh, things looked bad. *Then* the off-site backup storage device died. *Then* things looked bad.
So it was an unpleasant week filled with long hours of failing to pull the data from the RAID array, running test after test (each which assured us the hard drives and data were fine). The new card from Dell was seeming to have as many (if different) issues as the old one (can you say "Kernel Panic?"). So Monday afternoon (after the requisite House realization), I called Dell again, and they agreed to swap the motherboard and the card, and they had the new hardware on site in about an hour (I didn't know UPS had a courier service). The tech swapped the motherboard and card, and we rebuilt the system, calling it a night about 3AM Tuesday morning.
"We" constituted Austin and me. He did great work. And now everything is up and running again. At least it was when I left Cornerstone at 2PM yesterday.
God was gracious. Despite everything that happened, we lost no data. The old array had been fine. The failed motherboard just could not communicate with it.
Anyway, I learned from this that working on a server is *not* more fun than going sailing. I figured that was the case, but it's good to have empirical evidence.