CSH Mail Server Problems
Whitefox got rebooted due to a power problem, then the raid card refused to boot up because its hold-up battery is dead. We've been thus-far unsuccessful at convincing it to not give a damn. Seriously shitty hardware.
Our options:
1) Get the card to not care about the failed battery and boot anyway
Can't get the card to load its bios to even play around with settings/etc.
2) Get a new card
Can't find one.
3) Trick the card into thinking there's a battery there
Tried that, still complains.
4) Reinterleave the blocks by hand on another machine
Question marks.
5) Restore from backups onto a new array definition.
Possible loss of data.
It's looking like we're going to be forced to restore from backups, which I'm not looking forward to because we'll lose a little bit of mail and whatnot since the last backup.
Chris Lockfort has been working on it since it happened last night, but hasn't been able to recover it yet. I'll be going in today after work to help.
Update 6:10 pm:
We're going ahead with bare-metal restore from backups. The raid card is starting to respond, after some magic, so while we acquire backups from tape onto spare disk, we're going to continue to try to magic up the raid card.
Opcomm also has a twitter account now:
http://twitter.com/cshrtp
Our options:
1) Get the card to not care about the failed battery and boot anyway
Can't get the card to load its bios to even play around with settings/etc.
2) Get a new card
Can't find one.
3) Trick the card into thinking there's a battery there
Tried that, still complains.
4) Reinterleave the blocks by hand on another machine
Question marks.
5) Restore from backups onto a new array definition.
Possible loss of data.
It's looking like we're going to be forced to restore from backups, which I'm not looking forward to because we'll lose a little bit of mail and whatnot since the last backup.
Chris Lockfort has been working on it since it happened last night, but hasn't been able to recover it yet. I'll be going in today after work to help.
Update 6:10 pm:
We're going ahead with bare-metal restore from backups. The raid card is starting to respond, after some magic, so while we acquire backups from tape onto spare disk, we're going to continue to try to magic up the raid card.
Opcomm also has a twitter account now:
http://twitter.com/cshrtp