Yay! My server is back online. First off, **Much Thanks to
ertach for helping me through this mess!!** Without you, I have NO CLUE what I would've done.
In the end, it looks like the IDE controller on the motherboard got damaged somehow, and was working hard to corrupt/damage my hard drives. It succeeded on two of them, and may have been starting on a third. Not sure about the third. The disk is doing perfectly now & passed every test we tossed at it the past two days on a new mobo. The two that died were both on the IDE2 chain. The 100GB disk is shot so far as ever being able to be used again - it has bad blocks, including the superblock (where it stores the partition table). It was replaced with a shiny new 250GB Seagate disk I purchased Thursday (or was it Friday?) Night. At this point, I don't really care all that much about the data on the old disk - what I wanted back I can get back, and the rest is just meh... didn't really need that anymore. Thanks to Tom's ingenious idea, I do have a complete list of what was on the drive: When locate hasn't updated since the drive failed, you can use it to get a list of what was on the now dead drive. Hooray for locate not updating nightly on my server like it should be! :P
Yesterday, when we started really digging into the server, we realized another disk is also damaged, one of the 160GB drives. DD couldn't read the disk, and errored out on trying to back it up. So we got a program called
ddrescue and used that. ddrescue got all the data off, with a small bit of corruption in 1 or 2 files (haven't tracked down which files yet.. will do that eventually). This disk also has bad blocks... 34 of them, but not the superblock. I've marked the disk as bad & pulled it (replacing it with a
WD 250GB that was purchased in a SWEET deal at CompUSA today - $59.95 after rebates!), but I'm not sure if it's totally unusable. Any of you out there with hard drive knowledge have any input on this? If it was the controller causing the damage, is it likely the rest of the drive will fail still when in another computer.
To resolve the controller issue, I was forced to downgrade back to the old XP1600+ chip and a MSI KT3 board that I had lying around. Sadly this board doesn't support FSB333 for the Barton core Athlon XP chips, so now I get to buy a new motherboard for this computer. In the end I'll do this, but not right away. Also, we moved the server to a different case... but it has no sides :-( So now i'm on the look out for sides for the case - anybody know where I can get sides for a case?
Next up: Converting this bloody server over to use RAID 5. I'm *DONE* losing data.
[crashes out to bed]