Unexpected failure mode

[brad]

Dear Lazyweb,

Let's imagine for a second, hypothetically, that some drunk girl sees Blinkenlights on the RAID array in your garage and says (reportedly) "Oh neat, hot swap!", and proceeds to remove 3 drives from a particular 5-disk RAID-5 array, instantly killing the party music and a quite a bit of other data [accessibility].

Hypothetically,

Comments

[evan]

RAID HOWTO: 8.1 Recovery from a multiple disk failure

[matthew]

The howto makes sense, but I'd make one small change since this is a cascade failure and not simultaneous. I'd look at syslog to find which drive went offline first. Take the rest of the drives and force them back into a broken array with mdadm --assemble --force.

In theory this will put your array back to the state before the 2nd drive was yanked and your data will still be there (since the array stops once you lose 2 disks)

After that, add the last disk back in as a spare and rebuild.

ymmv

[dakus]

the raid was in the garage? seems like the heat would get to it?

[brad]

a) it's SF... brrr
b) underground

[henry]

just wrap that sucker in thermite cord and keep one ear to the scanner, all they'll get is a pile of molten metal. fuckin pigs!

[ext_53955]

I think you have just sold me on getting lockable trays. This is the second time I've heard about this routine.

[wcu]

ive seen her before, shes hot. party time!

[ywong]

Some people, when they have a complex technical problem to solve, think "I know, I'll ask for advice on LiveJournal!"

Now they have two problems.

[brad]

Hah. nice jwz quote variant.

[ywong]

My wife and I say that a lot when people talk about relationship problems, since asking for advice on LiveJournal (or rather, the internet) rarely leads to real solutions so much as it more often leads to HILARIOUS drama. It is slightly more useful for technical advice.