dannyman.toldme.com

Forcing Together a Broken RAID5 with mdadm

Scenario:

• a 4-disk mdadm-based software RAID5
• one disk fails: the RAID5 continues to function in a degraded state
• one attempts to replace the failed disk, but pulls (hot swap) an active disk instead

Ideally you pull from backups into a new or rebuilt system. But what if you wanted to get at the data on the broken RAID? I recently ran through this exercise.

To check out the health of the RAID you would say, for example mdadm --detail /dev/md0 which gives you plenty of detail, including, for example:

[...]
           State : active, FAILED, Not Started
[...]
    Number   Major   Minor   RaidDevice State
       -       0       49        1      sync   /dev/sda1
       -       0       33        2      sync   /dev/sdb1

Let’s say /dev/sdd1 had failed and /dev/sdc1 had been pulled in error. You can inspect each disk with, for example: mdadmn --examine /dev/sda1 and note useful stuff near the bottom. The sample output here is heavily trimmed for clarity.

/dev/sda1:
   Update Time : Fri Jul 17 14:42:15 2026
        Events : 1100008
   Device Role : Active device 0
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdb1:
   Update Time : Fri Jul 17 14:42:15 2026
        Events : 1100008
   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdc1:
   Update Time : Fri Jul 17 14:42:00 2026
        Events : 1100000
   Device Role : Active device 2
   Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)

/dev/sdd1:
   Update Time : Thu Jul 16 14:42:00 2026
        Events : 1000000
   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

This says:

• sda1 and sdb1 were last active in an array with two active members
• sdc1 was active in a three-member array
• sdc1’s update time ends 15 seconds before the last update for sda1 and sdb1
• sdc1 is missing 8 events that sda1 and sdb1 enjoyed together
• sdd1 was last active yesterday, 100,008 events ago, and when it died it was in an array with four active members

So, sdd1 is the dead disk and sdc1 was errantly pulled. But they can’t just form an array because their metadata do not line up.

# mdadm --run /dev/md0
mdadm: failed to start array: Input/output error

We can try manual assembly. First, I run --stop to “forget” the two-element array, then try mdadm --assemble:

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1
mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.

“But I asked it to use three drives! Oh yeah, the event count differs on sdc.”

Next, you spend some quality time reading the man page, and you perform your own risk analysis.

If approriate, you might be able to “fix” your RAID thus:

# mdadm --stop /dev/md0
mdadm: stopped /dev/md0
# mdadm --assemble --force /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1
mdadm: forcing event count in /dev/sdc1(2) from 1100000 to 1100008
mdadm: /dev/md0 assembled from 3 drives (out of 4).

At this point, you may find yourself with a working RAID and a filesystem you can mount. Copy off whatever data you value to an array that hasn’t been “forced” together!