Forcing Together a Broken RAID5 with mdadm
Link: https://dannyman.toldme.com/2026/07/17/mdadm-force/
Scenario:
- a 4-disk
mdadm-based software RAID5 - one disk fails: the RAID5 continues to function in a degraded state
- one attempts to replace the failed disk, but pulls (hot swap) an active disk instead
Ideally you pull from backups into a new or rebuilt system. But what if you wanted to get at the data on the broken RAID? I recently ran through this exercise.
To check out the health of the RAID you would say, for example mdadm --detail /dev/md0 which gives you plenty of detail, including, for example:
[...]
State : active, FAILED, Not Started
[...]
Number Major Minor RaidDevice State
- 0 49 1 sync /dev/sda1
- 0 33 2 sync /dev/sdb1
Let’s say /dev/sdd1 had failed and /dev/sdc1 had been pulled in error. You can inspect each disk with, for example: mdadmn --examine /dev/sda1 and note useful stuff near the bottom. The sample output here is heavily trimmed for clarity.
/dev/sda1:
Update Time : Fri Jul 17 14:42:15 2026
Events : 1100008
Device Role : Active device 0
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb1:
Update Time : Fri Jul 17 14:42:15 2026
Events : 1100008
Device Role : Active device 1
Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
Update Time : Fri Jul 17 14:42:00 2026
Events : 1100000
Device Role : Active device 2
Array State : AAA. ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
Update Time : Thu Jul 16 14:42:00 2026
Events : 1000000
Device Role : Active device 3
Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
This says:
- sda1 and sdb1 were last active in an array with two active members
- sdc1 was active in a three-member array
- sdc1’s update time ends 15 seconds before the last update for sda1 and sdb1
- sdc1 is missing 8 events that sda1 and sdb1 enjoyed together
- sdd1 was last active yesterday, 100,008 events ago, and when it died it was in an array with four active members
So, sdd1 is the dead disk and sdc1 was errantly pulled. But they can’t just form an array because their metadata do not line up.
# mdadm --run /dev/md0 mdadm: failed to start array: Input/output error
We can try manual assembly. First, I run --stop to “forget” the two-element array, then try mdadm --assemble:
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 mdadm: /dev/md0 assembled from 2 drives - not enough to start the array.
“But I asked it to use three drives! Oh yeah, the event count differs on sdc.”
Next, you spend some quality time reading the man page, and you perform your own risk analysis.
If approriate, you might be able to “fix” your RAID thus:
# mdadm --stop /dev/md0 mdadm: stopped /dev/md0 # mdadm --assemble --force /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 mdadm: forcing event count in /dev/sdc1(2) from 1100000 to 1100008 mdadm: /dev/md0 assembled from 3 drives (out of 4).
At this point, you may find yourself with a working RAID and a filesystem you can mount. Copy off whatever data you value to an array that hasn’t been “forced” together!
