FreeBSD HOWTO: Software Mirror System Disk
A new feature of FreeBSD 5.3 is the ability to set up a software mirror of your system disk. This allows you to boot off either of a pair of hard disks, which will then function as a RAID1, which will ensure system uptime in the face of a single disk failure.
STOP THE PRESS! The instructions below work great, but I have just stumbled upon an easier way:
http://www.onlamp.com/pub/a/bsd/2005/11/10/FreeBSD_Basics.html
If, for some reason that doesn’t work for you so well, here’s my cheat sheet for setting this up with gmirror:
(This crib sheet assumes you have a pair of identical IDE (in my case, SATA) drives identified as ad4 and ad6.)
- Install FreeBSD on to ad4.
- Reboot with the Install CD.
- Enter Fixit mode. (For FreeBSD less than 5.4, use Install CD disc2 as the “live filesystem”)
-
# chroot /dist
# mount_devfs devfs /dev
# gmirror load
# gmirror label -v -b round-robin gm0 /dev/ad4
# gmirror insert gm0 /dev/ad6
# mount /dev/mirror/gm0s1a /mnt
# echo ‘geom_mirror_load=”YES”‘ >> /mnt/boot/loader.conf
# echo ‘swapoff=”YES”‘ >> /mnt/etc/rc.conf - Edit /mnt/etc/fstab to convert ad4 -> mirror/gm0 (The
eecommand may be useful here – press ESC when you are done.)
takhun suggests the following:
# sed “s%ad4%mirror/gm0%” /mnt/etc/fstab > /mnt/etc/fstab.new
# mv /mnt/etc/fstab.new /mnt/etc/fstab - Reboot
Thanks to the few dozen people who have come before me, and posted crucial hints to the mailing lists. Thanks also to Ralf S. Engelschall who has a far more verbose explanation of how to do this sort of thing without console access and with mis-matched disks.
You should definately look over the gmirror man page, and review the output of gmirror list gm0 when swapping out drives. You can disable automatic rebuild, etc. It is quite nice.
Last note: If you need support with gmirror, Check out freebsd-questions — your question may be answered if you add a comment here, but asking on a mailing list will probably serve you best.
Responses
usbenny
Hi,
Thanks for the great howto. I am having a problem when I goto insert ad6 into the mirror it gives me a “Class not found” error. Any help would be greatly appreciated.
usbenny
usbenny
Ok, I got around this problem by doing a
gmirror loadbefore the insert.Thanks,
-usbenny
Don
I tried it but couldn’t do the last edit of
mnt/etc/fstabto convert ad4 -> mirror/gm0 because it was a read only file. I shouldn’t have done it because now the server will not boot up at all giving the error:mountroot> ufsmounting root from ufs
ffs_mountroot: can't find rootvp
Root mount failed: 6
Manual root filesystem specification:
Don
No worries, mate – git it in the end. For some reason the only editor I could invoke was ed so I had to learn that to edit the fstab file.
Thanks for the guide
andrew
hi!
i hust want to know the exact procedure how do i edit the fstab?
thanks!
andrew
Leffe
I copied
/usr/bin/vito/mntand then did another chroot to/mntto be able to modify/etc/fstabwithvi.Cheers!
/Leif
Dave Webster
When I type the command:
$ gmirror insert gm0 /dev/ad6the response is:
"Provider ad6 already inserted"then the next cmd:
$ mount /dev/mirror/gm0s1a /mntgives the response:
"/dev/mirror/gm0s1a: no such file or directory"Any help would be greatly appreciated.
regards,
Dave
tahkun
Oh! that’s great!
It thought about gmirror by which I also had used LiveFileSystem.
However, it did not make it to good, because devfs had not been mounted.
It became reference very much.
Thank you!
PS
By the way, I changed fstab by using
sed.# sed "s%ad4%mirror/gm0%" /mnt/etc/fstab > /mnt/etc/fstab.newanteeee
Hi all,
This seems great! So I don’t need to buy any hardware to set this up? And it is also possible to boot from both partitions?
Do I have to make any adjustments inside my server before doing this (connecting the SATA drives somehow?
manoj
Hi all.
I’m a starter on both BSD and software RAID stuff. I’ve been going through tons of materials on web searching for solution to create a good software mirror RAID0 setup on freeBSD and the resources on this site turned to be an angel from heaven :)
My configurations:
gmirror name gm0
first provider ad4
second provider ad6
As suggested in this site, I’ve successfully created mirror on my 2x74GB SATA drives using livefs CD boot and tested to be fault tolorent when I plugged out the first drive ad4 while live. The system was running fine in degraded state.
The question is:
Can I synchronize from the second drive (i.e. ad6) should I require to replace the first drive on ad4. I reconnected the first drive and tried to synchronize from the second drive but it wouldn’t let me. Is there a way to do that?
I know this is not a mailing list. But since I found this site more helpful that the lists I ran through, I hoped I would get short and to-the-point answer.
Thank you all.
dannyman
Manoj,
If you ran in to some specific errors, I’d hit a mailing list with those and see.
If you simply remove the drive and plug it back in, there may be issues because gmirror will see that you gave it its component back. It SHOULD mark it dirty and rebuild …
If you removed the drive, erased the data, then reinserted it, you’d have to do the “gmirror insert” command to add that drive to the gmirror. It would then rebuild.
gmirror can be configured to automatically rebuild, or manually. Check the man page.
Sincerely,
-danny
manoj
Thanks for the response. I tested it today and it is working flawlessly. And i’m very happy with the setup. I’m gonna keep it running for few days and put to production after that.
Just to add to your good setup notes:
If a drive fails (eg. drive /dev/ad4):
#gmirror forget gm0#shutdown -h
replace the drive with a fresh one
#gmirror insert gm0 /dev/ad4And it should start synchronizing if autosynchronization is turned on.
Thanks.
-Manoj
michi
Please help me:) I’m trying to do a gmirror for three days.. and then I’ve found this tutorial. But I stopped at 3.rd point, what does it mean to ‘Enter fix-it mode’? I’ve got 5.4 installed. Thanks.
SunSet
Ok got a question:
I still get the error..
Mountroot does any1 knows what i must do there?
agouti
Please can you tell us the difference between gmirror and vinum ?? i am a newbi and i read that vinum do raid mirroring too …
thanks.
agouti
sorry i found the answer: gmirror it’s raid1 spécific and it’s a part of geom framwork .vinum it’ s old and came from freebsd4 and will be replace by geom vinum …
thanks and ” place à gmirror “
urfx
hey big thanks, manged to get this working first time on a supermicro…
http://www.supermicro.com/products/system/1U/5014/SYS-5014C-T.cfm
I couldn’t get the hardware SATA raid controller to work with FreeBSD hence eventually came across this crib.
The experience I would like to share is, I couldn’t use this crib on a box where I had already tried to set up a gmirror [failed] using the method oulined by Engelschall.
http://people.freebsd.org/~rse/mirror/
Then I experienced problems similar to Dave Webster. Long and short of it I was unable to clean the disks on that first box and try again, i tried using dd but failed because at this point I was out of my depth.
Seems to me the key to succes is clean disks.
i.e. ad4 and ad6 so if anyone has a definitive way to make sure each and every sector is clean, it is probably a useful first step before installing FreeBsd on ad4.
Anyhow big thanks again. I’m now going to play with rebuilding a broken mirror before going for this a production solution
urfx
I had a hunch my dd failed because it was cleaning the first few sectors of the harddives ad4 and ad6, after RTFM…
The gmirror utility is used for mirror (RAID1) configurations. After a mirror's creation, all components are detected and configured automati- cally. All operations like failure detection, stale component detection, rebuild of stale components, etc. are also done automatically. The gmirror utility uses on-disk metadata (stored in the provider's last sec- tor) to store all needed information. Since the last sector is used for this purpose, it is possible to place a root file system on a mirror.… the hunch is confirmed so I think what is needed is a method to erase the last sectors of the HDD’s
I’ll man dd.
urfx
ok gmirror can do the work… here is how,
if you have tried and failed using various methods of gmirror, you will likely have data tagged at the end sectors of ad4 and ad6 or whatever the first the disks are in your array.
before trying danny’s method above, you need to clear these sectors. To do this succesfully you must be logged in as root, you can’t clear the sectors on a live system so you must install a minimal instance of freebsd [if you haven’t already] on as least two disks on your server, then use the following command…
### while booted from ad4
### while logged in as root
gmirror clear ad6
### while booted from ad6
### while logged in as root
gmirror clear ad4
### then you will be able to proceed with each and every step in Danny’s crib, to be safe do a clean install on ad4, [this is in the crib]
### urfx
Tim Provencio
Awesome walkthrough! Just bought a Dell SC420 and realized the “mirror set” was two different drives in FreeBSD. Disabled RAID in the BIOS and followed these instructions without a hitch. Thanks much!
boredhacker
Ugh, I guess my first post didn’t make it through (for one reason or another). Maybe I forgot to hit the submit button, or maybe the lengthy url I originally specified as my web-site screwed things up, I’m just not sure…
Anyway, the HOWTO is great and I used it to set up a duplexed raid mirror on my own machine (yay!).
I just have one question: why is the ‘-b round-robin’ option suggested? I used ‘-b load’ instead assuming that performance would be better.
Can anyone comment on the differences between the balance algorithms that are available (i.e. load, prefer, round-robin, and split). The gmirror manpage is a bit terse when describing the options, and IMHO it would be nice to have a better idea of what those options do without having to dive into the source code.
Once again, thanks for a great HOWTO!
dannyman
boredhacker,
Although comments from new people have to be moderated, I only got the one comment from you.
I chose round-robin because I figured if you have two disks in a mirror, they’re both under the same “load” constraints, and it is best to KISS.
If you really want to know, you should ask on FreeBSD-hackers@ …
Thanks,
-danny
ari braginsky
On bootup I am seeing:
GEOM_MIRROR: Device gm0 created (id=xxxx)
GEOM_MIRROR: Device gm0: provider ad0 detected.
GEOM_MIRROR: Force device gm0 start due to timeout.
Note: no media was in the CDROM at the time of bootup. CDROM is slave to IDE1 channel.
ari braginsky
Forgot to add that the system locks up (even keyboard) after:
GEOM_MIRROR: Force device gm0 start due to timeout.
There is no disk activity.
Thoughts?
ari braginsky
Solution:
See http://www.freebsd.org/cgi/query-pr.cgi?pr=73660.
Turns out I was able to fix my problem after removing the CDR that was a slave on IDE1, the same channel that the gmirror provider ad0 was on.
Possible fix is to increase the mirror timeout or put the CDR on a different IDE channel than then mirror provider.
Downtown
“I know this is a case of trying to Walk before Crawling”
Installing Geom on 5.4 and I enter all the required commands in the fixit mode. Then rebooting, nothing seems to have changed. Don’t I have to mount my changes from the fixit mode from /mnt2, problem is I am not sure of the commands to achieve this.
Can anyone help please?
Krisna
Great & Nice Danny.
But how about remote administrator? That doesn’t have acces to server hardware, only network connection with SSH access?
boredhacker
>>>I chose round-robin because I figured if you have two disks in a mirror, they’re both under the same “load” constraints, and it is best to KISS.
I can’t say that I agree with you on this one. From a user perspective typing ‘load’ or ‘prefer’ are both easier to use than ’round-robin’; fewer characters ;-) But seriously, in my desktop machine I have a duplexed gmirror built off of 2 disks that have the same geometry but one is udma66 and the other is udma100, I’m pretty sure using ‘load’ is better in this situation and probably others too.
Regardless, the how-to is still a wonderful thing. I certainly don’t want to take anything away from it. However, a little constructive criticism can be a good thing, so….
If I may suggest, although the how-to tells you how to set-up the mirror, it never mentions what to do when an actual disaster happens (i.e. when a disk needs to be changed).
Some things to consider:
1. Most modern disks will not be the same size even if “you have a pair of identical IDE … drives”. This is because the integrated drive electronics account for imperfections in the media (bad tracks/sectors). Thus the command “gmirror label -v -b round-robin gm0 /dev/ad4” in step 4 should *i think* be more like “gmirror label -v -b round-robin gm0 /dev/ad6 /dev/ad4”. This will create a mirror only as big as the smallest drive. If you don’t do this, you just might see a message that says you can’t insert ad6 because it’s too small (even though the disks are ‘identical’). In addition, to quote the gmirror manpage: “The order of components is important, because a component’s priority is based on its position (starting from 0). The component with the biggest priority is used by the prefer balance algorithm and is also used as a master component when resynchronization is needed, e.g. after a power failure when the device was open for writing.” In light of this, I’m not sure if the command should be “gmirror label -v -b round-robin gm0 /dev/ad6 /dev/ad4” or “gmirror label -v -b round-robin gm0 /dev/ad4 /dev/ad6” – but one may be better than the other. I mean… the gmirror manpage does say order is important, right?
2. When installing the system (in step 1) it might not be a bad idea to create a slice that is actually smaller (by say 32-128mb) than the drive allows – or to install the system to the truly smallest drive. This will insure that when/if a drive needs to be replaced you can simply use the same make and model (or a drive with the same physical geometry) and not worry about size discrepencies.
3. Replacing a drive can be simple if you have a new one on hand. If it isn’t the boot drive that has failed (or the mirror is already running and you have hot-swap hardware) then it’s just a matter of ‘gmirror forgeting/removing’ the old disk and ‘gmirror inserting’ the new disk. If you have hot-swappable hardware you shouldn’t even have to reboot. If the boot drive failed (and you can’t [re]boot) you’ll need the fix-it cd as you recommend to ‘forget/remove’ the drive, and ‘insert’ another. Use ‘gmirror status’ to wait until the mirror rebuilds itself before exiting fixit (since it was the boot drive that failed). Booting a partially rebuilt disk simply can’t be a good idea imho.
4. The crib sheet assumes you have a pair of identical IDE disk drives, which is a good thing. Assuming otherwise would introduce too much complexity for a simple how-to. However, this just makes the previous three considerations that much more important. After all, what good is having a mirror if you can’t rebuild it after a failure?
Now, as you can probably tell, I’m not 100% sure about a lot of this. And I may not have written it out completely or perfectly clear. But hopefully these suggestions can make a good how-to even better!
dannyman
bh,
I’d ask around on -hackers but I’m pretty sure bad block mapping is done mostly within the drives these days: they present a series of blocks to the OS that is smaller than their absolute physical capacity, and internally map around any bad blocks as they are encountered. An OS may in turn encounter bad blocks, and does a similar thing. If I were implementing GEOM and gmirror, that easiest thing to do is that if you have to remap a bad block on one drive, you re-map it in the virtual drive. So, if someone slips you a replacement drive with many bad blocks, you just map those into the virtual drive. Maybe if you drop drives, you can free up those bad blocks from the OS’ list.
I think it is good to think about these things, but I’d suggest a bit more research before you start making your life more inconvenient to solve troubles that have hopefully been addressed already. :)
Cheers,
-danny
★ On.the.nite.like.this ★ » Blog Archive » FreeBSD HOWTO: Software Mirror System Disk
A new feature of FreeBSD 5.3 is the ability to set up a software mirror of your system disk. This allows you to boot off either of a pair of hard disks, which will then function as a RAID1, which will ensure system uptime in the face of a single disk failure . . .
Shtir
Thanx for the HOWTO step by step instrauctions. But after doing everything FreeBSD dosn’t load stopping at:
mounting root from ufs:/dev/mirror/gm0s1a
setrootbyname failed
ffs_mountroot: can’t find rootvp
How can I fix this prob to boot FreeBSD.
Mitch Walker
Brilliant work, buddy. Thanks a million! Literally as easy as 1-2-3. It’s HOWTOs like this that make open-source work.
Andreas
Hi,
I have a mirror with two identically SATA drives. The mirror was build, as the system was installed a year ago.
This are the gemoetry for ad4/ad6: cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl)
And this is the geometry for gm4: cylinders=30401 heads=255 sectors/track=63 (16065 blks/cyl)
The one partition has this size (is for ad4/ad6 and gm4):
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 488397042 (238475 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 168/ head 14/ sector 63
You can see, that the partition is bigger than the gm4 geometry.
Any idea, what I can do? A fdisk -u mirror/gm4 will not work, because fdisk prins:
fdisk: Geom not found
fdisk: Failed to write sector zero
Andreas
ooops, forget something:
# gmirror list
Geom name: gm4
State: COMPLETE
Components: 1
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3556157093
Providers:
1. Name: mirror/gm4
Mediasize: 250059349504 (233G)
Sectorsize: 512
Mode: r4w4e5
Consumers:
1. Name: ad4
Mediasize: 250059350016 (233G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 3808931721
The difference ist exactly 512 bytes. Is this the last sector, where geom stores the meta data?
Is the difference with the geometry critical?
sara lidgey
Hi All,
I’ve been running a server using FreeBSD 5.3 and gmirror to mirror two identical IDE hard drives. This howto was very helpful when setting up. Its been running great for over a year. But recently everything went down and when I reboot and put a monitor on it I get the following errors on screen:
GEOM_MIRROR: Device gm0: provider ad1 disconnected
GEOM_MIRROR: Device gm0: provider mirror/gm0 destroyed
GEOM_MIRROR: Device gm0: rebuilding provider ad0 stopped
Fatal trap 12: page fault while in kernel mode… (this is followed by details about the fault)
These errors are preceded by other related error information that flys by on the screen and I have no way of seeing them again.
Does anyone now what steps I should take to figure what is going on and try to recover data or get the machine to boot?
Thanks so much,
S.
Fabien Langlet
Hi,
I have a computer with one hard disk (single slice ad0s1) running with FreeBSD. This computer is used as a WEB server, mail server, DNS server, … For a best security, I would like to establish a RAID-1 for the system partitions using your “howto†with a second new hard drive (mirror). My question is :
Is it possible to establish a RAID-1 on my system using your “howto†and keeping all my first drive data (WEB server, mail server,…) ? In fact, I really need a mirror to upgrade my security, but I don’t want to loose my data on my first hard drive.
Thanks for your tutorial.
Best regards.
F. Langlet-Cauet.
bones
Be aware that this doesn’t work with BSD 6.3 because the CD’s gmirror command somehow doesn’t list most options. This is also described in the url below.
As as workaround I installed a minimal freebsd system on a temporarily third disk, boot from it and created the raid. Then take it out again.
http://www.nabble.com/misc-120169:-gmirror-commands-error-with-“Class-not-found”-or-other-td15190746.html
Bishal
Hi,
I am trying to do RAID1 mirror on live freebsd6.0 system. Both harddisk are 40GB of size. When I type gmirror insert gm0 /dev/ad2 it gaves me “Provider ad2 too small” message. What’s wrong? Earlier ad2 hardisk was running Winxp and freebsd dual OS. I completly deleted the partition and tried to do RAID1 but that errror “Provider ad2 too small” message comes up.
Sysinstall command shows both hardisk size of 38162MB. Output of the fdisk command on both hardisk:
computer1# fdisk /dev/ad0
******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=77545 heads=16 sectors/track=63 (1008 blks/cyl)
Figures below won’t work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=77545 heads=16 sectors/track=63 (1008 blks/cyl)
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 78156162 (38162 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:
The data for partition 3 is:
The data for partition 4 is:
bhw-proxy.hons.com.np# fdisk -s /dev/ad2
fdisk: invalid fdisk partition table found
fdisk: read_s0: Unknown error: 0
computer1# fdisk /dev/ad2
******* Working on device /dev/ad2 *******
parameters extracted from in-core disklabel are:
cylinders=77542 heads=16 sectors/track=63 (1008 blks/cyl)
Figures below won’t work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=77542 heads=16 sectors/track=63 (1008 blks/cyl)
fdisk: invalid fdisk partition table found
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 78162273 (38165 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 741/ head 15/ sector 63
The data for partition 2 is:
The data for partition 3 is:
The data for partition 4 is:
Any suggestions? Please
likeygmirror
In case it might be of use to anyone, here is the gmirror cheat sheet I’ve been using for the last couple years. Its known to work on 6.0,6.1,6.2 and 6.3. It was originally cribbed from a website and has been updated for personal use since.
FreeBSD Duplex mirror using GEOM
— CD OS Install
* Partition : Commit entire disk (ad4)
* Partition : / 512M
SWAP 2GB
/var 2GB
/tmp 512M
/usr remainder
* Set up duplex mirror using GEOM (before CD reboot)
– INSTALL
– ALT-F4
– # csh
– # sysctl kern.geom.debugflags=16
result : kern.geom.debugflags: 0 -> 16
– # gmirror label -v -b round-robin gm0 /dev/ad4
result : Metadata value stored on /dev/ad4
– # echo geom_mirror_load=”YES” > /boot/loader.conf
– # cp /etc/fstab /etc/fstab.orig
– # vi /etc/fstab (replace /dev/ad4 with /dev/mirror/gm0)
– reboot
– # vi /etc/rc.conf
# need to do this to make dumping cores happy since it won’t use a gmirror’ed drive
dumpdev=”NO”
– SYNC
– # swapinfo
– # gmirror insert gm0 /dev/ad6
result : GEOM_MIRROR: Device gm0: provider ad6 detected.
result : GEOM_MIRROR: Device gm0: rebuilding provider ad6.
– # gmirror list | more
– # gmirror status
– FAILURE RECOVERY
– REMOVE A LIVE DISK FROM A MIRROR (but keep in machine)
– # gmirror deactivate gm0
– # gmirror forget gm0
(forget about removed devices)
– # gmirror status
– BRING A NEW DISK ONLINE
– ***insert drive***
– # atacontrol detach ***ata-channel-to_remove ex : ata3 for /dev/ad6***
(make sure channel is not initialized)
– # atacontrol attach ***ata-channel-to_remove ex : ata3 for /dev/ad6***
– # gmirror forget gm0
– # gmirror insert gm0
– # gmirror status
– HOT UNPLUG A LIVE DISK FROM A MIRROR !!!BROKEN IN 6.0
– # gmirror deactivate gm0 ***disk-to-remove devname ex : ad6***
– # gmirror forget gm0
(forget about removed devices)
– # atacontrol detach ***ata-channel-to_remove ex : ata3 for /dev/ad6***
– ***remove drive***
– # atacontrol list
dannyman
Likey,
I merged your comments. Please let me know if there’s anything amiss.
-d
Copycat Arts » Blog Archive » Soft-RAID1 for FreeBSD6
[…] http://dannyman.toldme.com/2005/01/24/freebsd-howto-gmirror-system/ […]
Comment
Tiny Print:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>