dannyman.toldme.com


FreeBSD, Technical

FreeBSD HOWTO: Software Mirror System Disk

A new feature of FreeBSD 5.3 is the ability to set up a software mirror of your system disk. This allows you to boot off either of a pair of hard disks, which will then function as a RAID1, which will ensure system uptime in the face of a single disk failure.

STOP THE PRESS! The instructions below work great, but I have just stumbled upon an easier way:
http://www.onlamp.com/pub/a/bsd/2005/11/10/FreeBSD_Basics.html

If, for some reason that doesn’t work for you so well, here’s my cheat sheet for setting this up with gmirror:

(This crib sheet assumes you have a pair of identical IDE (in my case, SATA) drives identified as ad4 and ad6.)

  1. Install FreeBSD on to ad4.
  2. Reboot with the Install CD.
  3. Enter Fixit mode. (For FreeBSD less than 5.4, use Install CD disc2 as the “live filesystem”)
  4. # chroot /dist
    # mount_devfs devfs /dev
    # gmirror load
    # gmirror label -v -b round-robin gm0 /dev/ad4
    # gmirror insert gm0 /dev/ad6
    # mount /dev/mirror/gm0s1a /mnt
    # echo ‘geom_mirror_load=”YES”‘ >> /mnt/boot/loader.conf
    # echo ‘swapoff=”YES”‘ >> /mnt/etc/rc.conf
  5. Edit /mnt/etc/fstab to convert ad4 -> mirror/gm0 (The ee command may be useful here – press ESC when you are done.)
    takhun suggests the following:
    # sed “s%ad4%mirror/gm0%” /mnt/etc/fstab > /mnt/etc/fstab.new
    # mv /mnt/etc/fstab.new /mnt/etc/fstab
  6. Reboot

Thanks to the few dozen people who have come before me, and posted crucial hints to the mailing lists. Thanks also to Ralf S. Engelschall who has a far more verbose explanation of how to do this sort of thing without console access and with mis-matched disks.

You should definately look over the gmirror man page, and review the output of gmirror list gm0 when swapping out drives. You can disable automatic rebuild, etc. It is quite nice.

Last note: If you need support with gmirror, Check out freebsd-questions — your question may be answered if you add a comment here, but asking on a mailing list will probably serve you best.

Read More

Next:
Previous:
Categories: FreeBSD, Technical
Possibly-Related Posts

Responses

February 3rd, 2005

usbenny

Hi,

Thanks for the great howto. I am having a problem when I goto insert ad6 into the mirror it gives me a “Class not found” error. Any help would be greatly appreciated.

usbenny

February 3rd, 2005

usbenny

Ok, I got around this problem by doing a gmirror load before the insert.

Thanks,
-usbenny

March 9th, 2005

Don

I tried it but couldn’t do the last edit of mnt/etc/fstab to convert ad4 -> mirror/gm0 because it was a read only file. I shouldn’t have done it because now the server will not boot up at all giving the error:
mountroot> ufs
mounting root from ufs
ffs_mountroot: can't find rootvp
Root mount failed: 6

Manual root filesystem specification:

March 11th, 2005

Don

No worries, mate – git it in the end. For some reason the only editor I could invoke was ed so I had to learn that to edit the fstab file.

Thanks for the guide

April 13th, 2005

andrew

hi!

i hust want to know the exact procedure how do i edit the fstab?

thanks!
andrew

April 17th, 2005

Leffe

I copied /usr/bin/vi to /mnt and then did another chroot to /mnt to be able to modify /etc/fstab with vi.

Cheers!

/Leif

May 2nd, 2005

Dave Webster

When I type the command:
$ gmirror insert gm0 /dev/ad6
the response is:
"Provider ad6 already inserted"
then the next cmd:
$ mount /dev/mirror/gm0s1a /mnt
gives the response:
"/dev/mirror/gm0s1a: no such file or directory"

Any help would be greatly appreciated.

regards,
Dave

May 15th, 2005

tahkun

Oh! that’s great!
It thought about gmirror by which I also had used LiveFileSystem.
However, it did not make it to good, because devfs had not been mounted.
It became reference very much.
Thank you!

PS
By the way, I changed fstab by using sed.
# sed "s%ad4%mirror/gm0%" /mnt/etc/fstab > /mnt/etc/fstab.new

June 2nd, 2005

anteeee

Hi all,

This seems great! So I don’t need to buy any hardware to set this up? And it is also possible to boot from both partitions?

Do I have to make any adjustments inside my server before doing this (connecting the SATA drives somehow?

June 3rd, 2005

manoj

Hi all.

I’m a starter on both BSD and software RAID stuff. I’ve been going through tons of materials on web searching for solution to create a good software mirror RAID0 setup on freeBSD and the resources on this site turned to be an angel from heaven :)

My configurations:
gmirror name gm0
first provider ad4
second provider ad6

As suggested in this site, I’ve successfully created mirror on my 2x74GB SATA drives using livefs CD boot and tested to be fault tolorent when I plugged out the first drive ad4 while live. The system was running fine in degraded state.

The question is:

Can I synchronize from the second drive (i.e. ad6) should I require to replace the first drive on ad4. I reconnected the first drive and tried to synchronize from the second drive but it wouldn’t let me. Is there a way to do that?

I know this is not a mailing list. But since I found this site more helpful that the lists I ran through, I hoped I would get short and to-the-point answer.

Thank you all.

June 3rd, 2005

dannyman

Manoj,

If you ran in to some specific errors, I’d hit a mailing list with those and see.

If you simply remove the drive and plug it back in, there may be issues because gmirror will see that you gave it its component back. It SHOULD mark it dirty and rebuild …

If you removed the drive, erased the data, then reinserted it, you’d have to do the “gmirror insert” command to add that drive to the gmirror. It would then rebuild.

gmirror can be configured to automatically rebuild, or manually. Check the man page.

Sincerely,
-danny

June 5th, 2005

manoj

Thanks for the response. I tested it today and it is working flawlessly. And i’m very happy with the setup. I’m gonna keep it running for few days and put to production after that.

Just to add to your good setup notes:

If a drive fails (eg. drive /dev/ad4):

#gmirror forget gm0
#shutdown -h

replace the drive with a fresh one

#gmirror insert gm0 /dev/ad4

And it should start synchronizing if autosynchronization is turned on.

Thanks.
-Manoj

June 13th, 2005

michi

Please help me:) I’m trying to do a gmirror for three days.. and then I’ve found this tutorial. But I stopped at 3.rd point, what does it mean to ‘Enter fix-it mode’? I’ve got 5.4 installed. Thanks.

July 2nd, 2005

SunSet

Ok got a question:

I still get the error..

Mountroot does any1 knows what i must do there?

July 12th, 2005

agouti

Please can you tell us the difference between gmirror and vinum ?? i am a newbi and i read that vinum do raid mirroring too …
thanks.

July 12th, 2005

agouti

sorry i found the answer: gmirror it’s raid1 spécific and it’s a part of geom framwork .vinum it’ s old and came from freebsd4 and will be replace by geom vinum …
thanks and ” place à gmirror “

July 15th, 2005

urfx

hey big thanks, manged to get this working first time on a supermicro…

http://www.supermicro.com/products/system/1U/5014/SYS-5014C-T.cfm

I couldn’t get the hardware SATA raid controller to work with FreeBSD hence eventually came across this crib.

The experience I would like to share is, I couldn’t use this crib on a box where I had already tried to set up a gmirror [failed] using the method oulined by Engelschall.

http://people.freebsd.org/~rse/mirror/

Then I experienced problems similar to Dave Webster. Long and short of it I was unable to clean the disks on that first box and try again, i tried using dd but failed because at this point I was out of my depth.

Seems to me the key to succes is clean disks.

i.e. ad4 and ad6 so if anyone has a definitive way to make sure each and every sector is clean, it is probably a useful first step before installing FreeBsd on ad4.

Anyhow big thanks again. I’m now going to play with rebuilding a broken mirror before going for this a production solution

July 15th, 2005

urfx

I had a hunch my dd failed because it was cleaning the first few sectors of the harddives ad4 and ad6, after RTFM…

     The gmirror utility is used for mirror (RAID1) configurations.  After a
     mirror's creation, all components are detected and configured automati-
     cally.  All operations like failure detection, stale component detection,
     rebuild of stale components, etc. are also done automatically.  The
     gmirror utility uses on-disk metadata (stored in the provider's last sec-
     tor) to store all needed information.  Since the last sector is used for
     this purpose, it is possible to place a root file system on a mirror.

… the hunch is confirmed so I think what is needed is a method to erase the last sectors of the HDD’s

I’ll man dd.

July 17th, 2005

urfx

ok gmirror can do the work… here is how,

if you have tried and failed using various methods of gmirror, you will likely have data tagged at the end sectors of ad4 and ad6 or whatever the first the disks are in your array.

before trying danny’s method above, you need to clear these sectors. To do this succesfully you must be logged in as root, you can’t clear the sectors on a live system so you must install a minimal instance of freebsd [if you haven’t already] on as least two disks on your server, then use the following command…

### while booted from ad4
### while logged in as root

gmirror clear ad6

### while booted from ad6
### while logged in as root

gmirror clear ad4

### then you will be able to proceed with each and every step in Danny’s crib, to be safe do a clean install on ad4, [this is in the crib]

### urfx

July 19th, 2005

Tim Provencio

Awesome walkthrough! Just bought a Dell SC420 and realized the “mirror set” was two different drives in FreeBSD. Disabled RAID in the BIOS and followed these instructions without a hitch. Thanks much!

July 23rd, 2005

boredhacker

Ugh, I guess my first post didn’t make it through (for one reason or another). Maybe I forgot to hit the submit button, or maybe the lengthy url I originally specified as my web-site screwed things up, I’m just not sure…

Anyway, the HOWTO is great and I used it to set up a duplexed raid mirror on my own machine (yay!).

I just have one question: why is the ‘-b round-robin’ option suggested? I used ‘-b load’ instead assuming that performance would be better.

Can anyone comment on the differences between the balance algorithms that are available (i.e. load, prefer, round-robin, and split). The gmirror manpage is a bit terse when describing the options, and IMHO it would be nice to have a better idea of what those options do without having to dive into the source code.

Once again, thanks for a great HOWTO!

July 23rd, 2005

dannyman

boredhacker,

Although comments from new people have to be moderated, I only got the one comment from you.

I chose round-robin because I figured if you have two disks in a mirror, they’re both under the same “load” constraints, and it is best to KISS.

If you really want to know, you should ask on FreeBSD-hackers@ …

Thanks,
-danny

July 27th, 2005

ari braginsky

On bootup I am seeing:

GEOM_MIRROR: Device gm0 created (id=xxxx)
GEOM_MIRROR: Device gm0: provider ad0 detected.
GEOM_MIRROR: Force device gm0 start due to timeout.

Note: no media was in the CDROM at the time of bootup. CDROM is slave to IDE1 channel.

July 28th, 2005

ari braginsky

Forgot to add that the system locks up (even keyboard) after:

GEOM_MIRROR: Force device gm0 start due to timeout.

There is no disk activity.

Thoughts?

July 30th, 2005

ari braginsky

Solution:

See http://www.freebsd.org/cgi/query-pr.cgi?pr=73660.

Turns out I was able to fix my problem after removing the CDR that was a slave on IDE1, the same channel that the gmirror provider ad0 was on.

Possible fix is to increase the mirror timeout or put the CDR on a different IDE channel than then mirror provider.

August 9th, 2005

Downtown

“I know this is a case of trying to Walk before Crawling”

Installing Geom on 5.4 and I enter all the required commands in the fixit mode. Then rebooting, nothing seems to have changed. Don’t I have to mount my changes from the fixit mode from /mnt2, problem is I am not sure of the commands to achieve this.

Can anyone help please?

August 10th, 2005

Krisna

Great & Nice Danny.

But how about remote administrator? That doesn’t have acces to server hardware, only network connection with SSH access?

August 12th, 2005

boredhacker

>>>I chose round-robin because I figured if you have two disks in a mirror, they’re both under the same “load” constraints, and it is best to KISS.

I can’t say that I agree with you on this one. From a user perspective typing ‘load’ or ‘prefer’ are both easier to use than ’round-robin’; fewer characters ;-) But seriously, in my desktop machine I have a duplexed gmirror built off of 2 disks that have the same geometry but one is udma66 and the other is udma100, I’m pretty sure using ‘load’ is better in this situation and probably others too.

Regardless, the how-to is still a wonderful thing. I certainly don’t want to take anything away from it. However, a little constructive criticism can be a good thing, so….

If I may suggest, although the how-to tells you how to set-up the mirror, it never mentions what to do when an actual disaster happens (i.e. when a disk needs to be changed).

Some things to consider:

1. Most modern disks will not be the same size even if “you have a pair of identical IDE … drives”. This is because the integrated drive electronics account for imperfections in the media (bad tracks/sectors). Thus the command “gmirror label -v -b round-robin gm0 /dev/ad4” in step 4 should *i think* be more like “gmirror label -v -b round-robin gm0 /dev/ad6 /dev/ad4”. This will create a mirror only as big as the smallest drive. If you don’t do this, you just might see a message that says you can’t insert ad6 because it’s too small (even though the disks are ‘identical’). In addition, to quote the gmirror manpage: “The order of components is important, because a component’s priority is based on its position (starting from 0). The component with the biggest priority is used by the prefer balance algorithm and is also used as a master component when resynchronization is needed, e.g. after a power failure when the device was open for writing.” In light of this, I’m not sure if the command should be “gmirror label -v -b round-robin gm0 /dev/ad6 /dev/ad4” or “gmirror label -v -b round-robin gm0 /dev/ad4 /dev/ad6” – but one may be better than the other. I mean… the gmirror manpage does say order is important, right?

2. When installing the system (in step 1) it might not be a bad idea to create a slice that is actually smaller (by say 32-128mb) than the drive allows – or to install the system to the truly smallest drive. This will insure that when/if a drive needs to be replaced you can simply use the same make and model (or a drive with the same physical geometry) and not worry about size discrepencies.

3. Replacing a drive can be simple if you have a new one on hand. If it isn’t the boot drive that has failed (or the mirror is already running and you have hot-swap hardware) then it’s just a matter of ‘gmirror forgeting/removing’ the old disk and ‘gmirror inserting’ the new disk. If you have hot-swappable hardware you shouldn’t even have to reboot. If the boot drive failed (and you can’t [re]boot) you’ll need the fix-it cd as you recommend to ‘forget/remove’ the drive, and ‘insert’ another. Use ‘gmirror status’ to wait until the mirror rebuilds itself before exiting fixit (since it was the boot drive that failed). Booting a partially rebuilt disk simply can’t be a good idea imho.

4. The crib sheet assumes you have a pair of identical IDE disk drives, which is a good thing. Assuming otherwise would introduce too much complexity for a simple how-to. However, this just makes the previous three considerations that much more important. After all, what good is having a mirror if you can’t rebuild it after a failure?

Now, as you can probably tell, I’m not 100% sure about a lot of this. And I may not have written it out completely or perfectly clear. But hopefully these suggestions can make a good how-to even better!

August 13th, 2005

dannyman

bh,

I’d ask around on -hackers but I’m pretty sure bad block mapping is done mostly within the drives these days: they present a series of blocks to the OS that is smaller than their absolute physical capacity, and internally map around any bad blocks as they are encountered. An OS may in turn encounter bad blocks, and does a similar thing. If I were implementing GEOM and gmirror, that easiest thing to do is that if you have to remap a bad block on one drive, you re-map it in the virtual drive. So, if someone slips you a replacement drive with many bad blocks, you just map those into the virtual drive. Maybe if you drop drives, you can free up those bad blocks from the OS’ list.

I think it is good to think about these things, but I’d suggest a bit more research before you start making your life more inconvenient to solve troubles that have hopefully been addressed already. :)

Cheers,
-danny

October 10th, 2005

★ On.the.nite.like.this ★ » Blog Archive » FreeBSD HOWTO: Software Mirror System Disk

A new feature of FreeBSD 5.3 is the ability to set up a software mirror of your system disk. This allows you to boot off either of a pair of hard disks, which will then function as a RAID1, which will ensure system uptime in the face of a single disk failure . . .

November 9th, 2005

Shtir

Thanx for the HOWTO step by step instrauctions. But after doing everything FreeBSD dosn’t load stopping at:

mounting root from ufs:/dev/mirror/gm0s1a
setrootbyname failed
ffs_mountroot: can’t find rootvp

How can I fix this prob to boot FreeBSD.

December 14th, 2005

Mitch Walker

Brilliant work, buddy. Thanks a million! Literally as easy as 1-2-3. It’s HOWTOs like this that make open-source work.

April 21st, 2006

Andreas

Hi,

I have a mirror with two identically SATA drives. The mirror was build, as the system was installed a year ago.

This are the gemoetry for ad4/ad6: cylinders=484521 heads=16 sectors/track=63 (1008 blks/cyl)

And this is the geometry for gm4: cylinders=30401 heads=255 sectors/track=63 (16065 blks/cyl)

The one partition has this size (is for ad4/ad6 and gm4):

The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 488397042 (238475 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 168/ head 14/ sector 63

You can see, that the partition is bigger than the gm4 geometry.

Any idea, what I can do? A fdisk -u mirror/gm4 will not work, because fdisk prins:

fdisk: Geom not found
fdisk: Failed to write sector zero

April 21st, 2006

Andreas

ooops, forget something:

# gmirror list
Geom name: gm4
State: COMPLETE
Components: 1
Balance: round-robin
Slice: 4096
Flags: NONE
GenID: 0
SyncID: 1
ID: 3556157093
Providers:
1. Name: mirror/gm4
Mediasize: 250059349504 (233G)
Sectorsize: 512
Mode: r4w4e5
Consumers:
1. Name: ad4
Mediasize: 250059350016 (233G)
Sectorsize: 512
Mode: r1w1e1
State: ACTIVE
Priority: 0
Flags: NONE
GenID: 0
SyncID: 1
ID: 3808931721

The difference ist exactly 512 bytes. Is this the last sector, where geom stores the meta data?

Is the difference with the geometry critical?

May 29th, 2006

sara lidgey

Hi All,

I’ve been running a server using FreeBSD 5.3 and gmirror to mirror two identical IDE hard drives. This howto was very helpful when setting up. Its been running great for over a year. But recently everything went down and when I reboot and put a monitor on it I get the following errors on screen:

GEOM_MIRROR: Device gm0: provider ad1 disconnected
GEOM_MIRROR: Device gm0: provider mirror/gm0 destroyed
GEOM_MIRROR: Device gm0: rebuilding provider ad0 stopped

Fatal trap 12: page fault while in kernel mode… (this is followed by details about the fault)

These errors are preceded by other related error information that flys by on the screen and I have no way of seeing them again.

Does anyone now what steps I should take to figure what is going on and try to recover data or get the machine to boot?

Thanks so much,
S.

September 6th, 2006

Fabien Langlet

Hi,

I have a computer with one hard disk (single slice ad0s1) running with FreeBSD. This computer is used as a WEB server, mail server, DNS server, … For a best security, I would like to establish a RAID-1 for the system partitions using your “howto” with a second new hard drive (mirror). My question is :

Is it possible to establish a RAID-1 on my system using your “howto” and keeping all my first drive data (WEB server, mail server,…) ? In fact, I really need a mirror to upgrade my security, but I don’t want to loose my data on my first hard drive.

Thanks for your tutorial.

Best regards.

F. Langlet-Cauet.

February 12th, 2008

bones

Be aware that this doesn’t work with BSD 6.3 because the CD’s gmirror command somehow doesn’t list most options. This is also described in the url below.

As as workaround I installed a minimal freebsd system on a temporarily third disk, boot from it and created the raid. Then take it out again.

http://www.nabble.com/misc-120169:-gmirror-commands-error-with-“Class-not-found”-or-other-td15190746.html

August 7th, 2008

Bishal

Hi,
I am trying to do RAID1 mirror on live freebsd6.0 system. Both harddisk are 40GB of size. When I type gmirror insert gm0 /dev/ad2 it gaves me “Provider ad2 too small” message. What’s wrong? Earlier ad2 hardisk was running Winxp and freebsd dual OS. I completly deleted the partition and tried to do RAID1 but that errror “Provider ad2 too small” message comes up.
Sysinstall command shows both hardisk size of 38162MB. Output of the fdisk command on both hardisk:

computer1# fdisk /dev/ad0
******* Working on device /dev/ad0 *******
parameters extracted from in-core disklabel are:
cylinders=77545 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won’t work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=77545 heads=16 sectors/track=63 (1008 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 78156162 (38162 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 1023/ head 254/ sector 63
The data for partition 2 is:

The data for partition 3 is:

The data for partition 4 is:

bhw-proxy.hons.com.np# fdisk -s /dev/ad2
fdisk: invalid fdisk partition table found
fdisk: read_s0: Unknown error: 0
computer1# fdisk /dev/ad2
******* Working on device /dev/ad2 *******
parameters extracted from in-core disklabel are:
cylinders=77542 heads=16 sectors/track=63 (1008 blks/cyl)

Figures below won’t work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=77542 heads=16 sectors/track=63 (1008 blks/cyl)

fdisk: invalid fdisk partition table found
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 165 (0xa5),(FreeBSD/NetBSD/386BSD)
start 63, size 78162273 (38165 Meg), flag 80 (active)
beg: cyl 0/ head 1/ sector 1;
end: cyl 741/ head 15/ sector 63
The data for partition 2 is:

The data for partition 3 is:

The data for partition 4 is:

Any suggestions? Please

January 8th, 2009

likeygmirror

In case it might be of use to anyone, here is the gmirror cheat sheet I’ve been using for the last couple years. Its known to work on 6.0,6.1,6.2 and 6.3. It was originally cribbed from a website and has been updated for personal use since.

FreeBSD Duplex mirror using GEOM

— CD OS Install
* Partition : Commit entire disk (ad4)
* Partition : / 512M
SWAP 2GB
/var 2GB
/tmp 512M
/usr remainder
* Set up duplex mirror using GEOM (before CD reboot)
– INSTALL
– ALT-F4
– # csh
– # sysctl kern.geom.debugflags=16
result : kern.geom.debugflags: 0 -> 16
– # gmirror label -v -b round-robin gm0 /dev/ad4
result : Metadata value stored on /dev/ad4
– # echo geom_mirror_load=”YES” > /boot/loader.conf
– # cp /etc/fstab /etc/fstab.orig
– # vi /etc/fstab (replace /dev/ad4 with /dev/mirror/gm0)
– reboot
– # vi /etc/rc.conf
# need to do this to make dumping cores happy since it won’t use a gmirror’ed drive
dumpdev=”NO”

– SYNC
– # swapinfo
– # gmirror insert gm0 /dev/ad6
result : GEOM_MIRROR: Device gm0: provider ad6 detected.
result : GEOM_MIRROR: Device gm0: rebuilding provider ad6.
– # gmirror list | more
– # gmirror status

– FAILURE RECOVERY
– REMOVE A LIVE DISK FROM A MIRROR (but keep in machine)
– # gmirror deactivate gm0
– # gmirror forget gm0
(forget about removed devices)
– # gmirror status

– BRING A NEW DISK ONLINE
– ***insert drive***
– # atacontrol detach ***ata-channel-to_remove ex : ata3 for /dev/ad6***
(make sure channel is not initialized)
– # atacontrol attach ***ata-channel-to_remove ex : ata3 for /dev/ad6***
– # gmirror forget gm0
– # gmirror insert gm0
– # gmirror status

– HOT UNPLUG A LIVE DISK FROM A MIRROR !!!BROKEN IN 6.0
– # gmirror deactivate gm0 ***disk-to-remove devname ex : ad6***
– # gmirror forget gm0
(forget about removed devices)
– # atacontrol detach ***ata-channel-to_remove ex : ata3 for /dev/ad6***
– ***remove drive***
– # atacontrol list

January 8th, 2009

dannyman

Likey,

I merged your comments. Please let me know if there’s anything amiss.

-d

Comment

Leave a comment . . .

Tiny Print:

  1. For private messages, e-mail me: dannyman@toldme.com.
  2. You must provide an e-mail address.
  3. You can use a bogus e-mail address, but I like to know who you are.
  4. I will not spam you. I will not publish or share your e-mail address.
  5. First-time commenters will be held for review.
  6. You can use these HTML tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>