These instructions are based on Debian Sarge 3.1, with some stuff from backports, using mdadm
Some of the software versions I used were:
kernel version: linux-image-2.6.18-4-686 (from backports)
initramfs-tools (version 0.85g~bpo.1) from backports
module-init-tools 3.2-pre1-2 (from backports)
udev 0.056-3 (from backports)
So with a 'pure' Sarge system, it may be different.
The hardware:
IBM x3250 with 1 x 160GB SATA disk, to which an identical one was added. These servers accept hot swap drives.
The initial install was done on /dev/sda. The extra (identical) drive was added and machine was rebooted, so that /dev/sdb was now available, though you may use /proc or /sys interface to find the new drive.
After adding the disk, you need to create the same partition table as on the original disk. One easy way to repartition is:
sfdisk -d /dev/sda | sfdisk /dev/sdb
Next, the kernel needs to support software raid. I was using a stock Debian 2.6.18 kernel, a modular kernel that includes the software raid drivers, and uses an initrd. The modules for raid-1 are 'md_mod' and 'raid1'. Use modprobe to load the modules:
modprobe md_mod raid1
There are a number of steps detailed on various other HOWTOs (that I used as a guide) that I found were not necessary for my particular case. In particular, I found that I didn't need to use MAKEDEV to create the device nodes for the raid devices (/dev/md0, /dev/md1, etc)7: udev takes care of that. Also, I did not need to manually configure raid modules to be loaded at boot time, and nor did I need to create or update the initrd to ensure the raid modules were loaded so that the root filesystem could be mounted. The Debian kernels include the raid drivers, but I am not sure what does the autodetection to load the raid drivers for the initrd. At a guess, I would say it finds a raid signature on the drive. I don't think it uses fdisk labels or info from grub config, since I have booted raid systems with no /dev/md0 in the grub config, or disklabel 'fd' on the disks. Obviously, it can't use anything on the filesystem, like /etc/mdadm/mdadm.conf or /etc/modules, as they aren't available before the root filesystem is mounted.
Anyway, on to the next steps. Since we are running on /dev/sda already, the raid needs to be constructed on using only the unused device (we don't want to obliterate the contents of the disk the system is currently using!). In effect, we initially create a degraded array. The original disk will be added later, once we have copied the data to the new disk and booted from it (detailed later).
My disk layout was as follows:
/dev/sda1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sda2 125 327 1630597+ fd Linux raid autodetect
/dev/sda3 328 1215 7132860 fd Linux raid autodetect
/dev/sda4 1216 19457 146528865 5 Extended
/dev/sda5 1216 1944 5855661 fd Linux raid autodetect
/dev/sda6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sda7 3525 19457 127981791 fd Linux raid autodetect
Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sdb2 125 327 1630597+ fd Linux raid autodetect
/dev/sdb3 328 1215 7132860 fd Linux raid autodetect
/dev/sdb4 1216 19457 146528865 5 Extended
/dev/sdb5 1216 1944 5855661 fd Linux raid autodetect
/dev/sdb6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sdb7 3525 19457 127981791 fd Linux raid autodetect
Initially, the partition type was 'Linux' (code 83 in fdisk), but I changed it to fdisk code 'fd'. As before, I don't think this is necessary at least initially (and the system was able to boot fine without those labels on a raid setup), but it might be useful or needed for other things anyway.
Create the raid devices:
mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md1 --level 1 --raid-devices=2 missing /dev/sdb3
and so on.
Once each raid device was created, I then made a filesystem 'mkfs.ext3 /dev/md0'. Do this for each raid device. Also, run a 'mkswap' on the new swap partition on /dev/sdb. If a disk fails, I think the machine should be able to boot ok, despite one of the swap partitions being dead. Before copying data, /etc/fstab and /boot/grub/menu.lst will need to be modified, and it may not hurt to put some details in /etc/mdadm/mdadm.conf (I am not sure if /etc/mdadm/mdadm.conf is actually needed for anything. test without it)
/etc/fstab - the new mountpoints + swap space:
### END DEBIAN AUTOMAGIC KERNELS LIST
NEW-mydns01:/var/lib# cat /etc/fstab
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/md0 / ext3 defaults,errors=remount-ro 0 1
/dev/md2 /home ext3 defaults 0 2
/dev/md1 /usr ext3 defaults 0 2
/dev/md3 /var ext3 defaults 0 2
/dev/md4 /var/lib/mysql ext3 defaults 0 2
/dev/sda1 none swap sw 0 0
/dev/sdb1 none swap sw 0 0
/dev/hda /media/cdrom0 udf,iso9660 user,noauto 0 0
/boot/grub/menu.lst - Important to have root filesystem as the raid device (in my case, /dev/md0), else you won't be able to hotadd the old disk (see further below for explanation). So the relevant part of /boot/grub/menu.lst looks like:
title Debian GNU/Linux, kernel 2.6.18-4-686
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro
initrd /boot/initrd.img-2.6.18-4-686
savedefault
title Debian GNU/Linux, kernel 2.6.18-4-686 (single-user mode)
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro single
initrd /boot/initrd.img-2.6.18-4-686
savedefault
/etc/mdadm/mdadm.conf:
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b196c373:f65bec91:01072554:c62097c9
devices=/dev/sdb2
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=47053389:84f72deb:b47832ba:1ec3ebc1
devices=/dev/sdb3
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=1c8ff28f:f4dae79a:6cbc8acd:caaf42cb
devices=/dev/sdb5
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=ae3882ea:df0f6494:d6b68701:1f922018
devices=/dev/sdb6
ARRAY /dev/md4 level=raid1 num-devices=2 UUID=ab8471a4:1d7d7959:922ec705:4b988a14
devices=/dev/sdb7
Note that there is only /dev/sdb listed here, not /dev/sda. That will be added later.
I am not sure what actually uses this file.
Copying data
rsync is good for this, though you could otherwise use tar or cp. I found one oversight on my part, which was that the mount point /var/lib/mysql was copied with ownership root:root and mode 755. This breaks mysql, so I think something in my rsync was incorrect
Assuming /dev/md0 is mounted at /mnt/root:
rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/mnt/* / /mnt/root
/dev/md1 will be the new /usr partition, and is mounted at /mnt/usr:
rsync -auHxv /usr/* /mnt/usr
And so on.
Boot block needs to be installed on each disk. Install it on /dev/sdb with grub
grub>root (hd1,1)
grub>setup (hd1)
It will need to find the root of the grub install, otherwise it will complain. But the rsync we did earlier should have taken car of that.
Reboot with changed fstab. Server should be running on 'degraded' raid running off /dev/sdb. Now /dev/sda needs to be added to the array. Use mdadm to do this, and this will copy the data from /dev/sdb over to /dev/sda (wiping whatever was on there)
mdadm /dev/md0 -a /dev/sda2
Due to a step I missed, I actually was using /dev/sda2 as the root filesystem in grub, though the raid was mounting '/' on /dev/md0. This caused an error when I tried to add /dev/sda to the raid:
mdadm: hot add failed for /dev/sda2: Invalid argument
/var/log/messages:
Feb 18 12:41:51 mydns01 kernel: md: error, md_import_device() returned -16
After much searching, I found that the issue was that the machine had root=/dev/sda2 in the grub config, and it will not allow you to use this device. I have read that this is because the device is in use, even though it appears not to be (since you have booted from /dev/sdb). Someone with a similar issue pointed out in dmesg:
Kernel command line: ro root=/dev/sda2
Anyway, change it to /dev/md0 in grub, and problem solved!
