initrd images created from kickstart

Update: initrd files are now 'xz' archives.



I wondered whether a kickstart of a RHEL/CentOS distro would provide a customised initrd image for the hardware it was being installed on. To check this, I compared the initrd from a server with a particular (Marvell) SAS disk controller with another server without that controller.

initrd images are gzipped cpio files, so to compare the two, I copied each initrd to another location, so as not to interfere with needed boot files, moved each to a filename with a '.gz' extension, then gunzipped them and ran 'cpio -t < initrd_file_name' to list the files. The SAS disk controller kernel module was present in the initrd from the host that had that particular hardware, but not on the other host which didn't. It wasn't present in the kernel image (vmlinuz) on either, though it was present in config as a loadable module, so would have to be in the initrd, since it needs a filesystem to load from, presumably:

CONFIG_SCSI_MVSAS=m

replacing a space with a newline

contents of file.txt:

abc def ghi

cat file.txt | tr ' ' '\012'

abc
def
ghi


I originally tried this with

cat file.txt | sed 's/ /\n/g'

and it didn't work. I am told that '\n' isn't always recognised as a newline

Migrating (expanding) an existing array from RAID 1 to RAID 10 on HP SmartArray 400i

I added a couple of 72GB drives to an existing 2-drive array on an HP DL360 G5, which has a SmartArray P400i controller. The driver was the HP cciss driver, and the server was running CentOS 5.5 x86_64.

The expansion of the array can be done while the server is online, over ssh using hpacucli. While the underlying drive can be expanded, I have not yet found a way to notify the kernel about the changed drive size, so it appears a reboot is still required (or reloading the cciss driver, which in most cases means a reboot).

Using hpacucli:


=> controller all show

Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )

=> ctrl slot=0 show config

Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )

array A (SAS, Unused Space: 0 MB)


logicaldrive 1 (68.3 GB, RAID 1, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)

unassigned

physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)

=> controller slot=0 logicaldrive 1 add drives=1I:1:3,1I:1:4
=> ctrl slot=0 show config

Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )

array A (SAS, Unused Space: 139953 MB)


logicaldrive 1 (68.3 GB, RAID 1+0, Transforming, 0% complete)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)

=>

Then you need to expand the logical drive into this space:

=> controller slot=0 show config

Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )

array A (SAS, Unused Space: 139953 MB)


logicaldrive 1 (68.3 GB, RAID 1+0, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)

=>
=> controller slot=0 logicaldrive 1 modify size=max

Warning: Extension may not be supported on certain operating systems.
Performing extension on these operating systems can cause data to
become inaccessible. See ACU documentation for details. Continue?
(y/n) y

=> controller slot=0 show config

Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )

array A (SAS, Unused Space: 0 MB)


logicaldrive 1 (136.7 GB, RAID 1+0, OK)

physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)

=>

The size is correctly reported in /sys:

# cat /sys/block/cciss\!c0d0/size
286611840
#

However, fdisk does not see it, even after a partprobe, and I could find no way of re-scanning in /proc or /sys to get it to see the increased size.

One suggestion I haven't tried is to use 'sfdisk -R'

So for now a reboot, then resize partitions via resize2fs.

How to resize (grow) a Xen VM partition

I was faced with a situation whereby a partition on a Xen VM was running out of space. Luckily, there was space available on the Xen host (i.e., Dom0). Also, the partition was at the end of the virtual disk, which simplifies things. So below are the steps I took to resize the partition.

NB: This assumes a Xen host (Dom0) is not running Logical Volume Management, and the VM (DomU) has an ext3 filesystem.

First of all, stop the VM and take a backup. This is in case anything goes wrong! You will need somewhere with enough space to copy the VM .img file, taking into consideration that the partition that you copy to, may well need free space to create the larger partition :-)

Once the VM has been stopped and the backup made, it's time to add the extra space to our VM. There are several HOWTOs on this, but what worked for me was to just concatenate a file onto the VM image. When creating a VM initially, the filesystem is built on a file in the same way. Using 'dd' is easy for this.

If we can add another 10GB to the partition, dd a file onto the end of the existing image:

Assuming a VM image called 'mc3.img', do:

dd if=/dev/zero bs=1G count=10 >> /path/to/mc3.img

NB: Be very careful to make sure you have '>>'

Now the filesystem needs to be expanded onto this image. Use 'losetup' to associate the file with a device.

'losetup -f' to search for the first free loopback device

Assuming this is '/dev/loop3', associate it with the image file

'losetup /dev/loop3 /path/to/mc3.img'

You will be able to see the old partitions if you run an 'fdisk -l' on /dev/loop3.

# fdisk -l /dev/loop0

Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 1019 1895670 83 Linux

However, the partition device nodes don't actually exist for us to be able to operate on them:

[root@xen001 vm]# fsck /dev/loop0p3
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
fsck.ext2: No such file or directory while trying to open /dev/loop0p3

The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193

So use 'kpartx' to create devices in /dev/mapper. List them:

[root@xen001 vm]# kpartx -l /dev/loop0
loop0p1 : 0 8385867 /dev/loop0 63
loop0p2 : 0 4192965 /dev/loop0 8385930
loop0p3 : 0 3791340 /dev/loop0 12578895
[root@xen001 vm]# kpartx -av /dev/loop0
add map loop0p1 : 0 8385867 linear /dev/loop0 63
add map loop0p2 : 0 4192965 linear /dev/loop0 8385930
add map loop0p3 : 0 3791340 linear /dev/loop0 12578895
[root@xen001 vm]# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 62 Jan 18 12:12 control
brw-r----- 1 root disk 253, 0 Jan 22 14:36 loop0p1
brw-r----- 1 root disk 253, 1 Jan 22 14:36 loop0p2
brw-r----- 1 root disk 253, 2 Jan 22 14:36 loop0p3
[root@xen001 vm]#

We need to delete the existing partition (in our case, partition 3, /dev/mapper/loop0p3), and then recreate it to take advantage of the extra space:

[root@xen001 vm]# fdisk /dev/loop0

The number of cylinders for this disk is set to 2325.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 1019 1895670 83 Linux

Command (m for help): d
Partition number (1-4): 3

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (784-2325, default 784):
Using default value 784
Last cylinder or +size or +sizeM or +sizeK (784-2325, default 2325):
Using default value 2325

Command (m for help): p

Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 2325 12386115 83 Linux

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@xen001 vm]#

Note how the 'End' cylinder has changed from 1019 to 2325

Now, resize filesystem (resize2fs supports resizing ext3 in 2.6.x kernels):

root@xen001 vm]# resize2fs /dev/mapper/loop0p3
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/mapper/loop0p3 to 3096528 (4k) blocks.
The filesystem on /dev/mapper/loop0p3 is now 3096528 blocks long.

Fsck it:

[root@xen001 vm]# e2fsck -f /dev/mapper/loop0p3
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/u01: 9474/3003520 files (17.4% non-contiguous), 527247/3096528 blocks
[root@xen001 vm]#

Remove it from device mapper:

[root@xen001 vm]# kpartx -d /dev/loop0

Detach it from loop device:

[root@xen001 vm]# losetup -d /dev/loop0

Start up the VM:

[root@xen001 vm]# xm create mc3
Using config file "/etc/xen/mc3".
Started domain mc3
[root@xen001 vm]#

And you're done

ldapmodify - adding objectClass, specifically, samba objectClass

The task was to add samba attributes to existing users in LDAP. The easy way is to do it via 'smbpasswd -a', though this will require various samba packages to be installed, and a samba config, and the ability of the LDAP server to look up users locally as LDAP users...(changes to pam in other words). This server was a production master LDAP server, so I wanted to see if I could do it without additional software and other changes like the one to PAM, and the obvious way was via 'ldapmodify'. First attempts failed, as I thought I could just add the objectClass first, and then the mandatory attribute, which is 'sambaSID' (as it says in /etc/openldap/schema/samba.schema: MUST ( uid $ sambaSID ).

contents of ldif:

dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
add: objectClass
objectClass: sambaSAMAccount


ldapmodify -x -D "uid=my.user,ou=Oslo,ou=People,dc=example,dc=com" -W -f m8.ldif
Enter LDAP Password:
modifying entry "uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com"
ldapmodify: Object class violation (65)
additional info: object class 'sambaSamAccount' requires attribute 'sambaSID'


So, I tried just adding the necessary attribute to the ldif as an additional change, but it still failed

modifying entry "uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com"
ldapmodify: Object class violation (65)
additional info: attribute 'sambaSID' not allowed


There is a special way to add an objectClass, as I found on:

http://www.zytrax.com/books/ldap/ch8/index.html#samples


I had to include all the existing objectclasses in the modify, as well as the new one, plus the necessary attribute:


dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: specialObjectClassUser
objectClass: sambaSamAccount
sambaSID: S-1-5-21-1096379175-3498632652-1565186858-17768

To add all of the attributes (not sure whether it is worth including all of these, but anyway):

dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: specialObjectClassUser
objectClass: sambaSamAccount
sambaSID: S-1-5-21-1096379175-3498632652-1565186858-17768
sambaPasswordHistory: 0000000000000000000000000000000000000000000000000000000000000000
sambaAcctFlags: [U ]
sambaLogonTime: 0
sambaLogoffTime: 2147483647
sambaKickoffTime: 2147483647
sambaPwdCanChange: 0
sambaLMPassword: 7E9Q28366B3F5C82AAD3B435B51404EE
sambaNTPassword: 42AZ499C5146E4F9BF017649A4756013
sambaPwdLastSet: 1255018017
sambaPwdMustChange: 1258906017

I have written a script in perl which can generate LDIFs, complete with randomised password hashes, and lookup of a user's UID to calculate sambaSID

super cool way to move files with spaces in shell

I had a bunch of files named like;

IMG_2345 2.JPG

I wanted to remove the space and the 2 so that it would become like:

IMG_2345.JPG

How to do this in a one liner?

Can't use usual for file in `ls *\ 2.JPG`;do mv $file etc, as it will
treat the space as a newline (and quoting $file to "$file" won't do it
either.

There is a convenient way of doing this with 'read', as well as using built-in string replacement

ls *\ 2.JPG | while read line;do mv "$line" "${line/\ 2/}";done



Substring Replacement

${string/substring/replacement}

Replace first match of $substring with $replacement.
${string//substring/replacement}

Replace all matches of $substring with $replacement.

http://www.linuxtopia.org/online_books/advanced_bash_scripting_guide/string-manipulation.html

setting pager to 'less' for mysql

Ever wanted to have 'less' functionality when doing mysql queries?

mysql> \P less -S
PAGER set to 'less -S'

So when you do a 'select *', it will automatically output it to less :)

ip_conntrack, /proc and sysctl

sysctl -a shows settings for conntrack, amongst other things. This will show count of number of entries in table as well


ip_conntrack has a default timeout of 5 days (432000 seconds) for *established* connections. That is, the entry will be kept in the table for 5 days before expiring if there is no traffic (?). With a large amount of traffic, this could grow very large.

There also settings that control SYN, ACK etc for conntrack (check this and see below)

AFAIK, the only way to clear the current count is to unload the relevant modules (check this)

You can put customisations that override defaults in /etc/sysctl.conf, which are read when sysctl is run, usually at boot. Warning:

When the ip conntrack module(s) is/are (re)loaded, the defaults are used. You have to run sysctl to read any settings in /etc/sysctl.conf (sysctl -p /etc/sysctl.conf)


net.ipv4.ip_conntrack_max = 1000000
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_loose = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_log_invalid = 0
net.ipv4.netfilter.ip_conntrack_generic_timeout = 3
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 3
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 2
net.ipv4.netfilter.ip_conntrack_udp_timeout = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 3
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_buckets = 8192
net.ipv4.netfilter.ip_conntrack_count = 598236
net.ipv4.netfilter.ip_conntrack_max = 1000000

postfix and rate limiting, sender control, etc

postfix has builtin controls restricting certain kinds of things, such as rate at which clients can connect, number of recipients per message, no. of connections per client, etc. These can be seen in man 8 smtpd and anvil.

As I didn't want to interfere too much with what might use the localhost instance, I added an instance of postfix listening on port 10025 of the eth0 address (192.168.10.208). That required adding a line to master.cf (I just added it under the standard smtp line):

192.168.10.208:10025 inet n - n - - smtpd
-o smtpd_client_message_rate_limit=5

The '-o smtpd_client_message_rate_limit=5' bit was to specify an override to limit connections from one client to 5 per the default anvil time (60s). The '-o' simply means override what is set in main.cf

The mail that I was attempting to restrict was being generated by snmpttd reading traps. It calls the 'mail' command for each trap, which can lead to a lot of mail in a large network. Now, the problem was that the 'mail' MUA (which is part of 'mailx' package in RHEL5) does not actually connect to smtp over tcp by default (and nor can I find any way of changing this in RHEL version, since all of the .mailrc and /etc/mail.rc directives, where you can specify smtp server and port, do not apply seemingly. They aren't in the man pages anyway. So I used the Mail::Sendmail module, which is easy to configure. BTW: I have not tried using '-o smtpd_client_message_rate_limit=5' on the 'unix' socket listening in master.cf, I assumed that it would not call smtpd, but that is because I was too lazy to read all the documentation on how postfix works ;)

Anyway, this works well

connection rate limiting to apache and iptables

We needed to create a way of connection rate limiting to a particular web page, from any given IP. There used to be various methods available to do this in Apache, such as mod_throttle/mod_choke/mod_limit, but the problem was these modules only worked with apache 1 or they were not actively developed anymore. We have netscreen firewalls in front of the the servers, but they cannot limit the number of connections per IP. IPtables can do this, via a number of different ways. There are modules in IPTables, such as 'connlimit' (most recent) and 'iplimit' (replaced by connlimit). However, the OS we were using was RHEL5.1 (kernel 2.6.18, and IPT 1.3.5), which, although it appears to support connlimit (type 'iptables -m connlimit --help' and it shows you usage info, as the iptables connlimit library, libipt_connlimit.so, is in /lib/iptables), there is no actual kernel module for it. When attempting to insert a rule with the connlimit module you get 'iptables: Unknown error 4294967295' (this was on a 32-bit machine). So there is a mismatch between kernel and user-space support for modules. The 'connlimit' module became mainstream in kernel 2.6.23. The options were to patch the kernel source tree using 'patch-o-matic' from the netfilter website. But since some doubt has been cast on the stability of these patches by various people, I decided against it, particularly as there is another option. In any case, I am not sure as to whether I was able to deploy a non-standard kernel onto production machines. It certainly would make updating more of a hassle. Inserting a binary module compiled on another devel box could also pose issues, in that there appear to be a lot of changes in the netfilter source tree, and it may invoke functions defined elsewhere lower in the tree than just the xt_connlimit module.

Found this at http://www.debian-administration.org/articles/187

It describes a simple way of connection limiting, using two rules, without using any experimental or recent modules.

iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 50 -j DROP

filesystem has unsupported features - fix with debugfs

On occasion I need to build machines with old distributions - usually, this involves copying the disks using tar or rsync onto a server that has been booted off a live CD such as GRML. When making an ext3 filesystem via a newer distribution such as GRML for what will be an older system such as redhat 7.2, certain ext3 features are not supported by the kernel and/or libraries (not sure which), and so it will complain when starting up, though it will still boot. I am not sure if it affects stability of the system. You can however remove the newer features put by there by the new boot CD by using debugfs. debugfs will do this on mounted filesystems as well. As a basis for comparison, I look at what one of the older servers has as the supported features by running debugfs on filesystems there, and then remove the features on the newly-built machine. You may also be able to get supported features in other ways, perhaps via logs or debugfs or researching what features are supported in the particular kernel/libraries/user-space tools or whatever enables the feature. I took the lazy method that is available though!

There is context sensitive

debugfs comes with its own shell, a bit like grub. You can type 'help' to access list of supported commands/operations. For instance, type 'feature' to list supported fs features, and 'feature - etc..' to delete features. Alternatively, you can run commands via the '-R' option from standard bash or other shell, or you can create a file with the operations you want to perform and send them to debugfs.

Very cool :)

converting an existing single disk machine to software RAID-1

The following is a guide based on my experience of setting up RAID 1 (mirroring) on a Debian Sarge system, with many of the packages coming from backports. I needed to use Sarge with backports for various reasons (rather than just using Etch). Anyways, I was able to do this remotely via ssh. It requires reboots, but no need for a boot CD/DVD or relying on back up data stored off the server (though this is a good idea to have in case something goes wrong!!). *PLEASE NOTE*: I suggest it as a guide only as it worked for me, but YMMV and I take no responsibility if you break your system!

These instructions are based on Debian Sarge 3.1, with some stuff from backports, using mdadm

Some of the software versions I used were:

kernel version: linux-image-2.6.18-4-686 (from backports)
initramfs-tools (version 0.85g~bpo.1) from backports
module-init-tools 3.2-pre1-2 (from backports)
udev 0.056-3 (from backports)

So with a 'pure' Sarge system, it may be different.

The hardware:

IBM x3250 with 1 x 160GB SATA disk, to which an identical one was added. These servers accept hot swap drives.

The initial install was done on /dev/sda. The extra (identical) drive was added and machine was rebooted, so that /dev/sdb was now available, though you may use /proc or /sys interface to find the new drive.


After adding the disk, you need to create the same partition table as on the original disk. One easy way to repartition is:

sfdisk -d /dev/sda | sfdisk /dev/sdb

Next, the kernel needs to support software raid. I was using a stock Debian 2.6.18 kernel, a modular kernel that includes the software raid drivers, and uses an initrd. The modules for raid-1 are 'md_mod' and 'raid1'. Use modprobe to load the modules:

modprobe md_mod raid1

There are a number of steps detailed on various other HOWTOs (that I used as a guide) that I found were not necessary for my particular case. In particular, I found that I didn't need to use MAKEDEV to create the device nodes for the raid devices (/dev/md0, /dev/md1, etc)7: udev takes care of that. Also, I did not need to manually configure raid modules to be loaded at boot time, and nor did I need to create or update the initrd to ensure the raid modules were loaded so that the root filesystem could be mounted. The Debian kernels include the raid drivers, but I am not sure what does the autodetection to load the raid drivers for the initrd. At a guess, I would say it finds a raid signature on the drive. I don't think it uses fdisk labels or info from grub config, since I have booted raid systems with no /dev/md0 in the grub config, or disklabel 'fd' on the disks. Obviously, it can't use anything on the filesystem, like /etc/mdadm/mdadm.conf or /etc/modules, as they aren't available before the root filesystem is mounted.

Anyway, on to the next steps. Since we are running on /dev/sda already, the raid needs to be constructed on using only the unused device (we don't want to obliterate the contents of the disk the system is currently using!). In effect, we initially create a degraded array. The original disk will be added later, once we have copied the data to the new disk and booted from it (detailed later).

My disk layout was as follows:

/dev/sda1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sda2 125 327 1630597+ fd Linux raid autodetect
/dev/sda3 328 1215 7132860 fd Linux raid autodetect
/dev/sda4 1216 19457 146528865 5 Extended
/dev/sda5 1216 1944 5855661 fd Linux raid autodetect
/dev/sda6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sda7 3525 19457 127981791 fd Linux raid autodetect


Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sdb2 125 327 1630597+ fd Linux raid autodetect
/dev/sdb3 328 1215 7132860 fd Linux raid autodetect
/dev/sdb4 1216 19457 146528865 5 Extended
/dev/sdb5 1216 1944 5855661 fd Linux raid autodetect
/dev/sdb6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sdb7 3525 19457 127981791 fd Linux raid autodetect


Initially, the partition type was 'Linux' (code 83 in fdisk), but I changed it to fdisk code 'fd'. As before, I don't think this is necessary at least initially (and the system was able to boot fine without those labels on a raid setup), but it might be useful or needed for other things anyway.

Create the raid devices:

mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md1 --level 1 --raid-devices=2 missing /dev/sdb3
and so on.

Once each raid device was created, I then made a filesystem 'mkfs.ext3 /dev/md0'. Do this for each raid device. Also, run a 'mkswap' on the new swap partition on /dev/sdb. If a disk fails, I think the machine should be able to boot ok, despite one of the swap partitions being dead. Before copying data, /etc/fstab and /boot/grub/menu.lst will need to be modified, and it may not hurt to put some details in /etc/mdadm/mdadm.conf (I am not sure if /etc/mdadm/mdadm.conf is actually needed for anything. test without it)

/etc/fstab - the new mountpoints + swap space:

### END DEBIAN AUTOMAGIC KERNELS LIST
NEW-mydns01:/var/lib# cat /etc/fstab
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/md0 / ext3 defaults,errors=remount-ro 0 1
/dev/md2 /home ext3 defaults 0 2
/dev/md1 /usr ext3 defaults 0 2
/dev/md3 /var ext3 defaults 0 2
/dev/md4 /var/lib/mysql ext3 defaults 0 2
/dev/sda1 none swap sw 0 0
/dev/sdb1 none swap sw 0 0
/dev/hda /media/cdrom0 udf,iso9660 user,noauto 0 0


/boot/grub/menu.lst - Important to have root filesystem as the raid device (in my case, /dev/md0), else you won't be able to hotadd the old disk (see further below for explanation). So the relevant part of /boot/grub/menu.lst looks like:

title Debian GNU/Linux, kernel 2.6.18-4-686
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro
initrd /boot/initrd.img-2.6.18-4-686
savedefault

title Debian GNU/Linux, kernel 2.6.18-4-686 (single-user mode)
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro single
initrd /boot/initrd.img-2.6.18-4-686
savedefault


/etc/mdadm/mdadm.conf:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b196c373:f65bec91:01072554:c62097c9
devices=/dev/sdb2
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=47053389:84f72deb:b47832ba:1ec3ebc1
devices=/dev/sdb3
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=1c8ff28f:f4dae79a:6cbc8acd:caaf42cb
devices=/dev/sdb5
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=ae3882ea:df0f6494:d6b68701:1f922018
devices=/dev/sdb6
ARRAY /dev/md4 level=raid1 num-devices=2 UUID=ab8471a4:1d7d7959:922ec705:4b988a14
devices=/dev/sdb7

Note that there is only /dev/sdb listed here, not /dev/sda. That will be added later.
I am not sure what actually uses this file.

Copying data

rsync is good for this, though you could otherwise use tar or cp. I found one oversight on my part, which was that the mount point /var/lib/mysql was copied with ownership root:root and mode 755. This breaks mysql, so I think something in my rsync was incorrect

Assuming /dev/md0 is mounted at /mnt/root:

rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/mnt/* / /mnt/root

/dev/md1 will be the new /usr partition, and is mounted at /mnt/usr:

rsync -auHxv /usr/* /mnt/usr



And so on.

Boot block needs to be installed on each disk. Install it on /dev/sdb with grub

grub>root (hd1,1)
grub>setup (hd1)

It will need to find the root of the grub install, otherwise it will complain. But the rsync we did earlier should have taken car of that.

Reboot with changed fstab. Server should be running on 'degraded' raid running off /dev/sdb. Now /dev/sda needs to be added to the array. Use mdadm to do this, and this will copy the data from /dev/sdb over to /dev/sda (wiping whatever was on there)

mdadm /dev/md0 -a /dev/sda2

Due to a step I missed, I actually was using /dev/sda2 as the root filesystem in grub, though the raid was mounting '/' on /dev/md0. This caused an error when I tried to add /dev/sda to the raid:

mdadm: hot add failed for /dev/sda2: Invalid argument

/var/log/messages:

Feb 18 12:41:51 mydns01 kernel: md: error, md_import_device() returned -16

After much searching, I found that the issue was that the machine had root=/dev/sda2 in the grub config, and it will not allow you to use this device. I have read that this is because the device is in use, even though it appears not to be (since you have booted from /dev/sdb). Someone with a similar issue pointed out in dmesg:

Kernel command line: ro root=/dev/sda2

Anyway, change it to /dev/md0 in grub, and problem solved!

plesk config information

besides the usual text files in /etc/, /usr/local/psa, some of it is stored in mysql in the psa DB, which can be accessed via Plesk control panel via phpmyadmin

UUID grub fstab

I wanted to migrate my data to a SATA disk from an IDE disk on an Ubuntu 7.10 system. In the past, I have simply tarred the contents of the partitions to the new disk, updated fstab and bootloader configs, reinstalled the MBR, and rebooted. However, in this instance, since the disk was laid out the same, I didn't really need to update partition info in grub config and fstab. If I had, I would have noticed that Ubuntu uses UUIDs for fstab and the grub config for identifying partitions. So I found myself with a system unable to get past the bootloader. My 'quick' fix was to use an Ubuntu CD in rescue mode and change /etc/fstab and /boot/grub/menu.lst to the 'old' style of /dev/sda1 etc. I then acquainted myself with the concept of UUID ('apropos UUID') plus some web searches. I found that you can find the UUID of a partition by running 'vol_id -u ', e.g., 'vol_id -u /dev/sda1'. I then copied this to fstab and menu.lst.

I think that UUID is handy way of doing things, in that it is possible to attach another disk to a machine without worrying whether it will change the order of things (e.g., by becoming the first scsi disk, and your system disk becomes the second disk, whilst all the fstab and grub entries point to the first still). It is still a slight mystery to me why the BIOS is able to boot from the second disk without modification though. Also, I am not sure why grub, when configured with (hd0) after adding a new disk which then becomes primary, isn't bothered. Maybe because this is just in the initial setup, and the bootloader is installed there anyway.

running out of memory when copying large files

From coraid support:

5.19 Q: How can I avoid running out of memory when copying large files?

A: You can tell the Linux kernel not to wait so long before writing data out to backing storage.

echo 5 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio

These settings are even tighter and should help even on a system that is doing a large amount of larger-than-RAM data transfers.

echo 3 > /proc/sys/vm/dirty_ratio
echo 3 > /proc/sys/vm/dirty_background_ratio
echo 5120 > /proc/sys/vm/min_free_kbytes

mysql replication

If mysql replication breaks with:

070816 13:03:17 Slave I/O thread: connected to master 'repl@mydns01.securepod.com:3306', replication started in log 'mydns01-bin.025' at position 20533567
070816 13:03:17 Error reading packet from server: Client requested master to start replication from impossible position (server_errno=1236)
070816 13:03:17 Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log'
070816 13:03:17 Slave I/O thread exiting, read up to log 'mydns01-bin.025', position 20533567

you can check the binlog that it last read up to (mydns01-bin.025), using mysqlbinlog, and search for log position it read up to (in this case 20533567). In this case, the last entry in mydns01-bin.025 is 20533147

mysqlbinlog mydns01-bin.025 | tail -10

#070815 16:01:10 server id 1 log_pos 20533147 Query thread_id=20094 exec_time=0 error_code=0
SET TIMESTAMP=1187211670;

INSERT INTO ZoneRecord
( ZoneRecord.zone, ZoneRecord.name, ZoneRecord.type, ZoneRecord.data, ZoneRecord.aux, ZoneRecord.ttl, ZoneRecord.active, ZoneRecord.categoryid )
VALUES ( '167857' , '' , 'NS' , 'ns1.securepod.com.' , 'None' , '14400' , '1' , '1' );

The next bin log mydns01-bin.026 starts from a new position (this position actually corresponds to the file size of the binlog). So: you can run the query that the last entry has in bin.025 and see if it exists, and run the query in bin.026 and see if it exists. If the first does, but the second does not, then you can change the bin log start position from the first position in bin.026, which is as easy as changing the master.info on the slave(s) to the new value and restarting mysql (or you could do it on the mysql command line if you can be bothered)

debootstrap bootstraps a basic Debian system of SUITE (eg, sarge, etch, sid) into TARGET from MIRROR by running
SCRIPT. MIRROR can be an http:// URL or a file:/// URL. Notice that file:/ URLs are translated to file:/// (cor‐
rect scheme as described in RFC1738 for local filenames), and file:// will not work.

Debootstrap can be used to install Debian in a system without using an installation disk but can also be used to
run a different Debian flavor in a chroot environment. This way you can create a full (minimal) Debian installation
which can be used for testing purposes (see the EXAMPLES section). If you are looking for a chroot system to build
packages please take a look at pbuilder.

differences between openvz and virtuozzo

openvz doesn't have vzfs, for one

Virtuozzo File System (VZFS) :

VZFS is a file system that allows to share common files among multiple VPSs without sacrificing flexibility. It is possible for VPS users to modify, update, replace, and delete shared files. When a user modifies a shared file, VZFS creates a private copy of the file transparently for the user. Thus, the modifications do not affect the other users of the file. Main benefits of VZFS are the following:

- It saves memory required for executables and libraries. A typical VPS running a simple web site might consume around 20–30 MBytes of RAM just for executable images. Sharing this memory improves scalability and total system performance;

- It saves disk space. A typical Linux server installation occupies several hundred MBytes of disk space. Sharing the files allows you to save up to 90% of disk space;

- VZFS does not require having different physical partitions for different VPSs or creating a special “file system in a file” setup for a VPS. This significantly simplifies disk administration;

- Disk quota enables the administrator to limit disk resources available to a VPS on-the-fly, in the same manner as the standard disk quota system works on a per-user basis. Disk quota for users and groups inside VPSs is also supported.

blocking spam with sendmail

Can block by domain or by IP address or network

in /etc/mail/access:

localhost.localdomain RELAY
localhost RELAY
127.0.0.1 RELAY
192.168.1 RELAY
202.124.241.222 ERROR:"550 Take a hike"

As this is a hash, rebuild to db with:

hash /etc/mail/access < /etc/mail/access

Useful way of telling php module support

php -m

or

php5 -m

php -v shows version

Also, you can do: 'php /home/cm/phpinfo.php'

ubuntu - BUG: soft lockup detected on CPU#0!

My Ubuntu (Ubuntu 7.04, AMD 1800XP, 2.6.20-15-generic #2 SMP Sun Apr 15 07:36:31 UTC 2007 i686 GNU/Linux) box would get halfway through booting and just sit there, doing nothing. ctrl+alt+del didn't do anything, and I couldn't get a console. To get more information, I rebooted and edited grub cmd line, taking off the 'quiet splash' options. It would get most of the way through fscking the first partition and then report: BUG: soft lockup detected on CPU#0! Not sure what causes this exactly, but I did a quick search and on some of the forums, a lot of people mentioned wireless network card as a possible source. The machine has got a wireless network card (it is a RaLink RT2561/RT61 802.11g PCI chipset), but not sure if this has anything to do with it. After a couple of futile soft reboots, I pulled the power and checked that the wireless card was properly seated, which it was, and booted the machine. It worked fine. I should try and find more out about what the error means though.

setting up ldapv3 under Ubuntu 7.04

I had numerous problems getting LDAP running under Ubuntu 7.04. Ubuntu 7.04 comes with slapd v2.3.30 and libldap2.3. I was setting it up as a precursor to running postfix and courier with virtual domain support. The main issue I had was using SASL, which can be disabled I think, but I wanted to use it just as a challenge (and it may be required for LDAP to be fully v3 compliant). I have read that no MTA currently supports SASL anyway, they are all v2 compliant only. I previously used LDAP v2 under Debian Sarge, which does not require SASL, and is very easy to get running out of the box. In any case, as MTAs auth via v2 only, I think you need to enable v2 bind support in slapd.conf even if you are running LDAP v3.

It is a very good idea to read the guide at www.openldap.org, as well as the relevant man pages, of which there are numerous. The documentation requires very careful attention.

I found a few useful pages, beside the official openldap documentation. Here's one that has been translated from japanese:

http://www.tom.sfc.keio.ac.jp/~torry/ldap/ldap_en.html#doc4_15616

I didn't delete the sasldb2 file as the author suggests though

Another one:

http://defindit.com/readme_files/ldap.html

And a pdf called 'Surviving Cyrus SASL':

http://postfix.state-of-mind.de/patrick.koetter/surviving_cyrus_sasl.pdf

A bit about SASL. See the wikipedia entry for background.

LDAP specific:

SASL - you can authenticate requests via SASL (for e.g., ldapdelete or other write operations), which looks at a password in /etc/sasldb2, or you can authenticate from info in the LDAP db, as you would other users. I'm still not 100% clear on whether you need to use SASL or not, and exactly what is required to have it working (some of my config options like sasl-realm and sasl-host may be superfluous, and I didn't use a slapd.conf file in /usr/lib/sasl2, etc etc)
I am not sure what auth mechanisms you need to set up.

I found that if you do specify a dn (i.e., with the '-D' flag), it will attempt to bind against what is in the LDAP db, e.g.,:

ldapdelete -x -D "cn=admin,dc=cm,dc=net" -W "ou=people,dc=cm,dc=net" -h localhost

but if you don't specify a dn, it will use SASL:

root@cm:/home/cm# ldapdelete -U admin "ou=people,dc=cm,dc=net" -h localhost
SASL/DIGEST-MD5 authentication started
Please enter your password:
SASL username: admin
SASL SSF: 128
SASL installing layers

Note that while SASL provides an *authentication* mechanism, that user still needs permission to be able to do things such as read or write to the LDAP db (authorization). So, you still need entries in the LDAP db allowing authenticated users permission to read/write/etc to LDAP (this is the same as with earlier versions of LDAP of course). Under v3, you could set LDAP up so that it authenticates via SASL and then uses the ACLs in the LDAP db to do control access.

cm@cm:~$ ldapadd -f 2nd.ldif -h localhost
SASL/DIGEST-MD5 authentication started
Please enter your password:
SASL username: cm
SASL SSF: 128
SASL installing layers
adding new entry "ou=People, dc=cm,dc=net"
ldap_add: Insufficient access (50)
additional info: no write access to parent


Note that you must specify the username (unless you are that user)


direct mapping vs search-based mapping

direct mapping avoids having to do an LDAP lookup. It uses the 'authz-regexp' or 'sasl-regexp' to re-write requests sent to it ('authz-regexp' and 'sasl-regexp' are the same AFAIK. authz is short for authorization, probably to distinguish it from authentication. I could not find any reference anywhere to 'sasl-regexp' using an apropos. But I was able to use both in the slapd.conf interchangeably. And the debug log shows sasl-regexp as 'slap_authz_regexp'

LDAP SASL library calls from the debug log:

May 13 22:53:09 cm slapd[24029]: do_sasl_bind: dn () mech DIGEST-MD5
May 13 22:53:09 cm slapd[24029]: SASL [conn=8] Debug: DIGEST-MD5 server step 2
May 13 22:53:09 cm slapd[24029]: slap_sasl_getdn: u:id converted to uid=admin,cn=cm.net,cn=DIGEST-MD5,cn=auth

search-mapping binds to the LDAP server and looks up the user via LDAP calls. It is more 'expensive', as you have to search the DIT

Linux-HA and heartbeat

http://www.linux-ha.org/

The goal of the Linux HA project is to provide 'a high availability (clustering) solution for Linux which promotes reliability, availability...'

The Linux-HA project's main software product is Heartbeat, which is a cluster management program for high availability clustering. Some of the features of heartbeat include:

- no fixed number of nodes
- resource monitoring
- fencing

Linux-HA state that their software (Heartbeat) is well integrated with separate projects like LVS and DRBD

Heartbeatv2 can do resource monitoring of Lustre for example, by using /etc/init.d/lustre with the 'status' argument (I think it looks in /proc or /sys), see https://mail.clusterfs.com/pipermail/lustre-discuss/2005-September/000870.html

keepalived

keepalived has been primarily developed as a means to provide high availability for LVS. It does this via health checks of protocols and services, and informs the kernel in case of failure (or if it becomes available again), and it uses a version of the VRRP to handle failover for the LVS director. (VRRP is an open standard based on Cisco's HSRP)

-It runs as a user-space program

-It checks multiple layers of the TCP stack (i.e., based on the OSI 7 layer model). It does checks at the IP level, tcp level, transport dialogue layer (5), and application layer.


"Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5/7. This framework gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the linux kernel via a setsockopt call to remove this server entry from the LVS topology. In addition keepalived implements an independent VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover."

LVS

LVS is described on the home page of the project as a 'load balancer'. The aim is to provide scalability for services. The main product of the LVS project is IPVS, which provides Layer-4 (transport layer) load balancing inside the kernel.

From the home page:

"IPVS implements transport-layer load balancing (layer-4 switching) inside the Linux kernel"

They also have under development Layer-7 (application level) switching inside the kernel, using KTCPVS


To provide redundancy, their are several possibilities listed on their home page.

- pirhana
- ultramonkey
- keepalived
- heartbeat + mon + coda
- heartbeat + ldirectord

I have set up a cluster using the last option. The 2nd is what nr uses.

lustre failover experimentation

'lconf --failover' didn't seem to work, but 'lconf --cleanup --force --service=mds /root/config.xml' did. It removed all the modules. Once I was satisfied that mds-1 was not using the device, I started the failover device, mds-2, just by running 'lconf --node mds-2 /root/config.xml'.

On the client that was mounting the resource:

LustreError: 19533:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout (sen t at 1177507980, 0s ago) req@f7d55600 x8852471556/t0 o400->mds_UUID@mds-1_UUID:1 2 lens 64/64 ref 1 fl Rpc:N/0/0 rc 0/0
LustreError: 19533:0:(client.c:940:ptlrpc_expire_one_request()) Skipped 29 previ ous similar messages
Lustre: 12:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked LNET upcall /usr/lib /lustre/lnet_upcall ROUTER_NOTIFY,192.168.241.229@tcp,down,1177507956
LustreError: MDC_mds-1_mds_MNT_client-f70a0400: Connection to service mds via ni d 192.168.241.229@tcp was lost; in progress operations using this service will w ait for recovery to complete.
Lustre: Changing connection for MDC_mds-1_mds_MNT_client-f70a0400 to mds-2_UUID/192.168.241.227@tcp

The share was fine from the client after the switchover.

There were no messages on the OSSes. But I did this:

oss-4:/home/cmcleay/lustre-1.4.10# lctl ping mds-2
12345-0@lo
12345-192.168.241.227@tcp
oss-4:/home/cmcleay/lustre-1.4.10# lctl ping mds-1
failed to ping 192.168.241.229@tcp: Input/output error



I used the init scripts after I was satisfied with this - all worked well.
Doing an 'ls' on the share hung, but it came back after a little while after starting the failover mds-1

You need to have the config file in the right place for the init scripts to work properly

There were some messages on the oss:
Apr 25 23:40:48 oss-4 kernel: Lustre: 6:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,192.168.241.229@tcp,down,1176793786
Apr 25 23:44:11 oss-4 kernel: Lustre: 17433:0:(filter.c:3236:filter_set_info_async()) ost-beta: received MDS connection from 192.168.241.229@tcp


Then it all stopped working :(

lustre errors with a new config

The MDS writes a log file on the MDS device (if you mount a MDS volume, you can see it)

I tried re-writing it, but I decided to reformat it instead (I'd also tried a reboot, but that didn't help either)

MDSDEV: mds mds_UUID /dev/sdb1 ldiskfs no
! /usr/sbin/lctl (22): error: setup: Invalid argument


mds-1:~# lconf -v --node mds-1 /root/config.xml
configuring for host: ['mds-1']
Checking XML modification time
+ debugfs -c -R 'stat /LOGS' /dev/sdb1 2>&1 | grep mtime
xmtime 1177503678 > kmtime 1176793916
Error: MDS startup logs are older than config /root/config.xml. Please run --write_conf on stopped MDS to update. Use '--old_conf' to start anyways.
mds-1:~# lconf -v --node mds-1 --write-conf /root/config.xml
configuring for host: ['mds-1']
Service: network NET_mds-1_lnet NET_mds-1_lnet_UUID
loading module: libcfs srcdir None devdir libcfs
+ /sbin/modprobe libcfs
loading module: lnet srcdir None devdir lnet
+ /sbin/modprobe lnet
+ /sbin/modprobe lnet
loading module: ksocklnd srcdir None devdir klnds/socklnd
+ /sbin/modprobe ksocklnd
Service: ldlm ldlm ldlm_UUID
loading module: lvfs srcdir None devdir lvfs
+ /sbin/modprobe lvfs
loading module: obdclass srcdir None devdir obdclass
+ /sbin/modprobe obdclass
loading module: ptlrpc srcdir None devdir ptlrpc
+ /sbin/modprobe ptlrpc
Service: mdsdev MDD_mds_mds-1 MDD_mds_mds-1_UUID
original inode_size 0
stripe_count 1 inode_size 512
loading module: lquota srcdir None devdir quota
+ /sbin/modprobe lquota
loading module: mdc srcdir None devdir mdc
+ /sbin/modprobe mdc
loading module: osc srcdir None devdir osc
+ /sbin/modprobe osc
loading module: lov srcdir None devdir lov
+ /sbin/modprobe lov
loading module: mds srcdir None devdir mds
+ /sbin/modprobe mds
loading module: ldiskfs srcdir None devdir ldiskfs
+ /sbin/modprobe ldiskfs
loading module: fsfilt_ldiskfs srcdir None devdir lvfs
+ /sbin/modprobe fsfilt_ldiskfs
Service: mdsdev MDD_mds_mds-1 MDD_mds_mds-1_UUID
original inode_size 0
stripe_count 1 inode_size 512
MDSDEV: mds mds_UUID /dev/sdb1 ldiskfs no
+ /usr/sbin/lctl
attach mds mds mds_UUID
quit
+ /usr/sbin/lctl
cfg_device mds
setup /dev/sdb1 ldiskfs
quit
+ /usr/sbin/lctl
ignore_errors
cfg_device $mds
cleanup
detach
quit
! /usr/sbin/lctl (22): error: setup: Invalid argument

gotcha with /dev and udev when copying system

Usually I use

cd /
tar -clf - .|(cd /mnt;tar -xpf -)

to make a copy of a system. However, with newer systems running udev, this will cause problems as it does not copy /dev, which gets put on its own partition. So you will not be able to boot into a system unless you copy the /dev entries (it has a basic skeleton including vital files such as /dev/console and /dev/sda etc, stuff not needed for most boot environments will be created by udev dynamically)

Failover for Lustre nodes

Lustre does not provide the tool set for the system-level components necessary for a complete failover solution (node failure detection, power control, and so on), as this functionality has been available for some time from third party tools. CFS does provide the necessary scripts to interact with these packages, and exposes health information for system monitoring. The recommended choice is the Heartbeat package from linux-ha.org. Lustre will work with any HA software that supports resource (I/O) fencing. The Heartbeat software is responsible for detecting failure of the primary server node and controlling the failover.

linuxthreads being used instead of NPTL (native posix threading library)

Built a Debian machine which had an unusual problem whereby it was using linuxthreads instead of NPTL (native posix threading library) threads. Thus, when threaded services like nscd and java started, they would show multiple processes rather than a single (threaded) process. The /lib/tls directory was present, and all the right packages. But as a getconf showed, it was using linuxthreads:

# getconf GNU_LIBPTHREAD_VERSION
linuxthreads-0.10

ldconfig -v showed the /lib/tls libraries, it just wasn't using them

The clue was in an strace when starting nscd:

access("/etc/ld.so.nohwcap", F_OK) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f00000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=74936, ...}) = 0
mmap2(NULL, 74936, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7eed000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\345"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=263040, ...}) = 0
mmap2(NULL, 264196, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7eac000
mmap2(0xb7ee4000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x38) = 0xb7ee4000
mmap2(0xb7eec000, 2052, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7eec000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9592, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7eab000
mmap2(NULL, 12404, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7ea7000
mmap2(0xb7ea9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7ea9000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libc.so.6", O_RDONLY) = 3


On a machine using nptl:


execve("/etc/init.d/nscd", ["/etc/init.d/nscd", "start"], [/* 16 vars */]) = 0
uname({sys="Linux", node="ws-6", ...}) = 0
brk(0) = 0x80e6000
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=19421, ...}) = 0
old_mmap(NULL, 19421, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\342"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=252592, ...}) = 0
old_mmap(NULL, 257868, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4001d000
old_mmap(0x40053000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x35000) = 0x40053000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\32"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9872, ...}) = 0
old_mmap(NULL, 8632, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4005c000
old_mmap(0x4005e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x4005e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY) = 3


The presence of a file 'ld.so.nohwcap' was the problem. According to the man page of ld.so (on an Etch machine): "When this file is present the dynamic linker will load the non-optimized version of a library, even if the CPU supports the optimized version"

The reason for the presence of this file was because glibc was downgraded to an earlier version, which causes Debian to put this file in /etc. glibc was downgraded because this machine was mistakenly installed with 'etch' instead of 'sarge' and it was deemed easier at the time to downgrade via apt/aptitude etc, rather than do a fresh install. Perhaps a mistake :)

load balancing with f5s

I was wondering how f5s configured as active-active do load balancing. That is, for the nodes or real servers that they load balance, how do they a) accept packets from a source, without configuring two separate default routes on the router sending the packets, and have a different IP on each f5 (having static default routes would be problematic if one of the f5s failed, so it would have to be able to detect this), and b) how do they accept traffic from the servers they are load balance equally, as the servers only have one default route.

I poked around the web and found that the Ultramonkey project, using Saru, can do this, among others. I think the way it might work is to have a common MAC address shared between two f5s, so that the packet goes to both f5s, and then use some mechanism whereby the f5s compare the packets to make sure that they are the same, and one of them forwards the packet, and one of them drops it. In a way, this scenario is not really load balancing, as both machines still receive the packet and process it to some degree. It would only really be worthwhile if very little processing was done on each packet to decide which machine was to forward it. Otherwise, you get no real gain from having two machines active. Perhaps this could be a hash lookup, which, if the hashes match, then some very simple algorithm could then be used to decide which unit will forward and which unit will drop the packet. Maybe a lot more of the CPU and system resources would be dedicated to NATing the packet and applying various other rules to it, so this scheme would work. Anyway, I don't really know if that is how it does it. I'll have to research a bit more.

fixing broken/missing /var/lib/dpkg/available file

/var/lib/dpkg/available went missing on a machine, breaking commands like 'dpkg -l':

ws-1:/usr/local/bin# dpkg -l
dpkg-query: failed to open package info file `/var/lib/dpkg/available' for reading: No such file or directory

'dselect update' (from man dpkg) fixed this (may also need to run apt-get update prior)

ws-1:/usr/local/bin# dselect update
Hit http://192.168.241.146 stable/main Packages
Hit http://192.168.241.146 stable/main Release
Hit http://192.168.241.146 stable/contrib Packages
Hit http://192.168.241.146 stable/contrib Release
Hit http://192.168.241.146 stable/non-free Packages
Hit http://192.168.241.146 stable/non-free Release
Hit http://192.168.241.146 stable/updates/main Packages
Hit http://192.168.241.146 stable/updates/main Release
Hit http://192.168.241.146 stable/updates/contrib Packages
Hit http://192.168.241.146 stable/updates/contrib Release
Hit http://192.168.241.146 sarge-backports/main Packages
Hit http://192.168.241.146 sarge-backports/main Release
Hit http://192.168.241.146 sarge-backports/contrib Packages
Hit http://192.168.241.146 sarge-backports/contrib Release
Hit http://192.168.241.146 sarge-backports/non-free Packages
Hit http://192.168.241.146 sarge-backports/non-free Release
Reading Package Lists... Done
Merging Available information
Replacing available packages info, using /var/cache/apt/available.
Information about 17330 package(s) was updated.

This also get information about packages that were not retrieved through apt-get, i.e., locally made packages

spamassassin

spamassassin and OCR plugin

I wanted to install something to weed out spam that had gifs containing spam content. Normally, these
can't be parsed by standard methods. AC pointed me in the direction of an OCR plugin, which can scan
an image and recognise certain patterns. At first, I found OCR, which was based on some perl stuff as
well as gocr. I had some problems integrating it into the existing spamassassin setup. It just was not
running the scans. I just didn't know where to put the 'loadplugin' statements and perl modules.
I found running spamassassin in debug mode is the best way to find out exactly where it looks and what tests
it does. By this point I'd found that there was a newer plugin which claimed improvements on OCR

FuzzyOCR (wiki at: http://fuzzyocr.own-hero.net/wiki/Installation-3.x)

spamassassin --debug FuzzyOcr < ./gif_spam > /dev/null

So I ran it it in debug mode and it prints all the paths it searches, plugins it loads, and tests it does.

It turned out that it was using /var/lib/spamassassin, not the standard /usr/share/spamassassin (which also existed)

The wiki comes with good instructions and suggested to put it in /etc/spamassassin, where spamd just picked it up (after a reload). I had to set the db directory to something writeable by the nobody user though. Running it in debug (as above) reported all the issues it had.




http://spamassassin.apache.org/tests_3_1_x.html


AREA TESTED LOCALE DESCRIPTION OF TEST TEST NAME DEFAULT SCORES
(local, net, with bayes, with bayes+net) MORE INFO
(additional wiki docs)
body Generic Test for Unsolicited Bulk Email GTUBE 1000.000 Wiki
body Incorporates a tracking ID number TRACKER_ID 2.000 1.295 2.292 1.032 Wiki
body Weird repeated double-quotation marks WEIRD_QUOTING 1.120 1.200 1.295 1.341 Wiki

exim - removing frozen messages

Found this as a one-liner to remove frozen messages on an exim server
Haven't tried it - might need to replace 'exim' with 'exim4' though

mailq | awk '/frozen/ { print $3 }' | xargs exim -Mrm

Fixing timestamps on mail server

Messages that passed through one of our mailservers were stamped with UTC rather than EST, which caused some customers to complain. Looking at the hardware clock, it showed UTC instead of EST. I wasn't able to set it via the hwclock options. The problem was that it was missing a symlink to /etc/localtime:

Should have looked like:

# ls -l /etc/localtime
lrwxrwxrwx 1 root root 48 Mar 31 11:19 /etc/localtime -> /usr/share/zoneinfo/Australia/Sydney

mkinitrd, depmod

Note that mkinitrd does not work on Debian post 2.6.12 kernels. It has been replaced by other packages, e.g., initramfs-tools (which I have installed), or Yaird, or linux-initramfs-tool.



Problems trying to use a 3ware 9550-sx 4LP on a box running Lustre

The default 2.6.8-3 debian kernel image works fine, but could not get the SUSE 2.6.5 Lustre kernel to work. Tried using the default image, and then generating an initrd. Also tried using the kernel source tree of that version, with drivers compiled in, or as modules. No luck. It was not able to mount root fs. This was on a Sarge machine. Below is the process I used to mkinitrd for a Debian 'Sarge' box, using a lustre-patched redhat EL kernel (2.6.9).

First, I tried making initrd using default /etc/mkinitrd/mkinitrd.conf values. It was failing silently. Solution: change the following:

# Command to generate the initrd image.
MKIMAGE='mkcramfs %s %s > /dev/null'

I copied mkext2fs (obtained from another machine) into /usr/local/sbin and put the following line at the suggestion of a colleague:

# Command to generate the initrd image.
MKIMAGE='/usr/local/sbin/mkext2fs %s %s > /dev/null'

The command to generate the initrd.img was:

mkinitrd -d /etc/mkinitrd -o /boot/initrd.img- /lib/modules/

But this failed because it could not find a modules.dep file, which was not generated from the conversion of the .rpm kernel image I had downloaded from clusterfs.com (I used alien -c to generate the .deb). So I ran a 'depmod -a ', where 'kernel_ver' was equivalent to a uname -r of the kernel that I wanted to run. This generated my modules.dep

Tried again. This failed because it reported that /tmp was running out of space. The /tmp filesystem had 288GB free, so it wasn't actually the /tmp filesystem, but it must have made a loopback filesystem of a certain small size on /tmp. The solution to this was to change:

# What modules to install.
MODULES=most

to:

# What modules to install.
MODULES=dep

in the /etc/mkinitrd/mkinitrd.conf file, and list the essential modules in /etc/mkinitrd/modules

e.g.,

scsi_mod
libata
ata_piix
3w_9xxx
ext3

These must be modules otherwise it will complain that it can't find them (of course). If they are compiled into the kernel, or not at all, then it will complain. It may still boot ok if they are in the kernel. Even if it complains, it will probably still build the image.

One question on my mind is whether the order that they are put in /etc/mkinitrd/modules matters. I suspect it doesn't, which might be why having a 'modules.dep' file is important. Some modules need to be loaded before others. I think dpkg kernels run pre-inst scripts (e.g., /var/lib/dpkg/info/linux-image-2.6.16-2-686-smp.preinst. Maybe after they install the modules they run a mkinitrd against the module tree. Need to spend a bit more time mucking around to find out.

One thing I did find out was that copying driver source from a 2.6.17 kernel source tree to a 2.6.5 source tree and then trying to compile does not always work, though it can. I was doing this to get a later version of the driver into a lustre kernel source tree, which was a fairly old one (2.6.5). It has worked in some cases though.



debian:/etc/mkinitrd# cd
debian:~# mkinitrd -d /etc/mkinitrd -o /boot/initrd.img-2.6.5lustre.1.4.7 /lib/modules/2.6.5lustre.1.4.7
/usr/sbin/mkinitrd: add_modules_dep_2_5: modprobe failed
FATAL: Module libata not found.
FATAL: Module ata_piix not found.
WARNING: This failure MAY indicate that your kernel will not boot!
but it can also be triggered by needed modules being compiled into
the kernel.


initial ramdisk creation in Debian stock kernels (from /var/lib/dpkg/info/linux-image-2.6.16-2-686-smp.postinst):


my @ramdisklist;
@ramdisklist = find_inird_tool($hostversion, $version, $ramdisk) if $ramdisk;
die "Failed to find suitable ramdisk generation tool for kernel version \n" .
"$version on running kernel $hostversion in $ramdisk\n"
if $#ramdisklist < 0;
my $success = 0;
for $ramdisk_cmd (@ramdisklist) {
print STDERR "Using $ramdisk_cmd to build the ramdisk.\n";
print STDERR "Other valid candidates: @ramdisklist\n" if $#ramdisklist > 0;

my $initrd_path = $realimageloc . "initrd.img-$version";
my $ret = system("$ramdisk_cmd " .
($mkimage ? "-m '$mkimage' " : "") .
"-o $initrd_path.new $modules_base/$version");
if ($ret) {
warn "$ramdisk_cmd failed to create initrd image.\n";
}
else {
rename("$initrd_path.new", "$initrd_path")
or die("Failed to rename initrd ($initrd_path)\n");
$success = 1;
last;
}
}

shutdown stuff

Running the following sequence of commands is useful if you have a machine that tries to stop a lot of processes that take a long time to stop or might hang the shutdown process (e.g., unmounting NFS shares when there are NFS problems). Issuing this command assumes that it is ok to immediately terminate processes. You run the sync first to ensure that the local data is written to disk first (doesn't try syncing NFS data)

sync
reboot -fn

'shutdown -c' will kill a shutdown command that has been issued

slay and kill -9 -1

slay is like kill -9 -1
It will kill all processes belonging to a user

AC used this to kill some hung 'umount -f '

Very handy

dns statistics with dnstop

dnstop -t -s eth0

Toggle with 'c' - this appears to be undocumented. Below is some flags (from the man page):


s display the source address table

d display the destination address table

t display the breakdown of query types seen

o display the breakdown of opcodes seen

1 show the TLD table

2 show the SLD table

3 show the 3LD table

c show the SLD+source table

# show the 3LD+source table

^R reset the counters

^X exit the program

? help

Apache server used as a spam proxy via php bug

Whilst working at the datacentre on day, I got a call from the office
to say that the load on a few of our cPanel servers was very high. After
a bit of looking around, I noticed that there were lot of log entries
(from various IPs in Taiwan to port 80, which requested connections to port 25
of another IP:

201.63.4.219 - - [16/Jan/2005:14:03:47 +1100] "CONNECT 215.66.11.47:25 HTTP/1.0" 200 1243

What was happening was that Apache was proxying connections to port 80
to mail servers to send spam (presumably so it didn't look like the messages
came from their IP). I know that Apache can be used as a proxy (using mod_proxy), but
this was not enabled in the httpd.conf. After some quick checking via google, the
problem turned out to be a bug in php.


##### php (apparently) has a
##### vulnerability which allows Apache to be used as a
##### proxy without the mod_proxy or mod_proxy_connect
##### modules. To block this, we block 'CONNECT'



Order deny,allow
Deny from all

serveraid 8i aacraid x86_64

Recently built a new database server with a serveraid 8i raid controller, which is based on an adaptec chipset. Under linux, it uses the aacraid driver. AFAIK, it is not supported by any current Debian Etch AMD-64 boot CDs. A quick google turned up a site which provides tools to do a net boot using a very minimal CD image (http://kmuto.jp/b.cgi/debian/d-i-2615-amd64.htm) which saved me a lot of time (note: must use 'modprobe aacraid' to get it to see the controller). I installed the standard SMP Debian 2.6.17 AMD 64 kernel, which allows it to boot fine. However, this kernel was not suitable for our purposes for a number of reasons, so I wanted to roll my own. But that is proving extremely difficult. When compiling the driver into the kernel, it sees the controller just fine, but does not pick up the disks. Same when using an initrd. I have tried copying the kernel config from the debian kernel and deleting stuff that wasn't needed, like pcmcia, acpi, video for linux, sound card support (this is a server, after all!), but to no avail. I tried patching the source tree with the latest driver source (actually, a rather dodgy patch of just replacing the original .c and .h files with the newer one; it still compiles ok), and updating the firmware. No luck so far. I am thinking that I might try comparing the kernel messages on boot between the working debian kernel and my failed attempts. So far nothing is standing out.

sendfile() issue on lustre 1.4.x

The messages below relate to an issue with Lustre and sendfile() syscall. Lustre 1.4.x does not support it. The sendfile() syscall copies directly from disk to the network rather than doing multiple copies to memory or whatever. The messages are caused by proftp running on this server. You can disable sendfile() by recompiling proftp with it turned off

Oct 12 12:46:47 cthulhu kernel: Lustre: 1483:0:(rw.c:1380:ll_readpage()) ino 20592684 page 175 (716800) not covered by a lock (mmap?). check debug logs.
Oct 12 12:46:47 cthulhu kernel: Lustre: 1483:0:(rw.c:1380:ll_readpage()) previously skipped 286 similar messages
Oct 12 12:46:53 cthulhu kernel: Lustre: 1483:0:(rw.c:1380:ll_readpage()) ino 20592684 page 245 (1003520) not covered by a lock (mmap?). check debug logs.
Oct 12 12:46:53 cthulhu kernel: Lustre: 1483:0:(rw.c:1380:ll_readpage()) previously skipped 69 similar messages
Oct 12 13:56:26 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) ino 16526558 page 0 (0) not covered by a lock (mmap?). check debug logs.
Oct 12 13:56:26 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) previously skipped 15 similar messages
Oct 12 13:56:47 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) ino 16526526 page 0 (0) not covered by a lock (mmap?). check debug logs.
Oct 12 13:58:59 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) ino 27364595 page 0 (0) not covered by a lock (mmap?). check debug logs.
Oct 12 14:01:24 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) ino 13216322 page 0 (0) not covered by a lock (mmap?). check debug logs.
Oct 12 14:07:43 cthulhu kernel: Lustre: 3216:0:(rw.c:1380:ll_readpage()) ino 20459809 page 68 (278528) not covered by a lock (mmap?). check debug logs.
Oct 12 14:08:04 cthulhu kernel: Lustre: 3216:0:(rw.c:1380:ll_readpage()) ino 20459809 page 209 (856064) not covered by a lock (mmap?). check debug logs.
Oct 12 14:08:04 cthulhu kernel: Lustre: 3216:0:(rw.c:1380:ll_readpage()) previously skipped 140 similar messages
Oct 12 14:11:48 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) ino 19054807 page 42 (172032) not covered by a lock (mmap?). check debug logs.
Oct 12 14:11:48 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) previously skipped 37 similar messages
Oct 12 14:15:08 cthulhu kernel: Lustre: 3154:0:(rw.c:1380:ll_readpage()) ino 19054900 page 30 (122880) not covered by a lock (mmap?). check debug logs

more sed stuff

from http://www.oracle.com/technology/pub/articles/dulaney_sed.html

$ cat sample_one
one 1
two 1
three 1
one 1
two 1
two 1
three 1
$

Suppose that it would be desirable for "1" to be substituted with "2," but only after the word "two" and not throughout every line. This can be accomplished by specifying that a match is to be found before giving the substitute command:

$ sed '/two/ s/1/2/' sample_one
one 1
two 2
three 1
one 1
two 2
two 2
three 1
$

handy way in shell of moving files with spaces in them

Say you have a bunch of files with spaces in the names, e.g.,:

02 Workinonit.m4a

And you want to move them to names without spaces (e.g., convert them to underscores). Luckily, there's an easy way: use 'read'

Let's say we want to copy them to filenames with an underscore replacing the space. To check that we have it right, the following command will echo what the result would be

find . -type f | while read f ; do echo cp "$f" "`echo "$f" | sed 's/ */_/g'`";done

cp ./18 Don't Cry.m4a ./18_Don't_Cry.m4a

Then just remove the first 'echo':

find . -type f | while read f ; do cp "$f" "`echo "$f" | sed 's/ /_/g'`";done

and you are done :) Interestingly, the site I found this on used sed 's/ */_g' (i.e., two spaces followed by an asterisk. This works, but I don't know why. I also found that you can use alternation to pick up other characters, like !,(). However, it seems sed has no way of dealing with apostrophes (according to all the man pages etc I consulted. Escaping it with a '\' will not work, nor will two backslashes. If you want to replace spaces, commas, exclamation marks and brackets in filenames, use:

sed -r 's/ |\(|\)|\,|\!/_/g'

I haven't yet worked out how to do all this in perl

useful debian tools

update-rc.d

e.g., update-rc.d aveserver start 50 2 3 4 5 . stop 30 0 1 6 .

modconf - allows you to add/remove modules from start up

update-rc.d

Debian way of managing sysV init scripts

You can create links or delete them, among other things, with this script. It is better to manage service startup with this rather than, e.g., deleting symlinks manually, as they may get recreated when the service is updated. Thus, if you want to disable a service, use update-rc.d to set this (see man page). There a variety of other handy scripts for managing and configuring services on Debian under the name of update-*

I/O scheduler for linux

The kernel I/O scheduler (the available schedulers are set at kernel compile config)
There are schedulers optimized for servers and desktops. 'noop' and 'deadline' are best for server environments, whereas 'anticipatory' and 'cfq' are for desktops


If set, it can be seen in /sys/block//queue/scheduler in later kernels (2.6.??)
Otherwise, it uses whatever default for the vendor kernel (e.g., SUSE enterprise use CFQ for their kernels by default)

You need to set it for each device, e.g., sda, sdb, hda etc

sysctl -w sys.block.sda.queue.scheduler=deadline

OR

echo "deadline" > /sys/block/sda/queue/scheduler

cat /sys/block/sda/queue/scheduler
noop anticipatory [deadline] cfq

The one in the square brackets is what is set

It can be set while running in later kernels too by changing that parameter

Otherwise, it can be set at boot time. Debian has a utility called 'sysfsutils' which allows you to set it in /etc/sysfs.conf

sysfsutils - sysfs query tool and boot-time setup

# cat /etc/sysfs.conf | grep cfq
block/sda/queue/scheduler = cfq
block/sdc/queue/scheduler = cfq

# /etc/init.d/sysfsutils restart

Or can be done via parameter at boot:

GRUB:
root=/dev/sda1 noapic elevator=deadline

LILO:
append="elevator=deadline"

From Lustre documentation:

deadline – This is the 'old' scheduler, which works well for servers.

anticipatory I/O scheduler (AS) – It is designed for 'batching' I/O requests. It does not work well for servers and high IO loads.

cfq – It adds multiple scheduling classes. cfq also does not work well for servers and high I/O loads.

noop – This is the 'old' elevator. It works well for servers.

This seems quite different to what RedHat recommend http://www.redhat.com/magazine/008jun05/features/schedulers/

backporting with debian

The quick and simple way:

1. Add the repository to you /etc/apt/sources.list, e.g., if you are running 'sarge' and want later packages:

deb http://www.backports.org/debian sarge-backports main contrib non-free

2. apt-get update

3. apt-get -t sarge-backports install

udpcast for machine replication

http://www.udpcast.linux.lu/

One way of replicating many machines at once, if they are identical hardware, including disk drives etc, is to use udpcast. The idea is that one machine is used as a template for the others. Each machine is booted off a special boot disk (or a system image served over the network) which has some special programs that allow machines to be replicated over the network. One machine acts as a sender and the others are receivers.

I had to build a special kernel as the disk controllers used were not yet in the kernel source tree. It is actually extremely easy to build a boot disk with the instructions given at http://www.udpcast.linux.lu/mkimagedoc.html. For a CD image, it just involves having the kernel image (can be placed anywhere) and the modules (in /lib/modules, if you are using a modular kernel). Then run makeImage -k -c

It will replicate the disks (maybe using dd?) over the network. In my case, I just used a crossover cable as I was just doing one machine (it is really designed to do many machines at once)

Of course, you may need to modify some things, like IP addresses, hostnames etc...

Passing parameters for modules and kernel drivers

As with modules, it is possible to pass parameters to drivers at boot time. There is documentation in the kernel source tree under Documentation/kernel-parameters.txt. You can find what parameters a module will accept, use 'modinfo -p ' (for a binary module). Or you can also look at the source code of the driver (if for instance, your driver is in the kernel).

For instance, if the driver says that you can pass it the option:

adp94xx=access_HostRAID:1

Then you can create a file in /etc/modprobe.d/ with:

options adp94xx adp94xx=access_HostRAID:1

If the driver is in the kernel, you alter it to:

adp94xx.adp94xx=access_HostRAID:1

Assuming you are using GRUB, edit the boot parameter and add the line above

adp94xx kernel driver for linux

Had to install Debian Sarge on a IBM xSeries 306m. This machine comes with a adaptec 9405 SAS controller, which is not yet in the kernel source tree. There are rpms for redhat and SUSE available though, as well as the driver source code, so you can compile it yourself. Steps to add it to the kernel source tree:

-Create a subdir in drivers/scsi called adp94xx
-copy the source files (.h, .c, and Makefile) provided by adaptec into this directory
-add this directory to the top level Makefile in the scsi directory, basing the format on the aic7xxx stuff
-add a line to Kconfig so that menuconfig would pick it up

One question: at what point do you use 'patch' to modify the kernel source tree as opposed to e.g., just sticking in a driver? Is a patch just a way of more conveniently and consistently adding the same thing (as well as distributing it)?

This enabled me to compile it into the kernel, which avoids the necessity of using an initrd to boot the system, since the rootfs will be on a disk attached to that controller. The other alternative is to compile it as a module (haven't yet tried this), or use the binary (rpm) modules supplied by Adaptec. As I was using Debian, I couldn't use the rpms directly, so I converted them into .deb files with Alien, and then extracted them using dpkg -x to get a .ko. One problem I have had with using the driver compiled into the kernel is not being able to get the driver to see the controller when it is configured in RAID mode. The driver defaults to non-RAID mode, which can be switched to RAID mode by sending it a parameter at boot. From reading Documentation/kernel-parameters.txt, and the documentation supplied with the driver, I figured this would be:

adp94xx.adp94xx=attach_HostRAID=1

so I added this at boot by editing the grub boot line. However, it failed to see the controller, reporting:

"Probing AIC-94xx Controller(s)...
AIC-94xx controller(s) attached = 0."


I have got it to work fine when using the driver as a module on a generic Debian 2.6.8 kernel:

modprobe adp94xx adp94xx=attach_HostRAID=1

scsi1 : Adaptec AIC-9405W SAS/SATA Host Adapter
Vendor: IBM-ESXS Model GNA073C33ESTT0Z N Rev: BHOD
Type: Direct-Access
etc
AIC-94xx controller(s) attached = 1.

(also need to modprobe sd-mod so that there is a scsi transport layer so you can access the disk.)

My next attempt will be to change the line in the source code that makes the default to be off:

From adp94xx_osm.c:

/* By default we do not attach to HostRAID enabled controllers.
* You can turn this on by passing
* adp94xx=attach_HostRAID:1
* to the driver (kernel command line, module parameter line).
*/
static int asd_attach_HostRAID = 0;

to

static int asd_attach_HostRAID = 1;

I'll try this when I have more time.

UPDATE: I have tried the above, using a debian 2.6.8 sarge kernel source tree, and compiled it in to the kernel and as a module. It fails to detect the card when it is configured as RAID in the BIOS. I have tried passing it the option on the command line to set this, despite the default supposedly being '1' (modprobe adp94xx adp94xx=attach_HostRAID=1) when used as a module. The only other option I can think of right now is to install the OS on a USB disk (they are 1RU servers with no space for further internal storage). I have tried putting a single disk in and marking this disk as 'simple storage' in the BIOS of the controller which allows data migration, but this still requires RAID as enabled, and it forgets this if you disable RAID

The following packages cannot be authenticated!

When trying to install packages on an 'Etch' system (upgraded from 'Sarge')

WARNING: The following packages cannot be authenticated!

tic tac toe

Install these packages without verification [y/N]?

To fix:

apt-get install debian-archive-keyring

apt-get update

parameter substitution in shell

To delete pattern from variable:

Assuming our the value of our variable is 'abracadabra'

my_variable=abracadabra

${variable%pattern} - deletes pattern AFTER variable. That is, if the pattern starts with pattern, it will delete it. If the pattern appears anywhere else than the end, the entire contents of the variable are printed.

echo ${my_variable%ra}

gives:

abracadab


${variable#pattern} - deletes pattern BEFORE variable

echo ${my_variable#abr}

gives:

acadabra

However, if it doesn't begin with 'abr', it will just print the variable (see above)

echo ${my_variable#ra}

gives:

abracadabra


But you can use regular expressions within the match.

Thus:

echo ${my_variable#*ra}

will work from the beginning of the value of the variable and delete everything up until the first occurence of 'ra'. A single '#' tells to substitute the smallest match, but if you use '##', it uses longest possible match (greedy match). Thus

echo ${my_variable##*ra}

will print nothing (or a newline), since the variable ends with 'ra'

Example usage:

To rename a list of files that end in ".bak" to a name without the ".bak":

for xx in `ls *.bak`;do mv $xx ${xx%.bak};done


In the Korn shell, you can also use '?' to match a single character, '[]' to match a set of characters enclosed in the parentheses, e.g., [a-s] will match any letters from a-s, and the negation would be [!a-s] to not match any characters from a-s. A '*' or a '+' will match one or more, e.g., ${my_variable%%+([a-r])} will print a blank. The '()' seem to be needed around the square brackets. I am not sure what the equivalent for this is in bash.

Taken from: UNIX Shell programming, by Kochan & Wood

cross-compiling between x86 versions on Debian

Say you had a nice, fast Debian 'Sarge' box and you wanted to use it to compile a kernel (or something else) for a Debian 'Woody' box - will this work? What about vice-versa? Could you compile a kernel for a Sarge box on a Woody machine? When cross-compiling for different architectures, there are special things that need to be done (look at these - what do they tell you about the things that go into compiling). There are pitfalls. For instance, if you wanted to compile a 2.4 kernel for your currently running Debian 'Woody' machine on a distro that uses a very recent version of gcc (=< 3.4), it will fail. This is because gcc 3.4 and above will not compile a 2.4 kernel. You need to use an earlier version of gcc. Presumably, vice versa, early versions of gcc would not be able to compile recent kernels either.

Another question on my mind is whether the tools used to create kernel images on later/earlier versions of an OS environment will cause problems for that kernel to be used on earlier/later OS environments. E.g., would gcc, make, bin86, bzip2, etc create a binary kernel image that just could not be used on an environment different to the machine the kernel was compiled on? For example, the version of bzip2 that compresses the kernel image on e.g., a Sarge machine, may not create it in a format that the version of bzip2 on the 'potato' machine you install it on can understand. What about creating symbol tables? And linkers?

What about using recent kernels on old machines? Problems here could be the kernel code, i.e., scsi.c, adaptec.c, ext2.h, etc may not work with user-space utilities on an old environment (e.g., filesystems such as ext2 compiled into a 2.6 kernel may not be compatible with a Debian 'potato' system, as the code for ext2 has changed since potato was created, and all of the tools it uses to access ext2 filesystems). Another example is 2.6 kernel for Woody systems: the module tools on woody cannot load the modules of a 2.6 system (the 2.6 kernel uses a different module loading system to that of Woody. Sarge has the necessary tools however). Of course, you could dispense with modules all together.

Think about what the kernel does. It manages resources, provides access to hardware (device drivers) and so on. A quick list:

-device drivers
-filesystems
-support for executable file formats (e.g., ELF, a.out)
-security (e.g., LIDS SELinux, GRSecurity)
-crypto
-Misc other stuff like networking, firewalls, sysctl etc

Thus, all the user-space stuff that talks to all of this has to be able to talk to the interfaces provided by the kernel.

What about things beside the kernel? e.g., if I compiled 'tar' on a woody system, could I use it on a sarge system? One question here is whether the compiler looks at your OS environment. Dynamic linking would be an issue possibly. It may assume a certain version of libc for instance. Then there is the sort of format the compiler produces.

2.6 kernel under Debian 'woody'

From: http://marc.herbert.free.fr/linux/linux2.6_for_woody.txt

The mechanism for dynamically loading kernel modules has been rewritten between 2.4 and 2.6. As a consequence, the former "modutils" tools (insmod, modprobe,...) are not compatible with 2.6 You need the
new debian package "module-init-tools" instead.

More info: http://marc.herbert.free.fr/linux/linux2.6_for_woody.txt

Of course, you could always use a monolithic kernel...

boot blocks, partition tables

I couldn't be bothered summarising this. Here is a good overview:

http://en.wikipedia.org/wiki/Mbr

Inodes

Much of this is drawn from the excellent Linux Tutorials page (http://www.linux-tutorial.info/modules.php?name=Tutorial&pageid=224), and a other bits from wikipedia

Inodes are used in most unix filesystems, although there are differences in implementation. Also, some filesystems (such as ReiserFS) do not use them. In BSD, inodes are called 'vnodes', the 'v' standing for their role as part of the filesystem abstraction layer (VFS). Whether inode or vnode, they all play the same sort of role. The term 'inode' dates from unix pioneer Dennis Ritchie, and he hazards a guess that the term may have once stood for 'index node'.

The number of inodes on a disk is set when the disk filesystem is created (e.g., with mke2fs), and cannot be adjusted (at least for ext2). The list of inodes is stored in a table called the inode table near the start of the disk, along with the superblock, which contains the number of free inodes, as well as the location of the inode table (I am not sure whether, like the superblock, there can be backup copies of the inode table placed at other locations on the disk. I would presume so). One of the purposes of inodes is to store information about files, which in POSIX implementations includes:

* The length of the file in bytes.
* Device ID (this identifies the device containing the file).
* The User ID of the file's owner.
* The Group ID of the file.
* An inode number that identifies the file within the filesystem.
* The file mode, which determines what users can read, write, and execute the file.
* Timestamps telling when the inode itself was last changed (ctime), the file content last modified (mtime), and last accessed (atime).
* A reference count telling how many hard links point to the inode.

The 'stat' system call retrieves a file's inode number and some of the information in the inode.
As users, we mostly deal with files via name. However, filenames just point to inodes, which are referred to by number. The inode corresponding to a file can be shown in a variety of ways, including the '-i' switch with 'ls', which shows the inode number. Inodes are not files themselves, but data structures (taking a guess here). They do take up blocks on disk however. Inodes act as pointers to the actual data blocks that files take up on the disk. They can do this either directly or indirectly. There are a total of 15 references to datablocks stored in each inode (presumably, the number is kept low to keep things efficient and stop the inode getting too large). The first 12 blocks of a file (under the ext2 implementation) refer directly to the addresses of the blocks on disk. Assuming a data block size of 4096 bytes, this would allow a file size of 12 x 4096 bytes = 48k. If all references were direct, the remaining three references would only allow a maximum file size of 60k, so ext2 uses a system of indirect referencing. For the next (13th) reference, the inode points to a data block which contains a 4 byte value (allowing 128 references). Subsequently, there are double indirect and triple indirect references. This allow files of up to 4TB in size, assuming a 4k block size. However, as the maximum for a 32-bit integer is 4GB, this is the limit on ext2 on 32-bit systems.

One inode per file. There can also be inodes without files.

BSD (vnodes):
* the permissions of the file
* the file link count
* old user and group ids
* the inode number
* the size of the file in bytes
* the last time the file was accessed (atime)
* the last time the file was modified (mtime)
* the last inode change time for the file (ctime)
* direct disk blocks
* indirect disk blocks
* status flags (chflags)
* blocks actually held
* file generation number
* the owner of the file
* the primary group of the owner of the file
Notice that the name of the file is not part of the inode's metadata. The filesystem doesn't care what the name of the file is; it only needs to know what inode number is associated with that file.