Fixing timestamps on mail server

Messages that passed through one of our mailservers were stamped with UTC rather than EST, which caused some customers to complain. Looking at the hardware clock, it showed UTC instead of EST. I wasn't able to set it via the hwclock options. The problem was that it was missing a symlink to /etc/localtime:

Should have looked like:

# ls -l /etc/localtime
lrwxrwxrwx 1 root root 48 Mar 31 11:19 /etc/localtime -> /usr/share/zoneinfo/Australia/Sydney

mkinitrd, depmod

Note that mkinitrd does not work on Debian post 2.6.12 kernels. It has been replaced by other packages, e.g., initramfs-tools (which I have installed), or Yaird, or linux-initramfs-tool.



Problems trying to use a 3ware 9550-sx 4LP on a box running Lustre

The default 2.6.8-3 debian kernel image works fine, but could not get the SUSE 2.6.5 Lustre kernel to work. Tried using the default image, and then generating an initrd. Also tried using the kernel source tree of that version, with drivers compiled in, or as modules. No luck. It was not able to mount root fs. This was on a Sarge machine. Below is the process I used to mkinitrd for a Debian 'Sarge' box, using a lustre-patched redhat EL kernel (2.6.9).

First, I tried making initrd using default /etc/mkinitrd/mkinitrd.conf values. It was failing silently. Solution: change the following:

# Command to generate the initrd image.
MKIMAGE='mkcramfs %s %s > /dev/null'

I copied mkext2fs (obtained from another machine) into /usr/local/sbin and put the following line at the suggestion of a colleague:

# Command to generate the initrd image.
MKIMAGE='/usr/local/sbin/mkext2fs %s %s > /dev/null'

The command to generate the initrd.img was:

mkinitrd -d /etc/mkinitrd -o /boot/initrd.img- /lib/modules/

But this failed because it could not find a modules.dep file, which was not generated from the conversion of the .rpm kernel image I had downloaded from clusterfs.com (I used alien -c to generate the .deb). So I ran a 'depmod -a ', where 'kernel_ver' was equivalent to a uname -r of the kernel that I wanted to run. This generated my modules.dep

Tried again. This failed because it reported that /tmp was running out of space. The /tmp filesystem had 288GB free, so it wasn't actually the /tmp filesystem, but it must have made a loopback filesystem of a certain small size on /tmp. The solution to this was to change:

# What modules to install.
MODULES=most

to:

# What modules to install.
MODULES=dep

in the /etc/mkinitrd/mkinitrd.conf file, and list the essential modules in /etc/mkinitrd/modules

e.g.,

scsi_mod
libata
ata_piix
3w_9xxx
ext3

These must be modules otherwise it will complain that it can't find them (of course). If they are compiled into the kernel, or not at all, then it will complain. It may still boot ok if they are in the kernel. Even if it complains, it will probably still build the image.

One question on my mind is whether the order that they are put in /etc/mkinitrd/modules matters. I suspect it doesn't, which might be why having a 'modules.dep' file is important. Some modules need to be loaded before others. I think dpkg kernels run pre-inst scripts (e.g., /var/lib/dpkg/info/linux-image-2.6.16-2-686-smp.preinst. Maybe after they install the modules they run a mkinitrd against the module tree. Need to spend a bit more time mucking around to find out.

One thing I did find out was that copying driver source from a 2.6.17 kernel source tree to a 2.6.5 source tree and then trying to compile does not always work, though it can. I was doing this to get a later version of the driver into a lustre kernel source tree, which was a fairly old one (2.6.5). It has worked in some cases though.



debian:/etc/mkinitrd# cd
debian:~# mkinitrd -d /etc/mkinitrd -o /boot/initrd.img-2.6.5lustre.1.4.7 /lib/modules/2.6.5lustre.1.4.7
/usr/sbin/mkinitrd: add_modules_dep_2_5: modprobe failed
FATAL: Module libata not found.
FATAL: Module ata_piix not found.
WARNING: This failure MAY indicate that your kernel will not boot!
but it can also be triggered by needed modules being compiled into
the kernel.


initial ramdisk creation in Debian stock kernels (from /var/lib/dpkg/info/linux-image-2.6.16-2-686-smp.postinst):


my @ramdisklist;
@ramdisklist = find_inird_tool($hostversion, $version, $ramdisk) if $ramdisk;
die "Failed to find suitable ramdisk generation tool for kernel version \n" .
"$version on running kernel $hostversion in $ramdisk\n"
if $#ramdisklist < 0;
my $success = 0;
for $ramdisk_cmd (@ramdisklist) {
print STDERR "Using $ramdisk_cmd to build the ramdisk.\n";
print STDERR "Other valid candidates: @ramdisklist\n" if $#ramdisklist > 0;

my $initrd_path = $realimageloc . "initrd.img-$version";
my $ret = system("$ramdisk_cmd " .
($mkimage ? "-m '$mkimage' " : "") .
"-o $initrd_path.new $modules_base/$version");
if ($ret) {
warn "$ramdisk_cmd failed to create initrd image.\n";
}
else {
rename("$initrd_path.new", "$initrd_path")
or die("Failed to rename initrd ($initrd_path)\n");
$success = 1;
last;
}
}

shutdown stuff

Running the following sequence of commands is useful if you have a machine that tries to stop a lot of processes that take a long time to stop or might hang the shutdown process (e.g., unmounting NFS shares when there are NFS problems). Issuing this command assumes that it is ok to immediately terminate processes. You run the sync first to ensure that the local data is written to disk first (doesn't try syncing NFS data)

sync
reboot -fn

'shutdown -c' will kill a shutdown command that has been issued

slay and kill -9 -1

slay is like kill -9 -1
It will kill all processes belonging to a user

AC used this to kill some hung 'umount -f '

Very handy

dns statistics with dnstop

dnstop -t -s eth0

Toggle with 'c' - this appears to be undocumented. Below is some flags (from the man page):


s display the source address table

d display the destination address table

t display the breakdown of query types seen

o display the breakdown of opcodes seen

1 show the TLD table

2 show the SLD table

3 show the 3LD table

c show the SLD+source table

# show the 3LD+source table

^R reset the counters

^X exit the program

? help

Apache server used as a spam proxy via php bug

Whilst working at the datacentre on day, I got a call from the office
to say that the load on a few of our cPanel servers was very high. After
a bit of looking around, I noticed that there were lot of log entries
(from various IPs in Taiwan to port 80, which requested connections to port 25
of another IP:

201.63.4.219 - - [16/Jan/2005:14:03:47 +1100] "CONNECT 215.66.11.47:25 HTTP/1.0" 200 1243

What was happening was that Apache was proxying connections to port 80
to mail servers to send spam (presumably so it didn't look like the messages
came from their IP). I know that Apache can be used as a proxy (using mod_proxy), but
this was not enabled in the httpd.conf. After some quick checking via google, the
problem turned out to be a bug in php.


##### php (apparently) has a
##### vulnerability which allows Apache to be used as a
##### proxy without the mod_proxy or mod_proxy_connect
##### modules. To block this, we block 'CONNECT'



Order deny,allow
Deny from all