backporting with debian

The quick and simple way:

1. Add the repository to you /etc/apt/sources.list, e.g., if you are running 'sarge' and want later packages:

deb http://www.backports.org/debian sarge-backports main contrib non-free

2. apt-get update

3. apt-get -t sarge-backports install

udpcast for machine replication

http://www.udpcast.linux.lu/

One way of replicating many machines at once, if they are identical hardware, including disk drives etc, is to use udpcast. The idea is that one machine is used as a template for the others. Each machine is booted off a special boot disk (or a system image served over the network) which has some special programs that allow machines to be replicated over the network. One machine acts as a sender and the others are receivers.

I had to build a special kernel as the disk controllers used were not yet in the kernel source tree. It is actually extremely easy to build a boot disk with the instructions given at http://www.udpcast.linux.lu/mkimagedoc.html. For a CD image, it just involves having the kernel image (can be placed anywhere) and the modules (in /lib/modules, if you are using a modular kernel). Then run makeImage -k -c

It will replicate the disks (maybe using dd?) over the network. In my case, I just used a crossover cable as I was just doing one machine (it is really designed to do many machines at once)

Of course, you may need to modify some things, like IP addresses, hostnames etc...

Passing parameters for modules and kernel drivers

As with modules, it is possible to pass parameters to drivers at boot time. There is documentation in the kernel source tree under Documentation/kernel-parameters.txt. You can find what parameters a module will accept, use 'modinfo -p ' (for a binary module). Or you can also look at the source code of the driver (if for instance, your driver is in the kernel).

For instance, if the driver says that you can pass it the option:

adp94xx=access_HostRAID:1

Then you can create a file in /etc/modprobe.d/ with:

options adp94xx adp94xx=access_HostRAID:1

If the driver is in the kernel, you alter it to:

adp94xx.adp94xx=access_HostRAID:1

Assuming you are using GRUB, edit the boot parameter and add the line above

adp94xx kernel driver for linux

Had to install Debian Sarge on a IBM xSeries 306m. This machine comes with a adaptec 9405 SAS controller, which is not yet in the kernel source tree. There are rpms for redhat and SUSE available though, as well as the driver source code, so you can compile it yourself. Steps to add it to the kernel source tree:

-Create a subdir in drivers/scsi called adp94xx
-copy the source files (.h, .c, and Makefile) provided by adaptec into this directory
-add this directory to the top level Makefile in the scsi directory, basing the format on the aic7xxx stuff
-add a line to Kconfig so that menuconfig would pick it up

One question: at what point do you use 'patch' to modify the kernel source tree as opposed to e.g., just sticking in a driver? Is a patch just a way of more conveniently and consistently adding the same thing (as well as distributing it)?

This enabled me to compile it into the kernel, which avoids the necessity of using an initrd to boot the system, since the rootfs will be on a disk attached to that controller. The other alternative is to compile it as a module (haven't yet tried this), or use the binary (rpm) modules supplied by Adaptec. As I was using Debian, I couldn't use the rpms directly, so I converted them into .deb files with Alien, and then extracted them using dpkg -x to get a .ko. One problem I have had with using the driver compiled into the kernel is not being able to get the driver to see the controller when it is configured in RAID mode. The driver defaults to non-RAID mode, which can be switched to RAID mode by sending it a parameter at boot. From reading Documentation/kernel-parameters.txt, and the documentation supplied with the driver, I figured this would be:

adp94xx.adp94xx=attach_HostRAID=1

so I added this at boot by editing the grub boot line. However, it failed to see the controller, reporting:

"Probing AIC-94xx Controller(s)...
AIC-94xx controller(s) attached = 0."


I have got it to work fine when using the driver as a module on a generic Debian 2.6.8 kernel:

modprobe adp94xx adp94xx=attach_HostRAID=1

scsi1 : Adaptec AIC-9405W SAS/SATA Host Adapter
Vendor: IBM-ESXS Model GNA073C33ESTT0Z N Rev: BHOD
Type: Direct-Access
etc
AIC-94xx controller(s) attached = 1.

(also need to modprobe sd-mod so that there is a scsi transport layer so you can access the disk.)

My next attempt will be to change the line in the source code that makes the default to be off:

From adp94xx_osm.c:

/* By default we do not attach to HostRAID enabled controllers.
* You can turn this on by passing
* adp94xx=attach_HostRAID:1
* to the driver (kernel command line, module parameter line).
*/
static int asd_attach_HostRAID = 0;

to

static int asd_attach_HostRAID = 1;

I'll try this when I have more time.

UPDATE: I have tried the above, using a debian 2.6.8 sarge kernel source tree, and compiled it in to the kernel and as a module. It fails to detect the card when it is configured as RAID in the BIOS. I have tried passing it the option on the command line to set this, despite the default supposedly being '1' (modprobe adp94xx adp94xx=attach_HostRAID=1) when used as a module. The only other option I can think of right now is to install the OS on a USB disk (they are 1RU servers with no space for further internal storage). I have tried putting a single disk in and marking this disk as 'simple storage' in the BIOS of the controller which allows data migration, but this still requires RAID as enabled, and it forgets this if you disable RAID

The following packages cannot be authenticated!

When trying to install packages on an 'Etch' system (upgraded from 'Sarge')

WARNING: The following packages cannot be authenticated!

tic tac toe

Install these packages without verification [y/N]?

To fix:

apt-get install debian-archive-keyring

apt-get update

parameter substitution in shell

To delete pattern from variable:

Assuming our the value of our variable is 'abracadabra'

my_variable=abracadabra

${variable%pattern} - deletes pattern AFTER variable. That is, if the pattern starts with pattern, it will delete it. If the pattern appears anywhere else than the end, the entire contents of the variable are printed.

echo ${my_variable%ra}

gives:

abracadab


${variable#pattern} - deletes pattern BEFORE variable

echo ${my_variable#abr}

gives:

acadabra

However, if it doesn't begin with 'abr', it will just print the variable (see above)

echo ${my_variable#ra}

gives:

abracadabra


But you can use regular expressions within the match.

Thus:

echo ${my_variable#*ra}

will work from the beginning of the value of the variable and delete everything up until the first occurence of 'ra'. A single '#' tells to substitute the smallest match, but if you use '##', it uses longest possible match (greedy match). Thus

echo ${my_variable##*ra}

will print nothing (or a newline), since the variable ends with 'ra'

Example usage:

To rename a list of files that end in ".bak" to a name without the ".bak":

for xx in `ls *.bak`;do mv $xx ${xx%.bak};done


In the Korn shell, you can also use '?' to match a single character, '[]' to match a set of characters enclosed in the parentheses, e.g., [a-s] will match any letters from a-s, and the negation would be [!a-s] to not match any characters from a-s. A '*' or a '+' will match one or more, e.g., ${my_variable%%+([a-r])} will print a blank. The '()' seem to be needed around the square brackets. I am not sure what the equivalent for this is in bash.

Taken from: UNIX Shell programming, by Kochan & Wood

cross-compiling between x86 versions on Debian

Say you had a nice, fast Debian 'Sarge' box and you wanted to use it to compile a kernel (or something else) for a Debian 'Woody' box - will this work? What about vice-versa? Could you compile a kernel for a Sarge box on a Woody machine? When cross-compiling for different architectures, there are special things that need to be done (look at these - what do they tell you about the things that go into compiling). There are pitfalls. For instance, if you wanted to compile a 2.4 kernel for your currently running Debian 'Woody' machine on a distro that uses a very recent version of gcc (=< 3.4), it will fail. This is because gcc 3.4 and above will not compile a 2.4 kernel. You need to use an earlier version of gcc. Presumably, vice versa, early versions of gcc would not be able to compile recent kernels either.

Another question on my mind is whether the tools used to create kernel images on later/earlier versions of an OS environment will cause problems for that kernel to be used on earlier/later OS environments. E.g., would gcc, make, bin86, bzip2, etc create a binary kernel image that just could not be used on an environment different to the machine the kernel was compiled on? For example, the version of bzip2 that compresses the kernel image on e.g., a Sarge machine, may not create it in a format that the version of bzip2 on the 'potato' machine you install it on can understand. What about creating symbol tables? And linkers?

What about using recent kernels on old machines? Problems here could be the kernel code, i.e., scsi.c, adaptec.c, ext2.h, etc may not work with user-space utilities on an old environment (e.g., filesystems such as ext2 compiled into a 2.6 kernel may not be compatible with a Debian 'potato' system, as the code for ext2 has changed since potato was created, and all of the tools it uses to access ext2 filesystems). Another example is 2.6 kernel for Woody systems: the module tools on woody cannot load the modules of a 2.6 system (the 2.6 kernel uses a different module loading system to that of Woody. Sarge has the necessary tools however). Of course, you could dispense with modules all together.

Think about what the kernel does. It manages resources, provides access to hardware (device drivers) and so on. A quick list:

-device drivers
-filesystems
-support for executable file formats (e.g., ELF, a.out)
-security (e.g., LIDS SELinux, GRSecurity)
-crypto
-Misc other stuff like networking, firewalls, sysctl etc

Thus, all the user-space stuff that talks to all of this has to be able to talk to the interfaces provided by the kernel.

What about things beside the kernel? e.g., if I compiled 'tar' on a woody system, could I use it on a sarge system? One question here is whether the compiler looks at your OS environment. Dynamic linking would be an issue possibly. It may assume a certain version of libc for instance. Then there is the sort of format the compiler produces.

2.6 kernel under Debian 'woody'

From: http://marc.herbert.free.fr/linux/linux2.6_for_woody.txt

The mechanism for dynamically loading kernel modules has been rewritten between 2.4 and 2.6. As a consequence, the former "modutils" tools (insmod, modprobe,...) are not compatible with 2.6 You need the
new debian package "module-init-tools" instead.

More info: http://marc.herbert.free.fr/linux/linux2.6_for_woody.txt

Of course, you could always use a monolithic kernel...

boot blocks, partition tables

I couldn't be bothered summarising this. Here is a good overview:

http://en.wikipedia.org/wiki/Mbr

Inodes

Much of this is drawn from the excellent Linux Tutorials page (http://www.linux-tutorial.info/modules.php?name=Tutorial&pageid=224), and a other bits from wikipedia

Inodes are used in most unix filesystems, although there are differences in implementation. Also, some filesystems (such as ReiserFS) do not use them. In BSD, inodes are called 'vnodes', the 'v' standing for their role as part of the filesystem abstraction layer (VFS). Whether inode or vnode, they all play the same sort of role. The term 'inode' dates from unix pioneer Dennis Ritchie, and he hazards a guess that the term may have once stood for 'index node'.

The number of inodes on a disk is set when the disk filesystem is created (e.g., with mke2fs), and cannot be adjusted (at least for ext2). The list of inodes is stored in a table called the inode table near the start of the disk, along with the superblock, which contains the number of free inodes, as well as the location of the inode table (I am not sure whether, like the superblock, there can be backup copies of the inode table placed at other locations on the disk. I would presume so). One of the purposes of inodes is to store information about files, which in POSIX implementations includes:

* The length of the file in bytes.
* Device ID (this identifies the device containing the file).
* The User ID of the file's owner.
* The Group ID of the file.
* An inode number that identifies the file within the filesystem.
* The file mode, which determines what users can read, write, and execute the file.
* Timestamps telling when the inode itself was last changed (ctime), the file content last modified (mtime), and last accessed (atime).
* A reference count telling how many hard links point to the inode.

The 'stat' system call retrieves a file's inode number and some of the information in the inode.
As users, we mostly deal with files via name. However, filenames just point to inodes, which are referred to by number. The inode corresponding to a file can be shown in a variety of ways, including the '-i' switch with 'ls', which shows the inode number. Inodes are not files themselves, but data structures (taking a guess here). They do take up blocks on disk however. Inodes act as pointers to the actual data blocks that files take up on the disk. They can do this either directly or indirectly. There are a total of 15 references to datablocks stored in each inode (presumably, the number is kept low to keep things efficient and stop the inode getting too large). The first 12 blocks of a file (under the ext2 implementation) refer directly to the addresses of the blocks on disk. Assuming a data block size of 4096 bytes, this would allow a file size of 12 x 4096 bytes = 48k. If all references were direct, the remaining three references would only allow a maximum file size of 60k, so ext2 uses a system of indirect referencing. For the next (13th) reference, the inode points to a data block which contains a 4 byte value (allowing 128 references). Subsequently, there are double indirect and triple indirect references. This allow files of up to 4TB in size, assuming a 4k block size. However, as the maximum for a 32-bit integer is 4GB, this is the limit on ext2 on 32-bit systems.

One inode per file. There can also be inodes without files.

BSD (vnodes):
* the permissions of the file
* the file link count
* old user and group ids
* the inode number
* the size of the file in bytes
* the last time the file was accessed (atime)
* the last time the file was modified (mtime)
* the last inode change time for the file (ctime)
* direct disk blocks
* indirect disk blocks
* status flags (chflags)
* blocks actually held
* file generation number
* the owner of the file
* the primary group of the owner of the file
Notice that the name of the file is not part of the inode's metadata. The filesystem doesn't care what the name of the file is; it only needs to know what inode number is associated with that file.

linux hardware reboot from software

#/bin/sh

#This script does the same thing as hitting the "reboot" button
#on a machine.

echo 1 > /proc/sys/kernel/sysrq # turns it on
echo s > /proc/sysrq-trigger # sync
echo u > /proc/sysre-trigger # umount ro
echo s > /proc/sysrq-trigger # sync
echo b > /proc/sysrq-trigger # immediate hardware reboot

the 'replace' command

Got a file that has xxx.xxx.xx.xxx and want to replace it with yyy.yyy.yy.yyy?


shell> replace xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy -- myfile.txt

The "--" tells the replace command where the string replacements end and the file name starts.

Or you can pipe into it like so:

shell> replace xxx.xxx.xxx.xxx yyy.yyy.yyy.yyy < myfile.txt

This will replace x with y in the myfile1.txt:

shell> replace x.x.x.x y.y.y.y y.y.y.y x.x.x.x -- myfile1.txt

This will actually swap the two in the two files:

shell> replace x y y x -- myfile1.txt myfile2.txt

Fun with 'find'

find . -printf 'Name: %f Owner: %u %s bytes\n'

From: http://marcnapoli.com.au/osi.php

find files younger than 24h

find . -type f -mtime -0

find dirs younger than 60 minutes

find . -type d -mmin -60

find all SUID and SGID

find / -path '/proc' -prune -o -path '/www/db/' -prune -o -type f -perm +6000 -fls /root/findallsetuidgid.txt

find / -xdev -type f -perm +u=s -o -perm +g=s -print  

find all world writables on a system

find / -noleaf -path '/proc' -prune \

  -o -path '/sys' -prune \

  -o -path '/dev' -prune \

  -o -perm -2 ! -type l  ! -type s \

  ! \( -type d -perm -1000 \) -fls /root/findworldwritables.txt

## Search / skipping a few dirs. noleaf assumes not all mounted
filesystems are unix fs. It will skip sockets, symlinks, and any
directory with the sticky bit set.



 find all files owned by no one in particular 

find / -path '/proc' -prune -o -nouser -o -nogroup -fls /root/findownedbynoone.txt

useful stuff about tar

tar will generally overwrite already existing files when untarring. There is the '-k' switch which tells it to keep old files. It will not erase files that are not in the tarball being extracted though. From the '--help':

-k, --keep-old-files don't replace existing files when extracting
--keep-newer-files don't replace existing files that are newer
than their archive copies
--overwrite overwrite existing files when extracting
--no-overwrite-dir preserve metadata of existing directories



When you extract a tarball as root, it will extract the file with the same UID as it was created with. I found this out in an amusing way when attempting to extract a tarball I'd downloaded, and it would fail with the message 'quota exceeded'. This was because the UID of the files in the tarball were the same as one of the users on the box. '--no-same-owner' will extract them as your current ID, which is the default for non-root.

Tar across the network, using ssh:

tar -czf - ~/mp3s | ssh 192.168.1.163 tar -xvzf -