Friday, May 19, 2017
Recent oVirt errors when doing a minor update on a 4.1 engine host
Had two problems after doing an engine update on 4.1.0.4-1.el7.centos:
First was that the engine service would not start due missing python-dateutil (no idea why this package wasn't included if it was needed in the update, and the engine service was working fine before the update). ABRT and journalctl provided useful info as to what needed fixing:
ovirt-engine-dwhd.service loaded activating auto-restart oVirt Engine Data Warehouse
* ovirt-engine.service loaded failed failed oVirt Engine
* ovirt-fence-kdump-listener.service loaded failed failed oVirt Engine fence_kdump listener
ovirt-imageio-proxy.service loaded active running oVirt ImageIO Proxy
ovirt-vmconsole-proxy-sshd.service loaded active running oVirt VM Console SSH server daemon
* ovirt-websocket-proxy.service loaded failed failed oVirt Engine websockets proxy
polkit.service loaded active running Authorization Manager
ABRT has detected 4 problem(s). For more info run: abrt-cli list --since 1495130066
[root@ovirt-engine ~]# abrt-cli list --since 1495130066
id 07e7e054795adf76b6260aed93a81420fe16e4cc
reason: service.py:35:<module>:ImportError: No module named dateutil
time: Thu May 18 18:57:04 2017
cmdline: /usr/bin/python /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.py --redirect-output --systemd=notify start
package: ovirt-engine-dwh-4.1.0-1.el7.centos
uid: 108 (ovirt)
count: 1
Directory: /var/spool/abrt/Python-2017-05-18-18:57:04-2765
id 107b6035ce460c7c4cabdbb8b9b2c5cee02f7b2c
reason: service.py:35:<module>:ImportError: No module named dateutil
time: Thu May 18 18:57:04 2017
cmdline: /usr/bin/python /usr/share/ovirt-engine/services/ovirt-fence-kdump-listener/ovirt-fence-kdump-listener.py --systemd=notify start
package: ovirt-engine-tools-4.1.0.4-1.el7.centos
uid: 108 (ovirt)
count: 1
Directory: /var/spool/abrt/Python-2017-05-18-18:57:04-2766
id 02916623c009e9277ec757c70ca1e1bc32020c62
reason: service.py:35:<module>:ImportError: No module named dateutil
time: Thu May 18 18:57:04 2017
cmdline: /usr/bin/python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start
package: ovirt-engine-backend-4.1.0.4-1.el7.centos
uid: 108 (ovirt)
count: 1
Directory: /var/spool/abrt/Python-2017-05-18-18:57:04-2767
id 588e5eb1889bfde67787877bc8d1156953eeae1e
reason: service.py:35:<module>:ImportError: No module named dateutil
time: Thu May 18 18:57:04 2017
cmdline: /usr/bin/python /usr/share/ovirt-engine/services/ovirt-websocket-proxy/ovirt-websocket-proxy.py --systemd=notify start
package: ovirt-engine-websocket-proxy-4.1.0.4-1.el7.centos
uid: 108 (ovirt)
count: 1
Directory: /var/spool/abrt/Python-2017-05-18-18:57:04-1391
The Autoreporting feature is disabled. Please consider enabling it by issuing
'abrt-auto-reporting enabled' as a user with root privileges
[root@ovirt-engine ~]#
May 18 18:57:04 ovirt-engine systemd[1]: Starting oVirt Engine...
May 18 18:57:04 ovirt-engine python[2767]: detected unhandled Python exception in '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py'
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: Traceback (most recent call last):
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: File "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py", line 32, in <module>
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: from ovirt_engine import service
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: File "/usr/lib/python2.7/site-packages/ovirt_engine/service.py", line 35, in <module>
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: from dateutil import tz
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: ImportError: No module named dateutil
May 18 18:57:06 ovirt-engine systemd[1]: ovirt-engine.service: main process exited, code=exited, status=1/FAILURE
May 18 18:57:06 ovirt-engine systemd[1]: Failed to start oVirt Engine.
May 18 18:57:06 ovirt-engine systemd[1]: Unit ovirt-engine.service entered failed state.
May 18 18:57:06 ovirt-engine systemd[1]: ovirt-engine.service failed.
Installed python-dateutil:
yum install python-dateutil
The second issue was vdsmd not being able to start. Once again, journalctl provided useful information on what to do (logging on vdsmd is very helpful):
* vdsmd.service loaded failed failed Virtual Desktop Server Manager
May 18 19:02:42 ovirt-engine systemd[1]: Starting Virtual Desktop Server Manager...
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running mkdirs
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running configure_coredump
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running configure_vdsm_logs
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running wait_for_network
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running run_init_hooks
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running upgraded_version_check
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running check_is_configured
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: Error:
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: One of the modules is not configured to work with VDSM.
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: To configure the module use the following:
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: 'vdsm-tool configure [--module module-name]'.
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: If all modules are not configured try to use:
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: 'vdsm-tool configure --force'
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: (The force flag will stop the module's service and start it
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: afterwards automatically to load the new configuration.)
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: multipath requires configuration
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: libvirt is not configured for vdsm yet
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: Modules certificates, sebool, multipath, passwd, sanlock, libvirt are not configured
May 18 19:02:43 ovirt-engine systemd[1]: vdsmd.service: control process exited, code=exited status=1
May 18 19:02:43 ovirt-engine systemd[1]: Failed to start Virtual Desktop Server Manager.
May 18 19:02:43 ovirt-engine systemd[1]: Unit vdsmd.service entered failed state.
May 18 19:02:43 ovirt-engine systemd[1]: vdsmd.service failed.
So I used vdsm-tool to configure the services it listed as needing configuring, e.g.:
[root@ovirt-engine ~]# vdsm-tool configure --module certificates
Checking configuration status...
Running configure...
Reconfiguration of certificates is done.
Done configuring modules to VDSM.
[root@ovirt-engine ~]# vdsm-tool configure --module sebool
Checking configuration status...
Running configure...
Reconfiguration of sebool is done.
Done configuring modules to VDSM.
[root@ovirt-engine ~]# vdsm-tool configure --module multipath
Checking configuration status...
multipath requires configuration
Running configure...
Reconfiguration of multipath is done.
Done configuring modules to VDSM.
[root@ovirt-engine ~]# vdsm-tool configure --module passwd
Checking configuration status...
Running configure...
Reconfiguration of passwd is done.
Done configuring modules to VDSM.
Some services needed to be stopped first before they could be configured:
[root@ovirt-engine ~]# vdsm-tool configure --module sanlock
Checking configuration status...
Error:
Cannot configure while service 'sanlock' is running.
Stop the service manually or use the --force flag.
[root@ovirt-engine ~]#
e.g., stop sanlock:
[root@ovirt-engine ~]# vdsm-tool service-stop sanlock
Wednesday, February 01, 2017
virt-p2v error
guestfsd: main_loop: proc 334 (fstrim) took 1.94 seconds
libguestfs: trace: v2v: fstrim = 0
libguestfs: trace: v2v: umount_all
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=157896k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=393292k,nr_inodes=98323,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/sda2 dir=/sysroot type=ext3 opts=rw,relatime,data=ordered freq=0 passno=0
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: umount /sysroot
libguestfs: trace: v2v: umount_all = 0
libguestfs: trace: v2v: mount "/dev/sda1" "/"
guestfsd: main_loop: proc 47 (umount_all) took 0.01 seconds
guestfsd: main_loop: new request, len 0x40
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: mount -o /dev/sda1 /sysroot/
[ 330.601856] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[ 330.605430] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
guestfsd: main_loop: proc 1 (mount) took 0.01 seconds
libguestfs: trace: v2v: mount = 0
libguestfs: trace: v2v: fstrim "/"
guestfsd: main_loop: new request, len 0x48
fsync /dev/sda
fsync /dev/sdb
commandrvf: stdout=y stderr=y flags=0x0
commandrvf: fstrim -v /sysroot/
/sysroot/: 90.5 MiB (94919680 bytes) trimmed
guestfsd: main_loop: proc 334 (fstrim) took 0.26 seconds
libguestfs: trace: v2v: fstrim = 0
[ 335.4] Closing the overlay
libguestfs: trace: v2v: umount_all
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=157896k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=393292k,nr_inodes=98323,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/sda1 dir=/sysroot type=ext3 opts=rw,relatime,data=ordered freq=0 passno=0
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: umount /sysroot
libguestfs: trace: v2v: umount_all = 0
libguestfs: trace: v2v: shutdown
libguestfs: trace: v2v: internal_autosync
guestfsd: main_loop: proc 47 (umount_all) took 0.01 seconds
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=157896k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=393292k,nr_inodes=98323,mode=755 freq=0 passno=0
fsync /dev/sda
fsync /dev/sdb
guestfsd: main_loop: proc 282 (internal_autosync) took 0.00 seconds
libguestfs: trace: v2v: internal_autosync = 0
libguestfs: sending SIGTERM to process 3438
libguestfs: qemu maxrss 460488K
libguestfs: trace: v2v: shutdown = 0
libguestfs: trace: v2v: close
libguestfs: closing guestfs handle 0x1a13dc0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfso5eAFU
[ 335.4] Checking if the guest needs BIOS or UEFI to boot
[ 335.4] Assigning disks to buses
virtio-blk slot 0:
target_file = /var/tmp/rancid-sda
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl599e6e.qcow2
target_overlay.ov_source = nbd:localhost:40113
virtio-blk slot 1:
target_file = /var/tmp/rancid-sdb
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl811e67.qcow2
target_overlay.ov_source = nbd:localhost:36773
[ 335.4] Copying disk 1/2 to /var/tmp/rancid-sda (raw)
target_file = /var/tmp/rancid-sda
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl599e6e.qcow2
target_overlay.ov_source = nbd:localhost:40113
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_has_backing_file "/var/tmp/v2vovl599e6e.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ --help
libguestfs: which_parser: g->qemu_img_info_parser = 1
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /dev/fd/4
libguestfs: parse_json: qemu-img info JSON output:\n{\n "backing-filename-format": "raw",\n "virtual-size": 160005980160,\n "filename": "/dev/fd/4",\n "cluster-size": 65536,\n "format": "qcow2",\n "actual-size": 275845120,\n "format-specific": {\n "type": "qcow2",\n "data": {\n "compat": "1.1",\n "lazy-refcounts": false,\n "refcount-bits": 16,\n "corrupt": false\n }\n },\n "backing-filename": "nbd:localhost:40113",\n "dirty-flag": false\n}\n\n
libguestfs: trace: disk_has_backing_file = 1
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_create "/var/tmp/rancid-sda" "raw" 160005980160 "preallocation:sparse"
libguestfs: trace: disk_create = 0
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovl599e6e.qcow2' '/var/tmp/rancid-sda'
(100.00/100%)
du --block-size=1 '/var/tmp/rancid-sda' | awk '{print $1}'
virtual copying rate: 888.3 M bits/sec
real copying rate: 128.8 M bits/sec
[2053.8] Copying disk 2/2 to /var/tmp/rancid-sdb (raw)
target_file = /var/tmp/rancid-sdb
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl811e67.qcow2
target_overlay.ov_source = nbd:localhost:36773
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_has_backing_file "/var/tmp/v2vovl811e67.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ --help
libguestfs: which_parser: g->qemu_img_info_parser = 1
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /dev/fd/4
libguestfs: parse_json: qemu-img info JSON output:\n{\n "backing-filename-format": "raw",\n "virtual-size": 899899242496,\n "filename": "/dev/fd/4",\n "cluster-size": 65536,\n "format": "qcow2",\n "actual-size": 551632896,\n "format-specific": {\n "type": "qcow2",\n "data": {\n "compat": "1.1",\n "lazy-refcounts": false,\n "refcount-bits": 16,\n "corrupt": false\n }\n },\n "backing-filename": "nbd:localhost:36773",\n "dirty-flag": false\n}\n\n
libguestfs: trace: disk_has_backing_file = 1
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_create "/var/tmp/rancid-sdb" "raw" 899899242496 "preallocation:sparse"
libguestfs: trace: disk_create = 0
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovl811e67.qcow2' '/var/tmp/rancid-sdb'
qemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused
virt-v2v: error: qemu-img command failed, see earlier errors
rm -rf '/var/tmp/null.KCJqyM'
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1a08e70 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1a20a80 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x19f7f20 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1a13dc0 (state 0)
libguestfs: trace: v2v: fstrim = 0
libguestfs: trace: v2v: umount_all
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=157896k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=393292k,nr_inodes=98323,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/sda2 dir=/sysroot type=ext3 opts=rw,relatime,data=ordered freq=0 passno=0
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: umount /sysroot
libguestfs: trace: v2v: umount_all = 0
libguestfs: trace: v2v: mount "/dev/sda1" "/"
guestfsd: main_loop: proc 47 (umount_all) took 0.01 seconds
guestfsd: main_loop: new request, len 0x40
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: mount -o /dev/sda1 /sysroot/
[ 330.601856] EXT4-fs (sda1): mounting ext3 file system using the ext4 subsystem
[ 330.605430] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
guestfsd: main_loop: proc 1 (mount) took 0.01 seconds
libguestfs: trace: v2v: mount = 0
libguestfs: trace: v2v: fstrim "/"
guestfsd: main_loop: new request, len 0x48
fsync /dev/sda
fsync /dev/sdb
commandrvf: stdout=y stderr=y flags=0x0
commandrvf: fstrim -v /sysroot/
/sysroot/: 90.5 MiB (94919680 bytes) trimmed
guestfsd: main_loop: proc 334 (fstrim) took 0.26 seconds
libguestfs: trace: v2v: fstrim = 0
[ 335.4] Closing the overlay
libguestfs: trace: v2v: umount_all
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=157896k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=393292k,nr_inodes=98323,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev/sda1 dir=/sysroot type=ext3 opts=rw,relatime,data=ordered freq=0 passno=0
commandrvf: stdout=n stderr=y flags=0x0
commandrvf: umount /sysroot
libguestfs: trace: v2v: umount_all = 0
libguestfs: trace: v2v: shutdown
libguestfs: trace: v2v: internal_autosync
guestfsd: main_loop: proc 47 (umount_all) took 0.01 seconds
guestfsd: main_loop: new request, len 0x28
umount-all: /proc/mounts: fsname=/dev/root dir=/ type=ext2 opts=rw,noatime,block_validity,barrier,user_xattr,acl freq=0 passno=0
umount-all: /proc/mounts: fsname=/proc dir=/proc type=proc opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=/sys dir=/sys type=sysfs opts=rw,relatime freq=0 passno=0
umount-all: /proc/mounts: fsname=tmpfs dir=/run type=tmpfs opts=rw,nosuid,relatime,size=157896k,mode=755 freq=0 passno=0
umount-all: /proc/mounts: fsname=/dev dir=/dev type=devtmpfs opts=rw,relatime,size=393292k,nr_inodes=98323,mode=755 freq=0 passno=0
fsync /dev/sda
fsync /dev/sdb
guestfsd: main_loop: proc 282 (internal_autosync) took 0.00 seconds
libguestfs: trace: v2v: internal_autosync = 0
libguestfs: sending SIGTERM to process 3438
libguestfs: qemu maxrss 460488K
libguestfs: trace: v2v: shutdown = 0
libguestfs: trace: v2v: close
libguestfs: closing guestfs handle 0x1a13dc0 (state 0)
libguestfs: command: run: rm
libguestfs: command: run: \ -rf /tmp/libguestfso5eAFU
[ 335.4] Checking if the guest needs BIOS or UEFI to boot
[ 335.4] Assigning disks to buses
virtio-blk slot 0:
target_file = /var/tmp/rancid-sda
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl599e6e.qcow2
target_overlay.ov_source = nbd:localhost:40113
virtio-blk slot 1:
target_file = /var/tmp/rancid-sdb
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl811e67.qcow2
target_overlay.ov_source = nbd:localhost:36773
[ 335.4] Copying disk 1/2 to /var/tmp/rancid-sda (raw)
target_file = /var/tmp/rancid-sda
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl599e6e.qcow2
target_overlay.ov_source = nbd:localhost:40113
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_has_backing_file "/var/tmp/v2vovl599e6e.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ --help
libguestfs: which_parser: g->qemu_img_info_parser = 1
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /dev/fd/4
libguestfs: parse_json: qemu-img info JSON output:\n{\n "backing-filename-format": "raw",\n "virtual-size": 160005980160,\n "filename": "/dev/fd/4",\n "cluster-size": 65536,\n "format": "qcow2",\n "actual-size": 275845120,\n "format-specific": {\n "type": "qcow2",\n "data": {\n "compat": "1.1",\n "lazy-refcounts": false,\n "refcount-bits": 16,\n "corrupt": false\n }\n },\n "backing-filename": "nbd:localhost:40113",\n "dirty-flag": false\n}\n\n
libguestfs: trace: disk_has_backing_file = 1
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_create "/var/tmp/rancid-sda" "raw" 160005980160 "preallocation:sparse"
libguestfs: trace: disk_create = 0
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovl599e6e.qcow2' '/var/tmp/rancid-sda'
(100.00/100%)
du --block-size=1 '/var/tmp/rancid-sda' | awk '{print $1}'
virtual copying rate: 888.3 M bits/sec
real copying rate: 128.8 M bits/sec
[2053.8] Copying disk 2/2 to /var/tmp/rancid-sdb (raw)
target_file = /var/tmp/rancid-sdb
target_format = raw
target_estimated_size = None
target_overlay = /var/tmp/v2vovl811e67.qcow2
target_overlay.ov_source = nbd:localhost:36773
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_has_backing_file "/var/tmp/v2vovl811e67.qcow2"
libguestfs: command: run: qemu-img
libguestfs: command: run: \ --help
libguestfs: which_parser: g->qemu_img_info_parser = 1
libguestfs: command: run: qemu-img
libguestfs: command: run: \ info
libguestfs: command: run: \ --output json
libguestfs: command: run: \ /dev/fd/4
libguestfs: parse_json: qemu-img info JSON output:\n{\n "backing-filename-format": "raw",\n "virtual-size": 899899242496,\n "filename": "/dev/fd/4",\n "cluster-size": 65536,\n "format": "qcow2",\n "actual-size": 551632896,\n "format-specific": {\n "type": "qcow2",\n "data": {\n "compat": "1.1",\n "lazy-refcounts": false,\n "refcount-bits": 16,\n "corrupt": false\n }\n },\n "backing-filename": "nbd:localhost:36773",\n "dirty-flag": false\n}\n\n
libguestfs: trace: disk_has_backing_file = 1
libguestfs: trace: set_verbose true
libguestfs: trace: set_verbose = 0
libguestfs: trace: disk_create "/var/tmp/rancid-sdb" "raw" 899899242496 "preallocation:sparse"
libguestfs: trace: disk_create = 0
qemu-img 'convert' '-p' '-n' '-f' 'qcow2' '-O' 'raw' '/var/tmp/v2vovl811e67.qcow2' '/var/tmp/rancid-sdb'
qemu-img: Could not open '/var/tmp/v2vovl811e67.qcow2': Could not open backing file: Failed to connect socket: Connection refused
virt-v2v: error: qemu-img command failed, see earlier errors
rm -rf '/var/tmp/null.KCJqyM'
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1a08e70 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1a20a80 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x19f7f20 (state 0)
libguestfs: trace: close
libguestfs: closing guestfs handle 0x1a13dc0 (state 0)
Friday, September 02, 2016
Importing Windows 2012 VMs into oVirt from VMware
I had quite a bit of trouble importing Windows 2012 server VMs into an oVirt cluster I had. The cluster is running 4.0.2.7-1.el7, and ships with a version of virt-v2v that does not support conversion of 2012 or Windows 8. I could never get the GUI tools to connect to VMware properly (they could auth, but couldn't find the VM at the URI I specified, and this also happened with virt-v2v). I could list the VMs no problem using ovftool however. Anyway, here are the steps I used:
1. I downloaded ovftool from VMWare (a login is needed to download
this). I installed it on a Linux machine with enough space for importing
VMs.
2. Create an .ova file via ovftool: the VM has to be shut off when doing this. e.g.,
2. Create an .ova file via ovftool: the VM has to be shut off when doing this. e.g.,
ovftool vi://<user>@<esxi_manager_ host>/<cluster_name>/<vm_ folder>/<vm_name> /path/to/vm.ova
ovftool allows you to browse the directory structure so you can build the URL as needed.
3.
Once you have an .ova, you'll need to get a recent version of virt-v2v
to convert it, plus the virtio-win package for RHEL 7 (I used
virtio-win-1.8.0-4.el7.noarch. rpm). I also created an ISO domain on
my oVirt cluster with the oVirt guest tools, as I needed to install
another driver after importing the VM. I used
virt-v2v-1.32.7-1.fc23.x86_64 on Fedora 23, and the later version on
Fedora 24 also worked. There is also a version for RHEL 7.3 (7.3 is
currently beta) which should also work.
4. I found
that I could not successfully import a 2012 VM when using virt-v2v to
directly export it to the export domain. It worked fine that way with
2008 and Windows 7, but failed immediately in the ovirt GUI with 2012.
So what I did was use another very useful tool from the same author of
virt-v2v which is designed to import a KVM image to oVirt. So I did the
following to convert the ova:
# export LIBGUEST_BACKEND=direct
# virt-v2v -i ova /mnt/import-vm/w2012-test.ova -o local -os /var/tmp/v2v/ -of qcow2
# export LIBGUEST_BACKEND=direct
# virt-v2v -i ova /mnt/import-vm/w2012-test.ova -o local -os /var/tmp/v2v/ -of qcow2
This
should install most of the drivers you need from /usr/share/virtio-win
(from the virtio-win rpm), though it complains about
/usr/share/virt-tools/rhev- apt.exe and a QXL driver as missing. These are not needed to get the VM running however.
5. Once converted, you should have a disk image in your output directory, e.g.,
/var/tmp/v2v/w2012-test-sda
/var/tmp/v2v/w2012-test-sda
6. Run 'import-to-ovirt.pl' to copy it to the export domain, e.g.,
import-to-ovirt.pl /var/tmp/v2v/w2012-test-sda ovirt-engine:/mnt/export-vm
import-to-ovirt.pl /var/tmp/v2v/w2012-test-sda ovirt-engine:/mnt/export-vm
7.
The VM should show up on the export storage and can then be imported. I
found I had to install one driver from ovirt-guest-tools by attaching
it to the VM.
Any questions, please ask and I will do my best to answer them!
C
Friday, June 22, 2012
Getting WinXP (or Win 7, Server etc) to see all cores as a KVM guest
I'd installed a Win XP Pro x86_64 guest on a Fedora 13 x86_64 KVM host. The host had 8 hyperthreaded CPUs (16 virtual cores in total). I had assigned all 16 vcpus to the WinXP guest, but it only showed 2 in the task manager, though in the device manager, it showed 16. The problem had to do with licensing: XP Pro is limited to 2 physical CPUs (even though it can see 16 cpus in the device manager). It can support (at least) 4 cores per cpu however. So the solution is to specify this in the config, which I found (via googling) can be done by modifying the xml defining the guest. To find the layout of the host:
virsh capabilities | grep topology
<topology sockets='2' cores='4' threads='2'/>
Using 'virsh edit <domain>', find where the vcpus are set:
<vcpu>16</vcpu>
Then add the following below:
<cpu>
<topology sockets='2' cores='4' threads='2'/>
</cpu>
Save it, and restart the guest. It now sees all vcpus.
I notice that from looking at the xml, libvirt reorders it slightly:
<vcpu>16</vcpu>
<os>
<type arch='x86_64' machine='pc-0.13'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu>
<topology sockets='2' cores='4' threads='2'/>
</cpu>
virsh capabilities | grep topology
<topology sockets='2' cores='4' threads='2'/>
Using 'virsh edit <domain>', find where the vcpus are set:
<vcpu>16</vcpu>
Then add the following below:
<cpu>
<topology sockets='2' cores='4' threads='2'/>
</cpu>
Save it, and restart the guest. It now sees all vcpus.
I notice that from looking at the xml, libvirt reorders it slightly:
<vcpu>16</vcpu>
<os>
<type arch='x86_64' machine='pc-0.13'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu>
<topology sockets='2' cores='4' threads='2'/>
</cpu>
Tuesday, February 14, 2012
editfiles cfengine v2
I had the task of disabling swap on a class of machines managed by cfengine2
The swap entry could have started with 'UUID=', or possibly '/dev/sda3'
One way of doing it:
editfiles:
{ /etc/fstab
CommentLinesMatching ".*swap(\s+)swap(\s+).*"
}
another, more complicated way:
editfiles:
{ /etc/fstab
ReplaceAll "swap(\s+)swap(\s+)defaults(\s+)0(\s+)0$" With "swap swap noauto,defaults 0 0"
}
Normally, with editfiles, you must match the whole line (hence the .* at the beginning and end of the first way using 'CommentLines'), but when matching a quoted regex (as with 'ReplaceAll'), you can just match (and replace) that part of the line
The swap entry could have started with 'UUID=', or possibly '/dev/sda3'
One way of doing it:
editfiles:
{ /etc/fstab
CommentLinesMatching ".*swap(\s+)swap(\s+).*"
}
another, more complicated way:
editfiles:
{ /etc/fstab
ReplaceAll "swap(\s+)swap(\s+)defaults(\s+)0(\s+)0$" With "swap swap noauto,defaults 0 0"
}
Normally, with editfiles, you must match the whole line (hence the .* at the beginning and end of the first way using 'CommentLines'), but when matching a quoted regex (as with 'ReplaceAll'), you can just match (and replace) that part of the line
Wednesday, January 25, 2012
Unpack and edit initramfs (for blacklisting modules, but can be used for other reasons)
I did this originally to blacklist the loading a kernel module, which may have been loaded by initramfs (adding rdblacklist to the kernel command line didn't work). However, the module still got loaded and what I should have done in the first instance was check modules.dep, as it was a dependency for another module that I wanted anyway. You also must make sure any other processes in init don't cause it to get loaded. Anyway, it was a useful exercise in how to modify initramfs!
How to unpack and modify a initramfs image
(tested with fedora 13)
Make a directory to unpack it in
Make a backup of the original initramfs image
cd to your unpack dir and unpack with: 'gzip -dc initramfs-2.6.34.9-69.fc13.x86_64.img | cpio -id'
The 'i' flag is for taking input, and the 'd' makes dirs
Modify the files as required
Recreate the image (I like to do it in 2 separate steps cos I'm superstitous):
find . | cpio -H newc -o > /some/dir/initramfs-2.6.34.9-69.fc13.x86_64.cpio
'-H' - format
newc - is the newc format type
'-o' - output
cat /tmp/initramfs-2.6.34.9-69.fc13.x86_64.cpio | gzip -9 > /tmp/initramfs-2.6.34.9-69.fc13.x86_64.img
Maximum compression is selected as that is what it is used in the original
Then copy it back
I haven't yet tried dracut
How to unpack and modify a initramfs image
(tested with fedora 13)
Make a directory to unpack it in
Make a backup of the original initramfs image
cd to your unpack dir and unpack with: 'gzip -dc initramfs-2.6.34.9-69.fc13.x86_64.img | cpio -id'
The 'i' flag is for taking input, and the 'd' makes dirs
Modify the files as required
Recreate the image (I like to do it in 2 separate steps cos I'm superstitous):
find . | cpio -H newc -o > /some/dir/initramfs-2.6.34.9-69.fc13.x86_64.cpio
'-H' - format
newc - is the newc format type
'-o' - output
cat /tmp/initramfs-2.6.34.9-69.fc13.x86_64.cpio | gzip -9 > /tmp/initramfs-2.6.34.9-69.fc13.x86_64.img
Maximum compression is selected as that is what it is used in the original
Then copy it back
I haven't yet tried dracut
Friday, November 25, 2011
broken ps
ps, w and a bunch of other things were not working (they would just hang uninterruptedly). I checked in proc. 'ls' worked fine, and so I thought I'd cat the cmdline of each process. I did this as a 'for' loop and found that it stopped at one particular PID. As I was able to read the /proc dir, I could see that the perms were ok, and that the process owner was someone else than the logged in user or root. I could not find anything in /sys (not that I would know really what to look for, since that is mostly device stuff). I ran tcpdump in case it had a keylogger sending stuff back (netstat would also hang)
cd /proc
ls | grep -E "([0-9]){1}" > /tmp/cm.txt
for pid in `cat /tmp/cm.txt`;do echo "checking PID $pid...";cat /proc/${pid}/cmdline;echo;done
I found that I could not kill the process either. Tried all 64 signals with a for loop.
Nothing in /var/run with a PID matching that.
The machine gets loads of 'BUG: Bad page state in process' errors for a variety of processes. These don't seem to result in crashes.
Perhaps there is a region of memory that the process has written to that is unreadable?
cd /proc
ls | grep -E "([0-9]){1}" > /tmp/cm.txt
for pid in `cat /tmp/cm.txt`;do echo "checking PID $pid...";cat /proc/${pid}/cmdline;echo;done
I found that I could not kill the process either. Tried all 64 signals with a for loop.
Nothing in /var/run with a PID matching that.
The machine gets loads of 'BUG: Bad page state in process' errors for a variety of processes. These don't seem to result in crashes.
Perhaps there is a region of memory that the process has written to that is unreadable?
Wednesday, August 31, 2011
initrd images created from kickstart
Update: initrd files are now 'xz' archives.
I wondered whether a kickstart of a RHEL/CentOS distro would provide a customised initrd image for the hardware it was being installed on. To check this, I compared the initrd from a server with a particular (Marvell) SAS disk controller with another server without that controller.
initrd images are gzipped cpio files, so to compare the two, I copied each initrd to another location, so as not to interfere with needed boot files, moved each to a filename with a '.gz' extension, then gunzipped them and ran 'cpio -t < initrd_file_name' to list the files. The SAS disk controller kernel module was present in the initrd from the host that had that particular hardware, but not on the other host which didn't. It wasn't present in the kernel image (vmlinuz) on either, though it was present in config as a loadable module, so would have to be in the initrd, since it needs a filesystem to load from, presumably:
CONFIG_SCSI_MVSAS=m
I wondered whether a kickstart of a RHEL/CentOS distro would provide a customised initrd image for the hardware it was being installed on. To check this, I compared the initrd from a server with a particular (Marvell) SAS disk controller with another server without that controller.
initrd images are gzipped cpio files, so to compare the two, I copied each initrd to another location, so as not to interfere with needed boot files, moved each to a filename with a '.gz' extension, then gunzipped them and ran 'cpio -t < initrd_file_name' to list the files. The SAS disk controller kernel module was present in the initrd from the host that had that particular hardware, but not on the other host which didn't. It wasn't present in the kernel image (vmlinuz) on either, though it was present in config as a loadable module, so would have to be in the initrd, since it needs a filesystem to load from, presumably:
CONFIG_SCSI_MVSAS=m
Wednesday, February 16, 2011
replacing a space with a newline
contents of file.txt:
abc def ghi
cat file.txt | tr ' ' '\012'
abc
def
ghi
I originally tried this with
cat file.txt | sed 's/ /\n/g'
and it didn't work. I am told that '\n' isn't always recognised as a newline
abc def ghi
cat file.txt | tr ' ' '\012'
abc
def
ghi
I originally tried this with
cat file.txt | sed 's/ /\n/g'
and it didn't work. I am told that '\n' isn't always recognised as a newline
Wednesday, February 02, 2011
Migrating (expanding) an existing array from RAID 1 to RAID 10 on HP SmartArray 400i
I added a couple of 72GB drives to an existing 2-drive array on an HP DL360 G5, which has a SmartArray P400i controller. The driver was the HP cciss driver, and the server was running CentOS 5.5 x86_64.
The expansion of the array can be done while the server is online, over ssh using hpacucli. While the underlying drive can be expanded, I have not yet found a way to notify the kernel about the changed drive size, so it appears a reboot is still required (or reloading the cciss driver, which in most cases means a reboot).
Using hpacucli:
=> controller all show
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
=> ctrl slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 0 MB)
logicaldrive 1 (68.3 GB, RAID 1, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
unassigned
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=> controller slot=0 logicaldrive 1 add drives=1I:1:3,1I:1:4
=> ctrl slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 139953 MB)
logicaldrive 1 (68.3 GB, RAID 1+0, Transforming, 0% complete)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=>
Then you need to expand the logical drive into this space:
=> controller slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 139953 MB)
logicaldrive 1 (68.3 GB, RAID 1+0, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=>
=> controller slot=0 logicaldrive 1 modify size=max
Warning: Extension may not be supported on certain operating systems.
Performing extension on these operating systems can cause data to
become inaccessible. See ACU documentation for details. Continue?
(y/n) y
=> controller slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 0 MB)
logicaldrive 1 (136.7 GB, RAID 1+0, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=>
The size is correctly reported in /sys:
# cat /sys/block/cciss\!c0d0/size
286611840
#
However, fdisk does not see it, even after a partprobe, and I could find no way of re-scanning in /proc or /sys to get it to see the increased size.
One suggestion I haven't tried is to use 'sfdisk -R'
So for now a reboot, then resize partitions via resize2fs.
The expansion of the array can be done while the server is online, over ssh using hpacucli. While the underlying drive can be expanded, I have not yet found a way to notify the kernel about the changed drive size, so it appears a reboot is still required (or reloading the cciss driver, which in most cases means a reboot).
Using hpacucli:
=> controller all show
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
=> ctrl slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 0 MB)
logicaldrive 1 (68.3 GB, RAID 1, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
unassigned
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=> controller slot=0 logicaldrive 1 add drives=1I:1:3,1I:1:4
=> ctrl slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 139953 MB)
logicaldrive 1 (68.3 GB, RAID 1+0, Transforming, 0% complete)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=>
Then you need to expand the logical drive into this space:
=> controller slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 139953 MB)
logicaldrive 1 (68.3 GB, RAID 1+0, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=>
=> controller slot=0 logicaldrive 1 modify size=max
Warning: Extension may not be supported on certain operating systems.
Performing extension on these operating systems can cause data to
become inaccessible. See ACU documentation for details. Continue?
(y/n) y
=> controller slot=0 show config
Smart Array P400i in Slot 0 (Embedded) (sn: PH81MQ6085 )
array A (SAS, Unused Space: 0 MB)
logicaldrive 1 (136.7 GB, RAID 1+0, OK)
physicaldrive 1I:1:1 (port 1I:box 1:bay 1, SAS, 72 GB, OK)
physicaldrive 1I:1:2 (port 1I:box 1:bay 2, SAS, 72 GB, OK)
physicaldrive 1I:1:3 (port 1I:box 1:bay 3, SAS, 72 GB, OK)
physicaldrive 1I:1:4 (port 1I:box 1:bay 4, SAS, 72 GB, OK)
=>
The size is correctly reported in /sys:
# cat /sys/block/cciss\!c0d0/size
286611840
#
However, fdisk does not see it, even after a partprobe, and I could find no way of re-scanning in /proc or /sys to get it to see the increased size.
One suggestion I haven't tried is to use 'sfdisk -R'
So for now a reboot, then resize partitions via resize2fs.
Friday, January 22, 2010
How to resize (grow) a Xen VM partition
I was faced with a situation whereby a partition on a Xen VM was running out of space. Luckily, there was space available on the Xen host (i.e., Dom0). Also, the partition was at the end of the virtual disk, which simplifies things. So below are the steps I took to resize the partition.
NB: This assumes a Xen host (Dom0) is not running Logical Volume Management, and the VM (DomU) has an ext3 filesystem.
First of all, stop the VM and take a backup. This is in case anything goes wrong! You will need somewhere with enough space to copy the VM .img file, taking into consideration that the partition that you copy to, may well need free space to create the larger partition :-)
Once the VM has been stopped and the backup made, it's time to add the extra space to our VM. There are several HOWTOs on this, but what worked for me was to just concatenate a file onto the VM image. When creating a VM initially, the filesystem is built on a file in the same way. Using 'dd' is easy for this.
If we can add another 10GB to the partition, dd a file onto the end of the existing image:
Assuming a VM image called 'mc3.img', do:
dd if=/dev/zero bs=1G count=10 >> /path/to/mc3.img
NB: Be very careful to make sure you have '>>'
Now the filesystem needs to be expanded onto this image. Use 'losetup' to associate the file with a device.
'losetup -f' to search for the first free loopback device
Assuming this is '/dev/loop3', associate it with the image file
'losetup /dev/loop3 /path/to/mc3.img'
You will be able to see the old partitions if you run an 'fdisk -l' on /dev/loop3.
# fdisk -l /dev/loop0
Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 1019 1895670 83 Linux
However, the partition device nodes don't actually exist for us to be able to operate on them:
[root@xen001 vm]# fsck /dev/loop0p3
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
fsck.ext2: No such file or directory while trying to open /dev/loop0p3
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193
So use 'kpartx' to create devices in /dev/mapper. List them:
[root@xen001 vm]# kpartx -l /dev/loop0
loop0p1 : 0 8385867 /dev/loop0 63
loop0p2 : 0 4192965 /dev/loop0 8385930
loop0p3 : 0 3791340 /dev/loop0 12578895
[root@xen001 vm]# kpartx -av /dev/loop0
add map loop0p1 : 0 8385867 linear /dev/loop0 63
add map loop0p2 : 0 4192965 linear /dev/loop0 8385930
add map loop0p3 : 0 3791340 linear /dev/loop0 12578895
[root@xen001 vm]# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 62 Jan 18 12:12 control
brw-r----- 1 root disk 253, 0 Jan 22 14:36 loop0p1
brw-r----- 1 root disk 253, 1 Jan 22 14:36 loop0p2
brw-r----- 1 root disk 253, 2 Jan 22 14:36 loop0p3
[root@xen001 vm]#
We need to delete the existing partition (in our case, partition 3, /dev/mapper/loop0p3), and then recreate it to take advantage of the extra space:
[root@xen001 vm]# fdisk /dev/loop0
The number of cylinders for this disk is set to 2325.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): p
Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 1019 1895670 83 Linux
Command (m for help): d
Partition number (1-4): 3
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (784-2325, default 784):
Using default value 784
Last cylinder or +size or +sizeM or +sizeK (784-2325, default 2325):
Using default value 2325
Command (m for help): p
Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 2325 12386115 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@xen001 vm]#
Note how the 'End' cylinder has changed from 1019 to 2325
Now, resize filesystem (resize2fs supports resizing ext3 in 2.6.x kernels):
root@xen001 vm]# resize2fs /dev/mapper/loop0p3
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/mapper/loop0p3 to 3096528 (4k) blocks.
The filesystem on /dev/mapper/loop0p3 is now 3096528 blocks long.
Fsck it:
[root@xen001 vm]# e2fsck -f /dev/mapper/loop0p3
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/u01: 9474/3003520 files (17.4% non-contiguous), 527247/3096528 blocks
[root@xen001 vm]#
Remove it from device mapper:
[root@xen001 vm]# kpartx -d /dev/loop0
Detach it from loop device:
[root@xen001 vm]# losetup -d /dev/loop0
Start up the VM:
[root@xen001 vm]# xm create mc3
Using config file "/etc/xen/mc3".
Started domain mc3
[root@xen001 vm]#
And you're done
NB: This assumes a Xen host (Dom0) is not running Logical Volume Management, and the VM (DomU) has an ext3 filesystem.
First of all, stop the VM and take a backup. This is in case anything goes wrong! You will need somewhere with enough space to copy the VM .img file, taking into consideration that the partition that you copy to, may well need free space to create the larger partition :-)
Once the VM has been stopped and the backup made, it's time to add the extra space to our VM. There are several HOWTOs on this, but what worked for me was to just concatenate a file onto the VM image. When creating a VM initially, the filesystem is built on a file in the same way. Using 'dd' is easy for this.
If we can add another 10GB to the partition, dd a file onto the end of the existing image:
Assuming a VM image called 'mc3.img', do:
dd if=/dev/zero bs=1G count=10 >> /path/to/mc3.img
NB: Be very careful to make sure you have '>>'
Now the filesystem needs to be expanded onto this image. Use 'losetup' to associate the file with a device.
'losetup -f' to search for the first free loopback device
Assuming this is '/dev/loop3', associate it with the image file
'losetup /dev/loop3 /path/to/mc3.img'
You will be able to see the old partitions if you run an 'fdisk -l' on /dev/loop3.
# fdisk -l /dev/loop0
Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 1019 1895670 83 Linux
However, the partition device nodes don't actually exist for us to be able to operate on them:
[root@xen001 vm]# fsck /dev/loop0p3
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
fsck.ext2: No such file or directory while trying to open /dev/loop0p3
The superblock could not be read or does not describe a correct ext2
filesystem. If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
e2fsck -b 8193
So use 'kpartx' to create devices in /dev/mapper. List them:
[root@xen001 vm]# kpartx -l /dev/loop0
loop0p1 : 0 8385867 /dev/loop0 63
loop0p2 : 0 4192965 /dev/loop0 8385930
loop0p3 : 0 3791340 /dev/loop0 12578895
[root@xen001 vm]# kpartx -av /dev/loop0
add map loop0p1 : 0 8385867 linear /dev/loop0 63
add map loop0p2 : 0 4192965 linear /dev/loop0 8385930
add map loop0p3 : 0 3791340 linear /dev/loop0 12578895
[root@xen001 vm]# ls -l /dev/mapper/
total 0
crw------- 1 root root 10, 62 Jan 18 12:12 control
brw-r----- 1 root disk 253, 0 Jan 22 14:36 loop0p1
brw-r----- 1 root disk 253, 1 Jan 22 14:36 loop0p2
brw-r----- 1 root disk 253, 2 Jan 22 14:36 loop0p3
[root@xen001 vm]#
We need to delete the existing partition (in our case, partition 3, /dev/mapper/loop0p3), and then recreate it to take advantage of the extra space:
[root@xen001 vm]# fdisk /dev/loop0
The number of cylinders for this disk is set to 2325.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Command (m for help): p
Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 1019 1895670 83 Linux
Command (m for help): d
Partition number (1-4): 3
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (784-2325, default 784):
Using default value 784
Last cylinder or +size or +sizeM or +sizeK (784-2325, default 2325):
Using default value 2325
Command (m for help): p
Disk /dev/loop0: 19.1 GB, 19126026240 bytes
255 heads, 63 sectors/track, 2325 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/loop0p1 * 1 522 4192933+ 83 Linux
/dev/loop0p2 523 783 2096482+ 82 Linux swap / Solaris
/dev/loop0p3 784 2325 12386115 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
[root@xen001 vm]#
Note how the 'End' cylinder has changed from 1019 to 2325
Now, resize filesystem (resize2fs supports resizing ext3 in 2.6.x kernels):
root@xen001 vm]# resize2fs /dev/mapper/loop0p3
resize2fs 1.39 (29-May-2006)
Resizing the filesystem on /dev/mapper/loop0p3 to 3096528 (4k) blocks.
The filesystem on /dev/mapper/loop0p3 is now 3096528 blocks long.
Fsck it:
[root@xen001 vm]# e2fsck -f /dev/mapper/loop0p3
e2fsck 1.39 (29-May-2006)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/u01: 9474/3003520 files (17.4% non-contiguous), 527247/3096528 blocks
[root@xen001 vm]#
Remove it from device mapper:
[root@xen001 vm]# kpartx -d /dev/loop0
Detach it from loop device:
[root@xen001 vm]# losetup -d /dev/loop0
Start up the VM:
[root@xen001 vm]# xm create mc3
Using config file "/etc/xen/mc3".
Started domain mc3
[root@xen001 vm]#
And you're done
Thursday, November 19, 2009
ldapmodify - adding objectClass, specifically, samba objectClass
The task was to add samba attributes to existing users in LDAP. The easy way is to do it via 'smbpasswd -a', though this will require various samba packages to be installed, and a samba config, and the ability of the LDAP server to look up users locally as LDAP users...(changes to pam in other words). This server was a production master LDAP server, so I wanted to see if I could do it without additional software and other changes like the one to PAM, and the obvious way was via 'ldapmodify'. First attempts failed, as I thought I could just add the objectClass first, and then the mandatory attribute, which is 'sambaSID' (as it says in /etc/openldap/schema/samba.schema: MUST ( uid $ sambaSID ).
contents of ldif:
dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
add: objectClass
objectClass: sambaSAMAccount
ldapmodify -x -D "uid=my.user,ou=Oslo,ou=People,dc=example,dc=com" -W -f m8.ldif
Enter LDAP Password:
modifying entry "uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com"
ldapmodify: Object class violation (65)
additional info: object class 'sambaSamAccount' requires attribute 'sambaSID'
So, I tried just adding the necessary attribute to the ldif as an additional change, but it still failed
modifying entry "uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com"
ldapmodify: Object class violation (65)
additional info: attribute 'sambaSID' not allowed
There is a special way to add an objectClass, as I found on:
http://www.zytrax.com/books/ldap/ch8/index.html#samples
I had to include all the existing objectclasses in the modify, as well as the new one, plus the necessary attribute:
dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: specialObjectClassUser
objectClass: sambaSamAccount
sambaSID: S-1-5-21-1096379175-3498632652-1565186858-17768
To add all of the attributes (not sure whether it is worth including all of these, but anyway):
dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: specialObjectClassUser
objectClass: sambaSamAccount
sambaSID: S-1-5-21-1096379175-3498632652-1565186858-17768
sambaPasswordHistory: 0000000000000000000000000000000000000000000000000000000000000000
sambaAcctFlags: [U ]
sambaLogonTime: 0
sambaLogoffTime: 2147483647
sambaKickoffTime: 2147483647
sambaPwdCanChange: 0
sambaLMPassword: 7E9Q28366B3F5C82AAD3B435B51404EE
sambaNTPassword: 42AZ499C5146E4F9BF017649A4756013
sambaPwdLastSet: 1255018017
sambaPwdMustChange: 1258906017
I have written a script in perl which can generate LDIFs, complete with randomised password hashes, and lookup of a user's UID to calculate sambaSID
contents of ldif:
dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
add: objectClass
objectClass: sambaSAMAccount
ldapmodify -x -D "uid=my.user,ou=Oslo,ou=People,dc=example,dc=com" -W -f m8.ldif
Enter LDAP Password:
modifying entry "uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com"
ldapmodify: Object class violation (65)
additional info: object class 'sambaSamAccount' requires attribute 'sambaSID'
So, I tried just adding the necessary attribute to the ldif as an additional change, but it still failed
modifying entry "uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com"
ldapmodify: Object class violation (65)
additional info: attribute 'sambaSID' not allowed
There is a special way to add an objectClass, as I found on:
http://www.zytrax.com/books/ldap/ch8/index.html#samples
I had to include all the existing objectclasses in the modify, as well as the new one, plus the necessary attribute:
dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: specialObjectClassUser
objectClass: sambaSamAccount
sambaSID: S-1-5-21-1096379175-3498632652-1565186858-17768
To add all of the attributes (not sure whether it is worth including all of these, but anyway):
dn: uid=jon.smith,ou=Oslo,ou=People,dc=example,dc=com
changetype: modify
objectClass: person
objectClass: organizationalPerson
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
objectClass: specialObjectClassUser
objectClass: sambaSamAccount
sambaSID: S-1-5-21-1096379175-3498632652-1565186858-17768
sambaPasswordHistory: 0000000000000000000000000000000000000000000000000000000000000000
sambaAcctFlags: [U ]
sambaLogonTime: 0
sambaLogoffTime: 2147483647
sambaKickoffTime: 2147483647
sambaPwdCanChange: 0
sambaLMPassword: 7E9Q28366B3F5C82AAD3B435B51404EE
sambaNTPassword: 42AZ499C5146E4F9BF017649A4756013
sambaPwdLastSet: 1255018017
sambaPwdMustChange: 1258906017
I have written a script in perl which can generate LDIFs, complete with randomised password hashes, and lookup of a user's UID to calculate sambaSID
Saturday, December 20, 2008
super cool way to move files with spaces in shell
I had a bunch of files named like;
IMG_2345 2.JPG
I wanted to remove the space and the 2 so that it would become like:
IMG_2345.JPG
How to do this in a one liner?
Can't use usual for file in `ls *\ 2.JPG`;do mv $file etc, as it will
treat the space as a newline (and quoting $file to "$file" won't do it
either.
There is a convenient way of doing this with 'read', as well as using built-in string replacement
ls *\ 2.JPG | while read line;do mv "$line" "${line/\ 2/}";done
Substring Replacement
${string/substring/replacement}
Replace first match of $substring with $replacement.
${string//substring/replacement}
Replace all matches of $substring with $replacement.
http://www.linuxtopia.org/online_books/advanced_bash_scripting_guide/string-manipulation.html
IMG_2345 2.JPG
I wanted to remove the space and the 2 so that it would become like:
IMG_2345.JPG
How to do this in a one liner?
Can't use usual for file in `ls *\ 2.JPG`;do mv $file etc, as it will
treat the space as a newline (and quoting $file to "$file" won't do it
either.
There is a convenient way of doing this with 'read', as well as using built-in string replacement
ls *\ 2.JPG | while read line;do mv "$line" "${line/\ 2/}";done
Substring Replacement
${string/substring/replacement}
Replace first match of $substring with $replacement.
${string//substring/replacement}
Replace all matches of $substring with $replacement.
http://www.linuxtopia.org/online_books/advanced_bash_scripting_guide/string-manipulation.html
Saturday, October 04, 2008
setting pager to 'less' for mysql
Ever wanted to have 'less' functionality when doing mysql queries?
mysql> \P less -S
PAGER set to 'less -S'
So when you do a 'select *', it will automatically output it to less :)
mysql> \P less -S
PAGER set to 'less -S'
So when you do a 'select *', it will automatically output it to less :)
Monday, September 15, 2008
ip_conntrack, /proc and sysctl
sysctl -a shows settings for conntrack, amongst other things. This will show count of number of entries in table as well
ip_conntrack has a default timeout of 5 days (432000 seconds) for *established* connections. That is, the entry will be kept in the table for 5 days before expiring if there is no traffic (?). With a large amount of traffic, this could grow very large.
There also settings that control SYN, ACK etc for conntrack (check this and see below)
AFAIK, the only way to clear the current count is to unload the relevant modules (check this)
You can put customisations that override defaults in /etc/sysctl.conf, which are read when sysctl is run, usually at boot. Warning:
When the ip conntrack module(s) is/are (re)loaded, the defaults are used. You have to run sysctl to read any settings in /etc/sysctl.conf (sysctl -p /etc/sysctl.conf)
net.ipv4.ip_conntrack_max = 1000000
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_loose = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_log_invalid = 0
net.ipv4.netfilter.ip_conntrack_generic_timeout = 3
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 3
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 2
net.ipv4.netfilter.ip_conntrack_udp_timeout = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 3
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_buckets = 8192
net.ipv4.netfilter.ip_conntrack_count = 598236
net.ipv4.netfilter.ip_conntrack_max = 1000000
ip_conntrack has a default timeout of 5 days (432000 seconds) for *established* connections. That is, the entry will be kept in the table for 5 days before expiring if there is no traffic (?). With a large amount of traffic, this could grow very large.
There also settings that control SYN, ACK etc for conntrack (check this and see below)
AFAIK, the only way to clear the current count is to unload the relevant modules (check this)
You can put customisations that override defaults in /etc/sysctl.conf, which are read when sysctl is run, usually at boot. Warning:
When the ip conntrack module(s) is/are (re)loaded, the defaults are used. You have to run sysctl to read any settings in /etc/sysctl.conf (sysctl -p /etc/sysctl.conf)
net.ipv4.ip_conntrack_max = 1000000
net.ipv4.netfilter.ip_conntrack_tcp_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_tcp_be_liberal = 0
net.ipv4.netfilter.ip_conntrack_tcp_loose = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_max_retrans = 3
net.ipv4.netfilter.ip_conntrack_log_invalid = 0
net.ipv4.netfilter.ip_conntrack_generic_timeout = 3
net.ipv4.netfilter.ip_conntrack_icmp_timeout = 3
net.ipv4.netfilter.ip_conntrack_udp_timeout_stream = 2
net.ipv4.netfilter.ip_conntrack_udp_timeout = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_time_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_last_ack = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_close_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_fin_wait = 2
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 432000
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_recv = 3
net.ipv4.netfilter.ip_conntrack_tcp_timeout_syn_sent = 3
net.ipv4.netfilter.ip_conntrack_checksum = 1
net.ipv4.netfilter.ip_conntrack_buckets = 8192
net.ipv4.netfilter.ip_conntrack_count = 598236
net.ipv4.netfilter.ip_conntrack_max = 1000000
Thursday, July 10, 2008
postfix and rate limiting, sender control, etc
postfix has builtin controls restricting certain kinds of things, such as rate at which clients can connect, number of recipients per message, no. of connections per client, etc. These can be seen in man 8 smtpd and anvil.
As I didn't want to interfere too much with what might use the localhost instance, I added an instance of postfix listening on port 10025 of the eth0 address (192.168.10.208). That required adding a line to master.cf (I just added it under the standard smtp line):
192.168.10.208:10025 inet n - n - - smtpd
-o smtpd_client_message_rate_limit=5
The '-o smtpd_client_message_rate_limit=5' bit was to specify an override to limit connections from one client to 5 per the default anvil time (60s). The '-o' simply means override what is set in main.cf
The mail that I was attempting to restrict was being generated by snmpttd reading traps. It calls the 'mail' command for each trap, which can lead to a lot of mail in a large network. Now, the problem was that the 'mail' MUA (which is part of 'mailx' package in RHEL5) does not actually connect to smtp over tcp by default (and nor can I find any way of changing this in RHEL version, since all of the .mailrc and /etc/mail.rc directives, where you can specify smtp server and port, do not apply seemingly. They aren't in the man pages anyway. So I used the Mail::Sendmail module, which is easy to configure. BTW: I have not tried using '-o smtpd_client_message_rate_limit=5' on the 'unix' socket listening in master.cf, I assumed that it would not call smtpd, but that is because I was too lazy to read all the documentation on how postfix works ;)
Anyway, this works well
As I didn't want to interfere too much with what might use the localhost instance, I added an instance of postfix listening on port 10025 of the eth0 address (192.168.10.208). That required adding a line to master.cf (I just added it under the standard smtp line):
192.168.10.208:10025 inet n - n - - smtpd
-o smtpd_client_message_rate_limit=5
The '-o smtpd_client_message_rate_limit=5' bit was to specify an override to limit connections from one client to 5 per the default anvil time (60s). The '-o' simply means override what is set in main.cf
The mail that I was attempting to restrict was being generated by snmpttd reading traps. It calls the 'mail' command for each trap, which can lead to a lot of mail in a large network. Now, the problem was that the 'mail' MUA (which is part of 'mailx' package in RHEL5) does not actually connect to smtp over tcp by default (and nor can I find any way of changing this in RHEL version, since all of the .mailrc and /etc/mail.rc directives, where you can specify smtp server and port, do not apply seemingly. They aren't in the man pages anyway. So I used the Mail::Sendmail module, which is easy to configure. BTW: I have not tried using '-o smtpd_client_message_rate_limit=5' on the 'unix' socket listening in master.cf, I assumed that it would not call smtpd, but that is because I was too lazy to read all the documentation on how postfix works ;)
Anyway, this works well
Friday, July 04, 2008
connection rate limiting to apache and iptables
We needed to create a way of connection rate limiting to a particular web page, from any given IP. There used to be various methods available to do this in Apache, such as mod_throttle/mod_choke/mod_limit, but the problem was these modules only worked with apache 1 or they were not actively developed anymore. We have netscreen firewalls in front of the the servers, but they cannot limit the number of connections per IP. IPtables can do this, via a number of different ways. There are modules in IPTables, such as 'connlimit' (most recent) and 'iplimit' (replaced by connlimit). However, the OS we were using was RHEL5.1 (kernel 2.6.18, and IPT 1.3.5), which, although it appears to support connlimit (type 'iptables -m connlimit --help' and it shows you usage info, as the iptables connlimit library, libipt_connlimit.so, is in /lib/iptables), there is no actual kernel module for it. When attempting to insert a rule with the connlimit module you get 'iptables: Unknown error 4294967295' (this was on a 32-bit machine). So there is a mismatch between kernel and user-space support for modules. The 'connlimit' module became mainstream in kernel 2.6.23. The options were to patch the kernel source tree using 'patch-o-matic' from the netfilter website. But since some doubt has been cast on the stability of these patches by various people, I decided against it, particularly as there is another option. In any case, I am not sure as to whether I was able to deploy a non-standard kernel onto production machines. It certainly would make updating more of a hassle. Inserting a binary module compiled on another devel box could also pose issues, in that there appear to be a lot of changes in the netfilter source tree, and it may invoke functions defined elsewhere lower in the tree than just the xt_connlimit module.
Found this at http://www.debian-administration.org/articles/187
It describes a simple way of connection limiting, using two rules, without using any experimental or recent modules.
iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 50 -j DROP
Found this at http://www.debian-administration.org/articles/187
It describes a simple way of connection limiting, using two rules, without using any experimental or recent modules.
iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --set
iptables -A INPUT -p tcp --dport 80 -i eth0 -m state --state NEW -m recent --update --seconds 60 --hitcount 50 -j DROP
Monday, March 03, 2008
filesystem has unsupported features - fix with debugfs
On occasion I need to build machines with old distributions - usually, this involves copying the disks using tar or rsync onto a server that has been booted off a live CD such as GRML. When making an ext3 filesystem via a newer distribution such as GRML for what will be an older system such as redhat 7.2, certain ext3 features are not supported by the kernel and/or libraries (not sure which), and so it will complain when starting up, though it will still boot. I am not sure if it affects stability of the system. You can however remove the newer features put by there by the new boot CD by using debugfs. debugfs will do this on mounted filesystems as well. As a basis for comparison, I look at what one of the older servers has as the supported features by running debugfs on filesystems there, and then remove the features on the newly-built machine. You may also be able to get supported features in other ways, perhaps via logs or debugfs or researching what features are supported in the particular kernel/libraries/user-space tools or whatever enables the feature. I took the lazy method that is available though!
There is context sensitive
debugfs comes with its own shell, a bit like grub. You can type 'help' to access list of supported commands/operations. For instance, type 'feature' to list supported fs features, and 'feature - etc..' to delete features. Alternatively, you can run commands via the '-R' option from standard bash or other shell, or you can create a file with the operations you want to perform and send them to debugfs.
Very cool :)
There is context sensitive
debugfs comes with its own shell, a bit like grub. You can type 'help' to access list of supported commands/operations. For instance, type 'feature' to list supported fs features, and 'feature -
Very cool :)
Monday, February 18, 2008
converting an existing single disk machine to software RAID-1
The following is a guide based on my experience of setting up RAID 1 (mirroring) on a Debian Sarge system, with many of the packages coming from backports. I needed to use Sarge with backports for various reasons (rather than just using Etch). Anyways, I was able to do this remotely via ssh. It requires reboots, but no need for a boot CD/DVD or relying on back up data stored off the server (though this is a good idea to have in case something goes wrong!!). *PLEASE NOTE*: I suggest it as a guide only as it worked for me, but YMMV and I take no responsibility if you break your system!
These instructions are based on Debian Sarge 3.1, with some stuff from backports, using mdadm
Some of the software versions I used were:
kernel version: linux-image-2.6.18-4-686 (from backports)
initramfs-tools (version 0.85g~bpo.1) from backports
module-init-tools 3.2-pre1-2 (from backports)
udev 0.056-3 (from backports)
So with a 'pure' Sarge system, it may be different.
The hardware:
IBM x3250 with 1 x 160GB SATA disk, to which an identical one was added. These servers accept hot swap drives.
The initial install was done on /dev/sda. The extra (identical) drive was added and machine was rebooted, so that /dev/sdb was now available, though you may use /proc or /sys interface to find the new drive.
After adding the disk, you need to create the same partition table as on the original disk. One easy way to repartition is:
sfdisk -d /dev/sda | sfdisk /dev/sdb
Next, the kernel needs to support software raid. I was using a stock Debian 2.6.18 kernel, a modular kernel that includes the software raid drivers, and uses an initrd. The modules for raid-1 are 'md_mod' and 'raid1'. Use modprobe to load the modules:
modprobe md_mod raid1
There are a number of steps detailed on various other HOWTOs (that I used as a guide) that I found were not necessary for my particular case. In particular, I found that I didn't need to use MAKEDEV to create the device nodes for the raid devices (/dev/md0, /dev/md1, etc)7: udev takes care of that. Also, I did not need to manually configure raid modules to be loaded at boot time, and nor did I need to create or update the initrd to ensure the raid modules were loaded so that the root filesystem could be mounted. The Debian kernels include the raid drivers, but I am not sure what does the autodetection to load the raid drivers for the initrd. At a guess, I would say it finds a raid signature on the drive. I don't think it uses fdisk labels or info from grub config, since I have booted raid systems with no /dev/md0 in the grub config, or disklabel 'fd' on the disks. Obviously, it can't use anything on the filesystem, like /etc/mdadm/mdadm.conf or /etc/modules, as they aren't available before the root filesystem is mounted.
Anyway, on to the next steps. Since we are running on /dev/sda already, the raid needs to be constructed on using only the unused device (we don't want to obliterate the contents of the disk the system is currently using!). In effect, we initially create a degraded array. The original disk will be added later, once we have copied the data to the new disk and booted from it (detailed later).
My disk layout was as follows:
/dev/sda1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sda2 125 327 1630597+ fd Linux raid autodetect
/dev/sda3 328 1215 7132860 fd Linux raid autodetect
/dev/sda4 1216 19457 146528865 5 Extended
/dev/sda5 1216 1944 5855661 fd Linux raid autodetect
/dev/sda6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sda7 3525 19457 127981791 fd Linux raid autodetect
Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sdb2 125 327 1630597+ fd Linux raid autodetect
/dev/sdb3 328 1215 7132860 fd Linux raid autodetect
/dev/sdb4 1216 19457 146528865 5 Extended
/dev/sdb5 1216 1944 5855661 fd Linux raid autodetect
/dev/sdb6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sdb7 3525 19457 127981791 fd Linux raid autodetect
Initially, the partition type was 'Linux' (code 83 in fdisk), but I changed it to fdisk code 'fd'. As before, I don't think this is necessary at least initially (and the system was able to boot fine without those labels on a raid setup), but it might be useful or needed for other things anyway.
Create the raid devices:
mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md1 --level 1 --raid-devices=2 missing /dev/sdb3
and so on.
Once each raid device was created, I then made a filesystem 'mkfs.ext3 /dev/md0'. Do this for each raid device. Also, run a 'mkswap' on the new swap partition on /dev/sdb. If a disk fails, I think the machine should be able to boot ok, despite one of the swap partitions being dead. Before copying data, /etc/fstab and /boot/grub/menu.lst will need to be modified, and it may not hurt to put some details in /etc/mdadm/mdadm.conf (I am not sure if /etc/mdadm/mdadm.conf is actually needed for anything. test without it)
/etc/fstab - the new mountpoints + swap space:
### END DEBIAN AUTOMAGIC KERNELS LIST
NEW-mydns01:/var/lib# cat /etc/fstab
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/md0 / ext3 defaults,errors=remount-ro 0 1
/dev/md2 /home ext3 defaults 0 2
/dev/md1 /usr ext3 defaults 0 2
/dev/md3 /var ext3 defaults 0 2
/dev/md4 /var/lib/mysql ext3 defaults 0 2
/dev/sda1 none swap sw 0 0
/dev/sdb1 none swap sw 0 0
/dev/hda /media/cdrom0 udf,iso9660 user,noauto 0 0
/boot/grub/menu.lst - Important to have root filesystem as the raid device (in my case, /dev/md0), else you won't be able to hotadd the old disk (see further below for explanation). So the relevant part of /boot/grub/menu.lst looks like:
title Debian GNU/Linux, kernel 2.6.18-4-686
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro
initrd /boot/initrd.img-2.6.18-4-686
savedefault
title Debian GNU/Linux, kernel 2.6.18-4-686 (single-user mode)
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro single
initrd /boot/initrd.img-2.6.18-4-686
savedefault
/etc/mdadm/mdadm.conf:
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b196c373:f65bec91:01072554:c62097c9
devices=/dev/sdb2
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=47053389:84f72deb:b47832ba:1ec3ebc1
devices=/dev/sdb3
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=1c8ff28f:f4dae79a:6cbc8acd:caaf42cb
devices=/dev/sdb5
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=ae3882ea:df0f6494:d6b68701:1f922018
devices=/dev/sdb6
ARRAY /dev/md4 level=raid1 num-devices=2 UUID=ab8471a4:1d7d7959:922ec705:4b988a14
devices=/dev/sdb7
Note that there is only /dev/sdb listed here, not /dev/sda. That will be added later.
I am not sure what actually uses this file.
Copying data
rsync is good for this, though you could otherwise use tar or cp. I found one oversight on my part, which was that the mount point /var/lib/mysql was copied with ownership root:root and mode 755. This breaks mysql, so I think something in my rsync was incorrect
Assuming /dev/md0 is mounted at /mnt/root:
rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/mnt/* / /mnt/root
/dev/md1 will be the new /usr partition, and is mounted at /mnt/usr:
rsync -auHxv /usr/* /mnt/usr
And so on.
Boot block needs to be installed on each disk. Install it on /dev/sdb with grub
grub>root (hd1,1)
grub>setup (hd1)
It will need to find the root of the grub install, otherwise it will complain. But the rsync we did earlier should have taken car of that.
Reboot with changed fstab. Server should be running on 'degraded' raid running off /dev/sdb. Now /dev/sda needs to be added to the array. Use mdadm to do this, and this will copy the data from /dev/sdb over to /dev/sda (wiping whatever was on there)
mdadm /dev/md0 -a /dev/sda2
Due to a step I missed, I actually was using /dev/sda2 as the root filesystem in grub, though the raid was mounting '/' on /dev/md0. This caused an error when I tried to add /dev/sda to the raid:
mdadm: hot add failed for /dev/sda2: Invalid argument
/var/log/messages:
Feb 18 12:41:51 mydns01 kernel: md: error, md_import_device() returned -16
After much searching, I found that the issue was that the machine had root=/dev/sda2 in the grub config, and it will not allow you to use this device. I have read that this is because the device is in use, even though it appears not to be (since you have booted from /dev/sdb). Someone with a similar issue pointed out in dmesg:
Kernel command line: ro root=/dev/sda2
Anyway, change it to /dev/md0 in grub, and problem solved!
These instructions are based on Debian Sarge 3.1, with some stuff from backports, using mdadm
Some of the software versions I used were:
kernel version: linux-image-2.6.18-4-686 (from backports)
initramfs-tools (version 0.85g~bpo.1) from backports
module-init-tools 3.2-pre1-2 (from backports)
udev 0.056-3 (from backports)
So with a 'pure' Sarge system, it may be different.
The hardware:
IBM x3250 with 1 x 160GB SATA disk, to which an identical one was added. These servers accept hot swap drives.
The initial install was done on /dev/sda. The extra (identical) drive was added and machine was rebooted, so that /dev/sdb was now available, though you may use /proc or /sys interface to find the new drive.
After adding the disk, you need to create the same partition table as on the original disk. One easy way to repartition is:
sfdisk -d /dev/sda | sfdisk /dev/sdb
Next, the kernel needs to support software raid. I was using a stock Debian 2.6.18 kernel, a modular kernel that includes the software raid drivers, and uses an initrd. The modules for raid-1 are 'md_mod' and 'raid1'. Use modprobe to load the modules:
modprobe md_mod raid1
There are a number of steps detailed on various other HOWTOs (that I used as a guide) that I found were not necessary for my particular case. In particular, I found that I didn't need to use MAKEDEV to create the device nodes for the raid devices (/dev/md0, /dev/md1, etc)7: udev takes care of that. Also, I did not need to manually configure raid modules to be loaded at boot time, and nor did I need to create or update the initrd to ensure the raid modules were loaded so that the root filesystem could be mounted. The Debian kernels include the raid drivers, but I am not sure what does the autodetection to load the raid drivers for the initrd. At a guess, I would say it finds a raid signature on the drive. I don't think it uses fdisk labels or info from grub config, since I have booted raid systems with no /dev/md0 in the grub config, or disklabel 'fd' on the disks. Obviously, it can't use anything on the filesystem, like /etc/mdadm/mdadm.conf or /etc/modules, as they aren't available before the root filesystem is mounted.
Anyway, on to the next steps. Since we are running on /dev/sda already, the raid needs to be constructed on using only the unused device (we don't want to obliterate the contents of the disk the system is currently using!). In effect, we initially create a degraded array. The original disk will be added later, once we have copied the data to the new disk and booted from it (detailed later).
My disk layout was as follows:
/dev/sda1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sda2 125 327 1630597+ fd Linux raid autodetect
/dev/sda3 328 1215 7132860 fd Linux raid autodetect
/dev/sda4 1216 19457 146528865 5 Extended
/dev/sda5 1216 1944 5855661 fd Linux raid autodetect
/dev/sda6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sda7 3525 19457 127981791 fd Linux raid autodetect
Disk /dev/sdb: 160.0 GB, 160041885696 bytes
255 heads, 63 sectors/track, 19457 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 * 1 124 995998+ 82 Linux swap / Solaris
/dev/sdb2 125 327 1630597+ fd Linux raid autodetect
/dev/sdb3 328 1215 7132860 fd Linux raid autodetect
/dev/sdb4 1216 19457 146528865 5 Extended
/dev/sdb5 1216 1944 5855661 fd Linux raid autodetect
/dev/sdb6 1945 3524 12691318+ fd Linux raid autodetect
/dev/sdb7 3525 19457 127981791 fd Linux raid autodetect
Initially, the partition type was 'Linux' (code 83 in fdisk), but I changed it to fdisk code 'fd'. As before, I don't think this is necessary at least initially (and the system was able to boot fine without those labels on a raid setup), but it might be useful or needed for other things anyway.
Create the raid devices:
mdadm --create /dev/md0 --level 1 --raid-devices=2 missing /dev/sdb2
mdadm --create /dev/md1 --level 1 --raid-devices=2 missing /dev/sdb3
and so on.
Once each raid device was created, I then made a filesystem 'mkfs.ext3 /dev/md0'. Do this for each raid device. Also, run a 'mkswap' on the new swap partition on /dev/sdb. If a disk fails, I think the machine should be able to boot ok, despite one of the swap partitions being dead. Before copying data, /etc/fstab and /boot/grub/menu.lst will need to be modified, and it may not hurt to put some details in /etc/mdadm/mdadm.conf (I am not sure if /etc/mdadm/mdadm.conf is actually needed for anything. test without it)
/etc/fstab - the new mountpoints + swap space:
### END DEBIAN AUTOMAGIC KERNELS LIST
NEW-mydns01:/var/lib# cat /etc/fstab
# /etc/fstab: static file system information.
#
#
proc /proc proc defaults 0 0
/dev/md0 / ext3 defaults,errors=remount-ro 0 1
/dev/md2 /home ext3 defaults 0 2
/dev/md1 /usr ext3 defaults 0 2
/dev/md3 /var ext3 defaults 0 2
/dev/md4 /var/lib/mysql ext3 defaults 0 2
/dev/sda1 none swap sw 0 0
/dev/sdb1 none swap sw 0 0
/dev/hda /media/cdrom0 udf,iso9660 user,noauto 0 0
/boot/grub/menu.lst - Important to have root filesystem as the raid device (in my case, /dev/md0), else you won't be able to hotadd the old disk (see further below for explanation). So the relevant part of /boot/grub/menu.lst looks like:
title Debian GNU/Linux, kernel 2.6.18-4-686
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro
initrd /boot/initrd.img-2.6.18-4-686
savedefault
title Debian GNU/Linux, kernel 2.6.18-4-686 (single-user mode)
root (hd0,1)
kernel /boot/vmlinuz-2.6.18-4-686 root=/dev/md0 ro single
initrd /boot/initrd.img-2.6.18-4-686
savedefault
/etc/mdadm/mdadm.conf:
mdadm --detail --scan >> /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=b196c373:f65bec91:01072554:c62097c9
devices=/dev/sdb2
ARRAY /dev/md1 level=raid1 num-devices=2 UUID=47053389:84f72deb:b47832ba:1ec3ebc1
devices=/dev/sdb3
ARRAY /dev/md2 level=raid1 num-devices=2 UUID=1c8ff28f:f4dae79a:6cbc8acd:caaf42cb
devices=/dev/sdb5
ARRAY /dev/md3 level=raid1 num-devices=2 UUID=ae3882ea:df0f6494:d6b68701:1f922018
devices=/dev/sdb6
ARRAY /dev/md4 level=raid1 num-devices=2 UUID=ab8471a4:1d7d7959:922ec705:4b988a14
devices=/dev/sdb7
Note that there is only /dev/sdb listed here, not /dev/sda. That will be added later.
I am not sure what actually uses this file.
Copying data
rsync is good for this, though you could otherwise use tar or cp. I found one oversight on my part, which was that the mount point /var/lib/mysql was copied with ownership root:root and mode 755. This breaks mysql, so I think something in my rsync was incorrect
Assuming /dev/md0 is mounted at /mnt/root:
rsync -auHxv --exclude=/proc/* --exclude=/sys/* --exclude=/mnt/* / /mnt/root
/dev/md1 will be the new /usr partition, and is mounted at /mnt/usr:
rsync -auHxv /usr/* /mnt/usr
And so on.
Boot block needs to be installed on each disk. Install it on /dev/sdb with grub
grub>root (hd1,1)
grub>setup (hd1)
It will need to find the root of the grub install, otherwise it will complain. But the rsync we did earlier should have taken car of that.
Reboot with changed fstab. Server should be running on 'degraded' raid running off /dev/sdb. Now /dev/sda needs to be added to the array. Use mdadm to do this, and this will copy the data from /dev/sdb over to /dev/sda (wiping whatever was on there)
mdadm /dev/md0 -a /dev/sda2
Due to a step I missed, I actually was using /dev/sda2 as the root filesystem in grub, though the raid was mounting '/' on /dev/md0. This caused an error when I tried to add /dev/sda to the raid:
mdadm: hot add failed for /dev/sda2: Invalid argument
/var/log/messages:
Feb 18 12:41:51 mydns01 kernel: md: error, md_import_device() returned -16
After much searching, I found that the issue was that the machine had root=/dev/sda2 in the grub config, and it will not allow you to use this device. I have read that this is because the device is in use, even though it appears not to be (since you have booted from /dev/sdb). Someone with a similar issue pointed out in dmesg:
Kernel command line: ro root=/dev/sda2
Anyway, change it to /dev/md0 in grub, and problem solved!
Tuesday, January 29, 2008
plesk config information
besides the usual text files in /etc/, /usr/local/psa, some of it is stored in mysql in the psa DB, which can be accessed via Plesk control panel via phpmyadmin
Saturday, December 15, 2007
UUID grub fstab
I wanted to migrate my data to a SATA disk from an IDE disk on an Ubuntu 7.10 system. In the past, I have simply tarred the contents of the partitions to the new disk, updated fstab and bootloader configs, reinstalled the MBR, and rebooted. However, in this instance, since the disk was laid out the same, I didn't really need to update partition info in grub config and fstab. If I had, I would have noticed that Ubuntu uses UUIDs for fstab and the grub config for identifying partitions. So I found myself with a system unable to get past the bootloader. My 'quick' fix was to use an Ubuntu CD in rescue mode and change /etc/fstab and /boot/grub/menu.lst to the 'old' style of /dev/sda1 etc. I then acquainted myself with the concept of UUID ('apropos UUID') plus some web searches. I found that you can find the UUID of a partition by running 'vol_id -u ', e.g., 'vol_id -u /dev/sda1'. I then copied this to fstab and menu.lst.
I think that UUID is handy way of doing things, in that it is possible to attach another disk to a machine without worrying whether it will change the order of things (e.g., by becoming the first scsi disk, and your system disk becomes the second disk, whilst all the fstab and grub entries point to the first still). It is still a slight mystery to me why the BIOS is able to boot from the second disk without modification though. Also, I am not sure why grub, when configured with (hd0) after adding a new disk which then becomes primary, isn't bothered. Maybe because this is just in the initial setup, and the bootloader is installed there anyway.
I think that UUID is handy way of doing things, in that it is possible to attach another disk to a machine without worrying whether it will change the order of things (e.g., by becoming the first scsi disk, and your system disk becomes the second disk, whilst all the fstab and grub entries point to the first still). It is still a slight mystery to me why the BIOS is able to boot from the second disk without modification though. Also, I am not sure why grub, when configured with (hd0) after adding a new disk which then becomes primary, isn't bothered. Maybe because this is just in the initial setup, and the bootloader is installed there anyway.
Wednesday, September 19, 2007
running out of memory when copying large files
From coraid support:
5.19 Q: How can I avoid running out of memory when copying large files?
A: You can tell the Linux kernel not to wait so long before writing data out to backing storage.
echo 5 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio
These settings are even tighter and should help even on a system that is doing a large amount of larger-than-RAM data transfers.
echo 3 > /proc/sys/vm/dirty_ratio
echo 3 > /proc/sys/vm/dirty_background_ratio
echo 5120 > /proc/sys/vm/min_free_kbytes
5.19 Q: How can I avoid running out of memory when copying large files?
A: You can tell the Linux kernel not to wait so long before writing data out to backing storage.
echo 5 > /proc/sys/vm/dirty_ratio
echo 5 > /proc/sys/vm/dirty_background_ratio
These settings are even tighter and should help even on a system that is doing a large amount of larger-than-RAM data transfers.
echo 3 > /proc/sys/vm/dirty_ratio
echo 3 > /proc/sys/vm/dirty_background_ratio
echo 5120 > /proc/sys/vm/min_free_kbytes
Thursday, August 16, 2007
mysql replication
If mysql replication breaks with:
070816 13:03:17 Slave I/O thread: connected to master 'repl@mydns01.securepod.com:3306', replication started in log 'mydns01-bin.025' at position 20533567
070816 13:03:17 Error reading packet from server: Client requested master to start replication from impossible position (server_errno=1236)
070816 13:03:17 Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log'
070816 13:03:17 Slave I/O thread exiting, read up to log 'mydns01-bin.025', position 20533567
you can check the binlog that it last read up to (mydns01-bin.025), using mysqlbinlog, and search for log position it read up to (in this case 20533567). In this case, the last entry in mydns01-bin.025 is 20533147
mysqlbinlog mydns01-bin.025 | tail -10
#070815 16:01:10 server id 1 log_pos 20533147 Query thread_id=20094 exec_time=0 error_code=0
SET TIMESTAMP=1187211670;
INSERT INTO ZoneRecord
( ZoneRecord.zone, ZoneRecord.name, ZoneRecord.type, ZoneRecord.data, ZoneRecord.aux, ZoneRecord.ttl, ZoneRecord.active, ZoneRecord.categoryid )
VALUES ( '167857' , '' , 'NS' , 'ns1.securepod.com.' , 'None' , '14400' , '1' , '1' );
The next bin log mydns01-bin.026 starts from a new position (this position actually corresponds to the file size of the binlog). So: you can run the query that the last entry has in bin.025 and see if it exists, and run the query in bin.026 and see if it exists. If the first does, but the second does not, then you can change the bin log start position from the first position in bin.026, which is as easy as changing the master.info on the slave(s) to the new value and restarting mysql (or you could do it on the mysql command line if you can be bothered)
070816 13:03:17 Slave I/O thread: connected to master 'repl@mydns01.securepod.com:3306', replication started in log 'mydns01-bin.025' at position 20533567
070816 13:03:17 Error reading packet from server: Client requested master to start replication from impossible position (server_errno=1236)
070816 13:03:17 Got fatal error 1236: 'Client requested master to start replication from impossible position' from master when reading data from binary log'
070816 13:03:17 Slave I/O thread exiting, read up to log 'mydns01-bin.025', position 20533567
you can check the binlog that it last read up to (mydns01-bin.025), using mysqlbinlog, and search for log position it read up to (in this case 20533567). In this case, the last entry in mydns01-bin.025 is 20533147
mysqlbinlog mydns01-bin.025 | tail -10
#070815 16:01:10 server id 1 log_pos 20533147 Query thread_id=20094 exec_time=0 error_code=0
SET TIMESTAMP=1187211670;
INSERT INTO ZoneRecord
( ZoneRecord.zone, ZoneRecord.name, ZoneRecord.type, ZoneRecord.data, ZoneRecord.aux, ZoneRecord.ttl, ZoneRecord.active, ZoneRecord.categoryid )
VALUES ( '167857' , '' , 'NS' , 'ns1.securepod.com.' , 'None' , '14400' , '1' , '1' );
The next bin log mydns01-bin.026 starts from a new position (this position actually corresponds to the file size of the binlog). So: you can run the query that the last entry has in bin.025 and see if it exists, and run the query in bin.026 and see if it exists. If the first does, but the second does not, then you can change the bin log start position from the first position in bin.026, which is as easy as changing the master.info on the slave(s) to the new value and restarting mysql (or you could do it on the mysql command line if you can be bothered)
Tuesday, July 10, 2007
debootstrap bootstraps a basic Debian system of SUITE (eg, sarge, etch, sid) into TARGET from MIRROR by running
SCRIPT. MIRROR can be an http:// URL or a file:/// URL. Notice that file:/ URLs are translated to file:/// (cor‐
rect scheme as described in RFC1738 for local filenames), and file:// will not work.
Debootstrap can be used to install Debian in a system without using an installation disk but can also be used to
run a different Debian flavor in a chroot environment. This way you can create a full (minimal) Debian installation
which can be used for testing purposes (see the EXAMPLES section). If you are looking for a chroot system to build
packages please take a look at pbuilder.
SCRIPT. MIRROR can be an http:// URL or a file:/// URL. Notice that file:/ URLs are translated to file:/// (cor‐
rect scheme as described in RFC1738 for local filenames), and file:// will not work.
Debootstrap can be used to install Debian in a system without using an installation disk but can also be used to
run a different Debian flavor in a chroot environment. This way you can create a full (minimal) Debian installation
which can be used for testing purposes (see the EXAMPLES section). If you are looking for a chroot system to build
packages please take a look at pbuilder.
Monday, June 25, 2007
differences between openvz and virtuozzo
openvz doesn't have vzfs, for one
Virtuozzo File System (VZFS) :
VZFS is a file system that allows to share common files among multiple VPSs without sacrificing flexibility. It is possible for VPS users to modify, update, replace, and delete shared files. When a user modifies a shared file, VZFS creates a private copy of the file transparently for the user. Thus, the modifications do not affect the other users of the file. Main benefits of VZFS are the following:
- It saves memory required for executables and libraries. A typical VPS running a simple web site might consume around 20–30 MBytes of RAM just for executable images. Sharing this memory improves scalability and total system performance;
- It saves disk space. A typical Linux server installation occupies several hundred MBytes of disk space. Sharing the files allows you to save up to 90% of disk space;
- VZFS does not require having different physical partitions for different VPSs or creating a special “file system in a file” setup for a VPS. This significantly simplifies disk administration;
- Disk quota enables the administrator to limit disk resources available to a VPS on-the-fly, in the same manner as the standard disk quota system works on a per-user basis. Disk quota for users and groups inside VPSs is also supported.
Virtuozzo File System (VZFS) :
VZFS is a file system that allows to share common files among multiple VPSs without sacrificing flexibility. It is possible for VPS users to modify, update, replace, and delete shared files. When a user modifies a shared file, VZFS creates a private copy of the file transparently for the user. Thus, the modifications do not affect the other users of the file. Main benefits of VZFS are the following:
- It saves memory required for executables and libraries. A typical VPS running a simple web site might consume around 20–30 MBytes of RAM just for executable images. Sharing this memory improves scalability and total system performance;
- It saves disk space. A typical Linux server installation occupies several hundred MBytes of disk space. Sharing the files allows you to save up to 90% of disk space;
- VZFS does not require having different physical partitions for different VPSs or creating a special “file system in a file” setup for a VPS. This significantly simplifies disk administration;
- Disk quota enables the administrator to limit disk resources available to a VPS on-the-fly, in the same manner as the standard disk quota system works on a per-user basis. Disk quota for users and groups inside VPSs is also supported.
Tuesday, June 12, 2007
blocking spam with sendmail
Can block by domain or by IP address or network
in /etc/mail/access:
localhost.localdomain RELAY
localhost RELAY
127.0.0.1 RELAY
192.168.1 RELAY
202.124.241.222 ERROR:"550 Take a hike"
As this is a hash, rebuild to db with:
hash /etc/mail/access < /etc/mail/access
in /etc/mail/access:
localhost.localdomain RELAY
localhost RELAY
127.0.0.1 RELAY
192.168.1 RELAY
202.124.241.222 ERROR:"550 Take a hike"
As this is a hash, rebuild to db with:
hash /etc/mail/access < /etc/mail/access
Useful way of telling php module support
php -m
or
php5 -m
php -v shows version
Also, you can do: 'php /home/cm/phpinfo.php'
or
php5 -m
php -v shows version
Also, you can do: 'php /home/cm/phpinfo.php'
Thursday, May 24, 2007
ubuntu - BUG: soft lockup detected on CPU#0!
My Ubuntu (Ubuntu 7.04, AMD 1800XP, 2.6.20-15-generic #2 SMP Sun Apr 15 07:36:31 UTC 2007 i686 GNU/Linux) box would get halfway through booting and just sit there, doing nothing. ctrl+alt+del didn't do anything, and I couldn't get a console. To get more information, I rebooted and edited grub cmd line, taking off the 'quiet splash' options. It would get most of the way through fscking the first partition and then report: BUG: soft lockup detected on CPU#0! Not sure what causes this exactly, but I did a quick search and on some of the forums, a lot of people mentioned wireless network card as a possible source. The machine has got a wireless network card (it is a RaLink RT2561/RT61 802.11g PCI chipset), but not sure if this has anything to do with it. After a couple of futile soft reboots, I pulled the power and checked that the wireless card was properly seated, which it was, and booted the machine. It worked fine. I should try and find more out about what the error means though.
Sunday, May 13, 2007
setting up ldapv3 under Ubuntu 7.04
I had numerous problems getting LDAP running under Ubuntu 7.04. Ubuntu 7.04 comes with slapd v2.3.30 and libldap2.3. I was setting it up as a precursor to running postfix and courier with virtual domain support. The main issue I had was using SASL, which can be disabled I think, but I wanted to use it just as a challenge (and it may be required for LDAP to be fully v3 compliant). I have read that no MTA currently supports SASL anyway, they are all v2 compliant only. I previously used LDAP v2 under Debian Sarge, which does not require SASL, and is very easy to get running out of the box. In any case, as MTAs auth via v2 only, I think you need to enable v2 bind support in slapd.conf even if you are running LDAP v3.
It is a very good idea to read the guide at www.openldap.org, as well as the relevant man pages, of which there are numerous. The documentation requires very careful attention.
I found a few useful pages, beside the official openldap documentation. Here's one that has been translated from japanese:
http://www.tom.sfc.keio.ac.jp/~torry/ldap/ldap_en.html#doc4_15616
I didn't delete the sasldb2 file as the author suggests though
Another one:
http://defindit.com/readme_files/ldap.html
And a pdf called 'Surviving Cyrus SASL':
http://postfix.state-of-mind.de/patrick.koetter/surviving_cyrus_sasl.pdf
A bit about SASL. See the wikipedia entry for background.
LDAP specific:
SASL - you can authenticate requests via SASL (for e.g., ldapdelete or other write operations), which looks at a password in /etc/sasldb2, or you can authenticate from info in the LDAP db, as you would other users. I'm still not 100% clear on whether you need to use SASL or not, and exactly what is required to have it working (some of my config options like sasl-realm and sasl-host may be superfluous, and I didn't use a slapd.conf file in /usr/lib/sasl2, etc etc)
I am not sure what auth mechanisms you need to set up.
I found that if you do specify a dn (i.e., with the '-D' flag), it will attempt to bind against what is in the LDAP db, e.g.,:
ldapdelete -x -D "cn=admin,dc=cm,dc=net" -W "ou=people,dc=cm,dc=net" -h localhost
but if you don't specify a dn, it will use SASL:
root@cm:/home/cm# ldapdelete -U admin "ou=people,dc=cm,dc=net" -h localhost
SASL/DIGEST-MD5 authentication started
Please enter your password:
SASL username: admin
SASL SSF: 128
SASL installing layers
Note that while SASL provides an *authentication* mechanism, that user still needs permission to be able to do things such as read or write to the LDAP db (authorization). So, you still need entries in the LDAP db allowing authenticated users permission to read/write/etc to LDAP (this is the same as with earlier versions of LDAP of course). Under v3, you could set LDAP up so that it authenticates via SASL and then uses the ACLs in the LDAP db to do control access.
cm@cm:~$ ldapadd -f 2nd.ldif -h localhost
SASL/DIGEST-MD5 authentication started
Please enter your password:
SASL username: cm
SASL SSF: 128
SASL installing layers
adding new entry "ou=People, dc=cm,dc=net"
ldap_add: Insufficient access (50)
additional info: no write access to parent
Note that you must specify the username (unless you are that user)
direct mapping vs search-based mapping
direct mapping avoids having to do an LDAP lookup. It uses the 'authz-regexp' or 'sasl-regexp' to re-write requests sent to it ('authz-regexp' and 'sasl-regexp' are the same AFAIK. authz is short for authorization, probably to distinguish it from authentication. I could not find any reference anywhere to 'sasl-regexp' using an apropos. But I was able to use both in the slapd.conf interchangeably. And the debug log shows sasl-regexp as 'slap_authz_regexp'
LDAP SASL library calls from the debug log:
May 13 22:53:09 cm slapd[24029]: do_sasl_bind: dn () mech DIGEST-MD5
May 13 22:53:09 cm slapd[24029]: SASL [conn=8] Debug: DIGEST-MD5 server step 2
May 13 22:53:09 cm slapd[24029]: slap_sasl_getdn: u:id converted to uid=admin,cn=cm.net,cn=DIGEST-MD5,cn=auth
search-mapping binds to the LDAP server and looks up the user via LDAP calls. It is more 'expensive', as you have to search the DIT
It is a very good idea to read the guide at www.openldap.org, as well as the relevant man pages, of which there are numerous. The documentation requires very careful attention.
I found a few useful pages, beside the official openldap documentation. Here's one that has been translated from japanese:
http://www.tom.sfc.keio.ac.jp/~torry/ldap/ldap_en.html#doc4_15616
I didn't delete the sasldb2 file as the author suggests though
Another one:
http://defindit.com/readme_files/ldap.html
And a pdf called 'Surviving Cyrus SASL':
http://postfix.state-of-mind.de/patrick.koetter/surviving_cyrus_sasl.pdf
A bit about SASL. See the wikipedia entry for background.
LDAP specific:
SASL - you can authenticate requests via SASL (for e.g., ldapdelete or other write operations), which looks at a password in /etc/sasldb2, or you can authenticate from info in the LDAP db, as you would other users. I'm still not 100% clear on whether you need to use SASL or not, and exactly what is required to have it working (some of my config options like sasl-realm and sasl-host may be superfluous, and I didn't use a slapd.conf file in /usr/lib/sasl2, etc etc)
I am not sure what auth mechanisms you need to set up.
I found that if you do specify a dn (i.e., with the '-D' flag), it will attempt to bind against what is in the LDAP db, e.g.,:
ldapdelete -x -D "cn=admin,dc=cm,dc=net" -W "ou=people,dc=cm,dc=net" -h localhost
but if you don't specify a dn, it will use SASL:
root@cm:/home/cm# ldapdelete -U admin "ou=people,dc=cm,dc=net" -h localhost
SASL/DIGEST-MD5 authentication started
Please enter your password:
SASL username: admin
SASL SSF: 128
SASL installing layers
Note that while SASL provides an *authentication* mechanism, that user still needs permission to be able to do things such as read or write to the LDAP db (authorization). So, you still need entries in the LDAP db allowing authenticated users permission to read/write/etc to LDAP (this is the same as with earlier versions of LDAP of course). Under v3, you could set LDAP up so that it authenticates via SASL and then uses the ACLs in the LDAP db to do control access.
cm@cm:~$ ldapadd -f 2nd.ldif -h localhost
SASL/DIGEST-MD5 authentication started
Please enter your password:
SASL username: cm
SASL SSF: 128
SASL installing layers
adding new entry "ou=People, dc=cm,dc=net"
ldap_add: Insufficient access (50)
additional info: no write access to parent
Note that you must specify the username (unless you are that user)
direct mapping vs search-based mapping
direct mapping avoids having to do an LDAP lookup. It uses the 'authz-regexp' or 'sasl-regexp' to re-write requests sent to it ('authz-regexp' and 'sasl-regexp' are the same AFAIK. authz is short for authorization, probably to distinguish it from authentication. I could not find any reference anywhere to 'sasl-regexp' using an apropos. But I was able to use both in the slapd.conf interchangeably. And the debug log shows sasl-regexp as 'slap_authz_regexp'
LDAP SASL library calls from the debug log:
May 13 22:53:09 cm slapd[24029]: do_sasl_bind: dn () mech DIGEST-MD5
May 13 22:53:09 cm slapd[24029]: SASL [conn=8] Debug: DIGEST-MD5 server step 2
May 13 22:53:09 cm slapd[24029]: slap_sasl_getdn: u:id converted to uid=admin,cn=cm.net,cn=DIGEST-MD5,cn=auth
search-mapping binds to the LDAP server and looks up the user via LDAP calls. It is more 'expensive', as you have to search the DIT
Tuesday, May 08, 2007
Linux-HA and heartbeat
http://www.linux-ha.org/
The goal of the Linux HA project is to provide 'a high availability (clustering) solution for Linux which promotes reliability, availability...'
The Linux-HA project's main software product is Heartbeat, which is a cluster management program for high availability clustering. Some of the features of heartbeat include:
- no fixed number of nodes
- resource monitoring
- fencing
Linux-HA state that their software (Heartbeat) is well integrated with separate projects like LVS and DRBD
Heartbeatv2 can do resource monitoring of Lustre for example, by using /etc/init.d/lustre with the 'status' argument (I think it looks in /proc or /sys), see https://mail.clusterfs.com/pipermail/lustre-discuss/2005-September/000870.html
The goal of the Linux HA project is to provide 'a high availability (clustering) solution for Linux which promotes reliability, availability...'
The Linux-HA project's main software product is Heartbeat, which is a cluster management program for high availability clustering. Some of the features of heartbeat include:
- no fixed number of nodes
- resource monitoring
- fencing
Linux-HA state that their software (Heartbeat) is well integrated with separate projects like LVS and DRBD
Heartbeatv2 can do resource monitoring of Lustre for example, by using /etc/init.d/lustre with the 'status' argument (I think it looks in /proc or /sys), see https://mail.clusterfs.com/pipermail/lustre-discuss/2005-September/000870.html
keepalived
keepalived has been primarily developed as a means to provide high availability for LVS. It does this via health checks of protocols and services, and informs the kernel in case of failure (or if it becomes available again), and it uses a version of the VRRP to handle failover for the LVS director. (VRRP is an open standard based on Cisco's HSRP)
-It runs as a user-space program
-It checks multiple layers of the TCP stack (i.e., based on the OSI 7 layer model). It does checks at the IP level, tcp level, transport dialogue layer (5), and application layer.
"Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5/7. This framework gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the linux kernel via a setsockopt call to remove this server entry from the LVS topology. In addition keepalived implements an independent VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover."
-It runs as a user-space program
-It checks multiple layers of the TCP stack (i.e., based on the OSI 7 layer model). It does checks at the IP level, tcp level, transport dialogue layer (5), and application layer.
"Keepalived implements a framework based on three family checks : Layer3, Layer4 & Layer5/7. This framework gives the daemon the ability of checking a LVS server pool states. When one of the server of the LVS server pool is down, keepalived informs the linux kernel via a setsockopt call to remove this server entry from the LVS topology. In addition keepalived implements an independent VRRPv2 stack to handle director failover. So in short keepalived is a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover."
LVS
LVS is described on the home page of the project as a 'load balancer'. The aim is to provide scalability for services. The main product of the LVS project is IPVS, which provides Layer-4 (transport layer) load balancing inside the kernel.
From the home page:
"IPVS implements transport-layer load balancing (layer-4 switching) inside the Linux kernel"
They also have under development Layer-7 (application level) switching inside the kernel, using KTCPVS
To provide redundancy, their are several possibilities listed on their home page.
- pirhana
- ultramonkey
- keepalived
- heartbeat + mon + coda
- heartbeat + ldirectord
I have set up a cluster using the last option. The 2nd is what nr uses.
From the home page:
"IPVS implements transport-layer load balancing (layer-4 switching) inside the Linux kernel"
They also have under development Layer-7 (application level) switching inside the kernel, using KTCPVS
To provide redundancy, their are several possibilities listed on their home page.
- pirhana
- ultramonkey
- keepalived
- heartbeat + mon + coda
- heartbeat + ldirectord
I have set up a cluster using the last option. The 2nd is what nr uses.
Wednesday, April 25, 2007
lustre failover experimentation
'lconf --failover' didn't seem to work, but 'lconf --cleanup --force --service=mds /root/config.xml' did. It removed all the modules. Once I was satisfied that mds-1 was not using the device, I started the failover device, mds-2, just by running 'lconf --node mds-2 /root/config.xml'.
On the client that was mounting the resource:
LustreError: 19533:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout (sen t at 1177507980, 0s ago) req@f7d55600 x8852471556/t0 o400->mds_UUID@mds-1_UUID:1 2 lens 64/64 ref 1 fl Rpc:N/0/0 rc 0/0
LustreError: 19533:0:(client.c:940:ptlrpc_expire_one_request()) Skipped 29 previ ous similar messages
Lustre: 12:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked LNET upcall /usr/lib /lustre/lnet_upcall ROUTER_NOTIFY,192.168.241.229@tcp,down,1177507956
LustreError: MDC_mds-1_mds_MNT_client-f70a0400: Connection to service mds via ni d 192.168.241.229@tcp was lost; in progress operations using this service will w ait for recovery to complete.
Lustre: Changing connection for MDC_mds-1_mds_MNT_client-f70a0400 to mds-2_UUID/192.168.241.227@tcp
The share was fine from the client after the switchover.
There were no messages on the OSSes. But I did this:
oss-4:/home/cmcleay/lustre-1.4.10# lctl ping mds-2
12345-0@lo
12345-192.168.241.227@tcp
oss-4:/home/cmcleay/lustre-1.4.10# lctl ping mds-1
failed to ping 192.168.241.229@tcp: Input/output error
I used the init scripts after I was satisfied with this - all worked well.
Doing an 'ls' on the share hung, but it came back after a little while after starting the failover mds-1
You need to have the config file in the right place for the init scripts to work properly
There were some messages on the oss:
Apr 25 23:40:48 oss-4 kernel: Lustre: 6:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,192.168.241.229@tcp,down,1176793786
Apr 25 23:44:11 oss-4 kernel: Lustre: 17433:0:(filter.c:3236:filter_set_info_async()) ost-beta: received MDS connection from 192.168.241.229@tcp
Then it all stopped working :(
On the client that was mounting the resource:
LustreError: 19533:0:(client.c:940:ptlrpc_expire_one_request()) @@@ timeout (sen t at 1177507980, 0s ago) req@f7d55600 x8852471556/t0 o400->mds_UUID@mds-1_UUID:1 2 lens 64/64 ref 1 fl Rpc:N/0/0 rc 0/0
LustreError: 19533:0:(client.c:940:ptlrpc_expire_one_request()) Skipped 29 previ ous similar messages
Lustre: 12:0:(linux-debug.c:96:libcfs_run_upcall()) Invoked LNET upcall /usr/lib /lustre/lnet_upcall ROUTER_NOTIFY,192.168.241.229@tcp,down,1177507956
LustreError: MDC_mds-1_mds_MNT_client-f70a0400: Connection to service mds via ni d 192.168.241.229@tcp was lost; in progress operations using this service will w ait for recovery to complete.
Lustre: Changing connection for MDC_mds-1_mds_MNT_client-f70a0400 to mds-2_UUID/192.168.241.227@tcp
The share was fine from the client after the switchover.
There were no messages on the OSSes. But I did this:
oss-4:/home/cmcleay/lustre-1.4.10# lctl ping mds-2
12345-0@lo
12345-192.168.241.227@tcp
oss-4:/home/cmcleay/lustre-1.4.10# lctl ping mds-1
failed to ping 192.168.241.229@tcp: Input/output error
I used the init scripts after I was satisfied with this - all worked well.
Doing an 'ls' on the share hung, but it came back after a little while after starting the failover mds-1
You need to have the config file in the right place for the init scripts to work properly
There were some messages on the oss:
Apr 25 23:40:48 oss-4 kernel: Lustre: 6:0:(linux-debug.c:98:libcfs_run_upcall()) Invoked LNET upcall /usr/lib/lustre/lnet_upcall ROUTER_NOTIFY,192.168.241.229@tcp,down,1176793786
Apr 25 23:44:11 oss-4 kernel: Lustre: 17433:0:(filter.c:3236:filter_set_info_async()) ost-beta: received MDS connection from 192.168.241.229@tcp
Then it all stopped working :(
lustre errors with a new config
The MDS writes a log file on the MDS device (if you mount a MDS volume, you can see it)
I tried re-writing it, but I decided to reformat it instead (I'd also tried a reboot, but that didn't help either)
MDSDEV: mds mds_UUID /dev/sdb1 ldiskfs no
! /usr/sbin/lctl (22): error: setup: Invalid argument
mds-1:~# lconf -v --node mds-1 /root/config.xml
configuring for host: ['mds-1']
Checking XML modification time
+ debugfs -c -R 'stat /LOGS' /dev/sdb1 2>&1 | grep mtime
xmtime 1177503678 > kmtime 1176793916
Error: MDS startup logs are older than config /root/config.xml. Please run --write_conf on stopped MDS to update. Use '--old_conf' to start anyways.
mds-1:~# lconf -v --node mds-1 --write-conf /root/config.xml
configuring for host: ['mds-1']
Service: network NET_mds-1_lnet NET_mds-1_lnet_UUID
loading module: libcfs srcdir None devdir libcfs
+ /sbin/modprobe libcfs
loading module: lnet srcdir None devdir lnet
+ /sbin/modprobe lnet
+ /sbin/modprobe lnet
loading module: ksocklnd srcdir None devdir klnds/socklnd
+ /sbin/modprobe ksocklnd
Service: ldlm ldlm ldlm_UUID
loading module: lvfs srcdir None devdir lvfs
+ /sbin/modprobe lvfs
loading module: obdclass srcdir None devdir obdclass
+ /sbin/modprobe obdclass
loading module: ptlrpc srcdir None devdir ptlrpc
+ /sbin/modprobe ptlrpc
Service: mdsdev MDD_mds_mds-1 MDD_mds_mds-1_UUID
original inode_size 0
stripe_count 1 inode_size 512
loading module: lquota srcdir None devdir quota
+ /sbin/modprobe lquota
loading module: mdc srcdir None devdir mdc
+ /sbin/modprobe mdc
loading module: osc srcdir None devdir osc
+ /sbin/modprobe osc
loading module: lov srcdir None devdir lov
+ /sbin/modprobe lov
loading module: mds srcdir None devdir mds
+ /sbin/modprobe mds
loading module: ldiskfs srcdir None devdir ldiskfs
+ /sbin/modprobe ldiskfs
loading module: fsfilt_ldiskfs srcdir None devdir lvfs
+ /sbin/modprobe fsfilt_ldiskfs
Service: mdsdev MDD_mds_mds-1 MDD_mds_mds-1_UUID
original inode_size 0
stripe_count 1 inode_size 512
MDSDEV: mds mds_UUID /dev/sdb1 ldiskfs no
+ /usr/sbin/lctl
attach mds mds mds_UUID
quit
+ /usr/sbin/lctl
cfg_device mds
setup /dev/sdb1 ldiskfs
quit
+ /usr/sbin/lctl
ignore_errors
cfg_device $mds
cleanup
detach
quit
! /usr/sbin/lctl (22): error: setup: Invalid argument
I tried re-writing it, but I decided to reformat it instead (I'd also tried a reboot, but that didn't help either)
MDSDEV: mds mds_UUID /dev/sdb1 ldiskfs no
! /usr/sbin/lctl (22): error: setup: Invalid argument
mds-1:~# lconf -v --node mds-1 /root/config.xml
configuring for host: ['mds-1']
Checking XML modification time
+ debugfs -c -R 'stat /LOGS' /dev/sdb1 2>&1 | grep mtime
xmtime 1177503678 > kmtime 1176793916
Error: MDS startup logs are older than config /root/config.xml. Please run --write_conf on stopped MDS to update. Use '--old_conf' to start anyways.
mds-1:~# lconf -v --node mds-1 --write-conf /root/config.xml
configuring for host: ['mds-1']
Service: network NET_mds-1_lnet NET_mds-1_lnet_UUID
loading module: libcfs srcdir None devdir libcfs
+ /sbin/modprobe libcfs
loading module: lnet srcdir None devdir lnet
+ /sbin/modprobe lnet
+ /sbin/modprobe lnet
loading module: ksocklnd srcdir None devdir klnds/socklnd
+ /sbin/modprobe ksocklnd
Service: ldlm ldlm ldlm_UUID
loading module: lvfs srcdir None devdir lvfs
+ /sbin/modprobe lvfs
loading module: obdclass srcdir None devdir obdclass
+ /sbin/modprobe obdclass
loading module: ptlrpc srcdir None devdir ptlrpc
+ /sbin/modprobe ptlrpc
Service: mdsdev MDD_mds_mds-1 MDD_mds_mds-1_UUID
original inode_size 0
stripe_count 1 inode_size 512
loading module: lquota srcdir None devdir quota
+ /sbin/modprobe lquota
loading module: mdc srcdir None devdir mdc
+ /sbin/modprobe mdc
loading module: osc srcdir None devdir osc
+ /sbin/modprobe osc
loading module: lov srcdir None devdir lov
+ /sbin/modprobe lov
loading module: mds srcdir None devdir mds
+ /sbin/modprobe mds
loading module: ldiskfs srcdir None devdir ldiskfs
+ /sbin/modprobe ldiskfs
loading module: fsfilt_ldiskfs srcdir None devdir lvfs
+ /sbin/modprobe fsfilt_ldiskfs
Service: mdsdev MDD_mds_mds-1 MDD_mds_mds-1_UUID
original inode_size 0
stripe_count 1 inode_size 512
MDSDEV: mds mds_UUID /dev/sdb1 ldiskfs no
+ /usr/sbin/lctl
attach mds mds mds_UUID
quit
+ /usr/sbin/lctl
cfg_device mds
setup /dev/sdb1 ldiskfs
quit
+ /usr/sbin/lctl
ignore_errors
cfg_device $mds
cleanup
detach
quit
! /usr/sbin/lctl (22): error: setup: Invalid argument
Wednesday, April 18, 2007
gotcha with /dev and udev when copying system
Usually I use
cd /
tar -clf - .|(cd /mnt;tar -xpf -)
to make a copy of a system. However, with newer systems running udev, this will cause problems as it does not copy /dev, which gets put on its own partition. So you will not be able to boot into a system unless you copy the /dev entries (it has a basic skeleton including vital files such as /dev/console and /dev/sda etc, stuff not needed for most boot environments will be created by udev dynamically)
cd /
tar -clf - .|(cd /mnt;tar -xpf -)
to make a copy of a system. However, with newer systems running udev, this will cause problems as it does not copy /dev, which gets put on its own partition. So you will not be able to boot into a system unless you copy the /dev entries (it has a basic skeleton including vital files such as /dev/console and /dev/sda etc, stuff not needed for most boot environments will be created by udev dynamically)
Tuesday, April 17, 2007
Failover for Lustre nodes
Lustre does not provide the tool set for the system-level components necessary for a complete failover solution (node failure detection, power control, and so on), as this functionality has been available for some time from third party tools. CFS does provide the necessary scripts to interact with these packages, and exposes health information for system monitoring. The recommended choice is the Heartbeat package from linux-ha.org. Lustre will work with any HA software that supports resource (I/O) fencing. The Heartbeat software is responsible for detecting failure of the primary server node and controlling the failover.
Thursday, February 22, 2007
linuxthreads being used instead of NPTL (native posix threading library)
Built a Debian machine which had an unusual problem whereby it was using linuxthreads instead of NPTL (native posix threading library) threads. Thus, when threaded services like nscd and java started, they would show multiple processes rather than a single (threaded) process. The /lib/tls directory was present, and all the right packages. But as a getconf showed, it was using linuxthreads:
# getconf GNU_LIBPTHREAD_VERSION
linuxthreads-0.10
ldconfig -v showed the /lib/tls libraries, it just wasn't using them
The clue was in an strace when starting nscd:
access("/etc/ld.so.nohwcap", F_OK) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f00000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=74936, ...}) = 0
mmap2(NULL, 74936, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7eed000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\345"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=263040, ...}) = 0
mmap2(NULL, 264196, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7eac000
mmap2(0xb7ee4000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x38) = 0xb7ee4000
mmap2(0xb7eec000, 2052, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7eec000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9592, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7eab000
mmap2(NULL, 12404, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7ea7000
mmap2(0xb7ea9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7ea9000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
On a machine using nptl:
execve("/etc/init.d/nscd", ["/etc/init.d/nscd", "start"], [/* 16 vars */]) = 0
uname({sys="Linux", node="ws-6", ...}) = 0
brk(0) = 0x80e6000
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=19421, ...}) = 0
old_mmap(NULL, 19421, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\342"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=252592, ...}) = 0
old_mmap(NULL, 257868, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4001d000
old_mmap(0x40053000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x35000) = 0x40053000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\32"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9872, ...}) = 0
old_mmap(NULL, 8632, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4005c000
old_mmap(0x4005e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x4005e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY) = 3
The presence of a file 'ld.so.nohwcap' was the problem. According to the man page of ld.so (on an Etch machine): "When this file is present the dynamic linker will load the non-optimized version of a library, even if the CPU supports the optimized version"
The reason for the presence of this file was because glibc was downgraded to an earlier version, which causes Debian to put this file in /etc. glibc was downgraded because this machine was mistakenly installed with 'etch' instead of 'sarge' and it was deemed easier at the time to downgrade via apt/aptitude etc, rather than do a fresh install. Perhaps a mistake :)
# getconf GNU_LIBPTHREAD_VERSION
linuxthreads-0.10
ldconfig -v showed the /lib/tls libraries, it just wasn't using them
The clue was in an strace when starting nscd:
access("/etc/ld.so.nohwcap", F_OK) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7f00000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=74936, ...}) = 0
mmap2(NULL, 74936, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7eed000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\345"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=263040, ...}) = 0
mmap2(NULL, 264196, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7eac000
mmap2(0xb7ee4000, 32768, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x38) = 0xb7ee4000
mmap2(0xb7eec000, 2052, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb7eec000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\20\f\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9592, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7eab000
mmap2(NULL, 12404, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb7ea7000
mmap2(0xb7ea9000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xb7ea9000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = 0
open("/lib/libc.so.6", O_RDONLY) = 3
On a machine using nptl:
execve("/etc/init.d/nscd", ["/etc/init.d/nscd", "start"], [/* 16 vars */]) = 0
uname({sys="Linux", node="ws-6", ...}) = 0
brk(0) = 0x80e6000
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.preload", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=19421, ...}) = 0
old_mmap(NULL, 19421, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40018000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/libncurses.so.5", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\220\342"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=252592, ...}) = 0
old_mmap(NULL, 257868, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4001d000
old_mmap(0x40053000, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x35000) = 0x40053000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\32"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=9872, ...}) = 0
old_mmap(NULL, 8632, PROT_READ|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x4005c000
old_mmap(0x4005e000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED, 3, 0x2000) = 0x4005e000
close(3) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
open("/lib/tls/libc.so.6", O_RDONLY) = 3
The presence of a file 'ld.so.nohwcap' was the problem. According to the man page of ld.so (on an Etch machine): "When this file is present the dynamic linker will load the non-optimized version of a library, even if the CPU supports the optimized version"
The reason for the presence of this file was because glibc was downgraded to an earlier version, which causes Debian to put this file in /etc. glibc was downgraded because this machine was mistakenly installed with 'etch' instead of 'sarge' and it was deemed easier at the time to downgrade via apt/aptitude etc, rather than do a fresh install. Perhaps a mistake :)
Friday, February 09, 2007
load balancing with f5s
I was wondering how f5s configured as active-active do load balancing. That is, for the nodes or real servers that they load balance, how do they a) accept packets from a source, without configuring two separate default routes on the router sending the packets, and have a different IP on each f5 (having static default routes would be problematic if one of the f5s failed, so it would have to be able to detect this), and b) how do they accept traffic from the servers they are load balance equally, as the servers only have one default route.
I poked around the web and found that the Ultramonkey project, using Saru, can do this, among others. I think the way it might work is to have a common MAC address shared between two f5s, so that the packet goes to both f5s, and then use some mechanism whereby the f5s compare the packets to make sure that they are the same, and one of them forwards the packet, and one of them drops it. In a way, this scenario is not really load balancing, as both machines still receive the packet and process it to some degree. It would only really be worthwhile if very little processing was done on each packet to decide which machine was to forward it. Otherwise, you get no real gain from having two machines active. Perhaps this could be a hash lookup, which, if the hashes match, then some very simple algorithm could then be used to decide which unit will forward and which unit will drop the packet. Maybe a lot more of the CPU and system resources would be dedicated to NATing the packet and applying various other rules to it, so this scheme would work. Anyway, I don't really know if that is how it does it. I'll have to research a bit more.
I poked around the web and found that the Ultramonkey project, using Saru, can do this, among others. I think the way it might work is to have a common MAC address shared between two f5s, so that the packet goes to both f5s, and then use some mechanism whereby the f5s compare the packets to make sure that they are the same, and one of them forwards the packet, and one of them drops it. In a way, this scenario is not really load balancing, as both machines still receive the packet and process it to some degree. It would only really be worthwhile if very little processing was done on each packet to decide which machine was to forward it. Otherwise, you get no real gain from having two machines active. Perhaps this could be a hash lookup, which, if the hashes match, then some very simple algorithm could then be used to decide which unit will forward and which unit will drop the packet. Maybe a lot more of the CPU and system resources would be dedicated to NATing the packet and applying various other rules to it, so this scheme would work. Anyway, I don't really know if that is how it does it. I'll have to research a bit more.
Wednesday, January 24, 2007
fixing broken/missing /var/lib/dpkg/available file
/var/lib/dpkg/available went missing on a machine, breaking commands like 'dpkg -l':
ws-1:/usr/local/bin# dpkg -l
dpkg-query: failed to open package info file `/var/lib/dpkg/available' for reading: No such file or directory
'dselect update' (from man dpkg) fixed this (may also need to run apt-get update prior)
ws-1:/usr/local/bin# dselect update
Hit http://192.168.241.146 stable/main Packages
Hit http://192.168.241.146 stable/main Release
Hit http://192.168.241.146 stable/contrib Packages
Hit http://192.168.241.146 stable/contrib Release
Hit http://192.168.241.146 stable/non-free Packages
Hit http://192.168.241.146 stable/non-free Release
Hit http://192.168.241.146 stable/updates/main Packages
Hit http://192.168.241.146 stable/updates/main Release
Hit http://192.168.241.146 stable/updates/contrib Packages
Hit http://192.168.241.146 stable/updates/contrib Release
Hit http://192.168.241.146 sarge-backports/main Packages
Hit http://192.168.241.146 sarge-backports/main Release
Hit http://192.168.241.146 sarge-backports/contrib Packages
Hit http://192.168.241.146 sarge-backports/contrib Release
Hit http://192.168.241.146 sarge-backports/non-free Packages
Hit http://192.168.241.146 sarge-backports/non-free Release
Reading Package Lists... Done
Merging Available information
Replacing available packages info, using /var/cache/apt/available.
Information about 17330 package(s) was updated.
This also get information about packages that were not retrieved through apt-get, i.e., locally made packages
ws-1:/usr/local/bin# dpkg -l
dpkg-query: failed to open package info file `/var/lib/dpkg/available' for reading: No such file or directory
'dselect update' (from man dpkg) fixed this (may also need to run apt-get update prior)
ws-1:/usr/local/bin# dselect update
Hit http://192.168.241.146 stable/main Packages
Hit http://192.168.241.146 stable/main Release
Hit http://192.168.241.146 stable/contrib Packages
Hit http://192.168.241.146 stable/contrib Release
Hit http://192.168.241.146 stable/non-free Packages
Hit http://192.168.241.146 stable/non-free Release
Hit http://192.168.241.146 stable/updates/main Packages
Hit http://192.168.241.146 stable/updates/main Release
Hit http://192.168.241.146 stable/updates/contrib Packages
Hit http://192.168.241.146 stable/updates/contrib Release
Hit http://192.168.241.146 sarge-backports/main Packages
Hit http://192.168.241.146 sarge-backports/main Release
Hit http://192.168.241.146 sarge-backports/contrib Packages
Hit http://192.168.241.146 sarge-backports/contrib Release
Hit http://192.168.241.146 sarge-backports/non-free Packages
Hit http://192.168.241.146 sarge-backports/non-free Release
Reading Package Lists... Done
Merging Available information
Replacing available packages info, using /var/cache/apt/available.
Information about 17330 package(s) was updated.
This also get information about packages that were not retrieved through apt-get, i.e., locally made packages
Subscribe to:
Posts (Atom)