Recent oVirt errors when doing a minor update on a 4.1 engine host

 Had two problems after doing an engine update on 4.1.0.4-1.el7.centos:

First was that the engine service would not start due missing python-dateutil (no idea why this package wasn't included if it was needed in the update, and the engine service was working fine before the update). ABRT and journalctl provided useful info as to what needed fixing:


   ovirt-engine-dwhd.service                                                                   loaded activating auto-restart oVirt Engine Data Warehouse
* ovirt-engine.service                                                                        loaded failed     failed       oVirt Engine
* ovirt-fence-kdump-listener.service                                                          loaded failed     failed       oVirt Engine fence_kdump listener
  ovirt-imageio-proxy.service                                                                 loaded active     running      oVirt ImageIO Proxy
  ovirt-vmconsole-proxy-sshd.service                                                          loaded active     running      oVirt VM Console SSH server daemon
* ovirt-websocket-proxy.service                                                               loaded failed     failed       oVirt Engine websockets proxy
  polkit.service                                                                              loaded active     running      Authorization Manager

ABRT has detected 4 problem(s). For more info run: abrt-cli list --since 1495130066
[root@ovirt-engine ~]# abrt-cli list --since 1495130066
id 07e7e054795adf76b6260aed93a81420fe16e4cc
reason:         service.py:35:<module>:ImportError: No module named dateutil
time:           Thu May 18 18:57:04 2017
cmdline:        /usr/bin/python /usr/share/ovirt-engine-dwh/services/ovirt-engine-dwhd/ovirt-engine-dwhd.py --redirect-output --systemd=notify start
package:        ovirt-engine-dwh-4.1.0-1.el7.centos
uid:            108 (ovirt)
count:          1
Directory:      /var/spool/abrt/Python-2017-05-18-18:57:04-2765

id 107b6035ce460c7c4cabdbb8b9b2c5cee02f7b2c
reason:         service.py:35:<module>:ImportError: No module named dateutil
time:           Thu May 18 18:57:04 2017
cmdline:        /usr/bin/python /usr/share/ovirt-engine/services/ovirt-fence-kdump-listener/ovirt-fence-kdump-listener.py --systemd=notify start
package:        ovirt-engine-tools-4.1.0.4-1.el7.centos
uid:            108 (ovirt)
count:          1
Directory:      /var/spool/abrt/Python-2017-05-18-18:57:04-2766

id 02916623c009e9277ec757c70ca1e1bc32020c62
reason:         service.py:35:<module>:ImportError: No module named dateutil
time:           Thu May 18 18:57:04 2017
cmdline:        /usr/bin/python /usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py --redirect-output --systemd=notify start
package:        ovirt-engine-backend-4.1.0.4-1.el7.centos
uid:            108 (ovirt)
count:          1
Directory:      /var/spool/abrt/Python-2017-05-18-18:57:04-2767

id 588e5eb1889bfde67787877bc8d1156953eeae1e
reason:         service.py:35:<module>:ImportError: No module named dateutil
time:           Thu May 18 18:57:04 2017
cmdline:        /usr/bin/python /usr/share/ovirt-engine/services/ovirt-websocket-proxy/ovirt-websocket-proxy.py --systemd=notify start
package:        ovirt-engine-websocket-proxy-4.1.0.4-1.el7.centos
uid:            108 (ovirt)
count:          1
Directory:      /var/spool/abrt/Python-2017-05-18-18:57:04-1391

The Autoreporting feature is disabled. Please consider enabling it by issuing
'abrt-auto-reporting enabled' as a user with root privileges
[root@ovirt-engine ~]#


 May 18 18:57:04 ovirt-engine systemd[1]: Starting oVirt Engine...
May 18 18:57:04 ovirt-engine python[2767]: detected unhandled Python exception in '/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py'
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: Traceback (most recent call last):
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: File "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.py", line 32, in <module>
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: from ovirt_engine import service
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: File "/usr/lib/python2.7/site-packages/ovirt_engine/service.py", line 35, in <module>
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: from dateutil import tz
May 18 18:57:06 ovirt-engine ovirt-engine.py[2767]: ImportError: No module named dateutil
May 18 18:57:06 ovirt-engine systemd[1]: ovirt-engine.service: main process exited, code=exited, status=1/FAILURE
May 18 18:57:06 ovirt-engine systemd[1]: Failed to start oVirt Engine.
May 18 18:57:06 ovirt-engine systemd[1]: Unit ovirt-engine.service entered failed state.
May 18 18:57:06 ovirt-engine systemd[1]: ovirt-engine.service failed.




Installed python-dateutil:

yum install python-dateutil

The second issue was vdsmd not being able to start. Once again, journalctl provided useful information on what to do (logging on vdsmd is very helpful):


* vdsmd.service                                                                                  loaded failed failed    Virtual Desktop Server Manager


May 18 19:02:42 ovirt-engine systemd[1]: Starting Virtual Desktop Server Manager...
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running mkdirs
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running configure_coredump
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running configure_vdsm_logs
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running wait_for_network
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running run_init_hooks
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running upgraded_version_check
May 18 19:02:42 ovirt-engine vdsmd_init_common.sh[2746]: vdsm: Running check_is_configured
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: Error:
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: One of the modules is not configured to work with VDSM.
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: To configure the module use the following:
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: 'vdsm-tool configure [--module module-name]'.
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: If all modules are not configured try to use:
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: 'vdsm-tool configure --force'
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: (The force flag will stop the module's service and start it
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: afterwards automatically to load the new configuration.)
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: multipath requires configuration
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: libvirt is not configured for vdsm yet
May 18 19:02:43 ovirt-engine vdsmd_init_common.sh[2746]: Modules certificates, sebool, multipath, passwd, sanlock, libvirt are not configured
May 18 19:02:43 ovirt-engine systemd[1]: vdsmd.service: control process exited, code=exited status=1
May 18 19:02:43 ovirt-engine systemd[1]: Failed to start Virtual Desktop Server Manager.
May 18 19:02:43 ovirt-engine systemd[1]: Unit vdsmd.service entered failed state.
May 18 19:02:43 ovirt-engine systemd[1]: vdsmd.service failed.

So I used vdsm-tool to configure the services it listed as needing configuring, e.g.:

[root@ovirt-engine ~]# vdsm-tool configure --module certificates

Checking configuration status...


Running configure...
Reconfiguration of certificates is done.

Done configuring modules to VDSM.
[root@ovirt-engine ~]# vdsm-tool configure --module sebool    

Checking configuration status...


Running configure...
Reconfiguration of sebool is done.

Done configuring modules to VDSM.
[root@ovirt-engine ~]# vdsm-tool configure --module multipath

Checking configuration status...

multipath requires configuration

Running configure...
Reconfiguration of multipath is done.

Done configuring modules to VDSM.
[root@ovirt-engine ~]# vdsm-tool configure --module passwd 

Checking configuration status...


Running configure...
Reconfiguration of passwd is done.

Done configuring modules to VDSM.

Some services needed to be stopped first before they could be configured:

[root@ovirt-engine ~]# vdsm-tool configure --module sanlock

Checking configuration status...

Error:

Cannot configure while service 'sanlock' is running.
 Stop the service manually or use the --force flag.
 

[root@ovirt-engine ~]#


 e.g., stop sanlock:

[root@ovirt-engine ~]# vdsm-tool service-stop sanlock