Oh, and I'd like to thank OST & team for catching my blunder.
On Tue, Aug 22, 2017 at 1:48 PM, Dan Kenigsberg <dan...@redhat.com> wrote: > This seems to be my fault, https://gerrit.ovirt.org/80908 should fix it. > > On Tue, Aug 22, 2017 at 1:14 PM, Nir Soffer <nsof...@redhat.com> wrote: >> >> >> On יום ג׳, 22 באוג׳ 2017, 12:57 Yedidyah Bar David <d...@redhat.com> wrote: >>> >>> On Tue, Aug 22, 2017 at 12:52 PM, Anton Marchukov <amarc...@redhat.com> >>> wrote: >>> > Hello All. >>> > >>> > Any news on this? I see the latest failures for vdsm is the same [1] >>> > and >>> > the job is still not working for it. >>> > >>> > [1] >>> > >>> > http://jenkins.ovirt.org/job/ovirt-master_change-queue-tester/1901/artifact/exported-artifacts/basic-suit-master-el7/test_logs/basic-suite-master/post-002_bootstrap.py/lago-basic-suite-master-engine/_var_log/ovirt-engine/host-deploy/ovirt-host-deploy-20170822035135-lago-basic-suite-master-host0-1f46d892.log >>> >>> This log has: >>> >>> 2017-08-22 03:51:28,272-0400 DEBUG otopi.context >>> context._executeMethod:128 Stage closeup METHOD >>> otopi.plugins.ovirt_host_deploy.vdsm.packages.Plugin._reconfigure >>> 2017-08-22 03:51:28,272-0400 DEBUG >>> otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.executeRaw:813 >>> execute: ('/bin/vdsm-tool', 'configure', '--force'), >>> executable='None', cwd='None', env=None >>> 2017-08-22 03:51:30,687-0400 DEBUG >>> otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.executeRaw:863 >>> execute-result: ('/bin/vdsm-tool', 'configure', '--force'), rc=1 >>> 2017-08-22 03:51:30,688-0400 DEBUG >>> otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:921 >>> execute-output: ('/bin/vdsm-tool', 'configure', '--force') stdout: >>> >>> Checking configuration status... >>> >>> abrt is not configured for vdsm >>> WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based >>> on vdsm configuration >>> lvm requires configuration >>> libvirt is not configured for vdsm yet >>> FAILED: conflicting vdsm and libvirt-qemu tls configuration. >>> vdsm.conf with ssl=True requires the following changes: >>> libvirtd.conf: listen_tcp=0, auth_tcp="sasl", listen_tls=1 >>> qemu.conf: spice_tls=1. >>> multipath requires configuration >>> >>> Running configure... >>> Reconfiguration of abrt is done. >>> Reconfiguration of passwd is done. >>> WARNING: LVM local configuration: /etc/lvm/lvmlocal.conf is not based >>> on vdsm configuration >>> Backing up /etc/lvm/lvmlocal.conf to /etc/lvm/lvmlocal.conf.201708220351 >>> Installing /usr/share/vdsm/lvmlocal.conf at /etc/lvm/lvmlocal.conf >>> Units need configuration: {'lvm2-lvmetad.service': {'LoadState': >>> 'loaded', 'ActiveState': 'active'}, 'lvm2-lvmetad.socket': >>> {'LoadState': 'loaded', 'ActiveState': 'active'}} >>> Reconfiguration of lvm is done. >>> Reconfiguration of sebool is done. >>> >>> 2017-08-22 03:51:30,688-0400 DEBUG >>> otopi.plugins.ovirt_host_deploy.vdsm.packages plugin.execute:926 >>> execute-output: ('/bin/vdsm-tool', 'configure', '--force') stderr: >>> Error: ServiceNotExistError: Tried all alternatives but failed: >>> ServiceNotExistError: dev-hugepages1G.mount is not native systemctl >>> service >>> ServiceNotExistError: dev-hugepages1G.mount is not a SysV service >>> >>> >>> 2017-08-22 03:51:30,689-0400 WARNING >>> otopi.plugins.ovirt_host_deploy.vdsm.packages >>> packages._reconfigure:155 Cannot configure vdsm >>> >>> Nir, any idea? >> >> >> Looks like some configurator has failed after sebool, but we don't have >> proper error message with the name of the configurator. >> >> Piotr, can you take a look? >> >> >>> >>> > >>> > >>> > >>> > On Sun, Aug 20, 2017 at 12:39 PM, Nir Soffer <nsof...@redhat.com> wrote: >>> >> >>> >> On Sun, Aug 20, 2017 at 11:08 AM Dan Kenigsberg <dan...@redhat.com> >>> >> wrote: >>> >>> >>> >>> On Sun, Aug 20, 2017 at 10:39 AM, Yaniv Kaul <yk...@redhat.com> wrote: >>> >>> > >>> >>> > >>> >>> > On Sun, Aug 20, 2017 at 8:48 AM, Daniel Belenky >>> >>> > <dbele...@redhat.com> >>> >>> > wrote: >>> >>> >> >>> >>> >> Failed test: basic_suite_master/002_bootstrap >>> >>> >> Version: oVirt Master >>> >>> >> Link to failed job: ovirt-master_change-queue-tester/1860/ >>> >>> >> Link to logs (Jenkins): test logs >>> >>> >> Suspected patch: https://gerrit.ovirt.org/#/c/80749/3 >>> >>> >> >>> >>> >> From what I was able to find, It seems that for some reason VDSM >>> >>> >> failed to >>> >>> >> start on host 1. The VDSM log is empty, and the only error I could >>> >>> >> find in >>> >>> >> supervdsm.log is that start of LLDP failed (Not sure if it's >>> >>> >> related) >>> >>> > >>> >>> > >>> >>> > Can you check the networking on the hosts? Something's very strange >>> >>> > there. >>> >>> > For example: >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 NetworkManager[685]: >>> >>> > <info> >>> >>> > [1503175122.2682] manager: (e7NZWeNDXwIjQia): new Bond device >>> >>> > (/org/freedesktop/NetworkManager/Devices/17) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting xmit hash policy to layer2+3 (2) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting xmit hash policy to encap2+3 (3) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting xmit hash policy to encap3+4 (4) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > option xmit_hash_policy: invalid value (5) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting primary_reselect to always (0) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting primary_reselect to better (1) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting primary_reselect to failure (2) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > option primary_reselect: invalid value (3) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting arp_all_targets to any (0) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > Setting arp_all_targets to all (1) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: >>> >>> > e7NZWeNDXwIjQia: >>> >>> > option arp_all_targets: invalid value (2) >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 kernel: bonding: >>> >>> > e7NZWeNDXwIjQia is being deleted... >>> >>> > Aug 19 16:38:42 lago-basic-suite-master-host0 lldpad: recvfrom(Event >>> >>> > interface): No buffer space available >>> >>> > >>> >>> > Y. >>> >>> >>> >>> >>> >>> >>> >>> The post-boot noise with funny-looking bonds is due to our calling of >>> >>> `vdsm-tool dump-bonding-options` every boot, in order to find the >>> >>> bonding defaults for the current kernel. >>> >>> >>> >>> > >>> >>> >> >>> >>> >> From host-deploy log: >>> >>> >> >>> >>> >> 2017-08-19 16:38:41,476-0400 DEBUG >>> >>> >> otopi.plugins.otopi.services.systemd >>> >>> >> systemd.state:130 starting service vdsmd >>> >>> >> 2017-08-19 16:38:41,476-0400 DEBUG >>> >>> >> otopi.plugins.otopi.services.systemd >>> >>> >> plugin.executeRaw:813 execute: ('/bin/systemctl', 'start', >>> >>> >> 'vdsmd.service'), >>> >>> >> executable='None', cwd='None', env=None >>> >>> >> 2017-08-19 16:38:44,628-0400 DEBUG >>> >>> >> otopi.plugins.otopi.services.systemd >>> >>> >> plugin.executeRaw:863 execute-result: ('/bin/systemctl', 'start', >>> >>> >> 'vdsmd.service'), rc=1 >>> >>> >> 2017-08-19 16:38:44,630-0400 DEBUG >>> >>> >> otopi.plugins.otopi.services.systemd >>> >>> >> plugin.execute:921 execute-output: ('/bin/systemctl', 'start', >>> >>> >> 'vdsmd.service') stdout: >>> >>> >> >>> >>> >> >>> >>> >> 2017-08-19 16:38:44,630-0400 DEBUG >>> >>> >> otopi.plugins.otopi.services.systemd >>> >>> >> plugin.execute:926 execute-output: ('/bin/systemctl', 'start', >>> >>> >> 'vdsmd.service') stderr: >>> >>> >> Job for vdsmd.service failed because the control process exited >>> >>> >> with >>> >>> >> error >>> >>> >> code. See "systemctl status vdsmd.service" and "journalctl -xe" for >>> >>> >> details. >>> >>> >> >>> >>> >> 2017-08-19 16:38:44,631-0400 DEBUG otopi.context >>> >>> >> context._executeMethod:142 method exception >>> >>> >> Traceback (most recent call last): >>> >>> >> File "/tmp/ovirt-dunwHj8Njn/pythonlib/otopi/context.py", line >>> >>> >> 132, >>> >>> >> in >>> >>> >> _executeMethod >>> >>> >> method['method']() >>> >>> >> File >>> >>> >> >>> >>> >> >>> >>> >> "/tmp/ovirt-dunwHj8Njn/otopi-plugins/ovirt-host-deploy/vdsm/packages.py", >>> >>> >> line 224, in _start >>> >>> >> self.services.state('vdsmd', True) >>> >>> >> File >>> >>> >> "/tmp/ovirt-dunwHj8Njn/otopi-plugins/otopi/services/systemd.py", >>> >>> >> line 141, in state >>> >>> >> service=name, >>> >>> >> RuntimeError: Failed to start service 'vdsmd' >>> >>> >> >>> >>> >> >>> >>> >> From /var/log/messages: >>> >>> >> >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> Error: >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> One of >>> >>> >> the modules is not configured to work with VDSM. >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> To >>> >>> >> configure the module use the following: >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> 'vdsm-tool configure [--module module-name]'. >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> If >>> >>> >> all >>> >>> >> modules are not configured try to use: >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> 'vdsm-tool configure --force' >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> (The >>> >>> >> force flag will stop the module's service and start it >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> afterwards automatically to load the new configuration.) >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> abrt >>> >>> >> is already configured for vdsm >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> lvm is >>> >>> >> configured for vdsm >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> libvirt is already configured for vdsm >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> multipath requires configuration >>> >>> >> Aug 19 16:38:44 lago-basic-suite-master-host0 vdsmd_init_common.sh: >>> >>> >> Modules sanlock, multipath are not configured >>> >> >>> >> >>> >> This means the host was not deployed correctly. When deploying vdsm >>> >> host deploy must run "vdsm-tool configure --force", which configures >>> >> multipath and sanlock. >>> >> >>> >> We did not change anything in multipath and sanlock configurators >>> >> lately. >>> >> >>> >> Didi, can you check this? >>> >> >>> >> _______________________________________________ >>> >> Devel mailing list >>> >> Devel@ovirt.org >>> >> http://lists.ovirt.org/mailman/listinfo/devel >>> > >>> > >>> > >>> > >>> > -- >>> > Anton Marchukov >>> > Team Lead - Release Management - RHV DevOps - Red Hat >>> > >>> >>> >>> >>> -- >>> Didi >> >> >> _______________________________________________ >> Devel mailing list >> Devel@ovirt.org >> http://lists.ovirt.org/mailman/listinfo/devel _______________________________________________ Devel mailing list Devel@ovirt.org http://lists.ovirt.org/mailman/listinfo/devel