+Eduardo Lima <etru...@redhat.com> and +Danilo Cesar Lemes de Paula
<ddepa...@redhat.com> FYI
Il giorno mar 16 nov 2021 alle ore 08:55 Yedidyah Bar David <d...@redhat.com>
ha scritto:
> Hi all,
>
> For a few days now we have failures in CI of the he-basic suite.
>
> At one point the failure seemed to have been around
> networking/routing/firewalling, but later it changed, and now the
> deploy process fails while trying to first start the engine vm after
> it's copied to the shared storage.
>
> I ran locally OST he-basic with current ost-images, reproduced the
> issue, and managed to "fix" it by enabling
> ovirt-master-centos-stream-advanced-virtualization-testing and
> downgrading qemu-kvm-* from 6.1.0 (from AppStream) to
> 15:6.0.0-33.el8s.
>
> Is this a known issue?
>
> How do we handle? Perhaps we should conflict with it somewhere until
> we find and fix the root cause.
>
> Please note that the flow is:
>
> 1. Create a local VM from the appliance image
> 2. Do stuff on this machine
> 3. Shut it down
> 4. Copy its disk to shared storage
> 5. Start the machine from the shared storage
>
> And that (1.) did work with 6.1.0, and also (5.) did work with 6.0.0
> (so the copying (using qemu-img) did work well) and the difference is
> elsewhere.
>
> Following is the diff between the qemu commands of (1.) and (5.) (as
> found in the respective logs). Any clue?
>
> --- localq 2021-11-16 08:48:01.230426260 +0100
> +++ sharedq 2021-11-16 08:48:46.884937598 +0100
> @@ -1,54 +1,79 @@
> -2021-11-14 15:09:56.430+0000: starting up libvirt version: 7.9.0,
> package: 1.module_el8.6.0+983+a7505f3f (CentOS Buildsys
> <b...@centos.org>, 2021-11-09-20:38:08, ), qemu version:
> 6.1.0qemu-kvm-6.1.0-4.module_el8.6.0+983+a7505f3f, kernel:
> 4.18.0-348.el8.x86_64, hostname:
> ost-he-basic-suite-master-host-0.lago.local
> +2021-11-14 15:29:10.686+0000: starting up libvirt version: 7.9.0,
> package: 1.module_el8.6.0+983+a7505f3f (CentOS Buildsys
> <b...@centos.org>, 2021-11-09-20:38:08, ), qemu version:
> 6.1.0qemu-kvm-6.1.0-4.module_el8.6.0+983+a7505f3f, kernel:
> 4.18.0-348.el8.x86_64, hostname:
> ost-he-basic-suite-master-host-0.lago.local
> LC_ALL=C \
> PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin \
> -HOME=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal \
> -XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/.local/share
> \
> -XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/.cache \
> -XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/.config \
> +HOME=/var/lib/libvirt/qemu/domain-2-HostedEngine \
> +XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-2-HostedEngine/.local/share \
> +XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-2-HostedEngine/.cache \
> +XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-2-HostedEngine/.config \
> /usr/libexec/qemu-kvm \
> --name guest=HostedEngineLocal,debug-threads=on \
> +-name guest=HostedEngine,debug-threads=on \
> -S \
> --object
> '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-HostedEngineLocal/master-key.aes"}'
> \
> --machine
> pc-q35-rhel8.5.0,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram
> \
> --cpu
> Cascadelake-Server,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvmclock=on
> \
> --m 3171 \
> --object
> '{"qom-type":"memory-backend-ram","id":"pc.ram","size":3325034496}' \
> +-object
> '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-HostedEngine/master-key.aes"}'
> \
> +-machine
> pc-q35-rhel8.4.0,accel=kvm,usb=off,dump-guest-core=off,graphics=off \
> +-cpu Cascadelake-Server-noTSX,mpx=off \
> +-m size=3247104k,slots=16,maxmem=12988416k \
> -overcommit mem-lock=off \
> --smp 2,sockets=2,cores=1,threads=1 \
> --uuid 716b26d9-982b-4c51-ac05-646f28346007 \
> +-smp 2,maxcpus=32,sockets=16,dies=1,cores=2,threads=1 \
> +-object '{"qom-type":"iothread","id":"iothread1"}' \
> +-object
> '{"qom-type":"memory-backend-ram","id":"ram-node0","size":3325034496}'
> \
> +-numa node,nodeid=0,cpus=0-31,memdev=ram-node0 \
> +-uuid a10f5518-1fc2-4aae-b7da-5d1d9875e753 \
> +-smbios
> type=1,manufacturer=oVirt,product=RHEL,version=8.6-1.el8,serial=d2f36f31-bb29-4e1f-b52d-8fddb632953c,uuid=a10f5518-1fc2-4aae-b7da-5d1d9875e753,family=oVirt
> \
> -no-user-config \
> -nodefaults \
> -chardev socket,id=charmonitor,fd=40,server=on,wait=off \
> -mon chardev=charmonitor,id=monitor,mode=control \
> --rtc base=utc \
> +-rtc base=2021-11-14T15:29:08,driftfix=slew \
> +-global kvm-pit.lost_tick_policy=delay \
> +-no-hpet \
> -no-shutdown \
> -global ICH9-LPC.disable_s3=1 \
> -global ICH9-LPC.disable_s4=1 \
> --boot menu=off,strict=on \
> +-boot strict=on \
> -device
> pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
> \
> -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1
> \
> -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2
> \
> -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3
> \
> -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4
> \
> --device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
> --blockdev
> '{"driver":"file","filename":"/var/tmp/localvm1hjkqhu2/images/b4985de8-fa7e-4b93-a93c-f348ef17d91e/b1614c86-bf90-44c4-9f5d-fd2b3c509934","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}'
> \
> --blockdev
> '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}'
> \
> --device
> virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1
> \
> --blockdev
> '{"driver":"file","filename":"/var/tmp/localvm1hjkqhu2/seed.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}'
> \
> --blockdev
> '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}'
> \
> --device ide-cd,bus=ide.0,drive=libvirt-1-format,id=sata0-0-0 \
> --netdev tap,fd=42,id=hostnet0,vhost=on,vhostfd=43 \
> --device
> virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:4d:89:07:dc,bus=pci.1,addr=0x0
> \
> --chardev pty,id=charserial0 \
> +-device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5
> \
> +-device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6
> \
> +-device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7
> \
> +-device
> pcie-root-port,port=24,chassis=9,id=pci.9,bus=pcie.0,multifunction=on,addr=0x3
> \
> +-device
> pcie-root-port,port=25,chassis=10,id=pci.10,bus=pcie.0,addr=0x3.0x1 \
> +-device
> pcie-root-port,port=26,chassis=11,id=pci.11,bus=pcie.0,addr=0x3.0x2 \
> +-device
> pcie-root-port,port=27,chassis=12,id=pci.12,bus=pcie.0,addr=0x3.0x3 \
> +-device
> pcie-root-port,port=28,chassis=13,id=pci.13,bus=pcie.0,addr=0x3.0x4 \
> +-device
> pcie-root-port,port=29,chassis=14,id=pci.14,bus=pcie.0,addr=0x3.0x5 \
> +-device
> pcie-root-port,port=30,chassis=15,id=pci.15,bus=pcie.0,addr=0x3.0x6 \
> +-device
> pcie-root-port,port=31,chassis=16,id=pci.16,bus=pcie.0,addr=0x3.0x7 \
> +-device pcie-root-port,port=32,chassis=17,id=pci.17,bus=pcie.0,addr=0x4 \
> +-device pcie-pci-bridge,id=pci.18,bus=pci.1,addr=0x0 \
> +-device
> qemu-xhci,p2=8,p3=8,id=ua-56e0dd42-5016-4a70-b2b6-7e3bfbc4002f,bus=pci.4,addr=0x0
> \
> +-device
> virtio-scsi-pci,iothread=iothread1,id=ua-1ba84ec0-6eb7-4e4c-9e5f-f446e0b2e67c,bus=pci.3,addr=0x0
> \
> +-device
> virtio-serial-pci,id=ua-5d6442e9-af6c-459a-8105-6d0cd90214a6,max_ports=16,bus=pci.5,addr=0x0
> \
> +-device
> ide-cd,bus=ide.2,id=ua-8a1b74dd-0b24-4f88-9df1-81d4cb7f404c,werror=report,rerror=report
> \
> +-blockdev
> '{"driver":"host_device","filename":"/run/vdsm/storage/8468bc65-907a-4c95-8f93-4d29fa722f62/5714a85b-8d09-4ba6-a89a-b39f98e664ff/68f4061e-a537-4051-af1e-baaf04929a25","aio":"native","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}'
> \
> +-blockdev
> '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":true,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}'
> \
> +-device
> virtio-blk-pci,iothread=iothread1,bus=pci.6,addr=0x0,drive=libvirt-1-format,id=ua-5714a85b-8d09-4ba6-a89a-b39f98e664ff,bootindex=1,write-cache=on,serial=5714a85b-8d09-4ba6-a89a-b39f98e664ff,werror=stop,rerror=stop
> \
> +-netdev
> tap,fds=44:45,id=hostua-33528c78-5281-4ebd-a5e2-5e8894d6a4aa,vhost=on,vhostfds=46:47
> \
> +-device
> virtio-net-pci,mq=on,vectors=6,host_mtu=1500,netdev=hostua-33528c78-5281-4ebd-a5e2-5e8894d6a4aa,id=ua-33528c78-5281-4ebd-a5e2-5e8894d6a4aa,mac=54:52:4d:89:07:dc,bus=pci.2,addr=0x0
> \
> +-chardev socket,id=charserial0,fd=48,server=on,wait=off \
> -device isa-serial,chardev=charserial0,id=serial0 \
> --chardev socket,id=charchannel0,fd=45,server=on,wait=off \
> --device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
> \
> --audiodev id=audio1,driver=none \
> --vnc 127.0.0.1:0,sasl=on,audiodev=audio1 \
> --device VGA,id=video0,vgamem_mb=16,bus=pcie.0,addr=0x1 \
> --object
> '{"qom-type":"rng-random","id":"objrng0","filename":"/dev/random"}' \
> --device virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.4,addr=0x0 \
> +-chardev socket,id=charchannel0,fd=49,server=on,wait=off \
> +-device
> virtserialport,bus=ua-5d6442e9-af6c-459a-8105-6d0cd90214a6.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
> \
> +-chardev spicevmc,id=charchannel1,name=vdagent \
> +-device
> virtserialport,bus=ua-5d6442e9-af6c-459a-8105-6d0cd90214a6.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0
> \
> +-chardev socket,id=charchannel2,fd=50,server=on,wait=off \
> +-device
> virtserialport,bus=ua-5d6442e9-af6c-459a-8105-6d0cd90214a6.0,nr=3,chardev=charchannel2,id=channel2,name=org.ovirt.hosted-engine-setup.0
> \
> +-audiodev id=audio1,driver=spice \
> +-spice
> port=5900,tls-port=5901,addr=192.168.200.3,x509-dir=/etc/pki/vdsm/libvirt-spice,tls-channel=main,tls-channel=display,tls-channel=inputs,tls-channel=cursor,tls-channel=playback,tls-channel=record,tls-channel=smartcard,tls-channel=usbredir,seamless-migration=on
> \
> +-device
> qxl-vga,id=ua-60b147e1-322a-4f49-bb16-0e7a76732396,ram_size=67108864,vram_size=33554432,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pcie.0,addr=0x1
> \
> +-device
> intel-hda,id=ua-fe1a3722-6c25-4719-9d0b-baeeb5d74a3e,bus=pci.18,addr=0x1
> \
> +-device
> hda-duplex,id=ua-fe1a3722-6c25-4719-9d0b-baeeb5d74a3e-codec0,bus=ua-fe1a3722-6c25-4719-9d0b-baeeb5d74a3e.0,cad=0,audiodev=audio1
> \
> +-device
> virtio-balloon-pci,id=ua-bd2a17f1-4d39-4d4d-8793-089081a2065c,bus=pci.7,addr=0x0
> \
> +-object
> '{"qom-type":"rng-random","id":"objua-7e3d85f3-15da-4f97-9434-90396750e2b2","filename":"/dev/urandom"}'
> \
> +-device
> virtio-rng-pci,rng=objua-7e3d85f3-15da-4f97-9434-90396750e2b2,id=ua-7e3d85f3-15da-4f97-9434-90396750e2b2,bus=pci.8,addr=0x0
> \
> +-device vmcoreinfo \
> -sandbox
> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
> \
> -msg timestamp=on
> -char device redirected to /dev/pts/1 (label charserial0)
> -2021-11-14 15:18:02.749+0000: Domain id=1 is tainted: custom-ga-command
> +2021-11-14T15:44:46.647989Z qemu-kvm: terminating on signal 15 from
> pid 21473 (<unknown process>)
>
> Best regards,
> --
> Didi
> _______________________________________________
> Devel mailing list -- devel@ovirt.org
> To unsubscribe send an email to devel-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/BDYP62MAJL2QVQZ7RHM2USZD4HXBGUA6/
>
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbona...@redhat.com
<https://www.redhat.com/>
*Red Hat respects your work life balance. Therefore there is no need to
answer this email out of your office hours.*
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/devel@ovirt.org/message/NHYBU7UHDDFRGYC4DVAESR6F3CLXH7EQ/
