Today I updated to qemu 4.0.1 since this was the latest version available for Alpine and I can confirm that I can repro the issue with this version as well. Not sure if relevant but I can also confirm that the problem happens with Windows Server 2012 R2 but also with Linux guests (it doesn't matter if the guest use uefi or bios firmware). I made this tests just to discard things.
Also as discussed I compiled qemu with debug symbols, repro the problem, collected a core dump and run both through gdb. This is the result: (gdb) thread apply all bt Thread 42 (LWP 33704): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fee02380b64 in ?? () #3 0x0000000000000000 in ?? () Thread 41 (LWP 33837): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1ad5b64 in ?? () #3 0x0000000000000000 in ?? () Thread 40 (LWP 33719): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fee02266b64 in ?? () #3 0x0000000000000000 in ?? () Thread 39 (LWP 33696): #0 0x00007fee04233171 in syscall () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee02be8b64 in ?? () #2 0x0000000000000030 in ?? () #3 0x00007fee02be2540 in ?? () #4 0x00007fee02be2500 in ?? () #5 0x00007fee02be2548 in ?? () #6 0x000055d7e4987f28 in rcu_gp_event () #7 0x0000000000000000 in ?? () Thread 38 (LWP 33839): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1a83b64 in ?? () #3 0x0000000000000000 in ?? () Thread 37 (LWP 33841): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1737b64 in ?? () #3 0x0000000000000000 in ?? () Thread 36 (LWP 33863): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8c83b64 in ?? () #3 0x0000000000000000 in ?? () Thread 35 (LWP 33842): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc170eb64 in ?? () #3 0x0000000000000000 in ?? () Thread 34 (LWP 33862): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8cacb64 in ?? () #3 0x0000000000000000 in ?? () Thread 33 (LWP 33843): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc16e5b64 in ?? () #3 0x0000000000000000 in ?? () Thread 32 (LWP 33861): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8cd5b64 in ?? () #3 0x0000000000000000 in ?? () Thread 31 (LWP 33844): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc16bcb64 in ?? () #3 0x0000000000000000 in ?? () Thread 30 (LWP 33858): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8e83b64 in ?? () #3 0x0000000000000000 in ?? () Thread 29 (LWP 33845): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1693b64 in ?? () #3 0x0000000000000000 in ?? () Thread 28 (LWP 33857): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8eacb64 in ?? () #3 0x0000000000000000 in ?? () Thread 27 (LWP 33846): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc166ab64 in ?? () #3 0x0000000000000000 in ?? () Thread 26 (LWP 33856): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8ed5b64 in ?? () #3 0x0000000000000000 in ?? () Thread 25 (LWP 33847): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc142ab64 in ?? () #3 0x0000000000000000 in ?? () Thread 24 (LWP 33855): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8efeb64 in ?? () #3 0x0000000000000000 in ?? () Thread 23 (LWP 33848): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedbd0feb64 in ?? () #3 0x0000000000000000 in ?? () Thread 22 (LWP 33854): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedbd031b64 in ?? () #3 0x0000000000000000 in ?? () Thread 21 (LWP 33849): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedbd0d5b64 in ?? () #3 0x0000000000000000 in ?? () Thread 20 (LWP 33852): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedbd05ab64 in ?? () #3 0x0000000000000000 in ?? () Thread 19 (LWP 33850): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedbd0acb64 in ?? () #3 0x0000000000000000 in ?? () Thread 18 (LWP 33851): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedbd083b64 in ?? () #3 0x0000000000000000 in ?? () Thread 17 (LWP 33836): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1afeb64 in ?? () #3 0x0000000000000000 in ?? () Thread 16 (LWP 33835): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1c5ab64 in ?? () #3 0x0000000000000000 in ?? () Thread 15 (LWP 33834): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1c83b64 in ?? () #3 0x0000000000000000 in ?? () Thread 14 (LWP 33833): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1cacb64 in ?? () #3 0x0000000000000000 in ?? () Thread 13 (LWP 33677): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x000055d7e516656c in ?? () #3 0x0000000000000000 in ?? () Thread 12 (LWP 33832): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1cd5b64 in ?? () #3 0x0000000000000000 in ?? () Thread 11 (LWP 33831): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1cfeb64 in ?? () #3 0x0000000000000000 in ?? () Thread 10 (LWP 33829): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1e67b64 in ?? () #3 0x0000000000000000 in ?? () Thread 9 (LWP 33828): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1e90b64 in ?? () #3 0x0000000000000000 in ?? () Thread 8 (LWP 33827): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fee02a95b64 in ?? () #3 0x0000000000000000 in ?? () Thread 7 (LWP 33732): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fee0223db64 in ?? () #3 0x0000000000000000 in ?? () Thread 6 (LWP 33706): #0 0x00007fee0423263d in ioctl () from /lib/ld-musl-x86_64.so.1 #1 0x0000000000000001 in ?? () #2 0x00007fee00000010 in ?? () #3 0x00007fee02351440 in ?? () #4 0x00007fee02351400 in ?? () #5 0x0000000000000000 in ?? () Thread 5 (LWP 33838): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1aacb64 in ?? () #3 0x0000000000000000 in ?? () Thread 4 (LWP 33860): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8cfeb64 in ?? () #3 0x0000000000000000 in ?? () Thread 3 (LWP 33859): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedb8e5ab64 in ?? () #3 0x0000000000000000 in ?? () Thread 2 (LWP 33840): #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 #2 0x00007fedc1a5ab64 in ?? () #3 0x0000000000000000 in ?? () Thread 1 (LWP 33701): #0 0x00007fee0421b7a1 in abort () from /lib/ld-musl-x86_64.so.1 #1 0x000055d7e6012b70 in ?? () #2 0x0000000000000020 in ?? () #3 0x0000000000000000 in ?? () (gdb) I'm not a developer nor skilled with gdb but if you provide me the debugging commands I can execute them and reply back with the results. I can also provide the binary and the core dump for analysis if needed. While waiting for replies I will check if I can upgrade to qemu 4.1.0, try to repro and provide the results. Thanks. Fernando On mié, oct 23, 2019 at 7:57 PM, Fernando Casas Schössow <casasferna...@outlook.com> wrote: > In virsh I would do this while the guest is running: > > virsh attach-disk <vmname> <path_to_ISO_file> <guest_device> --type > cdrom --mode readonly > > Following the example for guest from my first email: > > virsh attach-disk DCHOMENET01 > /resources/virtio-win-0.1.171-stable.iso sdb --type cdrom --mode > readonly > > Right after executing this, qemu crashes and log the assertion. > I can repro this also from virt-manager by selecting the cdrom device > -> Connect -> selecting the ISO file -> Choose volume -> Ok > (basically the same but in the gui). > > I may be able to try 4.1. Will look into it and report back. > > On mié, oct 23, 2019 at 7:33 PM, John Snow <js...@redhat.com> wrote: >> >> >> On 10/23/19 1:28 PM, Fernando Casas Schössow wrote: >>> Hi John, >>> >>> Thanks for looking into this. >>> I can quickly repro the problem with qemu 4.0 binary with debugging >>> symbols enabled as I have it available already. >>> Doing the same with qemu 4.1 or development head may be too much >>> hassle >>> but if it's really the only way I can give it try. >>> >>> Would it worth it to try with 4.0 first and get the strack trace >>> or it >>> will not help and the only way to go is with 4.1 (or dev)? >>> >>> Thanks, >>> >>> Fernando >>> >> >> If 4.0 is what you have access to, having the stack trace for that is >> better than not, but confirming it happens on the latest release >> would >> be nice. >> >> Can you share your workflow for virsh that reproduces the failure? >> >> --js >> >>> On mié, oct 23, 2019 at 5:34 PM, John Snow <js...@redhat.com> >>> wrote: >>>> On 10/18/19 5:41 PM, Fernando Casas Schössow wrote: >>>> >>>> Hi, >>>> >>>> Hi! Thanks for the report. >>>> >>>> Today while working with two different Windows Server 2012 R2 >>>> guests I found that when I try to attach an ISO file to a SCSI >>>> CD-ROM device through libvirt (virsh or virt-manager) while >>>> the >>>> guest is running, qemu crashes and the following message is >>>> logged: Assertion failed: blk_get_aio_context(d->conf.blk) == >>>> s->ctx >>>> >>>> (/home/buildozer/aports/main/qemu/src/qemu-4.0.0/hw/scsi/virtio-scsi.c: >>>> virtio_scsi_ctx_check: 246) I can repro this at will. All I >>>> have >>>> to do is to try to attach an ISO file to the SCSI CDROM while >>>> the >>>> guest is running. The SCSI controller model is virtio-scsi >>>> with >>>> iothread enabled. Please find below all the details about my >>>> setup >>>> that I considered relevant but I missed something please don't >>>> hesitate to let me know: >>>> >>>> Looks like we got aio_context management wrong with iothread for >>>> the >>>> media change events somewhere. Should be easy enough to fix if we >>>> figure out where the bad assumption is. >>>> >>>> Host arch: x86_64 Distro: Alpine Linux 3.10.2 qemu version: >>>> 4.0 >>>> >>>> Do you have the ability to try 4.1, or the latest development head >>>> with debugging symbols enabled? >>>> >>>> Linux kernel version: 4.19.67 libvirt: 5.5.0 Emulated SCSI >>>> controller: virtio-scsi (with iothread enabled) Guest >>>> firmware: >>>> OVMF-EFI Guest OS: Window Server 2012 R2 Guest virtio drivers >>>> version: 171 (current stable) qemu command line: >>>> /usr/bin/qemu-system-x86_64 -name >>>> guest=DCHOMENET01,debug-threads=on -S -object >>>> >>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-78-DCHOMENET01/master-key.aes >>>> -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=on >>>> -cpu >>>> >>>> IvyBridge,ss=on,vmx=off,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,umip=on,xsaveopt=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff >>>> -drive >>>> >>>> file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on >>>> -drive >>>> >>>> file=/var/lib/libvirt/qemu/nvram/DCHOMENET01_VARS.fd,if=pflash,format=raw,unit=1 >>>> -m 1536 -overcommit mem-lock=off -smp >>>> 1,sockets=1,cores=1,threads=1 -object iothread,id=iothread1 >>>> -uuid >>>> f06978ad-2734-44ab-a518-5dfcf71d625e -no-user-config >>>> -nodefaults >>>> -chardev socket,id=charmonitor,fd=33,server,nowait -mon >>>> chardev=charmonitor,id=monitor,mode=control -rtc >>>> base=localtime,driftfix=slew -global >>>> kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global >>>> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>> strict=on -device qemu-xhci,id=usb,bus=pci.0,addr=0x4 -device >>>> >>>> virtio-scsi-pci,iothread=iothread1,id=scsi0,num_queues=1,bus=pci.0,addr=0x5 >>>> -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 >>>> -drive >>>> >>>> file=/storage/storage-hdd-vms/virtual_machines_hdd/dchomenet01.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap,aio=threads >>>> -device >>>> >>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on >>>> -drive if=none,id=drive-scsi0-0-0-1,readonly=on -device >>>> >>>> scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=drive-scsi0-0-0-1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 >>>> -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device >>>> >>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b5:62,bus=pci.0,addr=0x3 >>>> -chardev >>>> >>>> socket,id=charserial0,host=127.0.0.1,port=4900,telnet,server,nowait >>>> -device >>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>> spicevmc,id=charchannel0,name=vdagent -device >>>> >>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 >>>> -chardev socket,id=charchannel1,fd=45,server,nowait -device >>>> >>>> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 >>>> -chardev >>>> spiceport,id=charchannel2,name=org.spice-space.webdav.0 >>>> -device >>>> >>>> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.spice-space.webdav.0 >>>> -device virtio-tablet-pci,id=input2,bus=pci.0,addr=0x7 -spice >>>> >>>> port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on >>>> -device >>>> >>>> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 >>>> -chardev spicevmc,id=charredir0,name=usbredir -device >>>> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 >>>> -chardev >>>> spicevmc,id=charredir1,name=usbredir -device >>>> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 >>>> -device >>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -sandbox >>>> >>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>> -msg timestamp=on I can provide a core dump of the process if >>>> needed for debugging and the guest XML as well. >>>> >>>> A backtrace is probably a great starting point (from gdb: `thread >>>> apply all bt`.) I don't know exactly what codepath is being >>>> exercised >>>> when you "attach an ISO file" through libvirt's interface. If you >>>> don't mind the hassle, trying on the 4.1 (or a development build >>>> would >>>> be even more luxurious) and giving a stacktrace would be nice. >>>> >>>> Thanks. Fernando >>>> >>>> Thanks! --js >>> >>> >> > >