BTW just to be clear, qemu is crashing in this scenario *only* if iothread is enabled for the guest. Without iothread enabled the operation is completed without any problems.
On jue, oct 24, 2019 at 11:07 PM, Fernando Casas Schössow <casasferna...@outlook.com> wrote: > Today I updated to qemu 4.0.1 since this was the latest version > available for Alpine and I can confirm that I can repro the issue > with this version as well. > Not sure if relevant but I can also confirm that the problem happens > with Windows Server 2012 R2 but also with Linux guests (it doesn't > matter if the guest use uefi or bios firmware). I made this tests > just to discard things. > > Also as discussed I compiled qemu with debug symbols, repro the > problem, collected a core dump and run both through gdb. This is the > result: > > (gdb) thread apply all bt > > Thread 42 (LWP 33704): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fee02380b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 41 (LWP 33837): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1ad5b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 40 (LWP 33719): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fee02266b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 39 (LWP 33696): > #0 0x00007fee04233171 in syscall () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee02be8b64 in ?? () > #2 0x0000000000000030 in ?? () > #3 0x00007fee02be2540 in ?? () > #4 0x00007fee02be2500 in ?? () > #5 0x00007fee02be2548 in ?? () > #6 0x000055d7e4987f28 in rcu_gp_event () > #7 0x0000000000000000 in ?? () > > Thread 38 (LWP 33839): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1a83b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 37 (LWP 33841): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1737b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 36 (LWP 33863): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8c83b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 35 (LWP 33842): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc170eb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 34 (LWP 33862): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8cacb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 33 (LWP 33843): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc16e5b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 32 (LWP 33861): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8cd5b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 31 (LWP 33844): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc16bcb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 30 (LWP 33858): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8e83b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 29 (LWP 33845): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1693b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 28 (LWP 33857): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8eacb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 27 (LWP 33846): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc166ab64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 26 (LWP 33856): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8ed5b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 25 (LWP 33847): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc142ab64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 24 (LWP 33855): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8efeb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 23 (LWP 33848): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedbd0feb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 22 (LWP 33854): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedbd031b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 21 (LWP 33849): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedbd0d5b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 20 (LWP 33852): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedbd05ab64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 19 (LWP 33850): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedbd0acb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 18 (LWP 33851): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedbd083b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 17 (LWP 33836): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1afeb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 16 (LWP 33835): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1c5ab64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 15 (LWP 33834): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1c83b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 14 (LWP 33833): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1cacb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 13 (LWP 33677): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x000055d7e516656c in ?? () > #3 0x0000000000000000 in ?? () > > Thread 12 (LWP 33832): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1cd5b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 11 (LWP 33831): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1cfeb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 10 (LWP 33829): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1e67b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 9 (LWP 33828): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1e90b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 8 (LWP 33827): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fee02a95b64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 7 (LWP 33732): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fee0223db64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 6 (LWP 33706): > #0 0x00007fee0423263d in ioctl () from /lib/ld-musl-x86_64.so.1 > #1 0x0000000000000001 in ?? () > #2 0x00007fee00000010 in ?? () > #3 0x00007fee02351440 in ?? () > #4 0x00007fee02351400 in ?? () > #5 0x0000000000000000 in ?? () > > Thread 5 (LWP 33838): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1aacb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 4 (LWP 33860): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8cfeb64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 3 (LWP 33859): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedb8e5ab64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 2 (LWP 33840): > #0 0x00007fee04252080 in ?? () from /lib/ld-musl-x86_64.so.1 > #1 0x00007fee0424f4b6 in ?? () from /lib/ld-musl-x86_64.so.1 > #2 0x00007fedc1a5ab64 in ?? () > #3 0x0000000000000000 in ?? () > > Thread 1 (LWP 33701): > #0 0x00007fee0421b7a1 in abort () from /lib/ld-musl-x86_64.so.1 > #1 0x000055d7e6012b70 in ?? () > #2 0x0000000000000020 in ?? () > #3 0x0000000000000000 in ?? () > (gdb) > > > I'm not a developer nor skilled with gdb but if you provide me the > debugging commands I can execute them and reply back with the results. > I can also provide the binary and the core dump for analysis if > needed. > > While waiting for replies I will check if I can upgrade to qemu > 4.1.0, try to repro and provide the results. > > Thanks. > > Fernando > > On mié, oct 23, 2019 at 7:57 PM, Fernando Casas Schössow > <casasferna...@outlook.com> wrote: >> In virsh I would do this while the guest is running: >> >> virsh attach-disk <vmname> <path_to_ISO_file> <guest_device> --type >> cdrom --mode readonly >> >> Following the example for guest from my first email: >> >> virsh attach-disk DCHOMENET01 >> /resources/virtio-win-0.1.171-stable.iso sdb --type cdrom --mode >> readonly >> >> Right after executing this, qemu crashes and log the assertion. >> I can repro this also from virt-manager by selecting the cdrom >> device -> Connect -> selecting the ISO file -> Choose volume -> Ok >> (basically the same but in the gui). >> >> I may be able to try 4.1. Will look into it and report back. >> >> On mié, oct 23, 2019 at 7:33 PM, John Snow <js...@redhat.com> wrote: >>> >>> >>> On 10/23/19 1:28 PM, Fernando Casas Schössow wrote: >>>> Hi John, >>>> >>>> Thanks for looking into this. >>>> I can quickly repro the problem with qemu 4.0 binary with >>>> debugging >>>> symbols enabled as I have it available already. >>>> Doing the same with qemu 4.1 or development head may be too much >>>> hassle >>>> but if it's really the only way I can give it try. >>>> >>>> Would it worth it to try with 4.0 first and get the strack trace >>>> or it >>>> will not help and the only way to go is with 4.1 (or dev)? >>>> >>>> Thanks, >>>> >>>> Fernando >>>> >>> >>> If 4.0 is what you have access to, having the stack trace for that >>> is >>> better than not, but confirming it happens on the latest release >>> would >>> be nice. >>> >>> Can you share your workflow for virsh that reproduces the failure? >>> >>> --js >>> >>>> On mié, oct 23, 2019 at 5:34 PM, John Snow <js...@redhat.com> >>>> wrote: >>>>> On 10/18/19 5:41 PM, Fernando Casas Schössow wrote: >>>>> >>>>> Hi, >>>>> >>>>> Hi! Thanks for the report. >>>>> >>>>> Today while working with two different Windows Server 2012 R2 >>>>> guests I found that when I try to attach an ISO file to a >>>>> SCSI >>>>> CD-ROM device through libvirt (virsh or virt-manager) while >>>>> the >>>>> guest is running, qemu crashes and the following message is >>>>> logged: Assertion failed: blk_get_aio_context(d->conf.blk) == >>>>> s->ctx >>>>> >>>>> (/home/buildozer/aports/main/qemu/src/qemu-4.0.0/hw/scsi/virtio-scsi.c: >>>>> virtio_scsi_ctx_check: 246) I can repro this at will. All I >>>>> have >>>>> to do is to try to attach an ISO file to the SCSI CDROM >>>>> while the >>>>> guest is running. The SCSI controller model is virtio-scsi >>>>> with >>>>> iothread enabled. Please find below all the details about my >>>>> setup >>>>> that I considered relevant but I missed something please >>>>> don't >>>>> hesitate to let me know: >>>>> >>>>> Looks like we got aio_context management wrong with iothread for >>>>> the >>>>> media change events somewhere. Should be easy enough to fix if we >>>>> figure out where the bad assumption is. >>>>> >>>>> Host arch: x86_64 Distro: Alpine Linux 3.10.2 qemu version: >>>>> 4.0 >>>>> >>>>> Do you have the ability to try 4.1, or the latest development >>>>> head >>>>> with debugging symbols enabled? >>>>> >>>>> Linux kernel version: 4.19.67 libvirt: 5.5.0 Emulated SCSI >>>>> controller: virtio-scsi (with iothread enabled) Guest >>>>> firmware: >>>>> OVMF-EFI Guest OS: Window Server 2012 R2 Guest virtio drivers >>>>> version: 171 (current stable) qemu command line: >>>>> /usr/bin/qemu-system-x86_64 -name >>>>> guest=DCHOMENET01,debug-threads=on -S -object >>>>> >>>>> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-78-DCHOMENET01/master-key.aes >>>>> -machine pc-i440fx-4.0,accel=kvm,usb=off,dump-guest-core=on >>>>> -cpu >>>>> >>>>> IvyBridge,ss=on,vmx=off,pcid=on,hypervisor=on,arat=on,tsc_adjust=on,umip=on,xsaveopt=on,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff >>>>> -drive >>>>> >>>>> file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd,if=pflash,format=raw,unit=0,readonly=on >>>>> -drive >>>>> >>>>> file=/var/lib/libvirt/qemu/nvram/DCHOMENET01_VARS.fd,if=pflash,format=raw,unit=1 >>>>> -m 1536 -overcommit mem-lock=off -smp >>>>> 1,sockets=1,cores=1,threads=1 -object iothread,id=iothread1 >>>>> -uuid >>>>> f06978ad-2734-44ab-a518-5dfcf71d625e -no-user-config >>>>> -nodefaults >>>>> -chardev socket,id=charmonitor,fd=33,server,nowait -mon >>>>> chardev=charmonitor,id=monitor,mode=control -rtc >>>>> base=localtime,driftfix=slew -global >>>>> kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global >>>>> PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot >>>>> strict=on -device qemu-xhci,id=usb,bus=pci.0,addr=0x4 -device >>>>> >>>>> virtio-scsi-pci,iothread=iothread1,id=scsi0,num_queues=1,bus=pci.0,addr=0x5 >>>>> -device >>>>> virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 >>>>> -drive >>>>> >>>>> file=/storage/storage-hdd-vms/virtual_machines_hdd/dchomenet01.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0,cache=none,discard=unmap,aio=threads >>>>> -device >>>>> >>>>> scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,device_id=drive-scsi0-0-0-0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1,write-cache=on >>>>> -drive if=none,id=drive-scsi0-0-0-1,readonly=on -device >>>>> >>>>> scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,device_id=drive-scsi0-0-0-1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1 >>>>> -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device >>>>> >>>>> virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:99:b5:62,bus=pci.0,addr=0x3 >>>>> -chardev >>>>> >>>>> socket,id=charserial0,host=127.0.0.1,port=4900,telnet,server,nowait >>>>> -device >>>>> isa-serial,chardev=charserial0,id=serial0 -chardev >>>>> spicevmc,id=charchannel0,name=vdagent -device >>>>> >>>>> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0 >>>>> -chardev socket,id=charchannel1,fd=45,server,nowait -device >>>>> >>>>> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 >>>>> -chardev >>>>> spiceport,id=charchannel2,name=org.spice-space.webdav.0 >>>>> -device >>>>> >>>>> virtserialport,bus=virtio-serial0.0,nr=3,chardev=charchannel2,id=channel2,name=org.spice-space.webdav.0 >>>>> -device virtio-tablet-pci,id=input2,bus=pci.0,addr=0x7 -spice >>>>> >>>>> port=5900,addr=127.0.0.1,disable-ticketing,seamless-migration=on >>>>> -device >>>>> >>>>> qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 >>>>> -chardev spicevmc,id=charredir0,name=usbredir -device >>>>> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 >>>>> -chardev >>>>> spicevmc,id=charredir1,name=usbredir -device >>>>> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 >>>>> -device >>>>> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -sandbox >>>>> >>>>> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny >>>>> -msg timestamp=on I can provide a core dump of the process if >>>>> needed for debugging and the guest XML as well. >>>>> >>>>> A backtrace is probably a great starting point (from gdb: `thread >>>>> apply all bt`.) I don't know exactly what codepath is being >>>>> exercised >>>>> when you "attach an ISO file" through libvirt's interface. If you >>>>> don't mind the hassle, trying on the 4.1 (or a development build >>>>> would >>>>> be even more luxurious) and giving a stacktrace would be nice. >>>>> >>>>> Thanks. Fernando >>>>> >>>>> Thanks! --js >>>> >>>> >>> >> >> > > >