On Mar 16, 2017 23:02, "Fam Zheng" <f...@redhat.com> wrote:
On Thu, 03/16 17:26, Ed Swierk wrote: > On Tue, Mar 14, 2017 at 8:36 AM, Fam Zheng <f...@redhat.com> wrote: > > After the AioContext lock push down, there is a race between > > virtio_scsi_dataplane_start and those "assert(s->ctx && > > s->dataplane_started)", because the latter doesn't isn't wrapped in > > aio_context_acquire. > > > > Reproducer is simply booting a Fedora guest with an empty > > virtio-scsi-dataplane controller: > > > > qemu-system-x86_64 \ > > -drive > > if=none,id=root,format=raw,file=Fedora-Cloud-Base-25-1.3.x86_64.raw \ > > -device virtio-scsi \ > > -device scsi-disk,drive=root,bootindex=1 \ > > -object iothread,id=io \ > > -device virtio-scsi-pci,iothread=io \ > > -net user,hostfwd=tcp::10022-:22 -net nic,model=virtio -m 2048 \ > > --enable-kvm > > > > Fix this by moving acquire/release pairs from virtio_scsi_handle_*_vq to > > their callers - and wrap the broken assertions in. > > > > Signed-off-by: Fam Zheng <f...@redhat.com> > > With this change on top of 2.9.0-rc0, I am able to boot a Linux guest > from a virtio-scsi drive with an iothread, e.g. > > qemu-system-x86_64 -nographic -enable-kvm -monitor > telnet:0.0.0.0:1234,server,nowait -m 1024 -object > iothread,id=iothread1 -device > virtio-scsi-pci,iothread=iothread1,id=scsi0 -drive > file=/x/drive.qcow2,format=qcow2,if=none,id=drive0,cache= directsync,aio=native > -device scsi-hd,drive=drive0,bootindex=1 > > But when I try to take a snapshot by running this in the monitor > > snapshot_blkdev drive0 /x/snap1.qcow2 > > qemu bombs with > > qemu-system-x86_64: /x/qemu/include/block/aio.h:457: > aio_enable_external: Assertion `ctx->external_disable_cnt > 0' failed. > > This does not occur if I don't use the iothread. > > I instrumented the code a bit, printing the value of bs, > bdrv_get_aio_context(bs), and > bdrv_get_aio_context(bs)->external_disable_cnt before and after > aio_{disable,enable}_external() in bdrv_drained_{begin,end}(). > > Without the iothread, nested calls to these functions cause the > counter to increase and decrease as you'd expect, and the context is > the same in each call. > > bdrv_drained_begin 0 bs=0x7fe9f5ad65a0 ctx=0x7fe9f5abc7b0 cnt=0 > bdrv_drained_begin 1 bs=0x7fe9f5ad65a0 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_begin 0 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_begin 1 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=3 > bdrv_drained_end 0 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=3 > bdrv_drained_end 1 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_begin 0 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_begin 1 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=3 > bdrv_drained_end 0 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=3 > bdrv_drained_end 1 bs=0x7fe9f67cfde0 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_begin 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 0 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=2 > bdrv_drained_end 1 bs=0x7fe9f5d12a00 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_end 0 bs=0x7fe9f5ad65a0 ctx=0x7fe9f5abc7b0 cnt=1 > bdrv_drained_end 1 bs=0x7fe9f5ad65a0 ctx=0x7fe9f5abc7b0 cnt=0 > > But with the iothread, there are at least two different context > pointers, and there is one extra call to bdrv_drained_end() without a > matching bdrv_drained_begin(). That last call comes from > external_snapshot_clean(). > > bdrv_drained_begin 0 bs=0x7fe4437545c0 ctx=0x7fe443749a00 cnt=0 > bdrv_drained_begin 1 bs=0x7fe4437545c0 ctx=0x7fe443749a00 cnt=1 > bdrv_drained_begin 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_begin 0 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_begin 1 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=2 > bdrv_drained_end 0 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=2 > bdrv_drained_end 1 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_begin 0 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_begin 1 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=2 > bdrv_drained_end 0 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=2 > bdrv_drained_end 1 bs=0x7fe44444de20 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 0 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=0 > bdrv_drained_begin 1 bs=0x7fe443990a00 ctx=0x7fe44373a7b0 cnt=1 > bdrv_drained_end 0 bs=0x7fe443990a00 ctx=0x7fe443749a00 cnt=1 > bdrv_drained_end 1 bs=0x7fe443990a00 ctx=0x7fe443749a00 cnt=0 > bdrv_drained_end 0 bs=0x7fe4437545c0 ctx=0x7fe443749a00 cnt=0 > qemu-system-x86_64: /x/qemu/include/block/aio.h:457: > aio_enable_external: Assertion `ctx->external_disable_cnt > 0' failed. This may indeed come from the left-behind virtio_scsi_release() you pointed out, I'll fix it in v3 and please see if it helps. Fam I tried that, but the 'ctx->external_disable_cnt > 0' assertion still occurs. --Ed