Re: virtio-scsi and another complex AioContext issue
On Wed, Jun 24, 2020 at 06:16:05PM +0200, Sergio Lopez wrote: > On Tue, Jun 23, 2020 at 03:24:54PM +0100, Stefan Hajnoczi wrote: > > On Mon, Jun 22, 2020 at 04:16:04PM +0200, Sergio Lopez wrote: > > > On Fri, Jun 19, 2020 at 02:04:06PM +0100, Stefan Hajnoczi wrote: > > > > On Thu, Jun 11, 2020 at 10:36:22AM +0200, Sergio Lopez wrote: > > > > > Hi, > > > > > > > > > > While debugging BZ#1844343, I managed to reproduce the issue which > > > > > leads to crash with a backtrace like this one: > > > > > > > > > > < snip > > > > > > Thread 2 (Thread 0x7fe208463f00 (LWP 1659571)): > > > > > #0 0x7fe2033b78ed in __lll_lock_wait () at /lib64/libpthread.so.0 > > > > > #1 0x7fe2033b0bd4 in pthread_mutex_lock () at > > > > > /lib64/libpthread.so.0 > > > > > #2 0x560caa8f1e6d in qemu_mutex_lock_impl > > > > > (mutex=0x560cacc68a10, file=0x560caaa9797f "util/async.c", > > > > > line=521) at util/qemu-thread-posix.c:78 > > > > > #3 0x560caa82414d in bdrv_set_aio_context_ignore > > > > > (bs=bs@entry=0x560cacc73570, > > > > > new_context=new_context@entry=0x560cacc5fed0, > > > > > ignore=ignore@entry=0x7ffe388b1cc0) at block.c:6192 > > > > > #4 0x560caa824503 in bdrv_child_try_set_aio_context > > > > > (bs=bs@entry=0x560cacc73570, ctx=0x560cacc5fed0, > > > > > ignore_child=, errp=) > > > > > at block.c:6272 > > > > > #5 0x560caa859e6b in blk_do_set_aio_context > > > > > (blk=0x560cacecf370, new_context=0x560cacc5fed0, > > > > > update_root_node=update_root_node@entry=true, errp=errp@entry=0x0) at > > > > > block/block-backend.c:1989 > > > > > #6 0x560caa85c501 in blk_set_aio_context > > > > > (blk=, new_context=, > > > > > errp=errp@entry=0x0) at block/block-backend.c:2010 > > > > > #7 0x560caa61db30 in virtio_scsi_hotunplug > > > > > (hotplug_dev=0x560cadaafbd0, dev=0x560cacec1210, > > > > > errp=0x7ffe388b1d80) > > > > > at > > > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:869 > > > > > #8 0x560caa6ccd1e in qdev_unplug (dev=0x560cacec1210, > > > > > errp=errp@entry=0x7ffe388b1db8) > > > > > at qdev-monitor.c:872 > > > > > #9 0x560caa6ccd9e in qmp_device_del (id=, > > > > > errp=errp@entry=0x7ffe388b1db8) > > > > > at qdev-monitor.c:884 > > > > > #10 0x560caa7ec4d3 in qmp_marshal_device_del > > > > > (args=, ret=, errp=0x7ffe388b1e18) > > > > > at qapi/qapi-commands-qdev.c:99 > > > > > #11 0x560caa8a45ec in do_qmp_dispatch > > > > > (errp=0x7ffe388b1e10, allow_oob=, > > > > > request=, cmds=0x560cab1928a0 ) at > > > > > qapi/qmp-dispatch.c:132 > > > > > #12 0x560caa8a45ec in qmp_dispatch > > > > > (cmds=0x560cab1928a0 , request=, > > > > > allow_oob=) > > > > > at qapi/qmp-dispatch.c:175 > > > > > #13 0x560caa7c2521 in monitor_qmp_dispatch (mon=0x560cacca2f00, > > > > > req=) > > > > > at monitor/qmp.c:145 > > > > > #14 0x560caa7c2bba in monitor_qmp_bh_dispatcher (data= > > > > out>) at monitor/qmp.c:234 > > > > > #15 0x560caa8ec716 in aio_bh_call (bh=0x560cacbd80e0) at > > > > > util/async.c:117 > > > > > #16 0x560caa8ec716 in aio_bh_poll (ctx=ctx@entry=0x560cacbd6da0) > > > > > at util/async.c:117 > > > > > #17 0x560caa8efb04 in aio_dispatch (ctx=0x560cacbd6da0) at > > > > > util/aio-posix.c:459 > > > > > #18 0x560caa8ec5f2 in aio_ctx_dispatch > > > > > (source=, callback=, > > > > > user_data=) at util/async.c:260 > > > > > #19 0x7fe2078d167d in g_main_context_dispatch () at > > > > > /lib64/libglib-2.0.so.0 > > > > > #20 0x560caa8eebb8 in glib_pollfds_poll () at util/main-loop.c:219 > > > > > #21 0x560caa8eebb8 in os_host_main_loop_wait (timeout= > > > > out>) at util/main-loop.c:242 > > > > > #22 0x560caa8eebb8 in main_loop_wait (nonblocking= > > > > out>) at util/main-loop.c:518 > > > > > #23 0x560caa6cfe51 in main_loop () at vl.c:1828 > > > > > #24 0x560caa57b322 in main (argc=, argv= > > > > out>, envp=) > > > > > at vl.c:4504 > > > > > > > > > > Thread 1 (Thread 0x7fe1fb059700 (LWP 1659573)): > > > > > #0 0x7fe20301b70f in raise () at /lib64/libc.so.6 > > > > > #1 0x7fe203005b25 in abort () at /lib64/libc.so.6 > > > > > #2 0x7fe2030059f9 in _nl_load_domain.cold.0 () at > > > > > /lib64/libc.so.6 > > > > > #3 0x7fe203013cc6 in .annobin_assert.c_end () at /lib64/libc.so.6 > > > > > #4 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > > > > block/block-backend.c:1968 > > > > > #5 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > > > > block/block-backend.c:1962 > > > > > #6 0x560caa61d79c in virtio_scsi_ctx_check (s=0x560cadaafbd0, > > > > > s=0x560cadaafbd0, d=0x560cacec1210) > > > > > at > > > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:250 > > > > > #7 0x560caa61d79c in virtio_scsi_handle_cmd_req_prepare > > > > >
Re: virtio-scsi and another complex AioContext issue
On Tue, Jun 23, 2020 at 03:24:54PM +0100, Stefan Hajnoczi wrote: > On Mon, Jun 22, 2020 at 04:16:04PM +0200, Sergio Lopez wrote: > > On Fri, Jun 19, 2020 at 02:04:06PM +0100, Stefan Hajnoczi wrote: > > > On Thu, Jun 11, 2020 at 10:36:22AM +0200, Sergio Lopez wrote: > > > > Hi, > > > > > > > > While debugging BZ#1844343, I managed to reproduce the issue which > > > > leads to crash with a backtrace like this one: > > > > > > > > < snip > > > > > Thread 2 (Thread 0x7fe208463f00 (LWP 1659571)): > > > > #0 0x7fe2033b78ed in __lll_lock_wait () at /lib64/libpthread.so.0 > > > > #1 0x7fe2033b0bd4 in pthread_mutex_lock () at > > > > /lib64/libpthread.so.0 > > > > #2 0x560caa8f1e6d in qemu_mutex_lock_impl > > > > (mutex=0x560cacc68a10, file=0x560caaa9797f "util/async.c", > > > > line=521) at util/qemu-thread-posix.c:78 > > > > #3 0x560caa82414d in bdrv_set_aio_context_ignore > > > > (bs=bs@entry=0x560cacc73570, > > > > new_context=new_context@entry=0x560cacc5fed0, > > > > ignore=ignore@entry=0x7ffe388b1cc0) at block.c:6192 > > > > #4 0x560caa824503 in bdrv_child_try_set_aio_context > > > > (bs=bs@entry=0x560cacc73570, ctx=0x560cacc5fed0, > > > > ignore_child=, errp=) > > > > at block.c:6272 > > > > #5 0x560caa859e6b in blk_do_set_aio_context > > > > (blk=0x560cacecf370, new_context=0x560cacc5fed0, > > > > update_root_node=update_root_node@entry=true, errp=errp@entry=0x0) at > > > > block/block-backend.c:1989 > > > > #6 0x560caa85c501 in blk_set_aio_context > > > > (blk=, new_context=, > > > > errp=errp@entry=0x0) at block/block-backend.c:2010 > > > > #7 0x560caa61db30 in virtio_scsi_hotunplug > > > > (hotplug_dev=0x560cadaafbd0, dev=0x560cacec1210, > > > > errp=0x7ffe388b1d80) > > > > at > > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:869 > > > > #8 0x560caa6ccd1e in qdev_unplug (dev=0x560cacec1210, > > > > errp=errp@entry=0x7ffe388b1db8) > > > > at qdev-monitor.c:872 > > > > #9 0x560caa6ccd9e in qmp_device_del (id=, > > > > errp=errp@entry=0x7ffe388b1db8) > > > > at qdev-monitor.c:884 > > > > #10 0x560caa7ec4d3 in qmp_marshal_device_del > > > > (args=, ret=, errp=0x7ffe388b1e18) at > > > > qapi/qapi-commands-qdev.c:99 > > > > #11 0x560caa8a45ec in do_qmp_dispatch > > > > (errp=0x7ffe388b1e10, allow_oob=, request= > > > out>, cmds=0x560cab1928a0 ) at qapi/qmp-dispatch.c:132 > > > > #12 0x560caa8a45ec in qmp_dispatch > > > > (cmds=0x560cab1928a0 , request=, > > > > allow_oob=) > > > > at qapi/qmp-dispatch.c:175 > > > > #13 0x560caa7c2521 in monitor_qmp_dispatch (mon=0x560cacca2f00, > > > > req=) > > > > at monitor/qmp.c:145 > > > > #14 0x560caa7c2bba in monitor_qmp_bh_dispatcher (data= > > > out>) at monitor/qmp.c:234 > > > > #15 0x560caa8ec716 in aio_bh_call (bh=0x560cacbd80e0) at > > > > util/async.c:117 > > > > #16 0x560caa8ec716 in aio_bh_poll (ctx=ctx@entry=0x560cacbd6da0) at > > > > util/async.c:117 > > > > #17 0x560caa8efb04 in aio_dispatch (ctx=0x560cacbd6da0) at > > > > util/aio-posix.c:459 > > > > #18 0x560caa8ec5f2 in aio_ctx_dispatch > > > > (source=, callback=, > > > > user_data=) at util/async.c:260 > > > > #19 0x7fe2078d167d in g_main_context_dispatch () at > > > > /lib64/libglib-2.0.so.0 > > > > #20 0x560caa8eebb8 in glib_pollfds_poll () at util/main-loop.c:219 > > > > #21 0x560caa8eebb8 in os_host_main_loop_wait (timeout= > > > out>) at util/main-loop.c:242 > > > > #22 0x560caa8eebb8 in main_loop_wait (nonblocking=) > > > > at util/main-loop.c:518 > > > > #23 0x560caa6cfe51 in main_loop () at vl.c:1828 > > > > #24 0x560caa57b322 in main (argc=, argv= > > > out>, envp=) > > > > at vl.c:4504 > > > > > > > > Thread 1 (Thread 0x7fe1fb059700 (LWP 1659573)): > > > > #0 0x7fe20301b70f in raise () at /lib64/libc.so.6 > > > > #1 0x7fe203005b25 in abort () at /lib64/libc.so.6 > > > > #2 0x7fe2030059f9 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 > > > > #3 0x7fe203013cc6 in .annobin_assert.c_end () at /lib64/libc.so.6 > > > > #4 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > > > block/block-backend.c:1968 > > > > #5 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > > > block/block-backend.c:1962 > > > > #6 0x560caa61d79c in virtio_scsi_ctx_check (s=0x560cadaafbd0, > > > > s=0x560cadaafbd0, d=0x560cacec1210) > > > > at > > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:250 > > > > #7 0x560caa61d79c in virtio_scsi_handle_cmd_req_prepare > > > > (req=0x7fe1ec013880, s=0x560cadaafbd0) > > > > at > > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:569 > > > > #8 0x560caa61d79c in virtio_scsi_handle_cmd_vq > > > > (s=s@entry=0x560cadaafbd0, vq=vq@entry=0x7fe1f82ac140)
Re: virtio-scsi and another complex AioContext issue
On Mon, Jun 22, 2020 at 04:16:04PM +0200, Sergio Lopez wrote: > On Fri, Jun 19, 2020 at 02:04:06PM +0100, Stefan Hajnoczi wrote: > > On Thu, Jun 11, 2020 at 10:36:22AM +0200, Sergio Lopez wrote: > > > Hi, > > > > > > While debugging BZ#1844343, I managed to reproduce the issue which > > > leads to crash with a backtrace like this one: > > > > > > < snip > > > > Thread 2 (Thread 0x7fe208463f00 (LWP 1659571)): > > > #0 0x7fe2033b78ed in __lll_lock_wait () at /lib64/libpthread.so.0 > > > #1 0x7fe2033b0bd4 in pthread_mutex_lock () at /lib64/libpthread.so.0 > > > #2 0x560caa8f1e6d in qemu_mutex_lock_impl > > > (mutex=0x560cacc68a10, file=0x560caaa9797f "util/async.c", line=521) > > > at util/qemu-thread-posix.c:78 > > > #3 0x560caa82414d in bdrv_set_aio_context_ignore > > > (bs=bs@entry=0x560cacc73570, > > > new_context=new_context@entry=0x560cacc5fed0, > > > ignore=ignore@entry=0x7ffe388b1cc0) at block.c:6192 > > > #4 0x560caa824503 in bdrv_child_try_set_aio_context > > > (bs=bs@entry=0x560cacc73570, ctx=0x560cacc5fed0, > > > ignore_child=, errp=) > > > at block.c:6272 > > > #5 0x560caa859e6b in blk_do_set_aio_context > > > (blk=0x560cacecf370, new_context=0x560cacc5fed0, > > > update_root_node=update_root_node@entry=true, errp=errp@entry=0x0) at > > > block/block-backend.c:1989 > > > #6 0x560caa85c501 in blk_set_aio_context > > > (blk=, new_context=, > > > errp=errp@entry=0x0) at block/block-backend.c:2010 > > > #7 0x560caa61db30 in virtio_scsi_hotunplug > > > (hotplug_dev=0x560cadaafbd0, dev=0x560cacec1210, errp=0x7ffe388b1d80) > > > at > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:869 > > > #8 0x560caa6ccd1e in qdev_unplug (dev=0x560cacec1210, > > > errp=errp@entry=0x7ffe388b1db8) > > > at qdev-monitor.c:872 > > > #9 0x560caa6ccd9e in qmp_device_del (id=, > > > errp=errp@entry=0x7ffe388b1db8) > > > at qdev-monitor.c:884 > > > #10 0x560caa7ec4d3 in qmp_marshal_device_del > > > (args=, ret=, errp=0x7ffe388b1e18) at > > > qapi/qapi-commands-qdev.c:99 > > > #11 0x560caa8a45ec in do_qmp_dispatch > > > (errp=0x7ffe388b1e10, allow_oob=, request= > > out>, cmds=0x560cab1928a0 ) at qapi/qmp-dispatch.c:132 > > > #12 0x560caa8a45ec in qmp_dispatch > > > (cmds=0x560cab1928a0 , request=, > > > allow_oob=) > > > at qapi/qmp-dispatch.c:175 > > > #13 0x560caa7c2521 in monitor_qmp_dispatch (mon=0x560cacca2f00, > > > req=) > > > at monitor/qmp.c:145 > > > #14 0x560caa7c2bba in monitor_qmp_bh_dispatcher (data= > > out>) at monitor/qmp.c:234 > > > #15 0x560caa8ec716 in aio_bh_call (bh=0x560cacbd80e0) at > > > util/async.c:117 > > > #16 0x560caa8ec716 in aio_bh_poll (ctx=ctx@entry=0x560cacbd6da0) at > > > util/async.c:117 > > > #17 0x560caa8efb04 in aio_dispatch (ctx=0x560cacbd6da0) at > > > util/aio-posix.c:459 > > > #18 0x560caa8ec5f2 in aio_ctx_dispatch > > > (source=, callback=, > > > user_data=) at util/async.c:260 > > > #19 0x7fe2078d167d in g_main_context_dispatch () at > > > /lib64/libglib-2.0.so.0 > > > #20 0x560caa8eebb8 in glib_pollfds_poll () at util/main-loop.c:219 > > > #21 0x560caa8eebb8 in os_host_main_loop_wait (timeout= > > out>) at util/main-loop.c:242 > > > #22 0x560caa8eebb8 in main_loop_wait (nonblocking=) at > > > util/main-loop.c:518 > > > #23 0x560caa6cfe51 in main_loop () at vl.c:1828 > > > #24 0x560caa57b322 in main (argc=, argv= > > out>, envp=) > > > at vl.c:4504 > > > > > > Thread 1 (Thread 0x7fe1fb059700 (LWP 1659573)): > > > #0 0x7fe20301b70f in raise () at /lib64/libc.so.6 > > > #1 0x7fe203005b25 in abort () at /lib64/libc.so.6 > > > #2 0x7fe2030059f9 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 > > > #3 0x7fe203013cc6 in .annobin_assert.c_end () at /lib64/libc.so.6 > > > #4 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > > block/block-backend.c:1968 > > > #5 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > > block/block-backend.c:1962 > > > #6 0x560caa61d79c in virtio_scsi_ctx_check (s=0x560cadaafbd0, > > > s=0x560cadaafbd0, d=0x560cacec1210) > > > at > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:250 > > > #7 0x560caa61d79c in virtio_scsi_handle_cmd_req_prepare > > > (req=0x7fe1ec013880, s=0x560cadaafbd0) > > > at > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:569 > > > #8 0x560caa61d79c in virtio_scsi_handle_cmd_vq > > > (s=s@entry=0x560cadaafbd0, vq=vq@entry=0x7fe1f82ac140) > > > at > > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:612 > > > #9 0x560caa61e48e in virtio_scsi_data_plane_handle_cmd > > > (vdev=, vq=0x7fe1f82ac140) > > > at > > >
Re: virtio-scsi and another complex AioContext issue
On Fri, Jun 19, 2020 at 02:04:06PM +0100, Stefan Hajnoczi wrote: > On Thu, Jun 11, 2020 at 10:36:22AM +0200, Sergio Lopez wrote: > > Hi, > > > > While debugging BZ#1844343, I managed to reproduce the issue which > > leads to crash with a backtrace like this one: > > > > < snip > > > Thread 2 (Thread 0x7fe208463f00 (LWP 1659571)): > > #0 0x7fe2033b78ed in __lll_lock_wait () at /lib64/libpthread.so.0 > > #1 0x7fe2033b0bd4 in pthread_mutex_lock () at /lib64/libpthread.so.0 > > #2 0x560caa8f1e6d in qemu_mutex_lock_impl > > (mutex=0x560cacc68a10, file=0x560caaa9797f "util/async.c", line=521) at > > util/qemu-thread-posix.c:78 > > #3 0x560caa82414d in bdrv_set_aio_context_ignore > > (bs=bs@entry=0x560cacc73570, > > new_context=new_context@entry=0x560cacc5fed0, > > ignore=ignore@entry=0x7ffe388b1cc0) at block.c:6192 > > #4 0x560caa824503 in bdrv_child_try_set_aio_context > > (bs=bs@entry=0x560cacc73570, ctx=0x560cacc5fed0, > > ignore_child=, errp=) > > at block.c:6272 > > #5 0x560caa859e6b in blk_do_set_aio_context > > (blk=0x560cacecf370, new_context=0x560cacc5fed0, > > update_root_node=update_root_node@entry=true, errp=errp@entry=0x0) at > > block/block-backend.c:1989 > > #6 0x560caa85c501 in blk_set_aio_context > > (blk=, new_context=, errp=errp@entry=0x0) > > at block/block-backend.c:2010 > > #7 0x560caa61db30 in virtio_scsi_hotunplug > > (hotplug_dev=0x560cadaafbd0, dev=0x560cacec1210, errp=0x7ffe388b1d80) > > at > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:869 > > #8 0x560caa6ccd1e in qdev_unplug (dev=0x560cacec1210, > > errp=errp@entry=0x7ffe388b1db8) > > at qdev-monitor.c:872 > > #9 0x560caa6ccd9e in qmp_device_del (id=, > > errp=errp@entry=0x7ffe388b1db8) > > at qdev-monitor.c:884 > > #10 0x560caa7ec4d3 in qmp_marshal_device_del > > (args=, ret=, errp=0x7ffe388b1e18) at > > qapi/qapi-commands-qdev.c:99 > > #11 0x560caa8a45ec in do_qmp_dispatch > > (errp=0x7ffe388b1e10, allow_oob=, request= > out>, cmds=0x560cab1928a0 ) at qapi/qmp-dispatch.c:132 > > #12 0x560caa8a45ec in qmp_dispatch > > (cmds=0x560cab1928a0 , request=, > > allow_oob=) > > at qapi/qmp-dispatch.c:175 > > #13 0x560caa7c2521 in monitor_qmp_dispatch (mon=0x560cacca2f00, > > req=) > > at monitor/qmp.c:145 > > #14 0x560caa7c2bba in monitor_qmp_bh_dispatcher (data=) > > at monitor/qmp.c:234 > > #15 0x560caa8ec716 in aio_bh_call (bh=0x560cacbd80e0) at > > util/async.c:117 > > #16 0x560caa8ec716 in aio_bh_poll (ctx=ctx@entry=0x560cacbd6da0) at > > util/async.c:117 > > #17 0x560caa8efb04 in aio_dispatch (ctx=0x560cacbd6da0) at > > util/aio-posix.c:459 > > #18 0x560caa8ec5f2 in aio_ctx_dispatch > > (source=, callback=, user_data= > out>) at util/async.c:260 > > #19 0x7fe2078d167d in g_main_context_dispatch () at > > /lib64/libglib-2.0.so.0 > > #20 0x560caa8eebb8 in glib_pollfds_poll () at util/main-loop.c:219 > > #21 0x560caa8eebb8 in os_host_main_loop_wait (timeout=) > > at util/main-loop.c:242 > > #22 0x560caa8eebb8 in main_loop_wait (nonblocking=) at > > util/main-loop.c:518 > > #23 0x560caa6cfe51 in main_loop () at vl.c:1828 > > #24 0x560caa57b322 in main (argc=, argv=, > > envp=) > > at vl.c:4504 > > > > Thread 1 (Thread 0x7fe1fb059700 (LWP 1659573)): > > #0 0x7fe20301b70f in raise () at /lib64/libc.so.6 > > #1 0x7fe203005b25 in abort () at /lib64/libc.so.6 > > #2 0x7fe2030059f9 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 > > #3 0x7fe203013cc6 in .annobin_assert.c_end () at /lib64/libc.so.6 > > #4 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > block/block-backend.c:1968 > > #5 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > > block/block-backend.c:1962 > > #6 0x560caa61d79c in virtio_scsi_ctx_check (s=0x560cadaafbd0, > > s=0x560cadaafbd0, d=0x560cacec1210) > > at > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:250 > > #7 0x560caa61d79c in virtio_scsi_handle_cmd_req_prepare > > (req=0x7fe1ec013880, s=0x560cadaafbd0) > > at > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:569 > > #8 0x560caa61d79c in virtio_scsi_handle_cmd_vq > > (s=s@entry=0x560cadaafbd0, vq=vq@entry=0x7fe1f82ac140) > > at > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:612 > > #9 0x560caa61e48e in virtio_scsi_data_plane_handle_cmd > > (vdev=, vq=0x7fe1f82ac140) > > at > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi-dataplane.c:60 > > #10 0x560caa62bfbe in virtio_queue_notify_aio_vq (vq=) > > at > > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/virtio/virtio.c:2243 > > #11 0x560caa8ef046
Re: virtio-scsi and another complex AioContext issue
On Thu, Jun 11, 2020 at 10:36:22AM +0200, Sergio Lopez wrote: > Hi, > > While debugging BZ#1844343, I managed to reproduce the issue which > leads to crash with a backtrace like this one: > > < snip > > Thread 2 (Thread 0x7fe208463f00 (LWP 1659571)): > #0 0x7fe2033b78ed in __lll_lock_wait () at /lib64/libpthread.so.0 > #1 0x7fe2033b0bd4 in pthread_mutex_lock () at /lib64/libpthread.so.0 > #2 0x560caa8f1e6d in qemu_mutex_lock_impl > (mutex=0x560cacc68a10, file=0x560caaa9797f "util/async.c", line=521) at > util/qemu-thread-posix.c:78 > #3 0x560caa82414d in bdrv_set_aio_context_ignore > (bs=bs@entry=0x560cacc73570, > new_context=new_context@entry=0x560cacc5fed0, > ignore=ignore@entry=0x7ffe388b1cc0) at block.c:6192 > #4 0x560caa824503 in bdrv_child_try_set_aio_context > (bs=bs@entry=0x560cacc73570, ctx=0x560cacc5fed0, ignore_child= out>, errp=) > at block.c:6272 > #5 0x560caa859e6b in blk_do_set_aio_context > (blk=0x560cacecf370, new_context=0x560cacc5fed0, > update_root_node=update_root_node@entry=true, errp=errp@entry=0x0) at > block/block-backend.c:1989 > #6 0x560caa85c501 in blk_set_aio_context > (blk=, new_context=, errp=errp@entry=0x0) > at block/block-backend.c:2010 > #7 0x560caa61db30 in virtio_scsi_hotunplug > (hotplug_dev=0x560cadaafbd0, dev=0x560cacec1210, errp=0x7ffe388b1d80) > at > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:869 > #8 0x560caa6ccd1e in qdev_unplug (dev=0x560cacec1210, > errp=errp@entry=0x7ffe388b1db8) > at qdev-monitor.c:872 > #9 0x560caa6ccd9e in qmp_device_del (id=, > errp=errp@entry=0x7ffe388b1db8) > at qdev-monitor.c:884 > #10 0x560caa7ec4d3 in qmp_marshal_device_del > (args=, ret=, errp=0x7ffe388b1e18) at > qapi/qapi-commands-qdev.c:99 > #11 0x560caa8a45ec in do_qmp_dispatch > (errp=0x7ffe388b1e10, allow_oob=, request=, > cmds=0x560cab1928a0 ) at qapi/qmp-dispatch.c:132 > #12 0x560caa8a45ec in qmp_dispatch > (cmds=0x560cab1928a0 , request=, > allow_oob=) > at qapi/qmp-dispatch.c:175 > #13 0x560caa7c2521 in monitor_qmp_dispatch (mon=0x560cacca2f00, > req=) > at monitor/qmp.c:145 > #14 0x560caa7c2bba in monitor_qmp_bh_dispatcher (data=) at > monitor/qmp.c:234 > #15 0x560caa8ec716 in aio_bh_call (bh=0x560cacbd80e0) at util/async.c:117 > #16 0x560caa8ec716 in aio_bh_poll (ctx=ctx@entry=0x560cacbd6da0) at > util/async.c:117 > #17 0x560caa8efb04 in aio_dispatch (ctx=0x560cacbd6da0) at > util/aio-posix.c:459 > #18 0x560caa8ec5f2 in aio_ctx_dispatch > (source=, callback=, user_data= out>) at util/async.c:260 > #19 0x7fe2078d167d in g_main_context_dispatch () at > /lib64/libglib-2.0.so.0 > #20 0x560caa8eebb8 in glib_pollfds_poll () at util/main-loop.c:219 > #21 0x560caa8eebb8 in os_host_main_loop_wait (timeout=) at > util/main-loop.c:242 > #22 0x560caa8eebb8 in main_loop_wait (nonblocking=) at > util/main-loop.c:518 > #23 0x560caa6cfe51 in main_loop () at vl.c:1828 > #24 0x560caa57b322 in main (argc=, argv=, > envp=) > at vl.c:4504 > > Thread 1 (Thread 0x7fe1fb059700 (LWP 1659573)): > #0 0x7fe20301b70f in raise () at /lib64/libc.so.6 > #1 0x7fe203005b25 in abort () at /lib64/libc.so.6 > #2 0x7fe2030059f9 in _nl_load_domain.cold.0 () at /lib64/libc.so.6 > #3 0x7fe203013cc6 in .annobin_assert.c_end () at /lib64/libc.so.6 > #4 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > block/block-backend.c:1968 > #5 0x560caa85bfe4 in blk_get_aio_context (blk=0x560cacecf370) at > block/block-backend.c:1962 > #6 0x560caa61d79c in virtio_scsi_ctx_check (s=0x560cadaafbd0, > s=0x560cadaafbd0, d=0x560cacec1210) > at > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:250 > #7 0x560caa61d79c in virtio_scsi_handle_cmd_req_prepare > (req=0x7fe1ec013880, s=0x560cadaafbd0) > at > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:569 > #8 0x560caa61d79c in virtio_scsi_handle_cmd_vq > (s=s@entry=0x560cadaafbd0, vq=vq@entry=0x7fe1f82ac140) > at > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi.c:612 > #9 0x560caa61e48e in virtio_scsi_data_plane_handle_cmd (vdev= out>, vq=0x7fe1f82ac140) > at > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/scsi/virtio-scsi-dataplane.c:60 > #10 0x560caa62bfbe in virtio_queue_notify_aio_vq (vq=) > at > /usr/src/debug/qemu-kvm-4.2.0-22.module+el8.2.1+6758+cb8d64c2.x86_64/hw/virtio/virtio.c:2243 > #11 0x560caa8ef046 in run_poll_handlers_once > (ctx=ctx@entry=0x560cacc689b0, timeout=timeout@entry=0x7fe1fb058658) at > util/aio-posix.c:517 > #12 0x560caa8efbc5 in try_poll_mode (timeout=0x7fe1fb058658, > ctx=0x560cacc689b0) > at util/aio-posix.c:607 > #13
