[PATCH v5 14/21] block/export: rewrite vduse-blk drain code

2023-05-04 Thread Stefan Hajnoczi
resume them in .drained_end(). This eliminates the need for the aio_set_fd_handler(is_external=true) flag, which is being removed from QEMU. This is a long list but splitting it into individual commits would probably lead to git bisect failures - the changes are all related. Signed-off-by:

[PATCH v5 11/21] block: drain from main loop thread in bdrv_co_yield_to_drain()

2023-05-04 Thread Stefan Hajnoczi
, and BlockDriver. Narrow IO_OR_GS_CODE() to GLOBAL_STATE_CODE() where appropriate. The test-bdrv-drain test case calls bdrv_drain() from an IOThread. This is now only allowed from coroutine context, so update the test case to run in a coroutine. Signed-off-by: Stefan Hajnoczi --- include/block

[PATCH v5 02/21] block-backend: split blk_do_set_aio_context()

2023-05-04 Thread Stefan Hajnoczi
oot_set_aio_ctx_commit() being invoked as part of the AioContext change propagation. This can be solved by temporarily setting blk->allow_aio_context_change to true. Future patches call blk_get_aio_context() from BlockDevOps->drained_end(), so this patch will become necessary. Signed-off-by: Stefa

[PATCH v5 05/21] virtio-scsi: stop using aio_disable_external() during unplug

2023-05-04 Thread Stefan Hajnoczi
ht requests remain once qdev_simple_device_unplug_cb() returns. Thanks to these two conditions we don't need aio_disable_external() anymore. Cc: Zhengui Li Reviewed-by: Paolo Bonzini Reviewed-by: Daniil Tatianin Signed-off-by: Stefan Hajnoczi --- hw/scsi/virtio-scsi.c | 3 --- 1 file changed, 3 d

[PATCH v5 13/21] hw/xen: do not set is_external=true on evtchn fds

2023-05-04 Thread Stefan Hajnoczi
o use BlockDevOps .drained_begin/end() callbacks. It no longer relies on is_external=true so it is safe to pass is_external=false. This is part of ongoing work to remove the aio_disable_external() API. Signed-off-by: Stefan Hajnoczi --- hw/xen/xen-bus.c | 8 1 file changed, 4 insertions(

[PATCH v5 12/21] xen-block: implement BlockDevOps->drained_begin()

2023-05-04 Thread Stefan Hajnoczi
depend on the dataplane instance, so move the blk_set_dev_ops() call after xen_block_dataplane_create(). Signed-off-by: Stefan Hajnoczi --- hw/block/dataplane/xen-block.h | 2 ++ hw/block/dataplane/xen-block.c | 42 +- hw/block/xen-block.c

[PATCH v5 09/21] hw/xen: do not use aio_set_fd_handler(is_external=true) in xen_xenstore

2023-05-04 Thread Stefan Hajnoczi
ff-by: Stefan Hajnoczi --- hw/i386/kvm/xen_xenstore.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/i386/kvm/xen_xenstore.c b/hw/i386/kvm/xen_xenstore.c index 900679af8a..6e81bc8791 100644 --- a/hw/i386/kvm/xen_xenstore.c +++ b/hw/i386/kvm/xen_xenstore.c @@ -133,7 +

[PATCH v5 18/21] virtio-blk: implement BlockDevOps->drained_begin()

2023-05-04 Thread Stefan Hajnoczi
attaching/detaching ioeventfds if the data plane is started/stopped during a drained section. This should be rare, but maybe the mirror block job can trigger it. Signed-off-by: Stefan Hajnoczi --- hw/block/dataplane/virtio-blk.c | 17 +-- hw/block/virtio-blk.c | 38

[PATCH v5 01/21] block: Fix use after free in blockdev_mark_auto_del()

2023-05-04 Thread Stefan Hajnoczi
. For both reasons, we can't just continue at block_job_next_locked(job). Instead, start at the head of the list again after job_cancel_locked() and skip those jobs that we already cancelled (or that are completing anyway). Signed-off-by: Kevin Wolf Reviewed-by: Stefan Hajnoczi Signed-off-by: S

[PATCH v5 00/21] block: remove aio_disable_external() API

2023-05-04 Thread Stefan Hajnoczi
ew I/O requests are not submitted in drained sections. Kevin Wolf (1): block: Fix use after free in blockdev_mark_auto_del() Stefan Hajnoczi (20): block-backend: split blk_do_set_aio_context() hw/qdev: introduce qdev_is_realized() helper virtio-scsi: avoid race between unplug and transp

Re: [PATCH v2 01/12] simpletrace: Improve parsing of sys.argv; fix files never closed.

2023-05-04 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 11:23:28AM +0200, Mads Ynddal wrote: > From: Mads Ynddal > > The arguments extracted from `sys.argv` named and unpacked to make it > clear what the arguments are and what they're used for. > > The two input files were opened, but never explicitly closed. File usage > chan

Re: [PATCH v2 00/12] simpletrace: refactor and general improvements

2023-05-04 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 11:23:27AM +0200, Mads Ynddal wrote: > From: Mads Ynddal > > I wanted to use simpletrace.py for an internal project, so I tried to update > and polish the code. Some of the commits resolve specific issues, while some > are more subjective. An internal project based on qem

Re: [PATCH] block: Fix use after free in blockdev_mark_auto_del()

2023-05-03 Thread Stefan Hajnoczi
> anyway). > > Signed-off-by: Kevin Wolf > --- > blockdev.c | 18 ++ > 1 file changed, 14 insertions(+), 4 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH] block/export: call blk_set_dev_ops(blk, NULL, NULL)

2023-05-03 Thread Stefan Hajnoczi
On Wed, May 03, 2023 at 10:43:16AM -0500, Eric Blake wrote: > On Tue, May 02, 2023 at 05:11:19PM -0400, Stefan Hajnoczi wrote: > > Most export types install BlockDeviceOps pointers. It is easy to forget > > to remove them because that happens automatically via the "drive"

Re: [PATCH] block/blkio: add 'fd' option to virtio-blk-vhost-vdpa driver

2023-05-03 Thread Stefan Hajnoczi
On Wed, May 03, 2023 at 11:15:56AM +0200, Stefano Garzarella wrote: > On Tue, May 02, 2023 at 03:02:32PM -0400, Stefan Hajnoczi wrote: > > On Tue, May 02, 2023 at 04:50:50PM +0200, Stefano Garzarella wrote: > > > The virtio-blk-vhost-vdpa driver in libblkio 1.3.0 supports

Re: [PATCH v4 03/20] virtio-scsi: avoid race between unplug and transport event

2023-05-03 Thread Stefan Hajnoczi
On Wed, May 03, 2023 at 10:00:57AM +0200, Kevin Wolf wrote: > Am 02.05.2023 um 20:56 hat Stefan Hajnoczi geschrieben: > > On Tue, May 02, 2023 at 05:19:46PM +0200, Kevin Wolf wrote: > > > Am 25.04.2023 um 19:26 hat Stefan Hajnoczi geschrieben: > > > > Only report

Re: [PATCH v4 07/20] block/export: stop using is_external in vhost-user-blk server

2023-05-03 Thread Stefan Hajnoczi
On Wed, May 03, 2023 at 10:08:46AM +0200, Kevin Wolf wrote: > Am 02.05.2023 um 22:06 hat Stefan Hajnoczi geschrieben: > > On Tue, May 02, 2023 at 06:04:24PM +0200, Kevin Wolf wrote: > > > Am 25.04.2023 um 19:27 hat Stefan Hajnoczi geschrieben: > > > > vhost-user act

Re: [PATCH v4 04/20] virtio-scsi: stop using aio_disable_external() during unplug

2023-05-03 Thread Stefan Hajnoczi
On Wed, May 03, 2023 at 01:40:49PM +0200, Kevin Wolf wrote: > Am 02.05.2023 um 22:02 hat Stefan Hajnoczi geschrieben: > > On Tue, May 02, 2023 at 03:19:52PM +0200, Kevin Wolf wrote: > > > Am 01.05.2023 um 17:09 hat Stefan Hajnoczi geschrieben: > > > > On Fri, Ap

[PATCH] block/export: call blk_set_dev_ops(blk, NULL, NULL)

2023-05-02 Thread Stefan Hajnoczi
eed to remember. This fixes the nbd and vhost-user-blk export types. Fixes: fd6afc501a01 ("nbd/server: Use drained block ops to quiesce the server") Fixes: ca858a5fe94c ("vhost-user-blk-server: notify client about disk resize") Signed-off-by: Stefan Hajnoczi --- block/export/exp

Re: [PATCH v4 10/20] block: drain from main loop thread in bdrv_co_yield_to_drain()

2023-05-02 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 06:21:20PM +0200, Kevin Wolf wrote: > Am 25.04.2023 um 19:27 hat Stefan Hajnoczi geschrieben: > > For simplicity, always run BlockDevOps .drained_begin/end/poll() > > callbacks in the main loop thread. This makes it easier to implement the > > callb

Re: [PATCH v4 07/20] block/export: stop using is_external in vhost-user-blk server

2023-05-02 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 06:04:24PM +0200, Kevin Wolf wrote: > Am 25.04.2023 um 19:27 hat Stefan Hajnoczi geschrieben: > > vhost-user activity must be suspended during bdrv_drained_begin/end(). > > This prevents new requests from interfering with whatever is happening > > i

Re: [PATCH v4 04/20] virtio-scsi: stop using aio_disable_external() during unplug

2023-05-02 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 03:19:52PM +0200, Kevin Wolf wrote: > Am 01.05.2023 um 17:09 hat Stefan Hajnoczi geschrieben: > > On Fri, Apr 28, 2023 at 04:22:55PM +0200, Kevin Wolf wrote: > > > Am 25.04.2023 um 19:27 hat Stefan Hajnoczi geschrieben: > > > > This patch is p

Re: [PATCH v4 06/20] block/export: wait for vhost-user-blk requests when draining

2023-05-02 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 05:42:51PM +0200, Kevin Wolf wrote: > Am 25.04.2023 um 19:27 hat Stefan Hajnoczi geschrieben: > > Each vhost-user-blk request runs in a coroutine. When the BlockBackend > > enters a drained section we need to enter a quiescent state. Currently > > a

Re: [PATCH] block/blkio: add 'fd' option to virtio-blk-vhost-vdpa driver

2023-05-02 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 04:50:50PM +0200, Stefano Garzarella wrote: > The virtio-blk-vhost-vdpa driver in libblkio 1.3.0 supports the new > 'fd' property. Let's expose this to the user, so the management layer > can pass the file descriptor of an already opened vhost-vdpa character > device. This i

Re: [PATCH v4 03/20] virtio-scsi: avoid race between unplug and transport event

2023-05-02 Thread Stefan Hajnoczi
On Tue, May 02, 2023 at 05:19:46PM +0200, Kevin Wolf wrote: > Am 25.04.2023 um 19:26 hat Stefan Hajnoczi geschrieben: > > Only report a transport reset event to the guest after the SCSIDevice > > has been unrealized by qdev_simple_device_unplug_cb(). > > > > qdev_sim

[PATCH 0/2] aio-posix: do not nest poll handlers

2023-05-02 Thread Stefan Hajnoczi
ce the bug, but I have included a test case that demonstrates the problem. Stefan Hajnoczi (2): aio-posix: do not nest poll handlers tested: add test for nested aio_poll() in poll handlers tests/unit/test-nested-aio-poll.c | 130 ++ util/aio-posix.c

[PATCH 2/2] tested: add test for nested aio_poll() in poll handlers

2023-05-02 Thread Stefan Hajnoczi
Signed-off-by: Stefan Hajnoczi --- tests/unit/test-nested-aio-poll.c | 130 ++ tests/unit/meson.build| 1 + 2 files changed, 131 insertions(+) create mode 100644 tests/unit/test-nested-aio-poll.c diff --git a/tests/unit/test-nested-aio-poll.c b/tests

[PATCH 1/2] aio-posix: do not nest poll handlers

2023-05-02 Thread Stefan Hajnoczi
ppe Esposito Cc: Paolo Bonzini Signed-off-by: Stefan Hajnoczi --- util/aio-posix.c | 11 +++ 1 file changed, 11 insertions(+) diff --git a/util/aio-posix.c b/util/aio-posix.c index a8be940f76..34bc2a64d8 100644 --- a/util/aio-posix.c +++ b/util/aio-posix.c @@ -353,8 +353,19 @@ static bool

Re: [PATCH] async: avoid use-after-free on re-entrancy guard

2023-05-02 Thread Stefan Hajnoczi
8513 > Fixes: 9c86c97f12 ("async: Add an optional reentrancy guard to the BH API") > Signed-off-by: Alexander Bulekov > --- > util/async.c | 14 -- > 1 file changed, 8 insertions(+), 6 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

[PATCH v2] vhost-user: send SET_STATUS 0 after GET_VRING_BASE

2023-05-02 Thread Stefan Hajnoczi
Tsirkin Cc: Cindy Lu Cc: Yajun Wu Signed-off-by: Stefan Hajnoczi --- v2: - Added VHOST_USER_PROTOCOL_F_STATUS check [Yajun Wu] - Added "Fixes:" tag [Michael] --- hw/virtio/vhost-user.c | 16 +++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/hw/virtio/vhost-us

Re: [PATCH] vhost-user: send SET_STATUS 0 after GET_VRING_BASE

2023-05-01 Thread Stefan Hajnoczi
On Fri, Apr 21, 2023 at 01:30:48PM +0800, Yajun Wu wrote: > > On 4/20/2023 9:07 PM, Stefan Hajnoczi wrote: > > > > Setting the VIRTIO Device Status Field to 0 resets the device. The > > device's state is lost, including the vring configuration. > > > >

Re: [PATCH 00/10] tracing: remove dynamic vcpu state

2023-05-01 Thread Stefan Hajnoczi
ce/trace-hmp-cmds.c| 17 +- > scripts/qapi/gen.py | 4 +- > scripts/tracetool/format/c.py | 6 -- > scripts/tracetool/format/h.py | 16 + > trace-events | 50 --- > 25 files changed, 43 insertions(+), 486 deletions(-) &g

Re: [PATCH 20/20] block: Mark bdrv_refresh_limits() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
- > include/block/block-global-state.h | 5 - > include/block/block_int-common.h | 3 ++- > block.c| 9 + > block/io.c | 1 - > 4 files changed, 15 insertions(+), 3 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 19/20] block: Mark bdrv_recurse_can_replace() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
quorum.c | 4 ++-- > blockdev.c | 3 +++ > 7 files changed, 19 insertions(+), 10 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 18/20] block: Mark bdrv_query_block_graph_info() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
gt; --- > include/block/qapi.h | 7 --- > qemu-img.c | 2 ++ > 2 files changed, 6 insertions(+), 3 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 17/20] block: Mark bdrv_query_bds_stats() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
gt; block/qapi.c | 6 -- > 1 file changed, 4 insertions(+), 2 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 16/20] block: Mark BlockDriver callbacks for amend job GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
sito > Signed-off-by: Kevin Wolf > --- > include/block/block_int-common.h | 12 ++-- > block/amend.c| 8 +++- > 2 files changed, 13 insertions(+), 7 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 15/20] block: Mark bdrv_co_debug_event() GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
> include/block/block_int-common.h | 4 ++-- > block.c | 2 ++ > 3 files changed, 9 insertions(+), 6 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 14/20] block: Mark bdrv_co_get_info() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
gt; block/mirror.c | 8 ++-- > block/raw-format.c | 2 +- > 7 files changed, 22 insertions(+), 14 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 13/20] block: Mark bdrv_co_get_allocated_file_size() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
On Tue, Apr 25, 2023 at 07:31:51PM +0200, Kevin Wolf wrote: > @@ -5778,6 +5779,7 @@ int64_t coroutine_fn > bdrv_co_get_allocated_file_size(BlockDriverState *bs) > { > BlockDriver *drv = bs->drv; > IO_CODE(); > +assert_bdrv_graph_readable(); Is there a need for runtime assertions in

Re: [PATCH 12/20] mirror: Take graph lock for accessing a node's parent list

2023-05-01 Thread Stefan Hajnoczi
; block/mirror.c | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) The commit message is misleading. This commit does not take the graph lock, it declares that the caller must already hold the graph lock. Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 11/20] vhdx: Take graph lock for accessing a node's parent list

2023-05-01 Thread Stefan Hajnoczi
gt; block/vhdx.c | 9 + > 1 file changed, 5 insertions(+), 4 deletions(-) The commit message is misleading. This commit does not take the graph lock, it declares that the caller must already hold the graph lock. Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 10/20] nbd: Mark nbd_co_do_establish_connection() and callers GRAPH_RDLOCK

2023-05-01 Thread Stefan Hajnoczi
pe Esposito > Signed-off-by: Kevin Wolf > --- > block/coroutines.h | 5 +++-- > block/nbd.c| 39 +-- > 2 files changed, 24 insertions(+), 20 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

[PATCH] block: compile out assert_bdrv_graph_readable() by default

2023-05-01 Thread Stefan Hajnoczi
general --enable-debug option. Signed-off-by: Stefan Hajnoczi --- meson_options.txt | 2 ++ configure | 1 + meson.build | 2 ++ block/graph-lock.c| 3 +++ scripts/meson-buildoptions.sh | 4 5 files changed, 12 insertions(+) diff

Re: [PATCH 09/20] nbd: Remove nbd_co_flush() wrapper function

2023-05-01 Thread Stefan Hajnoczi
| 11 +++ > 1 file changed, 3 insertions(+), 8 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 08/20] block: .bdrv_open is non-coroutine and unlocked

2023-05-01 Thread Stefan Hajnoczi
--- > 4 files changed, 21 insertions(+), 26 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 07/20] graph-lock: Fix GRAPH_RDLOCK_GUARD*() to be reader lock

2023-05-01 Thread Stefan Hajnoczi
On Tue, Apr 25, 2023 at 07:31:45PM +0200, Kevin Wolf wrote: > GRAPH_RDLOCK_GUARD() and GRAPH_RDLOCK_GUARD_MAINLOOP() only take a > reader lock for the graph, so the correct annotation for them to use is > TSA_ASSERT_SHARED rather than TSA_ASSERT. > > Signed-off-by: Kevin Wolf > --- > include/blo

Re: [PATCH 06/20] graph-lock: Add GRAPH_UNLOCKED(_PTR)

2023-05-01 Thread Stefan Hajnoczi
insertions(+) Modulo Eric's comment about the commit description: Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 05/20] test-bdrv-drain: Don't modify the graph in coroutines

2023-05-01 Thread Stefan Hajnoczi
On Tue, Apr 25, 2023 at 07:31:43PM +0200, Kevin Wolf wrote: > test-bdrv-drain contains a few test cases that are run both in coroutine > and non-coroutine context. Running the entire code including the setup > and shutdown in coroutines is incorrect because graph modifications can > generally not h

Re: [PATCH 04/20] block: Don't call no_coroutine_fns in qmp_block_resize()

2023-05-01 Thread Stefan Hajnoczi
On Tue, Apr 25, 2023 at 07:31:42PM +0200, Kevin Wolf wrote: > This QMP handler runs in a coroutine, so it must use the corresponding > no_co_wrappers instead. > > Signed-off-by: Kevin Wolf > --- > blockdev.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-)

Re: [PATCH 03/20] block: bdrv/blk_co_unref() for calls in coroutine context

2023-05-01 Thread Stefan Hajnoczi
| 18 +- > block/vpc.c | 6 +++--- > 12 files changed, 44 insertions(+), 40 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 02/20] block: Consistently call bdrv_activate() outside coroutine

2023-05-01 Thread Stefan Hajnoczi
ertions(+), 2 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH 01/20] qcow2: Don't call bdrv_getlength() in coroutine_fns

2023-05-01 Thread Stefan Hajnoczi
ck/qcow2.c | 19 +-- > 3 files changed, 13 insertions(+), 12 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH] test-aio-multithread: simplify test_multi_co_schedule

2023-05-01 Thread Stefan Hajnoczi
++-- > 1 file changed, 12 insertions(+), 6 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PATCH v4 04/20] virtio-scsi: stop using aio_disable_external() during unplug

2023-05-01 Thread Stefan Hajnoczi
On Fri, Apr 28, 2023 at 04:22:55PM +0200, Kevin Wolf wrote: > Am 25.04.2023 um 19:27 hat Stefan Hajnoczi geschrieben: > > This patch is part of an effort to remove the aio_disable_external() > > API because it does not fit in a multi-queue block layer world where > > ma

Re: [PATCH] test-aio-multithread: do not use mb_read/mb_set for simple flags

2023-05-01 Thread Stefan Hajnoczi
tests/unit/test-aio-multithread.c | 12 ++-- > 1 file changed, 6 insertions(+), 6 deletions(-) Reviewed-by: Stefan Hajnoczi signature.asc Description: PGP signature

Re: [PULL 00/17] Block patches

2023-05-01 Thread Stefan Hajnoczi
On Sat, Apr 29, 2023 at 11:05:06PM +0100, Richard Henderson wrote: > On 4/28/23 13:39, Stefan Hajnoczi wrote: > > The following changes since commit 05d50ba2d4668d43a835c5a502efdec9b92646e6: > > > >Merge tag 'migration-20230427-pull-request' of > > https

[PULL 08/17] docs/zoned-storage: add zoned device documentation

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Add the documentation about the zoned device support to virtio-blk emulation. Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Reviewed-by: Damien Le Moal Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Message-id: 20230427172019.3345-9

[PULL 09/17] file-posix: add tracking of the zone write pointers

2023-04-28 Thread Stefan Hajnoczi
zone - write to a zone - zone append Signed-off-by: Sam Li Message-id: 20230427172339.3709-2-faithilike...@gmail.com [Fix errno propagation from handle_aiocb_zone_mgmt() --Stefan] Signed-off-by: Stefan Hajnoczi --- include/block/block-common.h | 14 +++ include/block/block_int-common.h

[PULL 00/17] Block patches

2023-04-28 Thread Stefan Hajnoczi
The following changes since commit 05d50ba2d4668d43a835c5a502efdec9b92646e6: Merge tag 'migration-20230427-pull-request' of https://gitlab.com/juan.quintela/qemu into staging (2023-04-28 08:35:06 +0100) are available in the Git repository at: https://gitlab.com/stefanha/qemu.git tags/block-

Re: [PATCH v19 0/8] Add support for zoned device

2023-04-28 Thread Stefan Hajnoczi
On Fri, Apr 28, 2023 at 01:20:11AM +0800, Sam Li wrote: > Zoned Block Devices (ZBDs) devide the LBA space to block regions called zones > that are larger than the LBA size. It can only allow sequential writes, which > reduces write amplification in SSD, leading to higher throughput and increased >

[PULL 07/17] block: add some trace events for new block layer APIs

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Message-id: 20230427172019.3345-8-faithilike...@gmail.com Message-id: 20230324090605.28361-8-faithilike...@gmail.com Signed-off-by: Stefan

[PULL 04/17] block/raw-format: add zone operations to pass through requests

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li raw-format driver usually sits on top of file-posix driver. It needs to pass through requests of zone commands. Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed

Re: [PATCH v10 0/4] Add zone append write for zoned device

2023-04-28 Thread Stefan Hajnoczi
On Fri, Apr 28, 2023 at 01:23:35AM +0800, Sam Li wrote: > This patch series add zone append operation based on the previous > zoned device support part. The file-posix driver is modified to > add zone append emulation using regular writes. > > v9: > - address review comments [Stefan] > * fix get

[PULL 10/17] block: introduce zone append write for zoned devices

2023-04-28 Thread Stefan Hajnoczi
completion the device will respond with the position where the data has been written in the zone. Signed-off-by: Sam Li Reviewed-by: Dmitry Fomichev Reviewed-by: Stefan Hajnoczi Message-id: 20230427172339.3709-3-faithilike...@gmail.com Signed-off-by: Stefan Hajnoczi --- include/block/block

[PULL 13/17] include: update virtio_blk headers to v6.3-rc1

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Use scripts/update-linux-headers.sh to update headers to 6.3-rc1. Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Reviewed-by: Dmitry Fomichev Signed-off-by: Stefan Hajnoczi [Reran scripts/update-linux-headers.sh on Linux v6.3. The only change was the use of __virtioXX types

[PULL 15/17] block: add accounting for zone append operation

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Taking account of the new zone append write operation for zoned devices, BLOCK_ACCT_ZONE_APPEND enum is introduced as other I/O request type (read, write, flush). Signed-off-by: Sam Li Signed-off-by: Stefan Hajnoczi Message-Id: <20230407082528.18841-4-faithilike...@gmail.

[PULL 14/17] virtio-blk: add zoned storage emulation for zoned devices

2023-04-28 Thread Stefan Hajnoczi
Signed-off-by: Stefan Hajnoczi Message-Id: <20230407082528.18841-3-faithilike...@gmail.com> --- hw/block/virtio-blk-common.c | 2 + hw/block/virtio-blk.c| 389 +++ hw/virtio/virtio-qmp.c | 2 + 3 files changed, 393 insertions(+) diff --gi

[PULL 17/17] docs/zoned-storage:add zoned emulation use case

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Add the documentation about the example of using virtio-blk driver to pass the zoned block devices through to the guest. Signed-off-by: Sam Li Signed-off-by: Stefan Hajnoczi [Fix rST syntax --Stefan] Message-Id: <20230407082528.18841-6-faithilike...@gmail.com> --- docs

[PULL 16/17] virtio-blk: add some trace events for zoned emulation

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Signed-off-by: Stefan Hajnoczi Message-Id: <20230407082528.18841-5-faithilike...@gmail.com> --- hw/block/virtio-blk.c | 12 hw/block/trace-events | 7 +++ 2 files changed, 19 insertions(+) diff --gi

[PULL 05/17] block: add zoned BlockDriver check to block layer

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Putting zoned/non-zoned BlockDrivers on top of each other is not allowed. Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Reviewed-by: Hannes Reinecke Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Message-id: 20230427172019.3345-6

[PULL 02/17] block/file-posix: introduce helper functions for sysfs attributes

2023-04-28 Thread Stefan Hajnoczi
Reviewed-by: Stefan Hajnoczi Reviewed-by: Damien Le Moal Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Message-id: 20230427172019.3345-3-faithilike...@gmail.com Message-id: 20230324090605.28361-3-faithilike...@gmail.com [Adjust commit message prefix as suggested by

[PULL 06/17] iotests: test new zone operations

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li The new block layer APIs of zoned block devices can be tested by: $ tests/qemu-iotests/check zoned Run each zone operation on a newly created null_blk device and see whether it outputs the same zone information. Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Acked-by: Kevin

[PULL 11/17] qemu-iotests: test zone append operation

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li The patch tests zone append writes by reporting the zone wp after the completion of the call. "zap -p" option can print the sector offset value after completion, which should be the start sector where the append write begins. Signed-off-by: Sam Li Reviewed-by: Stefa

[PULL 12/17] block: add some trace events for zone append

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Signed-off-by: Sam Li Reviewed-by: Dmitry Fomichev Reviewed-by: Stefan Hajnoczi Message-id: 20230427172339.3709-5-faithilike...@gmail.com Signed-off-by: Stefan Hajnoczi --- block/file-posix.c | 3 +++ block/trace-events | 2 ++ 2 files changed, 5 insertions(+) diff --git a

[PULL 03/17] block/block-backend: add block layer APIs resembling Linux ZonedBlockDevice ioctls

2023-04-28 Thread Stefan Hajnoczi
=host_device, filename=/dev/nullb0 -c "zrp offset nr_zones" Signed-off-by: Sam Li Reviewed-by: Hannes Reinecke Reviewed-by: Stefan Hajnoczi Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Message-id: 20230427172019.3345-4-faithilike...@gmail.com

[PULL 01/17] block/block-common: add zoned device structs

2023-04-28 Thread Stefan Hajnoczi
From: Sam Li Signed-off-by: Sam Li Reviewed-by: Stefan Hajnoczi Reviewed-by: Damien Le Moal Reviewed-by: Hannes Reinecke Reviewed-by: Dmitry Fomichev Acked-by: Kevin Wolf Signed-off-by: Stefan Hajnoczi Message-id: 20230427172019.3345-2-faithilike...@gmail.com Message-id

[PATCH] aio-posix: zero out io_uring sqe user_data

2023-04-26 Thread Stefan Hajnoczi
le fdmon-io_uring in the future (and I'm working on that). Signed-off-by: Stefan Hajnoczi --- util/fdmon-io_uring.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/util/fdmon-io_uring.c b/util/fdmon-io_uring.c index ab43052dd7..35165bcb46 100644 --- a/util/fdmon-io_uring.c +++ b/util/fdm

[PATCH v4 17/20] virtio-blk: implement BlockDevOps->drained_begin()

2023-04-25 Thread Stefan Hajnoczi
attaching/detaching ioeventfds if the data plane is started/stopped during a drained section. This should be rare, but maybe the mirror block job can trigger it. Signed-off-by: Stefan Hajnoczi --- hw/block/dataplane/virtio-blk.c | 17 +-- hw/block/virtio-blk.c | 38

[PATCH v4 14/20] block/export: don't require AioContext lock around blk_exp_ref/unref()

2023-04-25 Thread Stefan Hajnoczi
The FUSE export calls blk_exp_ref/unref() without the AioContext lock. Instead of fixing the FUSE export, adjust blk_exp_ref/unref() so they work without the AioContext lock. This way it's less error-prone. Suggested-by: Paolo Bonzini Signed-off-by: Stefan Hajnoczi --- include/block/exp

[PATCH v4 03/20] virtio-scsi: avoid race between unplug and transport event

2023-04-25 Thread Stefan Hajnoczi
by: Daniil Tatianin Signed-off-by: Stefan Hajnoczi --- hw/scsi/scsi-bus.c| 3 ++- hw/scsi/virtio-scsi.c | 18 +- 2 files changed, 11 insertions(+), 10 deletions(-) diff --git a/hw/scsi/scsi-bus.c b/hw/scsi/scsi-bus.c index 07275fb631..64d7311757 100644 --- a/hw/scsi/scsi-b

[PATCH v4 16/20] virtio: make it possible to detach host notifier from any thread

2023-04-25 Thread Stefan Hajnoczi
: Stefan Hajnoczi --- hw/block/dataplane/virtio-blk.c | 2 ++ hw/scsi/virtio-scsi-dataplane.c | 9 + 2 files changed, 11 insertions(+) diff --git a/hw/block/dataplane/virtio-blk.c b/hw/block/dataplane/virtio-blk.c index b28d81737e..bd7cc6e76b 100644 --- a/hw/block/dataplane/virtio-blk.c +++ b

[PATCH v4 06/20] block/export: wait for vhost-user-blk requests when draining

2023-04-25 Thread Stefan Hajnoczi
when there are requests in flight. The in-flight counter needs to be atomic. Signed-off-by: Stefan Hajnoczi --- include/qemu/vhost-user-server.h | 4 +++- block/export/vhost-user-blk-server.c | 16 util/vhost-user-server.c | 14 ++ 3 files c

[PATCH v4 02/20] hw/qdev: introduce qdev_is_realized() helper

2023-04-25 Thread Stefan Hajnoczi
-off-by: Stefan Hajnoczi --- include/hw/qdev-core.h | 17 ++--- hw/scsi/scsi-bus.c | 3 +-- 2 files changed, 15 insertions(+), 5 deletions(-) diff --git a/include/hw/qdev-core.h b/include/hw/qdev-core.h index bd50ad5ee1..4d734cf35e 100644 --- a/include/hw/qdev-core.h +++ b/include

[PATCH v4 05/20] util/vhost-user-server: rename refcount to in_flight counter

2023-04-25 Thread Stefan Hajnoczi
e are no more requests. Avoid confusing by renaming refcount and ref/unref to in_flight and inc/dec. Reviewed-by: Paolo Bonzini Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Stefan Hajnoczi --- include/qemu/vhost-user-server.h | 6 +++--- block/export/vhost-user-blk-server.c

[PATCH v4 12/20] hw/xen: do not set is_external=true on evtchn fds

2023-04-25 Thread Stefan Hajnoczi
o use BlockDevOps .drained_begin/end() callbacks. It no longer relies on is_external=true so it is safe to pass is_external=false. This is part of ongoing work to remove the aio_disable_external() API. Signed-off-by: Stefan Hajnoczi --- hw/xen/xen-bus.c | 8 1 file changed, 4 insertions(

[PATCH v4 08/20] hw/xen: do not use aio_set_fd_handler(is_external=true) in xen_xenstore

2023-04-25 Thread Stefan Hajnoczi
ff-by: Stefan Hajnoczi --- hw/i386/kvm/xen_xenstore.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hw/i386/kvm/xen_xenstore.c b/hw/i386/kvm/xen_xenstore.c index 900679af8a..6e81bc8791 100644 --- a/hw/i386/kvm/xen_xenstore.c +++ b/hw/i386/kvm/xen_xenstore.c @@ -133,7 +

[PATCH v4 01/20] block-backend: split blk_do_set_aio_context()

2023-04-25 Thread Stefan Hajnoczi
oot_set_aio_ctx_commit() being invoked as part of the AioContext change propagation. This can be solved by temporarily setting blk->allow_aio_context_change to true. Future patches call blk_get_aio_context() from BlockDevOps->drained_end(), so this patch will become necessary. Signed-off-by: Stefa

[PATCH v4 10/20] block: drain from main loop thread in bdrv_co_yield_to_drain()

2023-04-25 Thread Stefan Hajnoczi
. Signed-off-by: Stefan Hajnoczi --- include/sysemu/block-backend-common.h | 25 + block/io.c| 3 ++- 2 files changed, 15 insertions(+), 13 deletions(-) diff --git a/include/sysemu/block-backend-common.h b/include/sysemu/block-backend-common.h index

[PATCH v4 11/20] xen-block: implement BlockDevOps->drained_begin()

2023-04-25 Thread Stefan Hajnoczi
depend on the dataplane instance, so move the blk_set_dev_ops() call after xen_block_dataplane_create(). Signed-off-by: Stefan Hajnoczi --- hw/block/dataplane/xen-block.h | 2 ++ hw/block/dataplane/xen-block.c | 42 +- hw/block/xen-block.c

[PATCH v4 09/20] block: add blk_in_drain() API

2023-04-25 Thread Stefan Hajnoczi
The BlockBackend quiesce_counter is greater than zero during drained sections. Add an API to check whether the BlockBackend is in a drained section. The next patch will use this API. Signed-off-by: Stefan Hajnoczi --- include/sysemu/block-backend-global-state.h | 1 + block/block-backend.c

[PATCH v4 20/20] aio: remove aio_disable_external() API

2023-04-25 Thread Stefan Hajnoczi
, is_external, io_read, io_poll, io_poll_ready; @@ - aio_set_event_notifier(ctx, notifier, is_external, io_read, io_poll, io_poll_ready) + aio_set_event_notifier(ctx, notifier, io_read, io_poll, io_poll_ready) Reviewed-by: Juan Quintela Reviewed-by: Philippe Mathieu-Daudé Signed-off-by: Stefan

[PATCH v4 19/20] virtio: do not set is_external=true on host notifiers

2023-04-25 Thread Stefan Hajnoczi
Host notifiers can now use is_external=false since virtio-blk and virtio-scsi no longer rely on is_external=true for drained sections. Signed-off-by: Stefan Hajnoczi --- hw/virtio/virtio.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/virtio/virtio.c b/hw/virtio

[PATCH v4 13/20] block/export: rewrite vduse-blk drain code

2023-04-25 Thread Stefan Hajnoczi
resume them in .drained_end(). This eliminates the need for the aio_set_fd_handler(is_external=true) flag, which is being removed from QEMU. This is a long list but splitting it into individual commits would probably lead to git bisect failures - the changes are all related. Signed-off-by:

[PATCH v4 18/20] virtio-scsi: implement BlockDevOps->drained_begin()

2023-04-25 Thread Stefan Hajnoczi
sees one pair of .drained_begin/end() calls. After this commit, virtio-scsi no longer depends on hw/virtio's ioeventfd aio_set_event_notifier(is_external=true). This commit is a step towards removing the aio_disable_external() API. Signed-off-by: Stefan Hajnoczi --- include/hw/scsi/s

[PATCH v4 15/20] block/fuse: do not set is_external=true on FUSE fd

2023-04-25 Thread Stefan Hajnoczi
This is part of ongoing work to remove the aio_disable_external() API. Use BlockDevOps .drained_begin/end/poll() instead of aio_set_fd_handler(is_external=true). As a side-effect the FUSE export now follows AioContext changes like the other export types. Signed-off-by: Stefan Hajnoczi

[PATCH v4 00/20] block: remove aio_disable_external() API

2023-04-25 Thread Stefan Hajnoczi
ily stop file descriptor handlers. This ensures that new I/O requests are not submitted in drained sections. The first two virtio-scsi patches were already sent as a separate series. I included them because they are necessary in order to fully remove aio_disable_external(). Stefan Hajnoczi (20):

[PATCH v4 07/20] block/export: stop using is_external in vhost-user-blk server

2023-04-25 Thread Stefan Hajnoczi
d the aio_disable_external() API cannot be used since multiple AioContext may be processing I/O, not just one. Switch to BlockDevOps->drained_begin/end() callbacks. Signed-off-by: Stefan Hajnoczi --- block/export/vhost-user-blk-server.c | 43 ++-- util/vhost-user-server.c

[PATCH v4 04/20] virtio-scsi: stop using aio_disable_external() during unplug

2023-04-25 Thread Stefan Hajnoczi
simple_device_unplug_cb() returns. Thanks to these two conditions we don't need aio_disable_external() anymore. Cc: Zhengui Li Reviewed-by: Paolo Bonzini Reviewed-by: Daniil Tatianin Signed-off-by: Stefan Hajnoczi --- hw/scsi/scsi-disk.c | 1 + hw/scsi/virtio-scsi.c | 3 --- 2 file

Re: [PATCH v3 13/20] block/export: rewrite vduse-blk drain code

2023-04-25 Thread Stefan Hajnoczi
On Fri, Apr 21, 2023 at 11:36:02AM +0800, Yongji Xie wrote: > Hi Stefan, > > On Thu, Apr 20, 2023 at 7:39 PM Stefan Hajnoczi wrote: > > > > vduse_blk_detach_ctx() waits for in-flight requests using > > AIO_WAIT_WHILE(). This is not allowe

<    10   11   12   13   14   15   16   17   18   19   >