Re: [PATCH qemu] docs/interop/qcow2.txt: fix description about "zlib" clusters

2023-05-16 Thread Eric Blake
On Tue, May 16, 2023 at 11:32:27PM +0900, ~akihirosuda wrote: > > From: Akihiro Suda > > "zlib" clusters are actually raw deflate (RFC1951) clusters without > zlib headers. > > Signed-off-by: Akihiro Suda > --- > docs/interop/qcow2.txt | 10 +++--- > 1 file changed, 7 insertions(+), 3 d

Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs

2023-05-16 Thread Olaf Hering
Am Tue, 16 May 2023 13:38:42 -0400 schrieb John Snow : > I haven't touched IDE or block code in quite a long while now -- I > don't think I can help land this fix, but I won't get in anyone's way, > either. Maybe just re-submit the patches with an improved commit > message / cover letter that help

[PATCH v6 19/20] virtio: do not set is_external=true on host notifiers

2023-05-16 Thread Stefan Hajnoczi
Host notifiers can now use is_external=false since virtio-blk and virtio-scsi no longer rely on is_external=true for drained sections. Signed-off-by: Stefan Hajnoczi --- hw/virtio/virtio.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/hw/virtio/virtio.c b/hw/virtio/vi

[PATCH v6 17/20] virtio-blk: implement BlockDevOps->drained_begin()

2023-05-16 Thread Stefan Hajnoczi
Detach ioeventfds during drained sections to stop I/O submission from the guest. virtio-blk is no longer reliant on aio_disable_external() after this patch. This will allow us to remove the aio_disable_external() API once all other code that relies on it is converted. Take extra care to avoid atta

[PATCH v6 16/20] virtio: make it possible to detach host notifier from any thread

2023-05-16 Thread Stefan Hajnoczi
virtio_queue_aio_detach_host_notifier() does two things: 1. It removes the fd handler from the event loop. 2. It processes the virtqueue one last time. The first step can be peformed by any thread and without taking the AioContext lock. The second step may need the AioContext lock (depending on t

[PATCH v6 14/20] block/export: don't require AioContext lock around blk_exp_ref/unref()

2023-05-16 Thread Stefan Hajnoczi
The FUSE export calls blk_exp_ref/unref() without the AioContext lock. Instead of fixing the FUSE export, adjust blk_exp_ref/unref() so they work without the AioContext lock. This way it's less error-prone. Suggested-by: Paolo Bonzini Signed-off-by: Stefan Hajnoczi Reviewed-by: Kevin Wolf ---

[PATCH v6 18/20] virtio-scsi: implement BlockDevOps->drained_begin()

2023-05-16 Thread Stefan Hajnoczi
The virtio-scsi Host Bus Adapter provides access to devices on a SCSI bus. Those SCSI devices typically have a BlockBackend. When the BlockBackend enters a drained section, the SCSI device must temporarily stop submitting new I/O requests. Implement this behavior by temporarily stopping virtio-scs

[PATCH v6 09/20] block: add blk_in_drain() API

2023-05-16 Thread Stefan Hajnoczi
The BlockBackend quiesce_counter is greater than zero during drained sections. Add an API to check whether the BlockBackend is in a drained section. The next patch will use this API. Signed-off-by: Stefan Hajnoczi Reviewed-by: Kevin Wolf --- include/sysemu/block-backend-global-state.h | 1 + b

[PATCH v6 04/20] virtio-scsi: stop using aio_disable_external() during unplug

2023-05-16 Thread Stefan Hajnoczi
This patch is part of an effort to remove the aio_disable_external() API because it does not fit in a multi-queue block layer world where many AioContexts may be submitting requests to the same disk. The SCSI emulation code is already in good shape to stop using aio_disable_external(). It was only

[PATCH v6 20/20] aio: remove aio_disable_external() API

2023-05-16 Thread Stefan Hajnoczi
All callers now pass is_external=false to aio_set_fd_handler() and aio_set_event_notifier(). The aio_disable_external() API that temporarily disables fd handlers that were registered is_external=true is therefore dead code. Remove aio_disable_external(), aio_enable_external(), and the is_external

[PATCH v6 10/20] block: drain from main loop thread in bdrv_co_yield_to_drain()

2023-05-16 Thread Stefan Hajnoczi
For simplicity, always run BlockDevOps .drained_begin/end/poll() callbacks in the main loop thread. This makes it easier to implement the callbacks and avoids extra locks. Move the function pointer declarations from the I/O Code section to the Global State section for BlockDevOps, BdrvChildClass,

[PATCH v6 15/20] block/fuse: do not set is_external=true on FUSE fd

2023-05-16 Thread Stefan Hajnoczi
This is part of ongoing work to remove the aio_disable_external() API. Use BlockDevOps .drained_begin/end/poll() instead of aio_set_fd_handler(is_external=true). As a side-effect the FUSE export now follows AioContext changes like the other export types. Signed-off-by: Stefan Hajnoczi Reviewed-

[PATCH v6 13/20] block/export: rewrite vduse-blk drain code

2023-05-16 Thread Stefan Hajnoczi
vduse_blk_detach_ctx() waits for in-flight requests using AIO_WAIT_WHILE(). This is not allowed according to a comment in bdrv_set_aio_context_commit(): /* * Take the old AioContex when detaching it from bs. * At this point, new_context lock is already acquired, and we are now * also ta

[PATCH v6 06/20] block/export: wait for vhost-user-blk requests when draining

2023-05-16 Thread Stefan Hajnoczi
Each vhost-user-blk request runs in a coroutine. When the BlockBackend enters a drained section we need to enter a quiescent state. Currently any in-flight requests race with bdrv_drained_begin() because it is unaware of vhost-user-blk requests. When blk_co_preadv/pwritev()/etc returns it wakes th

[PATCH v6 08/20] hw/xen: do not use aio_set_fd_handler(is_external=true) in xen_xenstore

2023-05-16 Thread Stefan Hajnoczi
There is no need to suspend activity between aio_disable_external() and aio_enable_external(), which is mainly used for the block layer's drain operation. This is part of ongoing work to remove the aio_disable_external() API. Reviewed-by: David Woodhouse Reviewed-by: Paul Durrant Signed-off-by:

[PATCH v6 11/20] xen-block: implement BlockDevOps->drained_begin()

2023-05-16 Thread Stefan Hajnoczi
Detach event channels during drained sections to stop I/O submission from the ring. xen-block is no longer reliant on aio_disable_external() after this patch. This will allow us to remove the aio_disable_external() API once all other code that relies on it is converted. Extend xen_device_set_event

[PATCH v6 12/20] hw/xen: do not set is_external=true on evtchn fds

2023-05-16 Thread Stefan Hajnoczi
is_external=true suspends fd handlers between aio_disable_external() and aio_enable_external(). The block layer's drain operation uses this mechanism to prevent new I/O from sneaking in between bdrv_drained_begin() and bdrv_drained_end(). The previous commit converted the xen-block device to use B

[PATCH v6 07/20] block/export: stop using is_external in vhost-user-blk server

2023-05-16 Thread Stefan Hajnoczi
vhost-user activity must be suspended during bdrv_drained_begin/end(). This prevents new requests from interfering with whatever is happening in the drained section. Previously this was done using aio_set_fd_handler()'s is_external argument. In a multi-queue block layer world the aio_disable_exter

[PATCH v6 05/20] util/vhost-user-server: rename refcount to in_flight counter

2023-05-16 Thread Stefan Hajnoczi
The VuServer object has a refcount field and ref/unref APIs. The name is confusing because it's actually an in-flight request counter instead of a refcount. Normally a refcount destroys the object upon reaching zero. The VuServer counter is used to wake up the vhost-user coroutine when there are n

[PATCH v6 03/20] virtio-scsi: avoid race between unplug and transport event

2023-05-16 Thread Stefan Hajnoczi
Only report a transport reset event to the guest after the SCSIDevice has been unrealized by qdev_simple_device_unplug_cb(). qdev_simple_device_unplug_cb() sets the SCSIDevice's qdev.realized field to false so that scsi_device_find/get() no longer see it. scsi_target_emulate_report_luns() also ne

[PATCH v6 01/20] block-backend: split blk_do_set_aio_context()

2023-05-16 Thread Stefan Hajnoczi
blk_set_aio_context() is not fully transactional because blk_do_set_aio_context() updates blk->ctx outside the transaction. Most of the time this goes unnoticed but a BlockDevOps.drained_end() callback that invokes blk_get_aio_context() fails assert(ctx == blk->ctx). This happens because blk->ctx i

[PATCH v6 02/20] hw/qdev: introduce qdev_is_realized() helper

2023-05-16 Thread Stefan Hajnoczi
Add a helper function to check whether the device is realized without requiring the Big QEMU Lock. The next patch adds a second caller. The goal is to avoid spreading DeviceState field accesses throughout the code. Suggested-by: Philippe Mathieu-Daudé Reviewed-by: Philippe Mathieu-Daudé Signed-o

[PATCH v6 00/20] block: remove aio_disable_external() API

2023-05-16 Thread Stefan Hajnoczi
v6: - Fix scsi_device_unrealize() -> scsi_qdev_unrealize() mistake in Patch 4 commit description [Kevin] - Explain why we don't schedule a BH in .drained_begin() in Patch 16 [Kevin] - Copy the comment explaining why the event notifier is tested and cleared in Patch 16 [Kevin] - Fix EPOLL_ENABLE

[PATCH qemu] docs/interop/qcow2.txt: fix description about "zlib" clusters

2023-05-16 Thread ~akihirosuda
From: Akihiro Suda "zlib" clusters are actually raw deflate (RFC1951) clusters without zlib headers. Signed-off-by: Akihiro Suda --- docs/interop/qcow2.txt | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt index f7dc3

Re: [PATCH v2] piix: fix regression during unplug in Xen HVM domUs

2023-05-16 Thread John Snow
On Fri, May 12, 2023 at 5:14 PM Stefano Stabellini wrote: > > On Wed, 10 May 2023, Olaf Hering wrote: > > Wed, 10 May 2023 00:58:27 +0200 Olaf Hering : > > > > > In my debugging (with v8.0.0) it turned out the three pci_set_word > > > causes the domU to hang. In fact, it is just the last one: > >

Re: [PATCH v3 1/1] block/blkio: use qemu_open() to support fd passing for virtio-blk

2023-05-16 Thread Jonathon Jongsma
On 5/15/23 5:10 AM, Stefano Garzarella wrote: On Thu, May 11, 2023 at 11:03:22AM -0500, Jonathon Jongsma wrote: On 5/11/23 4:15 AM, Stefano Garzarella wrote: The virtio-blk-vhost-vdpa driver in libblkio 1.3.0 supports the new 'fd' property. Let's expose this to the user, so the management layer

Re: [PATCH v5 12/21] xen-block: implement BlockDevOps->drained_begin()

2023-05-16 Thread Anthony PERARD via
On Thu, May 04, 2023 at 03:53:18PM -0400, Stefan Hajnoczi wrote: > @@ -819,11 +841,9 @@ void xen_block_dataplane_start(XenBlockDataPlane > *dataplane, > blk_set_aio_context(dataplane->blk, dataplane->ctx, NULL); > aio_context_release(old_context); > > -/* Only reason for failure is

Re: [PATCH v5 13/21] hw/xen: do not set is_external=true on evtchn fds

2023-05-16 Thread Anthony PERARD via
On Thu, May 04, 2023 at 03:53:19PM -0400, Stefan Hajnoczi wrote: > is_external=true suspends fd handlers between aio_disable_external() and > aio_enable_external(). The block layer's drain operation uses this > mechanism to prevent new I/O from sneaking in between > bdrv_drained_begin() and bdrv_dr

Re: [PATCH v5 12/21] xen-block: implement BlockDevOps->drained_begin()

2023-05-16 Thread Anthony PERARD via
On Thu, May 04, 2023 at 03:53:18PM -0400, Stefan Hajnoczi wrote: > Detach event channels during drained sections to stop I/O submission > from the ring. xen-block is no longer reliant on aio_disable_external() > after this patch. This will allow us to remove the > aio_disable_external() API once al

Re: [PATCH v2 01/16] migration: Don't use INT64_MAX for unlimited rate

2023-05-16 Thread Cédric Le Goater
On 5/16/23 11:24, Juan Quintela wrote: David Edmondson wrote: Juan Quintela writes: Define and use RATE_LIMIT_MAX instead. Suggest "RATE_LIMIT_MAX_NONE". Then even better RATE_LIMIT_DISABLED? I'd vote for RATE_LIMIT_DISABLED. RATE_LIMIT_NONE? Using MAX and NONE at the same time loo

Re: [PATCH v2 05/16] migration: Move rate_limit_max and rate_limit_used to migration_stats

2023-05-16 Thread Cédric Le Goater
On 5/15/23 21:56, Juan Quintela wrote: These way we can make them atomic and use this functions from any place. I also moved all functions that use rate_limit to migration-stats. Functions got renamed, they are not qemu_file anymore. qemu_file_rate_limit -> migration_rate_exceeded qemu_file_se

Re: [PATCH v2 03/16] migration: Move setup_time to mig_stats

2023-05-16 Thread David Edmondson
Juan Quintela writes: > David Edmondson wrote: >> Juan Quintela writes: >> >>> It is a time that needs to be cleaned each time cancel migration. >>> Once there create migration_time_since() to calculate how time since a >>> time in the past. >>> >>> Signed-off-by: Juan Quintela >>> >>> --- >>>

Re: [PATCH v2 03/16] migration: Move setup_time to mig_stats

2023-05-16 Thread Juan Quintela
David Edmondson wrote: > Juan Quintela writes: > >> It is a time that needs to be cleaned each time cancel migration. >> Once there create migration_time_since() to calculate how time since a >> time in the past. >> >> Signed-off-by: Juan Quintela >> >> --- >> >> Rename to migration_time_since (

Re: [PATCH v2 01/16] migration: Don't use INT64_MAX for unlimited rate

2023-05-16 Thread David Edmondson
Juan Quintela writes: > David Edmondson wrote: >> Juan Quintela writes: >> >>> Define and use RATE_LIMIT_MAX instead. >> >> Suggest "RATE_LIMIT_MAX_NONE". > > Then even better > > RATE_LIMIT_DISABLED? > RATE_LIMIT_NONE? RATE_LIMIT_NONE sounds good to me. > > Using MAX and NONE at the same tim

Re: [PATCH v2 03/16] migration: Move setup_time to mig_stats

2023-05-16 Thread David Edmondson
Juan Quintela writes: > It is a time that needs to be cleaned each time cancel migration. > Once there create migration_time_since() to calculate how time since a > time in the past. > > Signed-off-by: Juan Quintela > > --- > > Rename to migration_time_since (cédric) > --- > migration/migration

Re: [PATCH v2 02/16] migration: Correct transferred bytes value

2023-05-16 Thread David Edmondson
Juan Quintela writes: > We forget several places to add to trasferred amount of data. With "transferred". > this fixes I get: > >qemu_file_transferred() + multifd_bytes == transferred > > The only place whrer this is not true is during devices sending. But "where" > going all through th

Re: [PATCH v2 01/16] migration: Don't use INT64_MAX for unlimited rate

2023-05-16 Thread Juan Quintela
David Edmondson wrote: > Juan Quintela writes: > >> Define and use RATE_LIMIT_MAX instead. > > Suggest "RATE_LIMIT_MAX_NONE". Then even better RATE_LIMIT_DISABLED? RATE_LIMIT_NONE? Using MAX and NONE at the same time looks strange. >> Signed-off-by: Juan Quintela >> --- >> migration/migrati

Re: [PATCH v2 01/16] migration: Don't use INT64_MAX for unlimited rate

2023-05-16 Thread David Edmondson
Juan Quintela writes: > Define and use RATE_LIMIT_MAX instead. Suggest "RATE_LIMIT_MAX_NONE". > > Signed-off-by: Juan Quintela > --- > migration/migration-stats.h | 6 ++ > migration/migration.c | 4 ++-- > migration/qemu-file.c | 6 +- > 3 files changed, 13 insertions(+),