Re: [PATCH 4/4] iotests: make qemu_img raise on non-zero rc by default

2022-02-15 Thread John Snow
On Tue, Feb 15, 2022 at 6:09 PM Eric Blake wrote: > > On Tue, Feb 15, 2022 at 05:08:53PM -0500, John Snow wrote: > > re-configure qemu_img() into a function that will by default raise a > > VerboseProcessException (extended from CalledProcessException) on > > non-zero return codes. This will

Re: [PATCH 3/4] iotests: Remove explicit checks for qemu_img() == 0

2022-02-15 Thread John Snow
On Tue, Feb 15, 2022 at 6:05 PM Eric Blake wrote: > > On Tue, Feb 15, 2022 at 05:08:52PM -0500, John Snow wrote: > > qemu_img() returning zero ought to be the rule, not the > > exception. Remove all explicit checks against the condition in > > preparation for making non-zero returns an Exception.

Re: [PATCH 1/4] python/utils: add enboxify() text decoration utility

2022-02-15 Thread Philippe Mathieu-Daudé via
On 16/2/22 00:53, John Snow wrote: On Tue, Feb 15, 2022 at 5:55 PM Eric Blake wrote: On Tue, Feb 15, 2022 at 05:08:50PM -0500, John Snow wrote: print(enboxify(msg, width=72, name="commit message")) ┏━ commit message ━┓ ┃ enboxify() takes a

Re: [PATCH 2/4] iotests: add VerboseProcessError

2022-02-15 Thread John Snow
On Tue, Feb 15, 2022 at 5:58 PM Eric Blake wrote: > > On Tue, Feb 15, 2022 at 05:08:51PM -0500, John Snow wrote: > > This adds an Exception that extends the garden variety > > subprocess.CalledProcessError. When this exception is raised, it will > > still be caught when selecting for the stdlib

Re: [PATCH 1/4] python/utils: add enboxify() text decoration utility

2022-02-15 Thread John Snow
On Tue, Feb 15, 2022 at 5:55 PM Eric Blake wrote: > > On Tue, Feb 15, 2022 at 05:08:50PM -0500, John Snow wrote: > > >>> print(enboxify(msg, width=72, name="commit message")) > > ┏━ commit message ━┓ > > ┃ enboxify() takes a chunk of text and

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 09:23:36PM +0200, Nir Soffer wrote: > On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > > > According to the NBD spec, a server advertising > > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > > not see any cache inconsistencies: when properly

Re: [PATCH 4/4] iotests: make qemu_img raise on non-zero rc by default

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 05:08:53PM -0500, John Snow wrote: > re-configure qemu_img() into a function that will by default raise a > VerboseProcessException (extended from CalledProcessException) on > non-zero return codes. This will produce a stack trace that will show > the command line arguments

Re: [PATCH 3/4] iotests: Remove explicit checks for qemu_img() == 0

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 05:08:52PM -0500, John Snow wrote: > qemu_img() returning zero ought to be the rule, not the > exception. Remove all explicit checks against the condition in > preparation for making non-zero returns an Exception. > > Signed-off-by: John Snow > --- Reviewed-by: Eric

Re: [PATCH 2/4] iotests: add VerboseProcessError

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 05:08:51PM -0500, John Snow wrote: > This adds an Exception that extends the garden variety > subprocess.CalledProcessError. When this exception is raised, it will > still be caught when selecting for the stdlib variant. > > The difference is that the str() method of this

Re: [PATCH 1/4] python/utils: add enboxify() text decoration utility

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 05:08:50PM -0500, John Snow wrote: > >>> print(enboxify(msg, width=72, name="commit message")) > ┏━ commit message ━┓ > ┃ enboxify() takes a chunk of text and wraps it in a text art box that ┃ > ┃ adheres to a specified

Re: [PULL 0/3] Block patches

2022-02-15 Thread Peter Maydell
On Mon, 14 Feb 2022 at 17:44, Stefan Hajnoczi wrote: > > The following changes since commit cc5ce8b8b6be83e5fe3b668dbd061ad97c534e3f: > > Merge remote-tracking branch 'remotes/legoater/tags/pull-ppc-20220210' into > staging (2022-02-13 20:33:28 +) > > are available in the Git repository

Re: [PATCH 2/3] iotests: Allow using QMP with the QSD

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 02:57:26PM +0100, Hanna Reitz wrote: > Add a parameter to optionally open a QMP connection when creating a > QemuStorageDaemon instance. > > Signed-off-by: Hanna Reitz > --- > tests/qemu-iotests/iotests.py | 29 - > 1 file changed, 28

Re: [PATCH 1/3] block: Make bdrv_refresh_limits() non-recursive

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 02:57:25PM +0100, Hanna Reitz wrote: > bdrv_refresh_limits() recurses down to the node's children. That does > not seem necessary: When we refresh limits on some node, and then > recurse down and were to change one of its children's BlockLimits, then > that would mean we

Re: [PATCH] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 02:20:31PM +0100, Thomas Huth wrote: > Instead of failing the iotests if GNU sed is not available (or skipping > them completely in the check-block.sh script), it would be better to > simply skip the bash-based tests that rely on GNU sed, so that the other > tests could

Re: [PATCH 3/3] iotests/graph-changes-while-io: New test

2022-02-15 Thread Eric Blake
On Tue, Feb 15, 2022 at 02:57:27PM +0100, Hanna Reitz wrote: > Test the following scenario: > 1. Some block node (null-co) attached to a user (here: NBD server) that >performs I/O and keeps the node in an I/O thread > 2. Repeatedly run blockdev-add/blockdev-del to add/remove an overlay >

[PATCH 0/4] iotests: add detailed tracebacks to qemu_img() failures

2022-02-15 Thread John Snow
It came to my attention via th_huth that iotest 065 would crash in a way that was largely silent, except for the async QMP traces. The real cause turns out to be that iotest 065 does not support ztsd being compiled out of the build, so the qemu-img command fails ... silently. (And then everything

[PATCH 3/4] iotests: Remove explicit checks for qemu_img() == 0

2022-02-15 Thread John Snow
qemu_img() returning zero ought to be the rule, not the exception. Remove all explicit checks against the condition in preparation for making non-zero returns an Exception. Signed-off-by: John Snow --- tests/qemu-iotests/163 | 9 +++-- tests/qemu-iotests/216

[PATCH 2/4] iotests: add VerboseProcessError

2022-02-15 Thread John Snow
This adds an Exception that extends the garden variety subprocess.CalledProcessError. When this exception is raised, it will still be caught when selecting for the stdlib variant. The difference is that the str() method of this Exception also adds the stdout/stderr logs. In effect, if this

[PATCH 4/4] iotests: make qemu_img raise on non-zero rc by default

2022-02-15 Thread John Snow
re-configure qemu_img() into a function that will by default raise a VerboseProcessException (extended from CalledProcessException) on non-zero return codes. This will produce a stack trace that will show the command line arguments and return code from the failed process run. Users that want

[PATCH 1/4] python/utils: add enboxify() text decoration utility

2022-02-15 Thread John Snow
>>> print(enboxify(msg, width=72, name="commit message")) ┏━ commit message ━┓ ┃ enboxify() takes a chunk of text and wraps it in a text art box that ┃ ┃ adheres to a specified width. An optional title label may be given, ┃ ┃ and any of the

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
15.02.2022 20:18, Eric Blake wrote: According to the NBD spec, a server advertising NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another

Re: [PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Nir Soffer
On Tue, Feb 15, 2022 at 7:22 PM Eric Blake wrote: > According to the NBD spec, a server advertising > NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will > not see any cache inconsistencies: when properly separated by a single > flush, actions performed by one client will be

Re: [PULL 0/6] hw/nvme updates

2022-02-15 Thread Peter Maydell
On Mon, 14 Feb 2022 at 08:08, Klaus Jensen wrote: > > From: Klaus Jensen > > Hi Peter, > > The following changes since commit 48033ad678ae2def43bf0d543a2c4c3d2a93feaf: > > Merge remote-tracking branch > 'remotes/vsementsov/tags/pull-nbd-2022-02-09-v2' into staging (2022-02-12 > 22:04:07

Re: [PULL 0/4] Python patches

2022-02-15 Thread John Snow
On Tue, Feb 15, 2022 at 1:01 PM Peter Maydell wrote: > > On Tue, 15 Feb 2022 at 17:46, John Snow wrote: > > Just so I don't leave this thread hanging, I filed a GitLab issue and > > I'm working on it, but this one isn't as quick to solve as the other. > > > >

Re: [PULL 0/4] Python patches

2022-02-15 Thread Peter Maydell
On Tue, 15 Feb 2022 at 17:46, John Snow wrote: > Just so I don't leave this thread hanging, I filed a GitLab issue and > I'm working on it, but this one isn't as quick to solve as the other. > > https://gitlab.com/qemu-project/qemu/-/issues/874 Is there anything particular to NetBSD that means

Re: [PULL 18/38] Remove unnecessary minimum_version_id_old fields

2022-02-15 Thread Peter Maydell
On Tue, 15 Feb 2022 at 17:34, Cédric Le Goater wrote: > > On 2/15/22 18:13, Peter Maydell wrote: > > No, as the commit message notes, it deliberately did not change > > that one vmstate, because at the time of writing the patch > > that was the one vmstate that really was still using > >

[PATCH 4/4] block: simplify handling of try to merge different sized bitmaps

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
We have too much logic to simply check that bitmaps are of the same size. Let's just define that hbitmap_merge() and bdrv_dirty_bitmap_merge_internal() require their argument bitmaps be of same size, this simplifies things. Let's look through the callers: For backup_init_bcs_bitmap() we already

[PATCH 3/4] block: improve block_dirty_bitmap_merge(): don't allocate extra bitmap

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
We don't need extra bitmap. All we need is to backup the original bitmap when we do first merge. So, drop extra temporary bitmap and work directly with target and backup. Note that block_dirty_bitmap_merge() semantics changed: on failure target may be modified now, and caller should call

[PATCH 2/4] block: block_dirty_bitmap_merge(): fix error path

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
At the end we ignore failure of bdrv_merge_dirty_bitmap() and report success. And still set errp. That's wrong. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/monitor/bitmap-qmp-cmds.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/block/monitor/bitmap-qmp-cmds.c

[PATCH 1/4] block: bdrv_merge_dirty_bitmap: add return value

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
Add return value to bdrv_merge_dirty_bitmap() and use it to reduce error propagation. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/dirty-bitmap.h| 2 +- block/dirty-bitmap.c| 6 -- block/monitor/bitmap-qmp-cmds.c | 5 + 3 files changed, 6 insertions(+),

[PATCH 0/4] block/dirty-bitmaps: fix and improve bitmap merge

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
Hi all! Here are some good refactoring and fix of bitmap merge paths. Vladimir Sementsov-Ogievskiy (4): block: bdrv_merge_dirty_bitmap: add return value block: block_dirty_bitmap_merge(): fix error path block: improve block_dirty_bitmap_merge(): don't allocate extra bitmap block:

Re: [PULL 0/4] Python patches

2022-02-15 Thread John Snow
On Tue, Feb 8, 2022 at 9:40 AM Peter Maydell wrote: > > On Thu, 3 Feb 2022 at 23:22, John Snow wrote: > > > > On Thu, Feb 3, 2022 at 11:52 AM Peter Maydell > > wrote: > > > > > > On Thu, 3 Feb 2022 at 16:38, John Snow wrote: > > > > > > > On Thu, Feb 3, 2022, 11:20 AM Peter Maydell > > > >

Re: [PULL 18/38] Remove unnecessary minimum_version_id_old fields

2022-02-15 Thread Cédric Le Goater
On 2/15/22 18:13, Peter Maydell wrote: On Thu, 27 Jan 2022 at 15:14, Juan Quintela wrote: From: Peter Maydell The migration code will not look at a VMStateDescription's minimum_version_id_old field unless that VMSD has set the load_state_old field to something non-NULL. (The purpose of

[PATCH v2] nbd/server: Allow MULTI_CONN for shared writable exports

2022-02-15 Thread Eric Blake
According to the NBD spec, a server advertising NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the

Re: [PULL 18/38] Remove unnecessary minimum_version_id_old fields

2022-02-15 Thread Peter Maydell
On Thu, 27 Jan 2022 at 15:14, Juan Quintela wrote: > > From: Peter Maydell > > The migration code will not look at a VMStateDescription's > minimum_version_id_old field unless that VMSD has set the > load_state_old field to something non-NULL. (The purpose of > minimum_version_id_old is to

Re: [PATCH v2] ide: Increment BB in-flight counter for TRIM BH

2022-02-15 Thread Hanna Reitz
Ping (I can take it too, if you’d like, John, but you’re listed as the only maintainer for hw/ide, so...  Just say the word, though!) On 20.01.22 15:22, Hanna Reitz wrote: When we still have an AIOCB registered for DMA operations, we try to settle the respective operation by draining the

Re: [PATCH 1/6] tests/qemu-iotests: Improve the check for GNU sed

2022-02-15 Thread Thomas Huth
On 15/02/2022 14.51, Daniel P. Berrangé wrote: On Tue, Feb 15, 2022 at 02:28:24PM +0100, Thomas Huth wrote: On 11/02/2022 17.48, Thomas Huth wrote: On 11/02/2022 17.14, Eric Blake wrote: On Tue, Feb 08, 2022 at 03:52:19PM +0100, Thomas Huth wrote: The current code with $SED has been

[PATCH 1/3] block: Make bdrv_refresh_limits() non-recursive

2022-02-15 Thread Hanna Reitz
bdrv_refresh_limits() recurses down to the node's children. That does not seem necessary: When we refresh limits on some node, and then recurse down and were to change one of its children's BlockLimits, then that would mean we noticed the changed limits by pure chance. The fact that we refresh

[PATCH 3/3] iotests/graph-changes-while-io: New test

2022-02-15 Thread Hanna Reitz
Test the following scenario: 1. Some block node (null-co) attached to a user (here: NBD server) that performs I/O and keeps the node in an I/O thread 2. Repeatedly run blockdev-add/blockdev-del to add/remove an overlay to/from that node Each blockdev-add triggers bdrv_refresh_limits(), and

[PATCH 2/3] iotests: Allow using QMP with the QSD

2022-02-15 Thread Hanna Reitz
Add a parameter to optionally open a QMP connection when creating a QemuStorageDaemon instance. Signed-off-by: Hanna Reitz --- tests/qemu-iotests/iotests.py | 29 - 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/tests/qemu-iotests/iotests.py

[PATCH 0/3] block: Make bdrv_refresh_limits() non-recursive

2022-02-15 Thread Hanna Reitz
Hi, Most bdrv_refresh_limits() callers do not drain the subtree of the node whose limits are refreshed, so concurrent I/O requests to child nodes can occur (if the node is in an I/O thread). bdrv_refresh_limits() is recursive, so such requests can happen to a node whose limits are being

Re: [PATCH 1/6] tests/qemu-iotests: Improve the check for GNU sed

2022-02-15 Thread Daniel P . Berrangé
On Tue, Feb 15, 2022 at 02:28:24PM +0100, Thomas Huth wrote: > On 11/02/2022 17.48, Thomas Huth wrote: > > On 11/02/2022 17.14, Eric Blake wrote: > > > On Tue, Feb 08, 2022 at 03:52:19PM +0100, Thomas Huth wrote: > > > > > > The current code with $SED has been introduced almost three years > > > >

Re: [PATCH 1/6] tests/qemu-iotests: Improve the check for GNU sed

2022-02-15 Thread Thomas Huth
On 11/02/2022 17.48, Thomas Huth wrote: On 11/02/2022 17.14, Eric Blake wrote: On Tue, Feb 08, 2022 at 03:52:19PM +0100, Thomas Huth wrote: The current code with $SED has been introduced almost three years ago already...    Can’t we just do `alias sed=gsed`? Maybe ... but let's ask

[PATCH] tests/qemu-iotests: Rework the checks and spots using GNU sed

2022-02-15 Thread Thomas Huth
Instead of failing the iotests if GNU sed is not available (or skipping them completely in the check-block.sh script), it would be better to simply skip the bash-based tests that rely on GNU sed, so that the other tests could still be run. Thus we now explicitely use "gsed" (either as direct

[PATCH] block: fix preallocate filter: don't do unaligned preallocate requests

2022-02-15 Thread Vladimir Sementsov-Ogievskiy
There is a bug in handling BDRV_REQ_NO_WAIT flag: we still may wait in wait_serialising_requests() if request is unaligned. And this is possible for the only user of this flag (preallocate filter) if underlying file is unaligned to its request_alignment on start. So, we have to fix preallocate

[PATCH v2 6/6] libvduse: Add support for reconnecting

2022-02-15 Thread Xie Yongji
To support reconnecting after restart or crash, VDUSE backend might need to resubmit inflight I/Os. This stores the metadata such as the index of inflight I/O's descriptors to a shm file so that VDUSE backend can restore them during reconnecting. Signed-off-by: Xie Yongji ---

[PATCH v2 5/6] vduse-blk: Add vduse-blk resize support

2022-02-15 Thread Xie Yongji
To support block resize, this uses vduse_dev_update_config() to update the capacity field in configuration space and inject config interrupt on the block resize callback. Signed-off-by: Xie Yongji --- block/export/vduse-blk.c | 20 1 file changed, 20 insertions(+) diff

[PATCH v2 3/6] libvduse: Add VDUSE (vDPA Device in Userspace) library

2022-02-15 Thread Xie Yongji
VDUSE [1] is a linux framework that makes it possible to implement software-emulated vDPA devices in userspace. This adds a library as a subproject to help implementing VDUSE backends in QEMU. [1] https://www.kernel.org/doc/html/latest/userspace-api/vduse.html Signed-off-by: Xie Yongji ---

[PATCH v2 1/6] block: Support passing NULL ops to blk_set_dev_ops()

2022-02-15 Thread Xie Yongji
This supports passing NULL ops to blk_set_dev_ops() so that we can remove stale ops in some cases. Signed-off-by: Xie Yongji --- block/block-backend.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/block-backend.c b/block/block-backend.c index 4ff6b4d785..08dd0a3093

[PATCH v2 4/6] vduse-blk: implements vduse-blk export

2022-02-15 Thread Xie Yongji
This implements a VDUSE block backends based on the libvduse library. We can use it to export the BDSs for both VM and container (host) usage. The new command-line syntax is: $ qemu-storage-daemon \ --blockdev file,node-name=drive0,filename=test.img \ --export

[PATCH v2 0/6] Support exporting BDSs via VDUSE

2022-02-15 Thread Xie Yongji
Hi all, Last few months ago, VDUSE (vDPA Device in Userspace) [1] has been merged into Linux kernel as a framework that make it possible to emulate a vDPA device in userspace. This series aimed at implementing a VDUSE block backend based on the qemu-storage-daemon infrastructure. To support

[PATCH v2 2/6] linux-headers: Add vduse.h

2022-02-15 Thread Xie Yongji
This adds vduse header to linux headers so that the relevant VDUSE API can be used in subsequent patches. Signed-off-by: Xie Yongji --- linux-headers/linux/vduse.h | 306 scripts/update-linux-headers.sh | 2 +- 2 files changed, 307 insertions(+), 1

Re: [PATCH 0/4] Make qemu-img dd more flexible

2022-02-15 Thread Fabian Ebner
Am 11.02.22 um 17:42 schrieb Hanna Reitz: > On 11.02.22 17:31, Eric Blake wrote: >> On Thu, Feb 10, 2022 at 02:31:19PM +0100, Fabian Ebner wrote: >>> Adds support for reading from stdin and writing to stdout (when raw >>> format is used), as well as overriding the size of the output and >>> input

Re: [PATCH v2 7/8] tests/qemu-iotests/testrunner: Print diff to stderr in TAP mode

2022-02-15 Thread Paolo Bonzini
On 2/9/22 11:15, Thomas Huth wrote: When running in TAP mode, stdout is reserved for the TAP protocol. To see the "diff" of the failed test, we have to print it to stderr instead. Signed-off-by: Thomas Huth --- tests/qemu-iotests/testrunner.py | 5 - 1 file changed, 4 insertions(+), 1