[PATCH] qemu-img: Use "depth":-1 to make backing probes obvious

2021-06-10 Thread Eric Blake
The recently-added NBD context qemu:allocation-depth makes an obvious case for why it is important to distinguish between locally-present data (even with that data is sparse) [shown as depth 1 over NBD], and data that could not be found anywhere in the backing chain [shown as depth 0]. But

Re: iotest 233 failing

2021-06-10 Thread Daniel P . Berrangé
On Thu, Jun 10, 2021 at 10:31:14PM +0100, Daniel P. Berrangé wrote: > On Thu, Jun 10, 2021 at 03:34:46PM -0500, Eric Blake wrote: > > I'm now getting failures on iotest 233: > > > > 233 fail [15:26:01] [15:26:03] 2.1s (last: 1.3s) output > > mismatch (see 233.out.bad) > > ---

Re: iotest 233 failing

2021-06-10 Thread Daniel P . Berrangé
On Thu, Jun 10, 2021 at 03:34:46PM -0500, Eric Blake wrote: > I'm now getting failures on iotest 233: > > 233 fail [15:26:01] [15:26:03] 2.1s (last: 1.3s) output mismatch > (see 233.out.bad) > --- /home/eblake/qemu/tests/qemu-iotests/233.out > +++ 233.out.bad > @@ -65,6 +65,6 @@ >

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 11:09:05PM +0300, Nir Soffer wrote: > > But: > > > > $ qemu-img map --output=json -f qcow2 > > json:'{"driver":"qcow2","backing":null, \ > > "file":{"driver":"file","filename":"top.qcow2"}}' > > [{ "start": 0, "length": 65536, "depth": 0, "zero": true, "data": false}, >

iotest 233 failing

2021-06-10 Thread Eric Blake
I'm now getting failures on iotest 233: 233 fail [15:26:01] [15:26:03] 2.1s (last: 1.3s) output mismatch (see 233.out.bad) --- /home/eblake/qemu/tests/qemu-iotests/233.out +++ 233.out.bad @@ -65,6 +65,6 @@ == final server log == qemu-nbd: option negotiation failed: Verify failed:

Re: [PATCH 2/4] Python QEMU utils: introduce a generic feature list

2021-06-10 Thread Wainer dos Santos Moschetta
Hi, On 6/8/21 8:55 PM, Cleber Rosa Junior wrote: On Tue, Jun 8, 2021 at 5:42 PM Wainer dos Santos Moschetta mailto:waine...@redhat.com>> wrote: Hi, On 6/8/21 11:09 AM, Cleber Rosa wrote: > Which can be used to check for any "feature" that is available as a > QEMU command

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 9:35 PM Eric Blake wrote: > > On Tue, Jun 08, 2021 at 07:38:10PM +0300, Nir Soffer wrote: > > The example I provided was not detailed enough, what we actually do is: > > > > qemu-nbd .. 'json:{"driver": "qcow2", "backing": null, "file": > > {"driver": "file",

Re: [PATCH 2/4] Python QEMU utils: introduce a generic feature list

2021-06-10 Thread Willian Rampazzo
On Tue, Jun 8, 2021 at 11:09 AM Cleber Rosa wrote: > > Which can be used to check for any "feature" that is available as a > QEMU command line option, and that will return its list of available > options. > > This is a generalization of the list_accel() utility function, which > is itself

Re: [PATCH 3/4] Acceptance Tests: introduce method to require a feature and option

2021-06-10 Thread Willian Rampazzo
On Tue, Jun 8, 2021 at 11:09 AM Cleber Rosa wrote: > > In this context, and according to the qemu.utils.list_feature() utility > function, a feature is something is available as a QEMU command line > option that can take the "help" value. > > This builds on top of that utility function, and

Re: [PATCH 2/4] Python QEMU utils: introduce a generic feature list

2021-06-10 Thread Willian Rampazzo
On Tue, Jun 8, 2021 at 8:55 PM Cleber Rosa Junior wrote: > > > > On Tue, Jun 8, 2021 at 5:42 PM Wainer dos Santos Moschetta > wrote: >> >> Hi, >> >> On 6/8/21 11:09 AM, Cleber Rosa wrote: >> > Which can be used to check for any "feature" that is available as a >> > QEMU command line option, and

Re: [PATCH 4/4] Jobs based on custom runners: add CentOS Stream 8

2021-06-10 Thread Willian Rampazzo
On Tue, Jun 8, 2021 at 11:10 AM Cleber Rosa wrote: > > This introduces three different parts of a job designed to run > on a custom runner managed by Red Hat. The goals include: > > a) serve as a model for other organizations that want to onboard > their own runners, with their specific

Re: [PATCH 0/4] Jobs based on custom runners: add CentOS Stream 8

2021-06-10 Thread Willian Rampazzo
On Tue, Jun 8, 2021 at 11:09 AM Cleber Rosa wrote: > > This builds on top the "GitLab Custom Runners and Jobs (was: QEMU > Gating CI)" series, showing an example of how other entities can > add their own custom jobs to the GitLab CI pipeline. > > First of all, it may be useful to see an actual

Re: [PATCH v4 04/32] block/nbd: connect_thread_func(): do qio_channel_set_delay(false)

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 01:07:34PM +0300, Vladimir Sementsov-Ogievskiy wrote: > nbd_open() does it (through nbd_establish_connection()). > Actually we lost that call on reconnect path in 1dc4718d849e1a1fe > "block/nbd: use non-blocking connect: fix vm hang on connect()" > when we have introduced

Re: [PATCH] qemu-{img,nbd}: Don't report zeroed cluster as a hole

2021-06-10 Thread Eric Blake
On Tue, Jun 08, 2021 at 07:38:10PM +0300, Nir Soffer wrote: > The example I provided was not detailed enough, what we actually do is: > > qemu-nbd .. 'json:{"driver": "qcow2", "backing": null, "file": > {"driver": "file", "filename": "top.qcow2"}}' > > So there is no backing chain and

Re: [PATCH] hw/nvme: be more careful when deasserting IRQs

2021-06-10 Thread Klaus Jensen
On Jun 10 13:46, Jakub Jermář wrote: An IRQ vector used by a completion queue cannot be deasserted without first checking if the same vector does not need to stay asserted for some other completion queue. Signed-off-by: Jakub Jermar --- hw/nvme/ctrl.c | 21 +++-- 1 file changed,

Re: [Libguestfs] [libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 06:20:31PM +0100, Richard W.M. Jones wrote: > > I think you need to update the nbdinfo.pod file as well, since there > are some references there. Yep, thanks for catching. > > As Nir says, it's unfortunate, but I guess we've got to do it. If you > push it (with docs

Re: [PATCH v4 01/32] co-queue: drop extra coroutine_fn marks

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 20:22, Eric Blake wrote: On Thu, Jun 10, 2021 at 01:07:31PM +0300, Vladimir Sementsov-Ogievskiy wrote: qemu_co_queue_next() and qemu_co_queue_restart_all() just call aio_co_wake() which works well in non-coroutine context. So these functions can be called from non-coroutine context

Re: [PATCH] hw/nvme: be more careful when deasserting IRQs

2021-06-10 Thread Klaus Jensen
+cc qemu-block, maintainers On Jun 10 13:46, Jakub Jermář wrote: An IRQ vector used by a completion queue cannot be deasserted without first checking if the same vector does not need to stay asserted for some other completion queue. Signed-off-by: Jakub Jermar --- hw/nvme/ctrl.c | 21

Re: qemu: arm: mounting ubifs using nandsim on busybox

2021-06-10 Thread Pintu Agarwal
On Thu, 10 Jun 2021 at 19:22, Richard Weinberger wrote: > > Pintu, > > - Ursprüngliche Mail - > > Von: "Pintu Agarwal" > >> nandsim works as expected. It creates a new and *erased* NAND for you. > >> So you have no UBI volumes. Therfore UBIFS cannot be mounted. > >> I suggest creating a

Re: [PATCH v4 01/32] co-queue: drop extra coroutine_fn marks

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 01:07:31PM +0300, Vladimir Sementsov-Ogievskiy wrote: > qemu_co_queue_next() and qemu_co_queue_restart_all() just call > aio_co_wake() which works well in non-coroutine context. So these > functions can be called from non-coroutine context as well. And > actually

Re: [Libguestfs] [libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping

2021-06-10 Thread Richard W.M. Jones
I think you need to update the nbdinfo.pod file as well, since there are some references there. As Nir says, it's unfortunate, but I guess we've got to do it. If you push it (with docs fixed) soon then I can put it into the libnbd 1.8.1 stable release. Rich. -- Richard Jones, Virtualization

Re: [libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 6:03 PM Eric Blake wrote: > > Qemu has unfortunately used two distinct meanings for 'allocated' over > time, and that ambiguity has overloaded the word to make it difficult > to know which sense is meant in a given context. In one usage, > 'allocated' implies that data is

Re: [PATCH v3 2/5] block-copy: improve comments of BlockCopyTask and BlockCopyState types and functions

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 17:21, Emanuele Giuseppe Esposito wrote: On 10/06/2021 13:12, Vladimir Sementsov-Ogievskiy wrote: 10.06.2021 13:46, Emanuele Giuseppe Esposito wrote: On 10/06/2021 12:27, Vladimir Sementsov-Ogievskiy wrote: 10.06.2021 13:14, Emanuele Giuseppe Esposito wrote: On 09/06/2021

[libnbd PATCH] info: Avoid ambiguous 'allocated' terminology in mapping

2021-06-10 Thread Eric Blake
Qemu has unfortunately used two distinct meanings for 'allocated' over time, and that ambiguity has overloaded the word to make it difficult to know which sense is meant in a given context. In one usage, 'allocated' implies that data is present and occupies space on a block device (the opposite

Re: [PATCH v3 4/5] block-copy: add a CoMutex

2021-06-10 Thread Emanuele Giuseppe Esposito
On 09/06/2021 14:25, Vladimir Sementsov-Ogievskiy wrote: 08.06.2021 10:33, Emanuele Giuseppe Esposito wrote: Add a CoMutex to protect concurrent access of block-copy data structures. This mutex also protects .copy_bitmap, because its thread-safe API does not prevent it from assigning two

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 17:04, Eric Blake wrote: Maybe the thing to do is improve the documentation and try to avoid ambiguous terminalogy; in qemu:allocation-depth, a return of depth 0 should be called "absent", not "unallocated". And in libnbd, a base:allocation of 0 should be "data" or "normal", not

Re: [PATCH v3 2/5] block-copy: improve comments of BlockCopyTask and BlockCopyState types and functions

2021-06-10 Thread Emanuele Giuseppe Esposito
On 10/06/2021 13:12, Vladimir Sementsov-Ogievskiy wrote: 10.06.2021 13:46, Emanuele Giuseppe Esposito wrote: On 10/06/2021 12:27, Vladimir Sementsov-Ogievskiy wrote: 10.06.2021 13:14, Emanuele Giuseppe Esposito wrote: On 09/06/2021 11:12, Vladimir Sementsov-Ogievskiy wrote: 08.06.2021

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 16:47, Eric Blake wrote: On Thu, Jun 10, 2021 at 03:30:17PM +0300, Vladimir Sementsov-Ogievskiy wrote: The correct fix is for ovirt to additionally use the qemu:allocation-depth metadata context added in 5.2: after all, the actual determination for what is needed to recreate a qcow2

Re: [PATCH v2 17/18] modules: check arch and block load on mismatch

2021-06-10 Thread Gerd Hoffmann
Hi, > Is the JSON file completely static, listing all modules that were built > regardless of whether they are currently installed, or would it need to > be refreshed when installing/uninstalling RPMs with modules ? I would > think we can do the former and simply handle missing modules on disk

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 04:16:27PM +0300, Nir Soffer wrote: > On Thu, Jun 10, 2021 at 2:52 AM Nir Soffer wrote: > > > > On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: > > I posted a work in progress patch implementing support for > qemu:joint-allocaition > in oVirt: >

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 03:30:17PM +0300, Vladimir Sementsov-Ogievskiy wrote: > > > The correct fix is for ovirt to additionally use the > > > qemu:allocation-depth metadata context added in 5.2: after all, the > > > actual determination for what is needed to recreate a qcow2 file is > > > not

Re: [PATCH v3 0/7] block: file-posix queue

2021-06-10 Thread Paolo Bonzini
On 07/06/21 15:52, Maxim Levitsky wrote: I used to have a patch series that was about to fix the block limits of the scsi-block, which I think is similar to this patch series. Sorry that I kind of forgot about it for too much time.

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 02:52:10AM +0300, Nir Soffer wrote: > > So, as a convenience, we can provide yet another metadata context, > > "qemu:joint-allocation", which provides the bulk of the same > > information already available from using "base:allocation" and > > "qemu:allocation-depth" in

Re: [RFC libnbd PATCH] info: Add support for new qemu:joint-allocation

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 4:06 PM Eric Blake wrote: > > On Thu, Jun 10, 2021 at 01:20:13AM +0300, Nir Soffer wrote: > > > + else if (strcmp (metacontext, "qemu:joint-allocation") == 0) { > > > +/* Combo of base:allocation and stripped-down qemu:allocation-depth > > > */ > > > +const char

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Nir Soffer
On Thu, Jun 10, 2021 at 2:52 AM Nir Soffer wrote: > > On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: I posted a work in progress patch implementing support for qemu:joint-allocaition in oVirt: https://gerrit.ovirt.org/c/ovirt-imageio/+/115197 The most important part is the nbd client:

Re: [PATCH v2 17/18] modules: check arch and block load on mismatch

2021-06-10 Thread Daniel P . Berrangé
On Thu, Jun 10, 2021 at 02:57:21PM +0200, Gerd Hoffmann wrote: > Hi, > > > > for (modlist = modinfo->list; modlist != NULL; modlist = > > > modlist->next) { > > > +if (modlist->value->has_arch) { > > > +if (strcmp(modlist->value->name, module_name) == 0) { > > > +

Re: [RFC libnbd PATCH] info: Add support for new qemu:joint-allocation

2021-06-10 Thread Eric Blake
On Thu, Jun 10, 2021 at 01:20:13AM +0300, Nir Soffer wrote: > > + else if (strcmp (metacontext, "qemu:joint-allocation") == 0) { > > +/* Combo of base:allocation and stripped-down qemu:allocation-depth */ > > +const char *base, *depth; > > +switch (type & 3) { > > +case 0: base =

Re: [PATCH v2 17/18] modules: check arch and block load on mismatch

2021-06-10 Thread Gerd Hoffmann
Hi, > > for (modlist = modinfo->list; modlist != NULL; modlist = > > modlist->next) { > > +if (modlist->value->has_arch) { > > +if (strcmp(modlist->value->name, module_name) == 0) { > > +if (!module_arch || > > +

Re: qemu io test: spurious failures of 030

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Hi! 030 is known to crash sometimes The only thing we have (as far as I know) for now is my outdated "[PATCH RFC 0/5] Fix accidental crash in iotest 30" (6 moths ago, time flies fast :\ https://patchew.org/QEMU/20201120161622.1537-1-vsement...@virtuozzo.com/ I hope one day I'll return to

Re: [PATCH v2 17/18] modules: check arch and block load on mismatch

2021-06-10 Thread Daniel P . Berrangé
On Thu, Jun 10, 2021 at 07:57:54AM +0200, Gerd Hoffmann wrote: > Add module_allow_arch() to set the target architecture. > In case a module is limited to some arch verify arches > match and ignore the module if not. > > Signed-off-by: Gerd Hoffmann > --- > include/qemu/module.h | 1 + >

Re: [PATCH 2/2] nbd: Add new qemu:joint-allocation metadata context

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 02:52, Nir Soffer wrote: On Wed, Jun 9, 2021 at 9:01 PM Eric Blake wrote: When trying to reconstruct a qcow2 chain using information provided over NBD, ovirt had been relying on an unsafe assumption that any portion of the qcow2 file advertised as sparse would defer to the backing

Re: [PATCH 1/2] iotests: Improve and rename test 309 to nbd-qemu-allocation

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
09.06.2021 21:01, Eric Blake wrote: Enhance the test to inspect what qemu-nbd is advertising during handshake, and rename it now that we support useful iotest names. Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy -- Best regards, Vladimir

[PATCH v6 8/9] block: Allow changing bs->file on reopen

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
From: Alberto Garcia When the x-blockdev-reopen was added it allowed reconfiguring the graph by replacing backing files, but changing the 'file' option was forbidden. Because of this restriction some operations are not possible, notably inserting and removing block filters. This patch adds

[PATCH v6 9/9] iotests: Test replacing files with x-blockdev-reopen

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
From: Alberto Garcia This patch adds new tests in which we use x-blockdev-reopen to change bs->file Signed-off-by: Alberto Garcia --- tests/qemu-iotests/245 | 109 - tests/qemu-iotests/245.out | 11 +++- 2 files changed, 117 insertions(+), 3

[PATCH v6 4/9] block: bdrv_reopen_parse_backing(): don't check frozen child

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
bdrv_set_backing_noperm() takes care of it (actual check is in bdrv_set_file_or_backing_noperm()), so we don't need to check it here. While being here, improve error message a bit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block.c| 14 +-

Re: [PATCH v4 0/6] Allow changing bs->file on reopen

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
09.06.2021 19:40, Vladimir Sementsov-Ogievskiy wrote: 09.06.2021 18:53, Kevin Wolf wrote: Am 14.05.2021 um 17:53 hat Vladimir Sementsov-Ogievskiy geschrieben: Hi Alberto! What are your plans for v5? I'm now finishing a new series which makes backup-top filter public, and I want to base it on

[PATCH v6 3/9] block: bdrv_reopen_parse_backing(): don't check aio context

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We don't need this check: bdrv_set_backing_noperm() will do it anyway (actually in bdrv_attach_child_common()). Signed-off-by: Vladimir Sementsov-Ogievskiy --- block.c | 33 - 1 file changed, 33 deletions(-) diff --git a/block.c b/block.c index

[PATCH v6 7/9] block: BDRVReopenState: drop replace_backing_bs field

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
It's used only in bdrv_reopen_commit(). "backing" is covered by the loop through all children except for case when we removed backing child during reopen. Make it more obvious and drop extra boolean field: qdict_del will not fail if there is no such entry. Signed-off-by: Vladimir

[PATCH v6 5/9] block: bdrv_reopen_parse_backing(): simplify handling implicit filters

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
The logic around finding overlay here is not obvious. Actually it does two simple things: 1. If new bs is already in backing chain, split from parent bs by several implicit filters we are done, do nothing. 2. Otherwise, don't try to replace implicit filter. Let's rewrite this in more obvious

[PATCH v6 6/9] block: move supports_backing check to bdrv_set_file_or_backing_noperm()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Move supports_backing check of bdrv_reopen_parse_backing to called (through bdrv_set_backing_noperm()) bdrv_set_file_or_backing_noperm() function. The check applies to general case, so it's appropriate for bdrv_set_file_or_backing_noperm(). We have to declare backing support for two test drivers,

[PATCH v6 2/9] block: introduce bdrv_set_file_or_backing_noperm()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
To be used for reopen in future commit. Notes: - It seems OK to update inherits_from if new bs is recursively inherits from parent bs. Let's just not check for backing_chain_contains, to support file child of non-filters. - Simply check child->frozen instead of

[PATCH v6 1/9] block: introduce bdrv_remove_file_or_backing_child()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
To be used for reopen in future commit. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block.c | 24 +--- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/block.c b/block.c index 98673e788a..d21c9e4316 100644 --- a/block.c +++ b/block.c @@ -4795,17 +4795,16 @@

[PATCH v6 0/9] Allow changing bs->file on reopen

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Note that this is only "subject" part of original Alberto's "[PATCH v4 0/6] Allow changing bs->file on reopen" which also included multiple reopen and dropping x- prefix of reopen together with some refactoring. v6: rebased 08: changed error message in test 09: somehow, now '.' are spread in test

[PATCH v5 33/35] iotests/image-fleecing: rename tgt_node

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Actually target of backup(sync=None) is not a final backup target: image fleecing is intended to be used with external tool, which will copy data from fleecing node to some real backup target. Also, we are going to add a test case for "push backup with fleecing", where instead of exporting

[PATCH v5 29/35] iotests/222: constantly use single quotes for strings

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
The file use both single and double quotes for strings. Let's be consistent. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- tests/qemu-iotests/222 | 68 +- 1 file changed, 34 insertions(+), 34 deletions(-) diff --git

[PATCH v5 31/35] iotests.py: hmp_qemu_io: support qdev

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- tests/qemu-iotests/iotests.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py index 025e288ddd..9d0031a0e8 100644 ---

[PATCH v5 35/35] iotests/image-fleecing: add test-case for copy-before-write filter

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
New fleecing method becomes available: copy-before-write filter. Actually we don't need backup job to setup image fleecing. Add test for new recommended way of image fleecing. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- tests/qemu-iotests/tests/image-fleecing |

[PATCH v5 30/35] iotests: move 222 to tests/image-fleecing

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Give a good name to test file. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- tests/qemu-iotests/{222 => tests/image-fleecing} | 0 tests/qemu-iotests/{222.out => tests/image-fleecing.out} | 0 2 files changed, 0 insertions(+), 0 deletions(-) rename

[PATCH v5 28/35] iotests/222: fix pylint and mypy complains

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Here: - long line - move to new interface of vm.qmp() (direct passing dict), to avoid mypy false-positive, as it thinks that unpacked dict is a positional argument. - extra parenthesis - handle event_wait possible None value Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max

[PATCH v5 32/35] iotests/image-fleecing: proper source device

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Define scsi device to operate with it by qom-set in further patch. Give a new node-name to source block node, to not look like device name. Job now don't want to work without giving explicit id, so, let's call it "fleecing". Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz

[PATCH v5 25/35] python/qemu/machine.py: refactor _qemu_args()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
- use shorter construction - don't create new dict if not needed - drop extra unpacking key-val arguments - drop extra default values Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz Reviewed-by: John Snow --- python/qemu/machine/machine.py | 18 -- 1

[PATCH v5 34/35] iotests/image-fleecing: prepare for adding new test-case

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to add a test-case with some behavior modifications. So, let's prepare a function to be reused. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- tests/qemu-iotests/tests/image-fleecing | 19 +-- 1 file changed, 13 insertions(+), 6 deletions(-)

[PATCH v5 18/35] block/copy-before-write: cbw_init(): use file child after attaching

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
In the next commit we'll get rid of source argument of cbw_init(). Prepare to it now, to make next commit simpler: move the code block that uses source below attaching the child and use bs->file->bs instead of source variable. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz

[PATCH v5 23/35] block/copy-before-write: make public block driver

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Finally, copy-before-write gets own .bdrv_open and .bdrv_close handlers, block_init() call and becomes available through bdrv_open(). To achieve this: - cbw_init gets unused flags argument and becomes cbw_open - block_copy_state_free() call moved to new cbw_close() - in bdrv_cbw_append: -

[PATCH v5 15/35] block/copy-before-write: bdrv_cbw_append(): replace child at last

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Refactor the function to replace child at last. Thus we don't need to revert it and code is simplified. block-copy state initialization being done before replacing the child doesn't need any drained section. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz ---

[PATCH v5 27/35] iotests.py: VM: add own __enter__ method

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
In superclass __enter__ method is defined with return value type hint 'QEMUMachine'. So, mypy thinks that return value of VM.__enter__ is QEMUMachine. Let's redefine __enter__ in VM class, to give it correct type hint. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz ---

[PATCH v5 26/35] python/qemu/machine: QEMUMachine: improve qmp() method

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We often call qmp() with unpacking dict, like qmp('foo', **{...}). mypy don't really like it, it thinks that passed unpacked dict is a positional argument and complains that it type should be bool (because second argument of qmp() is conv_keys: bool). Allow passing dict directly, simplifying

[PATCH v5 21/35] block/copy-before-write: initialize block-copy bitmap

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to publish copy-before-write filter to be used in separate of backup. Future step would support bitmap for the filter. But let's start from full set bitmap. We have to modify backup, as bitmap is first initialized by copy-before-write filter, and then backup modifies it.

[PATCH v5 19/35] block/copy-before-write: bdrv_cbw_append(): drop unused compress arg

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.h | 1 - block/backup.c| 2 +- block/copy-before-write.c | 7 +++ 3 files changed, 4 insertions(+), 6 deletions(-) diff --git a/block/copy-before-write.h b/block/copy-before-write.h index

[PATCH v5 20/35] block/copy-before-write: cbw_init(): use options

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
One more step closer to .bdrv_open(): use options instead of plain arguments. Move to bdrv_open_child() calls, native for drive open handlers. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/copy-before-write.c | 29 +++-- 1 file changed, 15 insertions(+), 14

[PATCH v5 22/35] block/block-copy: make setting progress optional

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Now block-copy will crash if user don't set progress meter by block_copy_set_progress_meter(). copy-before-write filter will be used in separate of backup job, and it doesn't want any progress meter (for now). So, allow not setting it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max

[PATCH v5 12/35] block/copy-before-write: relax permission requirements when no parents

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to publish copy-before-write filter. So, user should be able to create it with blockdev-add first, specifying both filtered and target children. And then do blockdev-reopen, to actually insert the filter where needed. Currently, filter unshares write permission unconditionally on

[PATCH v5 17/35] block/copy-before-write: cbw_init(): rename variables

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
One more step closer to real .bdrv_open() handler: use more usual names for bs being initialized and its state. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/copy-before-write.c | 29 ++--- 1 file changed, 14 insertions(+), 15 deletions(-)

[PATCH v5 24/35] qapi: publish copy-before-write filter

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- qapi/block-core.json | 30 ++ 1 file changed, 26 insertions(+), 4 deletions(-) diff --git a/qapi/block-core.json b/qapi/block-core.json index 2ea294129e..6c1ce86235 100644 ---

[PATCH v5 09/35] block/block-copy: introduce block_copy_set_copy_opts()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We'll need a possibility to set compress and use_copy_range options after initialization of the state. So make corresponding part of block_copy_state_new() separate and public. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block-copy.h | 2 ++ block/block-copy.c | 56

[PATCH v5 11/35] block/backup: move cluster size calculation to block-copy

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
The main consumer of cluster-size is block-copy. Let's calculate it here instead of passing through backup-top. We are going to publish copy-before-write filter soon, so it will be created through options. But we don't want for now to make explicit option for cluster-size, let's continue to

[PATCH v5 05/35] qdev-properties: PropertyInfo: add realized_set_allowed field

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Add field, so property can declare support for setting the property when device is realized. To be used in the following commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- include/hw/qdev-properties.h | 1 + hw/core/qdev-properties.c| 6 +++--- 2 files changed,

[PATCH v5 04/35] block: introduce blk_replace_bs

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Add function to change bs inside blk. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- include/sysemu/block-backend.h | 1 + block/block-backend.c | 8 2 files changed, 9 insertions(+) diff --git a/include/sysemu/block-backend.h

[PATCH v5 16/35] block/copy-before-write: introduce cbw_init()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Move part of bdrv_cbw_append() to new function cbw_open(). It's an intermediate step for adding normal .bdrv_open() handler to the filter. With this commit no logic is changed, but we have a function which will be turned into .bdrv_open() handler in future commit. Signed-off-by: Vladimir

[PATCH v5 13/35] block/copy-before-write: drop extra bdrv_unref on failure path

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
bdrv_attach_child() do bdrv_unref() on failure, so we shouldn't do it by hand here. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/copy-before-write.c | 1 - 1 file changed, 1 deletion(-) diff --git a/block/copy-before-write.c b/block/copy-before-write.c index

[PATCH v5 14/35] block/copy-before-write: use file child instead of backing

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to publish copy-before-write filter, and there no public backing-child-based filter in Qemu. No reason to create a precedent, so let's refactor copy-before-write filter instead. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/copy-before-write.c | 39

[PATCH v5 10/35] block/backup: set copy_range and compress after filter insertion

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to publish copy-before-write filter, so it would be initialized through options. Still we don't want to publish compress and copy-range options, as 1. Modern way to enable compression is to use compress filter. 2. For copy-range it's unclean how to make proper interface: - it's has

[PATCH v5 03/35] block: introduce bdrv_replace_child_bs()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Add function to transactionally replace bs inside BdrvChild. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- include/block/block.h | 2 ++ block.c | 31 +++ 2 files changed, 33 insertions(+) diff --git a/include/block/block.h

[PATCH v5 08/35] block-copy: always set BDRV_REQ_SERIALISING flag

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
It won't hurt in common case, so let's not bother with detecting image fleecing. Also, we want to simplify initialization interface of copy-before-write filter as we are going to make it public. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/copy-before-write.h

[PATCH v5 07/35] block: rename backup-top to copy-before-write

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to convert backup_top to full featured public filter, which can be used in separate of backup job. Start from renaming from "how it used" to "what it does". While updating comments in 283 iotest, drop and rephrase also things about ".active", as this field is now dropped, and filter

[PATCH v5 00/35] block: publish backup-top filter

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Hi all! v5: rebase on master 25: fix pylint complain 'Unnecessary "else" after "return"' [John] 25-26: rebased on moved machine.py use Dict[str, object] typing instead of Dict[str, Any] [John] add John's r-b, keep Max's r-b Patches without r-b: 9, 10, 19, 20 v4: rebase on Kevin's

[PATCH v5 06/35] qdev: allow setting drive property for realized device

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We need an ability to insert filters above top block node, attached to block device. It can't be achieved with blockdev-reopen command. So, we want do it with help of qom-set. Intended usage: Assume there is a node A that is attached to some guest device. 1. blockdev-add to create a filter node

[PATCH v5 02/35] block: comment graph-modifying function not updating permissions

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block.c | 8 1 file changed, 8 insertions(+) diff --git a/block.c b/block.c index 34bfe4ffe8..98673e788a 100644 --- a/block.c +++ b/block.c @@ -2770,6 +2770,8 @@ static TransactionActionDrv

[PATCH v5 01/35] block: rename bdrv_replace_child to bdrv_replace_child_tran

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We have bdrv_replace_child() wrapper on bdrv_replace_child_noperm(). But bdrv_replace_child() doesn't update permissions. It's rather strange, as normally it's expected that foo() should call foo_noperm() and update permissions. Let's rename and add comment. Signed-off-by: Vladimir

Re: [PATCH v3 2/5] block-copy: improve comments of BlockCopyTask and BlockCopyState types and functions

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 13:46, Emanuele Giuseppe Esposito wrote: On 10/06/2021 12:27, Vladimir Sementsov-Ogievskiy wrote: 10.06.2021 13:14, Emanuele Giuseppe Esposito wrote: On 09/06/2021 11:12, Vladimir Sementsov-Ogievskiy wrote: 08.06.2021 10:33, Emanuele Giuseppe Esposito wrote: As done in

Re: [PATCH v3 2/5] block-copy: improve comments of BlockCopyTask and BlockCopyState types and functions

2021-06-10 Thread Emanuele Giuseppe Esposito
On 10/06/2021 12:27, Vladimir Sementsov-Ogievskiy wrote: 10.06.2021 13:14, Emanuele Giuseppe Esposito wrote: On 09/06/2021 11:12, Vladimir Sementsov-Ogievskiy wrote: 08.06.2021 10:33, Emanuele Giuseppe Esposito wrote: As done in BlockCopyCallState, categorize BlockCopyTask and

Re: [PATCH v3 2/5] block-copy: improve comments of BlockCopyTask and BlockCopyState types and functions

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
10.06.2021 13:14, Emanuele Giuseppe Esposito wrote: On 09/06/2021 11:12, Vladimir Sementsov-Ogievskiy wrote: 08.06.2021 10:33, Emanuele Giuseppe Esposito wrote: As done in BlockCopyCallState, categorize BlockCopyTask and BlockCopyState in IN, State and OUT fields. This is just to understand

Re: [PATCH v3 2/5] block-copy: improve comments of BlockCopyTask and BlockCopyState types and functions

2021-06-10 Thread Emanuele Giuseppe Esposito
On 09/06/2021 11:12, Vladimir Sementsov-Ogievskiy wrote: 08.06.2021 10:33, Emanuele Giuseppe Esposito wrote: As done in BlockCopyCallState, categorize BlockCopyTask and BlockCopyState in IN, State and OUT fields. This is just to understand which field has to be protected with a lock.

[PATCH v4 28/32] block/nbd: split nbd_co_do_establish_connection out of nbd_reconnect_attempt

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Split out the part that we want to reuse for nbd_open(). Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/nbd.c | 80 - 1 file changed, 42 insertions(+), 38 deletions(-) diff --git a/block/nbd.c b/block/nbd.c index 411435c155..8caeafc8d3

[PATCH v4 27/32] block-coroutine-wrapper: allow non bdrv_ prefix

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We are going to reuse the script to generate a nbd_ function in further commit. Prepare the script now. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- scripts/block-coroutine-wrapper.py | 7 --- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git

[PATCH v4 31/32] block/nbd: add nbd_client_connected() helper

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We already have two similar helpers for other state. Let's add another one for convenience. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- block/nbd.c | 25 ++--- 1 file changed, 14 insertions(+), 11 deletions(-) diff --git a/block/nbd.c

[PATCH v4 29/32] nbd/client-connection: add option for non-blocking connection attempt

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
We'll need a possibility of non-blocking nbd_co_establish_connection(), so that it returns immediately, and it returns success only if a connections was previously established in background. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- include/block/nbd.h | 2 +-

[PATCH v4 26/32] nbd/client-connection: return only one io channel

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
block/nbd doesn't need underlying sioc channel anymore. So, we can update nbd/client-connection interface to return only one top-most io channel, which is more straight forward. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- include/block/nbd.h | 4 ++--

[PATCH v4 22/32] block/nbd: split nbd_handle_updated_info out of nbd_client_handshake()

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
To be reused in the following patch. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Roman Kagan Reviewed-by: Eric Blake --- block/nbd.c | 100 ++-- 1 file changed, 58 insertions(+), 42 deletions(-) diff --git a/block/nbd.c

[PATCH v4 20/32] nbd/client-connection: implement connection retry

2021-06-10 Thread Vladimir Sementsov-Ogievskiy
Add an option for a thread to retry connection until succeeds. We'll use nbd/client-connection both for reconnect and for initial connection in nbd_open(), so we need a possibility to use same NBDClientConnection instance to connect once in nbd_open() and then use retry semantics for reconnect.

  1   2   >