Re: [Qemu-block] [PATCH] iotests: test clearing unknown autoclear_features by qcow2

10.11.2017 23:02, Kevin Wolf wrote: Am 10.11.2017 um 18:54 hat Vladimir Sementsov-Ogievskiy geschrieben: Test clearing unknown autoclear_features by qcow2 on incoming migration. Signed-off-by: Vladimir Sementsov-Ogievskiy --- Hi all! This patch shows degradation,

Re: [Qemu-block] [PATCH 01/10] qemu-iotests: make execution of tests agnostic to test type

On 16/11/2017 18:38, Cleber Rosa wrote: > check makes a distinction on how it runs Python based tests. The > current approach is inconsistent because: > > 1) a large number of Python tests are already set as executable files > (eg: 030, 040, 041, 044, 045, 055, 056, 057, 065, 093, 118, 147, 149,

Re: [Qemu-block] [PATCH 06/10] qemu-iotests: turn owner variable into a comment

On 16/11/2017 18:38, Cleber Rosa wrote: > This variables has no real use. To avoid pretending it does, while > still keeping the information, let's turn it into a comment. > > The format chosen is the one already being used on tests 149 and 194. I would just delete it... Paolo >

Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API

On 11/16/2017 03:17 AM, Vladimir Sementsov-Ogievskiy wrote: > 16.11.2017 00:20, John Snow wrote: >> >> On 11/13/2017 11:20 AM, Vladimir Sementsov-Ogievskiy wrote: >>> Hi all. >>> >>> There are three qmp commands, needed to implement external backup API. >>> >>> Using these three commands, client

Re: [Qemu-block] [Qemu-devel] [PATCH v8 10/14] migration: add postcopy migration of dirty bitmaps

On 11/16/2017 11:50 AM, Dr. David Alan Gilbert wrote: > * John Snow (js...@redhat.com) wrote: >> >> >> On 10/30/2017 12:33 PM, Vladimir Sementsov-Ogievskiy wrote: >>> Postcopy migration of dirty bitmaps. Only named dirty bitmaps, >>> associated with root nodes and non-root named nodes are

Re: [Qemu-block] [Qemu-devel] [PATCH v8 10/14] migration: add postcopy migration of dirty bitmaps

On 11/16/2017 05:24 AM, Vladimir Sementsov-Ogievskiy wrote: > 15.11.2017 04:58, John Snow wrote: >> >> On 10/30/2017 12:33 PM, Vladimir Sementsov-Ogievskiy wrote: >>> Postcopy migration of dirty bitmaps. Only named dirty bitmaps, >>> associated with root nodes and non-root named nodes are

Re: [Qemu-block] [Qemu-devel] [PATCH v8 10/14] migration: add postcopy migration of dirty bitmaps

On 11/16/2017 05:24 AM, Vladimir Sementsov-Ogievskiy wrote: >>> +    /* if a block is zero we need to flush here since the network >>> + * bandwidth is now a lot higher than the storage device bandwidth. >>> + * thus if we queue zero blocks we slow down the migration. */ >> Can you

Re: [Qemu-block] segfault in parallel blockjobs (iotest 30)

On 11/16/2017 10:54 AM, Alberto Garcia wrote: > On Wed 15 Nov 2017 05:31:20 PM CET, Anton Nefedov wrote: >>> I have the impression that one major source of headaches is the fact >>> that the reopen queue contains nodes that don't need to be reopened at >>> all. Ideally this should be detected

Re: [Qemu-block] [PATCH v2] qapi: block-core: Clarify events emitted by 'block-job-cancel'

On 11/16/2017 04:14 AM, Kashyap Chamarthy wrote: > On Wed, Nov 15, 2017 at 04:56:13PM -0500, John Snow wrote: >> >> >> On 11/15/2017 04:54 PM, Kashyap Chamarthy wrote: >>> On Wed, Nov 15, 2017 at 02:15:57PM -0500, John Snow wrote: > > [...] > is it covered sufficiently in

Re: [Qemu-block] [PATCH] iotests: test clearing unknown autoclear_features by qcow2

On 11/10/2017 03:02 PM, Kevin Wolf wrote: > Am 10.11.2017 um 18:54 hat Vladimir Sementsov-Ogievskiy geschrieben: >> Test clearing unknown autoclear_features by qcow2 on incoming >> migration. >> >> Signed-off-by: Vladimir Sementsov-Ogievskiy >> --- >> >> Hi all! >> >>

Re: [Qemu-block] [Qemu-devel] [PATCH v8 02/14] block/dirty-bitmap: add locked version of bdrv_release_dirty_bitmap

On 11/16/2017 03:56 AM, Vladimir Sementsov-Ogievskiy wrote: > 11.11.2017 01:52, John Snow wrote: >> >> On 10/30/2017 12:32 PM, Vladimir Sementsov-Ogievskiy wrote: >>> It is needed to realize bdrv_dirty_bitmap_release_successor in >>> the following patch. >>> >> OK, but... >> >>> Signed-off-by:

[Qemu-block] [PATCH 05/10] qemu-iotests: define functions used in _cleanup() before its use

The functions used in _cleanup() come from common.rc, which currently gets sourced after _cleanup() is defined and registered as a signal handler. When _cleanup() is executed, it has no valid references to those functions, as BASH won't resolve the reference at that time. While at it, also fix

[Qemu-block] [PATCH 09/10] qemu-iotests: remove unused "here" variable

Another legacy variable that did not convince me it has any purpose whatsoever. Signed-off-by: Cleber Rosa --- tests/qemu-iotests/001 | 1 - tests/qemu-iotests/002 | 1 - tests/qemu-iotests/003 | 1 - tests/qemu-iotests/004 | 1 - tests/qemu-iotests/005 | 1 -

[Qemu-block] [PATCH 10/10] qemu-iotests: add section on how to write a new I/O test

This adds some basic information on how to write a new test. I'm aware that some of the information in the wiki (Testing/QemuIoTests) could also belong here. Since copying content over won't generate much interesting feedback, the goal here is to get feedback on the sample_test_templates,

[Qemu-block] [PATCH 06/10] qemu-iotests: turn owner variable into a comment

This variables has no real use. To avoid pretending it does, while still keeping the information, let's turn it into a comment. The format chosen is the one already being used on tests 149 and 194. Signed-off-by: Cleber Rosa --- tests/qemu-iotests/001 | 5 ++---

[Qemu-block] [PATCH 07/10] qemu-iotests: remove the concept of $seq.full (and boiler plate code)

The $seq.full file, in theory, should contain the full output of a test error. In practice, it's only used on a single test, and the boiler plate code to clean it up plagues all other tests. Let's remove the concept altogether, and record the failure in the output itself for the one test using

[Qemu-block] [PATCH 04/10] qemu-iotests: include (source) filters from common.rc

There's an explicit dependency from common.rc on common.filters, that is, it consumes functions defined there. Just like common.config is included in common.rc, it makes sense to also sense common.filter. This drops the requirement on individual tests to include common.filter, reducing the

[Qemu-block] [PATCH 08/10] qemu-iotests: clean up double comment characters

This is a syntactic only change, just to make it consistent with the style used on all other tests. Signed-off-by: Cleber Rosa --- tests/qemu-iotests/048 | 37 ++--- tests/qemu-iotests/074 | 40 2 files

[Qemu-block] [PATCH 01/10] qemu-iotests: make execution of tests agnostic to test type

check makes a distinction on how it runs Python based tests. The current approach is inconsistent because: 1) a large number of Python tests are already set as executable files (eg: 030, 040, 041, 044, 045, 055, 056, 057, 065, 093, 118, 147, 149, 155, 165 and 194) 2) a smaller number of Python

[Qemu-block] [PATCH 03/10] qemu-iotests: be strict with expected output

The contract between runner (check) and test is one that accepts some minor differences in the expcted output, as the comparison method employed is a "diff -w". This is an exception to a clearer and more straightforward rule of just requiring the generated output to be *equal* (and not similar)

[Qemu-block] [PATCH 00/10] I/O tests cleanups

Hi all, This is a collection of cleanups, simplifications, and hopefully improvements to the I/O tests. Please don't mind the "change noise": this almost qualifies as a collection of trivial patches. It skips that category (of trivial patches) because it proposes a few conceptual changes to how

[Qemu-block] [PATCH 02/10] qemu-iotests: fix filename containing checks

Commit cce293a2945 moved some functions from common.config to common.rc, but the error messages still reference the old file location. Signed-off-by: Cleber Rosa --- tests/qemu-iotests/common.rc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git

[Qemu-block] [PATCH v2 4/4] iotests: 030: add compressed block-stream test

Signed-off-by: Anton Nefedov --- tests/qemu-iotests/030 | 66 +- tests/qemu-iotests/030.out | 4 +-- 2 files changed, 67 insertions(+), 3 deletions(-) diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030 index

[Qemu-block] [PATCH v2 1/4] qcow2: multiple clusters write compressed

From: Pavel Butsykin At the moment, qcow2_co_pwritev_compressed can process the requests size less than or equal to one cluster. This patch added possibility to write compressed data in the QCOW2 more than one cluster. The implementation is simple, we just split large

[Qemu-block] [PATCH v2 0/4] compressed block-stream

v2: - 1st patch omitted (applied to 2.11) - remarks applied v1: http://lists.nongnu.org/archive/html/qemu-devel/2017-11/msg02515.html It might be useful to compress images during block-stream; this way the user can merge compressed images of a backing chain and the result will remain

[Qemu-block] [PATCH v2 2/4] block: support compressed write for copy-on-read

Signed-off-by: Anton Nefedov Reviewed-by: Max Reitz --- block/io.c | 23 +-- block/trace-events | 2 +- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/block/io.c b/block/io.c index 3d5ef2c..2ba62a3

[Qemu-block] [PATCH v2 3/4] block-stream: add compress option

It might be useful to compress images during block-stream; this way the user can merge compressed images of a backing chain and the result will remain compressed. Signed-off-by: Anton Nefedov --- qapi/block-core.json | 4 include/block/block_int.h | 4

Re: [Qemu-block] [Qemu-devel] [PATCH v8 10/14] migration: add postcopy migration of dirty bitmaps

* John Snow (js...@redhat.com) wrote: > > > On 10/30/2017 12:33 PM, Vladimir Sementsov-Ogievskiy wrote: > > Postcopy migration of dirty bitmaps. Only named dirty bitmaps, > > associated with root nodes and non-root named nodes are migrated. > > > > If destination qemu is already containing a

Re: [Qemu-block] [Nbd] [Qemu-devel] How to online resize qemu disk with nbd protocol?

On Thu, Nov 16, 2017 at 09:30:41AM -0600, Eric Blake wrote: > On 11/16/2017 03:51 AM, Wouter Verhelst wrote: > > >> I also remember from talking with Vladimir during KVM Forum last month > >> that one of the shortfalls of the NBD protocol is that you can only ever > >> send a length of up to 32

Re: [Qemu-block] segfault in parallel blockjobs (iotest 30)

On 16/11/2017 6:54 PM, Alberto Garcia wrote: On Wed 15 Nov 2017 05:31:20 PM CET, Anton Nefedov wrote: I have the impression that one major source of headaches is the fact that the reopen queue contains nodes that don't need to be reopened at all. Ideally this should be detected early on in

Re: [Qemu-block] segfault in parallel blockjobs (iotest 30)

On Wed 15 Nov 2017 05:31:20 PM CET, Anton Nefedov wrote: >> I have the impression that one major source of headaches is the fact >> that the reopen queue contains nodes that don't need to be reopened at >> all. Ideally this should be detected early on in bdrv_reopen_queue(), so >> there's no

Re: [Qemu-block] [Nbd] [Qemu-devel] How to online resize qemu disk with nbd protocol?

On 11/16/2017 03:51 AM, Wouter Verhelst wrote: >> I also remember from talking with Vladimir during KVM Forum last month >> that one of the shortfalls of the NBD protocol is that you can only ever >> send a length of up to 32 bits on the command side (unless we introduce >> structured commands in

Re: [Qemu-block] [ovirt-users] Enabling libgfapi disk access with oVirt 4.2

Hi, I don't mean on the fly upgrade, just confused that i should stop all VMs at once, as i understood procedure. If it's possible to do per VM, it's perfectly OK for me. Thank you Nir for clarification. Best regards, Misak Khachatryan On Thu, Nov 16, 2017 at 2:05 AM, Nir Soffer

Re: [Qemu-block] [PATCH for-2.11] throttle-groups: forget timer and schedule next TGM on detach

On Thu, Nov 16, 2017 at 11:21:50AM +, Stefan Hajnoczi wrote: > tg->any_timer_armed[] must be cleared when detaching pending timers from > the AioContext. Failure to do so leads to hung I/O because it looks > like there are still timers pending when in fact they have been removed. > > Other

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

On 2017-11-16 11:08, Gandalf Corvotempesta wrote: > 2017-11-15 23:55 GMT+01:00 Max Reitz : >> https://xanclic.moe/convert-xva.rb -- does this work? >> (It seems to works on the two example images I found...) >> >> An example is in the code, you use it like this: >> >> $

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

I think it's even possible to modify that shell script to pipe into nbdkit and from there to ‘qemu-img convert’. I'm too lazy to actually do that right now, but the basic idea is here: https://rwmj.wordpress.com/2014/10/14/streaming-nbd-server/ You'll probably have to add ‘-m 1’ to the

Re: [Qemu-block] [PATCH 5/5] iotest 030: add compressed block-stream test

On 15/11/2017 10:51 PM, Max Reitz wrote: On 2017-11-14 11:16, Anton Nefedov wrote: Signed-off-by: Anton Nefedov --- tests/qemu-iotests/030 | 69 +- tests/qemu-iotests/030.out | 4 +-- 2 files changed, 70

Re: [Qemu-block] [PATCH for-2.11] throttle-groups: forget timer and schedule next TGM on detach

On Thu 16 Nov 2017 12:21:50 PM CET, Stefan Hajnoczi wrote: > tg->any_timer_armed[] must be cleared when detaching pending timers from > the AioContext. Failure to do so leads to hung I/O because it looks > like there are still timers pending when in fact they have been removed. > > Other

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

On Thu, 16 Nov 2017 13:56:16 +0100 Tomáš Golembiovský wrote: > On Wed, 15 Nov 2017 21:41:20 +0100 > Gandalf Corvotempesta wrote: > > > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones : > > > Gandalf, is there an XVA file

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

On Wed, 15 Nov 2017 21:41:20 +0100 Gandalf Corvotempesta wrote: > 2017-11-15 21:29 GMT+01:00 Richard W.M. Jones : > > Gandalf, is there an XVA file publically available (pref. not > > too big) that we can look at? > > I can try to provide one,

Re: [Qemu-block] [PATCH v4] throttle-groups: drain before detaching ThrottleState

On Mon, Nov 13, 2017 at 2:29 PM, Alberto Garcia wrote: > On Fri 10 Nov 2017 04:19:34 PM CET, Stefan Hajnoczi wrote: >> I/O requests hang after stop/cont commands at least since QEMU 2.10.0 >> with -drive iops=100: >> >> (guest)$ dd if=/dev/zero of=/dev/vdb oflag=direct

[Qemu-block] [PATCH for-2.11] throttle-groups: forget timer and schedule next TGM on detach

tg->any_timer_armed[] must be cleared when detaching pending timers from the AioContext. Failure to do so leads to hung I/O because it looks like there are still timers pending when in fact they have been removed. Other ThrottleGroupMembers might have requests pending too so it's necessary to

Re: [Qemu-block] [Qemu-devel] [PATCH v8 10/14] migration: add postcopy migration of dirty bitmaps

15.11.2017 04:58, John Snow wrote: On 10/30/2017 12:33 PM, Vladimir Sementsov-Ogievskiy wrote: Postcopy migration of dirty bitmaps. Only named dirty bitmaps, associated with root nodes and non-root named nodes are migrated. If destination qemu is already containing a dirty bitmap with the

Re: [Qemu-block] [PATCH 4/5] block-stream: add compress option

On 15/11/2017 10:16 PM, Max Reitz wrote: On 2017-11-14 11:16, Anton Nefedov wrote: Signed-off-by: Anton Nefedov --- qapi/block-core.json | 4 include/block/block_int.h | 4 +++- block/stream.c| 16 blockdev.c

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

On Thu, Nov 16, 2017 at 11:07:10AM +0100, Gandalf Corvotempesta wrote: > 2017-11-16 11:01 GMT+01:00 Richard W.M. Jones : > > As mentioned before you can use this to do a qemu-img convert using > > captive nbdkit: > > > > $ nbdkit -U - \ > > perl script=./xva-reader.pl

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

2017-11-16 11:01 GMT+01:00 Richard W.M. Jones : > As mentioned before you can use this to do a qemu-img convert using > captive nbdkit: > > $ nbdkit -U - \ > perl script=./xva-reader.pl file=./debian8cloud.xva size=4294967296 \ > --run 'qemu-img convert -f raw $nbd

Re: [Qemu-block] [Libguestfs] [Qemu-devel] [qemu-img] support for XVA

2017-11-15 23:55 GMT+01:00 Max Reitz : > https://xanclic.moe/convert-xva.rb -- does this work? > (It seems to works on the two example images I found...) > > An example is in the code, you use it like this: > > $ ./convert-xva.rb ~/Downloads/stats-appliance-2.36.020502.xva

Re: [Qemu-block] [PATCH 3/5] block: support compressed write for copy-on-read

On 15/11/2017 9:49 PM, Max Reitz wrote: On 2017-11-14 11:16, Anton Nefedov wrote: Signed-off-by: Anton Nefedov --- block/io.c | 30 -- block/trace-events | 2 +- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git

Re: [Qemu-block] [PATCH 0/5] compressed block-stream

On 16/11/2017 6:26 AM, Fam Zheng wrote: On Tue, 11/14 13:16, Anton Nefedov wrote: It might be useful to compress images during block-stream; this way the user can merge compressed images of a backing chain and the result will remain compressed. I haven't looked at the patches yet so maybe the

Re: [Qemu-block] [Qemu-devel] [Libguestfs] [qemu-img] support for XVA

Here's my solution, as a nbdkit plugin written in Perl. As with Max's solution I don't bother to parse the virtual size out of the XML file, so you need to specify that on the command line otherwise the disk will be truncated to the largest extent stored in the file. Also the ‘.xva’ file must

Re: [Qemu-block] [Nbd] [Qemu-devel] How to online resize qemu disk with nbd protocol?

On Tue, Nov 14, 2017 at 01:06:17PM -0600, Eric Blake wrote: > On 11/14/2017 11:37 AM, Wouter Verhelst wrote: > > On Tue, Nov 14, 2017 at 10:41:39AM -0600, Eric Blake wrote: > >> Another thought - with structured replies, we finally have a way to let > >> the client ask for the server to send

Re: [Qemu-block] [PATCH v2] qapi: block-core: Clarify events emitted by 'block-job-cancel'

On Wed, Nov 15, 2017 at 04:56:13PM -0500, John Snow wrote: > > > On 11/15/2017 04:54 PM, Kashyap Chamarthy wrote: > > On Wed, Nov 15, 2017 at 02:15:57PM -0500, John Snow wrote: [...] > >> is it covered sufficiently in live-block-operations.rst ? > > > > I looked in there[2] too. Short

Re: [Qemu-block] [Qemu-devel] [PATCH v8 02/14] block/dirty-bitmap: add locked version of bdrv_release_dirty_bitmap

11.11.2017 01:52, John Snow wrote: On 10/30/2017 12:32 PM, Vladimir Sementsov-Ogievskiy wrote: It is needed to realize bdrv_dirty_bitmap_release_successor in the following patch. OK, but... Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/dirty-bitmap.c |

Re: [Qemu-block] [PATCH v2 for-2.11] nbd/server: Fix error reporting for bad requests

16.11.2017 00:35, Eric Blake wrote: The NBD spec says an attempt to NBD_CMD_TRIM on a read-only export should fail with EPERM, as a trim has the potential to change disk contents, but we were relying on the block layer to catch that for us, which might not always give the right error (and even

Re: [Qemu-block] [Qemu-devel] [PATCH for-2.12 0/4] qmp dirty bitmap API

16.11.2017 00:20, John Snow wrote: On 11/13/2017 11:20 AM, Vladimir Sementsov-Ogievskiy wrote: Hi all. There are three qmp commands, needed to implement external backup API. Using these three commands, client may do all needed bitmap management by hand: on backup start we need to do a

Re: [Qemu-block] [PATCH 1/4 for-2.11?] block/dirty-bitmap: add lock to bdrv_enable/disable_dirty_bitmap

13.11.2017 20:50, Eric Blake wrote: On 11/13/2017 10:20 AM, Vladimir Sementsov-Ogievskiy wrote: Like other setters here these functions should take a lock. Signed-off-by: Vladimir Sementsov-Ogievskiy --- block/dirty-bitmap.c | 4 1 file changed, 4