Re: cluster_size got from backup_calculate_cluster_size()

2020-05-21 Thread Derek Su
On 2020/5/22 上午4:53, Vladimir Sementsov-Ogievskiy wrote: 21.05.2020 21:19, John Snow wrote: On 5/21/20 5:56 AM, Derek Su wrote: Hi, The cluster_size got from backup_calculate_cluster_size(), MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size), is 64K regardless of the target image's cluster

Re: [PATCH v3 07/17] block/io: support int64_t bytes in bdrv_co_do_copy_on_readv()

2020-05-21 Thread Eric Blake
On 4/30/20 6:10 AM, Vladimir Sementsov-Ogievskiy wrote: We are generally moving to int64_t for both offset and bytes parameters on all io paths. Main motivation is realization of 64-bit write_zeroes operation for fast zeroing large disk chunks, up to the whole disk. We chose signed type, to be

[PATCH v4 4/6] iotests: 194: test also migration of dirty bitmap

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
Test that dirty bitmap migration works when we deal with mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- tests/qemu-iotests/194 | 14 ++ tests/qemu-iotests/194.out | 6 ++ 2 files changed, 16 insertions(+), 4 deletions(-) diff --git

[PATCH v4 6/6] migration/block-dirty-bitmap: forbid migration by generated node-name

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
It actually never worked with libvirt, as auto-generated names are different on source and destination. It's unsafe and useless to migrate by auto-generated node-names, so let's forbid it. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Eric Blake ---

[PATCH v4 3/6] migration/block-dirty-bitmap: fix bitmaps pre-blockdev migration during mirror job

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
Important thing for bitmap migration is to select destination block node to obtain the migrated bitmap. Prepatch, on source we use bdrv_get_device_or_node_name() to identify the node, and on target we do bdrv_lookup_bs. bdrv_get_device_or_node_name() returns blk name only for direct children of

[PATCH v4 2/6] block/dirty-bitmap: add bdrv_has_named_bitmaps helper

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
To be used for bitmap migration in further commit. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Eric Blake --- include/block/dirty-bitmap.h | 1 + block/dirty-bitmap.c | 13 + 2 files changed, 14 insertions(+) diff --git

[PATCH v4 5/6] migration/block-dirty-bitmap: add_bitmaps_to_list: check disk name once

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich Reviewed-by: Eric Blake --- migration/block-dirty-bitmap.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c index

[PATCH v4 1/6] migration/block-dirty-bitmap: refactor init_dirty_bitmap_migration

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
Split out handling one bs, it is needed for the following commit, which will handle BlockBackends separately. Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Eric Blake --- migration/block-dirty-bitmap.c | 89 +++--- 1 file changed, 49 insertions(+), 40

[PATCH v4 0/6] fix migration with bitmaps and mirror

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
v4: (Max's patch marking filters as filters already in master) 03: careful select child of filter, avoid crash 04: add Eric's r-b 05-06: tweak error message, keep Andrey's r-b, add Eric's r-b Vladimir Sementsov-Ogievskiy (6): migration/block-dirty-bitmap: refactor init_dirty_bitmap_migration

Re: [PATCH v3 5/7] iotests: 194: test also migration of dirty bitmap

2020-05-21 Thread Eric Blake
On 5/15/20 7:40 AM, Vladimir Sementsov-Ogievskiy wrote: Test that dirty bitmap migration works when we deal with mirror. Signed-off-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/194 | 14 ++ tests/qemu-iotests/194.out | 6 ++ 2 files changed, 16

Re: [PATCH v3 6/7] migration/block-dirty-bitmap: add_bitmaps_to_list: check disk name once

2020-05-21 Thread Eric Blake
On 5/15/20 7:40 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Andrey Shinkevich --- migration/block-dirty-bitmap.c | 17 +++-- 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/migration/block-dirty-bitmap.c

Re: [PATCH v3 7/7] migration/block-dirty-bitmap: forbid migration by generated node-name

2020-05-21 Thread Eric Blake
On 5/15/20 7:40 AM, Vladimir Sementsov-Ogievskiy wrote: It actually never worked with libvirt, as auto-generated names are different on source and destination. It's unsafe and useless to migrate by auto-generated node-names, so let's forbid it. Signed-off-by: Vladimir Sementsov-Ogievskiy

Re: [PATCH v3 4/7] migration/block-dirty-bitmap: fix bitmaps pre-blockdev migration during mirror job

2020-05-21 Thread Eric Blake
On 5/19/20 5:51 AM, Vladimir Sementsov-Ogievskiy wrote: 18.05.2020 23:36, Eric Blake wrote: On 5/15/20 7:40 AM, Vladimir Sementsov-Ogievskiy wrote: Important thing for bitmap migration is to select destination block node to obtain the migrated bitmap. +    /* Skip filters without

Re: cluster_size got from backup_calculate_cluster_size()

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2020 21:19, John Snow wrote: On 5/21/20 5:56 AM, Derek Su wrote: Hi, The cluster_size got from backup_calculate_cluster_size(), MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size), is 64K regardless of the target image's cluster size. Not regardless -- but it is using 64K as a

[PATCH v6 5/5] iotests: Add test 291 to for qemu-img bitmap coverage

2020-05-21 Thread Eric Blake
Add a new test covering the 'qemu-img bitmap' subcommand, as well as 'qemu-img convert --bitmaps', both added in recent patches. Signed-off-by: Eric Blake Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/291 | 112

[PATCH v6 0/5] qemu-img: Add convert --bitmaps

2020-05-21 Thread Eric Blake
v5 was here: https://lists.gnu.org/archive/html/qemu-devel/2020-05/msg05958.html original cover letter here: https://lists.gnu.org/archive/html/qemu-devel/2020-04/msg03464.html Hopefully this version is close enough to do a pull request Monday. Since then: - patch 2: drop 'measure --bitmaps'

[PATCH v6 3/5] qemu-img: Factor out code for merging bitmaps

2020-05-21 Thread Eric Blake
The next patch will add another client that wants to merge dirty bitmaps; it will be easier to refactor the code to construct the QAPI struct correctly into a helper function. Signed-off-by: Eric Blake --- qemu-img.c | 34 +- 1 file changed, 21 insertions(+), 13

[PATCH v6 1/5] iotests: Fix test 178

2020-05-21 Thread Eric Blake
A recent change to qemu-img changed expected error message output, but 178 takes long enough to execute that it does not get run by 'make check' or './check -g quick'. Fixes: 43d589b074 Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy --- tests/qemu-iotests/178.out.qcow2 | 2

[PATCH v6 4/5] qemu-img: Add convert --bitmaps option

2020-05-21 Thread Eric Blake
Make it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically shorthand, as the same effect could be accomplished with a series of 'qemu-img bitmap --add' and

[PATCH v6 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Eric Blake
It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when measuring an existing image and output format that both support bitmaps. Update iotest 178

Re: [PATCH RFC 00/32] python/qemu: refactor as installable package

2020-05-21 Thread John Snow
Ping, please give this series a look-over. It looks big, but the changes themselves are actually fairly tiny. Here's a TOC: Patch 1 moves files and renames import statements. Patches 2-3 do some basic delinting. Patches 4-5 do more basic delinting and add flake8/pylintrc cfg Patches 6-10 add

Re: [PATCH v4 9/9] iotests: rename and move 169 and 199 tests

2020-05-21 Thread John Snow
On 5/19/20 7:41 AM, Kevin Wolf wrote: > Am 19.05.2020 um 13:32 hat Vladimir Sementsov-Ogievskiy geschrieben: >> 19.05.2020 12:07, Kevin Wolf wrote: >>> Am 18.05.2020 um 18:12 hat Thomas Huth geschrieben: On 15/05/2020 23.15, Vladimir Sementsov-Ogievskiy wrote: > Rename bitmaps

Re: cluster_size got from backup_calculate_cluster_size()

2020-05-21 Thread John Snow
On 5/21/20 5:56 AM, Derek Su wrote: > Hi, > > The cluster_size got from backup_calculate_cluster_size(), > MAX(BACKUP_CLUSTER_SIZE_DEFAULT, bdi.cluster_size), is 64K regardless > of the target image's cluster size. Not regardless -- but it is using 64K as a minimum. > > > For example: > >

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Nir Soffer
On Thu, May 21, 2020 at 6:09 PM Vladimir Sementsov-Ogievskiy wrote: > > 21.05.2020 16:29, Nir Soffer wrote: > > On Thu, May 21, 2020 at 2:40 PM Vladimir Sementsov-Ogievskiy > > wrote: > >> > >> 21.05.2020 02:00, Nir Soffer wrote: > >>> On Thu, May 21, 2020 at 1:01 AM Eric Blake wrote: > >

Re: [PATCH v2 1/4] Introduce yank feature

2020-05-21 Thread Daniel P . Berrangé
On Thu, May 21, 2020 at 05:42:41PM +0200, Lukas Straub wrote: > On Thu, 21 May 2020 16:03:35 +0100 > Stefan Hajnoczi wrote: > > > On Wed, May 20, 2020 at 11:05:39PM +0200, Lukas Straub wrote: > > > +void yank_generic_iochannel(void *opaque) > > > +{ > > > +QIOChannel *ioc =

Re: [PATCH v2 1/4] Introduce yank feature

2020-05-21 Thread Lukas Straub
On Thu, 21 May 2020 16:03:35 +0100 Stefan Hajnoczi wrote: > On Wed, May 20, 2020 at 11:05:39PM +0200, Lukas Straub wrote: > > +void yank_generic_iochannel(void *opaque) > > +{ > > +QIOChannel *ioc = QIO_CHANNEL(opaque); > > + > > +qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH,

Re: [PATCH v2 4/4] migration: Add yank feature

2020-05-21 Thread Lukas Straub
On Wed, 20 May 2020 23:05:50 +0200 Lukas Straub wrote: > Register yank functions on sockets to shut them down. > > Signed-off-by: Lukas Straub Don't review this commit for now, I'll have to revamp it anyway. Regards, Lukas Straub pgpitHe5HEV6g.pgp Description: OpenPGP digital signature

Re: [PATCH v5 4/5] qemu-img: Add convert --bitmaps option

2020-05-21 Thread Eric Blake
On 5/21/20 10:11 AM, Nir Soffer wrote: On Thu, May 21, 2020 at 1:01 AM Eric Blake wrote: Make it easier to copy all the persistent bitmaps of (the top layer of) a source image along with its guest-visible contents, by adding a boolean flag for use with qemu-img convert. This is basically

Re: [PATCH v5 4/5] qemu-img: Add convert --bitmaps option

2020-05-21 Thread Nir Soffer
On Thu, May 21, 2020 at 1:01 AM Eric Blake wrote: > > Make it easier to copy all the persistent bitmaps of (the top layer > of) a source image along with its guest-visible contents, by adding a > boolean flag for use with qemu-img convert. This is basically > shorthand, as the same effect could

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2020 16:29, Nir Soffer wrote: On Thu, May 21, 2020 at 2:40 PM Vladimir Sementsov-Ogievskiy wrote: 21.05.2020 02:00, Nir Soffer wrote: On Thu, May 21, 2020 at 1:01 AM Eric Blake wrote: It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such

Re: [PATCH v2 1/4] Introduce yank feature

2020-05-21 Thread Stefan Hajnoczi
On Wed, May 20, 2020 at 11:05:39PM +0200, Lukas Straub wrote: > +void yank_generic_iochannel(void *opaque) > +{ > +QIOChannel *ioc = QIO_CHANNEL(opaque); > + > +qio_channel_shutdown(ioc, QIO_CHANNEL_SHUTDOWN_BOTH, NULL); > +} > + > +void qmp_yank(strList *instances, Error **errp) > +{ > +

Re: [PATCH] io_uring: use io_uring_cq_ready() to check for ready cqes

2020-05-21 Thread Stefano Garzarella
On Thu, May 21, 2020 at 03:45:34PM +0100, Stefan Hajnoczi wrote: > On Tue, May 19, 2020 at 03:49:42PM +0200, Stefano Garzarella wrote: > > In qemu_luring_poll_cb() we are not using the cqe peeked from the > > CQ ring. We are using io_uring_peek_cqe() only to see if there > > are cqes ready, so we

Re: [PATCH] io_uring: use io_uring_cq_ready() to check for ready cqes

2020-05-21 Thread Stefan Hajnoczi
On Tue, May 19, 2020 at 03:49:42PM +0200, Stefano Garzarella wrote: > In qemu_luring_poll_cb() we are not using the cqe peeked from the > CQ ring. We are using io_uring_peek_cqe() only to see if there > are cqes ready, so we can replace it with io_uring_cq_ready(). > > Signed-off-by: Stefano

Re: [PATCH] io_uring: retry io_uring_submit() if it fails with errno=EINTR

2020-05-21 Thread Stefan Hajnoczi
On Tue, May 19, 2020 at 03:30:41PM +0200, Stefano Garzarella wrote: > As recently documented [1], io_uring_enter(2) syscall can return an > error (errno=EINTR) if the operation was interrupted by a delivery > of a signal before it could complete. > > This should happen when IORING_ENTER_GETEVENTS

Re: [PATCH 0/4] memory: Add memory_region_msync() & make NVMe emulated device generic

2020-05-21 Thread Philippe Mathieu-Daudé
On 5/21/20 2:46 PM, Paolo Bonzini wrote: On 21/05/20 14:32, Stefan Hajnoczi wrote: On Fri, May 08, 2020 at 08:24:52AM +0200, Philippe Mathieu-Daudé wrote: Let the NVMe emulated device be target-agnostic. It is not clear if dccvap_writefn() really needs memory_region_writeback() or could use

Re: [PATCH v3 0/3] various: Remove unnecessary casts

2020-05-21 Thread Paolo Bonzini
On 19/05/20 06:34, Markus Armbruster wrote: > Cédric Le Goater writes: > >> On 5/18/20 3:17 PM, Markus Armbruster wrote: >>> Paolo Bonzini writes: >>> On 15/05/20 07:58, Markus Armbruster wrote: > Philippe Mathieu-Daudé writes: > >> Remove unnecessary casts using coccinelle

Re: [PATCH 0/2] aio-posix: fix fdmon-io_uring memory leak

2020-05-21 Thread Stefan Hajnoczi
On Mon, May 11, 2020 at 07:36:28PM +0100, Stefan Hajnoczi wrote: > This bug was introduced in QEMU 5.0 and causes guests to slow down because > AioHandlers are not freed when the fdmon-io_uring file descriptor monitoring > implementation is used by the main loop thread's glib event loop. This

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Nir Soffer
On Thu, May 21, 2020 at 2:40 PM Vladimir Sementsov-Ogievskiy wrote: > > 21.05.2020 02:00, Nir Soffer wrote: > > On Thu, May 21, 2020 at 1:01 AM Eric Blake wrote: > >> > >> It's useful to know how much space can be occupied by qcow2 persistent > >> bitmaps, even though such metadata is unrelated

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Eric Blake
On 5/21/20 8:17 AM, Nir Soffer wrote: I'm ok with both options, can we have only one? That was the crux of the conversation after v3, where we were trying to figure out what interface you actually needed. I implemented both to show the difference, but if you want only one, then my preference

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Nir Soffer
On Thu, May 21, 2020 at 4:08 PM Eric Blake wrote: > > On 5/20/20 6:00 PM, Nir Soffer wrote: > > >> > >> On the command-line side, 'qemu-img measure' gains a new --bitmaps > >> flag. When present, the bitmap size is rolled into the two existing > >> measures (or errors if either the source image

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Eric Blake
On 5/20/20 6:00 PM, Nir Soffer wrote: On the command-line side, 'qemu-img measure' gains a new --bitmaps flag. When present, the bitmap size is rolled into the two existing measures (or errors if either the source image or destination format lacks bitmaps); when absent, there is never an

Re: [PATCH 0/4] memory: Add memory_region_msync() & make NVMe emulated device generic

2020-05-21 Thread Paolo Bonzini
On 21/05/20 14:32, Stefan Hajnoczi wrote: > On Fri, May 08, 2020 at 08:24:52AM +0200, Philippe Mathieu-Daudé wrote: >> Let the NVMe emulated device be target-agnostic. >> >> It is not clear if dccvap_writefn() really needs >> memory_region_writeback() or could use memory_region_msync(). >> >>

Re: [PATCH 0/4] memory: Add memory_region_msync() & make NVMe emulated device generic

2020-05-21 Thread Stefan Hajnoczi
On Fri, May 08, 2020 at 08:24:52AM +0200, Philippe Mathieu-Daudé wrote: > Let the NVMe emulated device be target-agnostic. > > It is not clear if dccvap_writefn() really needs > memory_region_writeback() or could use memory_region_msync(). > > Philippe Mathieu-Daudé (4): > memory: Rename

Re: [PATCH v5 2/5] qcow2: Expose bitmaps' size during measure

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2020 02:00, Nir Soffer wrote: On Thu, May 21, 2020 at 1:01 AM Eric Blake wrote: It's useful to know how much space can be occupied by qcow2 persistent bitmaps, even though such metadata is unrelated to the guest-visible data. Report this value as an additional QMP field, present when

Re: [PATCH v5 1/5] iotests: Fix test 178

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
21.05.2020 01:01, Eric Blake wrote: A recent change to qemu-img changed expected error message output, but 178 takes long enough to execute that it does not get run by 'make check' or './check -g quick'. Fixes: 43d589b074 Signed-off-by: Eric Blake Reviewed-by: Vladimir Sementsov-Ogievskiy

Re: [PATCH v3 5/5] iotests: add commit top->base cases to 274

2020-05-21 Thread Vladimir Sementsov-Ogievskiy
20.05.2020 19:39, Eric Blake wrote: On 5/20/20 3:35 AM, Vladimir Sementsov-Ogievskiy wrote: These cases are fixed by previous patches around block_status and is_allocated. Signed-off-by: Vladimir Sementsov-Ogievskiy ---   tests/qemu-iotests/274 | 20