[Qemu-block] [PATCH] bitmap: Update count after a merge

2018-09-26 Thread Eric Blake
We need an accurate count of the number of bits set in a bitmap after a merge. In particular, since the merge operation short-circuits a merge from an empty source, if you have bitmaps A, B, and C where B started empty, then merge C into B, and B into A, an inaccurate count meant that A did not

[Qemu-block] [PULL 1/3] nbd/server: fix bitmap export

2018-09-26 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy bitmap_to_extents function is broken: it switches dirty variable after every iteration, however it can process only part of dirty (or zero) area during one iteration in case when this area is too large for one extent. Fortunately, the bug doesn't produce wrong

[Qemu-block] [PULL 3/3] nbd/server: send more than one extent of base:allocation context

2018-09-26 Thread Eric Blake
From: Vladimir Sementsov-Ogievskiy This is necessary for efficient block-status export, for clients which support it. (qemu is not yet such a client, but could become one.) Signed-off-by: Vladimir Sementsov-Ogievskiy Message-Id: <20180704112302.471456-3-vsement...@virtuozzo.com> [eblake:

Re: [Qemu-block] [Qemu-devel] [PATCH] dirty-bitmaps: allow merging to disabled bitmaps

2018-09-26 Thread Eric Blake
On 9/19/18 4:16 PM, John Snow wrote: On 09/19/2018 05:08 PM, Eric Blake wrote: On 9/19/18 2:58 PM, John Snow wrote: We wish to prohibit merging to read-only bitmaps and frozen bitmaps, but "disabled" bitmaps only preclude their recording of live, new information. It does not prohibit them

Re: [Qemu-block] [PATCH v3 0/5] dirty-bitmaps: fix QMP command permissions

2018-09-26 Thread Eric Blake
On 9/25/18 6:49 PM, John Snow wrote: based on: jsnow/bitmaps staging branch This series builds on a previous standalone patch and adjusts the permission for all (or most) of the QMP bitmap commands. John Snow (5): block/dirty-bitmaps: add user_modifiable status checker

Re: [Qemu-block] [PATCH v3 1/5] block/dirty-bitmaps: add user_modifiable status checker

2018-09-26 Thread Eric Blake
On 9/26/18 7:19 AM, Vladimir Sementsov-Ogievskiy wrote: 26.09.2018 14:53, Vladimir Sementsov-Ogievskiy wrote: 26.09.2018 02:49, John Snow wrote: Instead of both frozen and qmp_locked checks, wrap it into one check. frozen implies the bitmap is split in two (for backup), and shouldn't be

Re: [Qemu-block] [PATCH v3 1/5] block/dirty-bitmaps: add user_modifiable status checker

2018-09-26 Thread Eric Blake
On 9/26/18 6:53 AM, Vladimir Sementsov-Ogievskiy wrote: 26.09.2018 02:49, John Snow wrote: Instead of both frozen and qmp_locked checks, wrap it into one check. frozen implies the bitmap is split in two (for backup), and shouldn't be modified. qmp_locked implies it's being used by another

Re: [Qemu-block] [PATCH 2/2] nbd/server: send more than one extent of base:allocation context

2018-09-26 Thread Eric Blake
On 9/17/18 11:15 AM, Vladimir Sementsov-Ogievskiy wrote: 05.07.2018 19:18, Vladimir Sementsov-Ogievskiy wrote: 05.07.2018 18:59, Eric Blake wrote: On 07/04/2018 06:23 AM, Vladimir Sementsov-Ogievskiy wrote: This is necessary for efficient block-status export, for clients which support it.

[Qemu-block] [PATCH] util/hbitmaps: recalculate count on merge

2018-09-26 Thread John Snow
We have been neglecting to do so, which results in wrong counts after merge. In the worst case, we may think the bitmap is empty when it has had new writes merged into it. Reported-by: Eric Blake Signed-off-by: John Snow --- util/hbitmap.c | 3 +++ 1 file changed, 3 insertions(+) diff --git

Re: [Qemu-block] [Qemu-devel] [PATCH v3 0/5] dirty-bitmaps: fix QMP command permissions

2018-09-26 Thread John Snow
On 09/26/2018 08:19 AM, Vladimir Sementsov-Ogievskiy wrote: > 26.09.2018 02:49, John Snow wrote: >> based on: jsnow/bitmaps staging branch >> >> This series builds on a previous standalone patch and adjusts >> the permission for all (or most) of the QMP bitmap commands. >> >> John Snow (5): >>  

Re: [Qemu-block] [Qemu-devel] [PATCH v3 2/5] block/dirty-bitmaps: fix merge permissions

2018-09-26 Thread John Snow
On 09/26/2018 08:08 AM, Vladimir Sementsov-Ogievskiy wrote: > 26.09.2018 14:55, Vladimir Sementsov-Ogievskiy wrote: >> 26.09.2018 02:49, John Snow wrote: >>> We wish to prohibit merging to read-only bitmaps and frozen bitmaps, >>> but "disabled" bitmaps only preclude their recording of live,

Re: [Qemu-block] [Qemu-devel] [PATCH v3 2/5] block/dirty-bitmaps: fix merge permissions

2018-09-26 Thread John Snow
On 09/26/2018 08:08 AM, Vladimir Sementsov-Ogievskiy wrote: > 26.09.2018 14:55, Vladimir Sementsov-Ogievskiy wrote: >> 26.09.2018 02:49, John Snow wrote: >>> We wish to prohibit merging to read-only bitmaps and frozen bitmaps, >>> but "disabled" bitmaps only preclude their recording of live,

[Qemu-block] [PATCH v2 1/2] MAINTAINERS: Replace myself with John Snow for block jobs

2018-09-26 Thread Jeff Cody
I'll not be involved with day-to-day qemu development, and John Snow is a block jobs wizard. Have him take over block job maintainership duties. Signed-off-by: Jeff Cody --- MAINTAINERS | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/MAINTAINERS b/MAINTAINERS index

[Qemu-block] [PATCH v2 0/2] Maintainership changes

2018-09-26 Thread Jeff Cody
Changes from v1: - cc'ed Peter (meant to for v1, forgot) - added John's 'jobs' branch to his git url - kept myself as maintainer for vhdx - moved sheepdog to 'Odd Fixes' I'm not going to be involved with day-to-day qemu development, so this necessitates some changes. Jeff Cody

[Qemu-block] [PATCH v2 2/2] MAINTAINERS: Remove myself as block maintainer

2018-09-26 Thread Jeff Cody
I'll not be involved in day-to-day qemu development. Remove myself as maintainer from the remainder of the network block drivers, and revert them to the general block layer maintainership. Move 'sheepdog' to the 'Odd Fixes' support level. For VHDX, added my personal email address as a

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] MAINTAINERS: Remove myself as block maintainer

2018-09-26 Thread Jeff Cody
On Tue, Sep 25, 2018 at 03:49:36PM +0800, Fam Zheng wrote: > On Tue, 09/25 09:37, Markus Armbruster wrote: > > Do we want to have a dedicated VHDX driver submaintainer again? Fam, > > you're maintaining VMDK, could you cover VHDX as well? > > I don't know a lot VHDX internals. Considering my

[Qemu-block] [PATCH v12 8/9] qcow2: Set the default cache-clean-interval to 10 minutes

2018-09-26 Thread Leonid Bloch
The default cache-clean-interval is set to 10 minutes, in order to lower the overhead of the qcow2 caches (before the default was 0, i.e. disabled). * For non-Linux platforms the default is kept at 0, because cache-clean-interval is not supported there yet. Signed-off-by: Leonid Bloch

[Qemu-block] [PATCH v12 4/9] qcow2: Avoid duplication in setting the refcount cache size

2018-09-26 Thread Leonid Bloch
The refcount cache size does not need to be set to its minimum value in read_cache_sizes(), as it is set to at least its minimum value in qcow2_update_options_prepare(). Signed-off-by: Leonid Bloch Reviewed-by: Alberto Garcia Reviewed-by: Kevin Wolf --- block/qcow2.c | 5 ++--- 1 file

[Qemu-block] [PATCH v12 7/9] qcow2: Resize the cache upon image resizing

2018-09-26 Thread Leonid Bloch
The caches are now recalculated upon image resizing. This is done because the new default behavior of assigning L2 cache relatively to the image size, implies that the cache will be adapted accordingly after an image resize. Signed-off-by: Leonid Bloch Reviewed-by: Alberto Garcia ---

[Qemu-block] [PATCH v12 6/9] qcow2: Increase the default upper limit on the L2 cache size

2018-09-26 Thread Leonid Bloch
The upper limit on the L2 cache size is increased from 1 MB to 32 MB on Linux platforms, and to 8 MB on other platforms (this difference is caused by the ability to set intervals for cache cleaning on Linux platforms only). This is done in order to allow default full coverage with the L2 cache

[Qemu-block] [PATCH v12 9/9] qcow2: Explicit number replaced by a constant

2018-09-26 Thread Leonid Bloch
Signed-off-by: Leonid Bloch Reviewed-by: Alberto Garcia Reviewed-by: Kevin Wolf --- block/qcow2.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index 95e1c98daa..7277feda13 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1324,7 +1324,7

[Qemu-block] [PATCH v12 5/9] qcow2: Assign the L2 cache relatively to the image size

2018-09-26 Thread Leonid Bloch
Sufficient L2 cache can noticeably improve the performance when using large images with frequent I/O. Previously, unless 'cache-size' was specified and was large enough, the L2 cache was set to a certain size without taking the virtual image size into account. Now, the L2 cache assignment is

[Qemu-block] [PATCH v12 3/9] qcow2: Make sizes more humanly readable

2018-09-26 Thread Leonid Bloch
Signed-off-by: Leonid Bloch Reviewed-by: Alberto Garcia Reviewed-by: Kevin Wolf --- block/qcow2.c | 2 +- block/qcow2.h | 9 + 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index c13153735a..d2c07ce9fe 100644 --- a/block/qcow2.c +++

[Qemu-block] [PATCH v12 1/9] qcow2: Options' documentation fixes

2018-09-26 Thread Leonid Bloch
Signed-off-by: Leonid Bloch Reviewed-by: Alberto Garcia Reviewed-by: Kevin Wolf --- docs/qcow2-cache.txt | 21 ++--- qemu-options.hx | 9 ++--- 2 files changed, 20 insertions(+), 10 deletions(-) diff --git a/docs/qcow2-cache.txt b/docs/qcow2-cache.txt index

[Qemu-block] [PATCH v12 2/9] include: Add a lookup table of sizes

2018-09-26 Thread Leonid Bloch
Adding a lookup table for the powers of two, with the appropriate size prefixes. This is needed when a size has to be stringified, in which case something like '(1 * KiB)' would become a literal '(1 * (1L << 10))' string. Powers of two are used very often for sizes, so such a table will also make

[Qemu-block] [PATCH v12 0/9] Take the image size into account when allocating the L2 cache

2018-09-26 Thread Leonid Bloch
This series makes the qcow2 L2 cache assignment aware of the image size, with the intention for it to cover the entire image. The importance of this change is in noticeable performance improvement, especially with heavy random I/O. The memory overhead is not big in most cases, as only 1 MB of

Re: [Qemu-block] [PATCH v11 7/9] qcow2: Resize the cache upon image resizing

2018-09-26 Thread Alberto Garcia
On Wed 26 Sep 2018 04:21:21 PM CEST, Leonid Bloch wrote: > Oops, sorry. Would something like that be OK: > > On 9/26/18 1:55 PM, Alberto Garcia wrote: >> On Wed 26 Sep 2018 12:43:03 PM CEST, Kevin Wolf wrote: +/* Update cache sizes */ +options = qdict_clone_shallow(bs->options);

Re: [Qemu-block] [PATCH v11 7/9] qcow2: Resize the cache upon image resizing

2018-09-26 Thread Leonid Bloch
Oops, sorry. Would something like that be OK: On 9/26/18 1:55 PM, Alberto Garcia wrote: > On Wed 26 Sep 2018 12:43:03 PM CEST, Kevin Wolf wrote: >>> +/* Update cache sizes */ >>> +options = qdict_clone_shallow(bs->options); >>> +ret = qcow2_update_options(bs, options, s->flags, errp);

Re: [Qemu-block] [PATCH v3 0/5] dirty-bitmaps: fix QMP command permissions

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 02:49, John Snow wrote: based on: jsnow/bitmaps staging branch This series builds on a previous standalone patch and adjusts the permission for all (or most) of the QMP bitmap commands. John Snow (5): block/dirty-bitmaps: add user_modifiable status checker block/dirty-bitmaps:

Re: [Qemu-block] [PATCH v3 1/5] block/dirty-bitmaps: add user_modifiable status checker

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 14:53, Vladimir Sementsov-Ogievskiy wrote: 26.09.2018 02:49, John Snow wrote: Instead of both frozen and qmp_locked checks, wrap it into one check. frozen implies the bitmap is split in two (for backup), and shouldn't be modified. qmp_locked implies it's being used by another

Re: [Qemu-block] [PATCH v3 2/5] block/dirty-bitmaps: fix merge permissions

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 14:55, Vladimir Sementsov-Ogievskiy wrote: 26.09.2018 02:49, John Snow wrote: We wish to prohibit merging to read-only bitmaps and frozen bitmaps, but "disabled" bitmaps only preclude their recording of live, new information. It does not prohibit them from manual writes at the behest

Re: [Qemu-block] [PATCH v3 5/5] block/backup: prohibit backup from using in-use bitmaps

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 02:49, John Snow wrote: If the bitmap is locked, we shouldn't touch it. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 12 ++-- 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/blockdev.c b/blockdev.c index

Re: [Qemu-block] [PATCH v3 4/5] block/dirty-bitmaps: prohibit enable/disable on locked/frozen bitmaps

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 02:49, John Snow wrote: We're not being consistent about this. If it's in use by an operation, the user should not be able to change the behavior of that bitmap. Signed-off-by: John Snow Reviewed-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 26 --

Re: [Qemu-block] [PATCH v3 3/5] block/dirty-bitmaps: allow clear on disabled bitmaps

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 02:49, John Snow wrote: Similarly to merge, it's OK to allow clear operations on disabled bitmaps, as this condition only means that they are not recording new writes. We are free to clear it if the user requests it. Signed-off-by: John Snow Reviewed-by: Vladimir

Re: [Qemu-block] [PATCH v3 2/5] block/dirty-bitmaps: fix merge permissions

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 02:49, John Snow wrote: We wish to prohibit merging to read-only bitmaps and frozen bitmaps, but "disabled" bitmaps only preclude their recording of live, new information. It does not prohibit them from manual writes at the behest of the user, as is the case for merge operations.

Re: [Qemu-block] [PATCH v3 1/5] block/dirty-bitmaps: add user_modifiable status checker

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
26.09.2018 02:49, John Snow wrote: Instead of both frozen and qmp_locked checks, wrap it into one check. frozen implies the bitmap is split in two (for backup), and shouldn't be modified. qmp_locked implies it's being used by another operation, like being exported over NBD. In both cases it

Re: [Qemu-block] [PATCH v3 10/10] block: Allow changing 'force-share' on reopen

2018-09-26 Thread Alberto Garcia
On Wed 26 Sep 2018 01:34:28 PM CEST, Kevin Wolf wrote: >> @@ -3353,6 +3370,7 @@ void bdrv_reopen_commit(BDRVReopenState *reopen_state) >> bs->open_flags = reopen_state->flags; >> bs->read_only = !(reopen_state->flags & BDRV_O_RDWR); >> bs->detect_zeroes =

Re: [Qemu-block] [PATCH v3 10/10] block: Allow changing 'force-share' on reopen

2018-09-26 Thread Kevin Wolf
Am 06.09.2018 um 11:37 hat Alberto Garcia geschrieben: > 'force-share' is one of the basic BlockdevOptions available for all > drivers, but it's not handled by bdrv_reopen_prepare() so any attempt > to change it results in a "Cannot change the option" error: > >(qemu) qemu-io virtio0 "reopen

Re: [Qemu-block] [PATCH] qapi: drop x- from x-block-latency-histogram-set

2018-09-26 Thread Vladimir Sementsov-Ogievskiy
20.06.2018 19:07, John Snow wrote: On 06/20/2018 11:28 AM, Vladimir Sementsov-Ogievskiy wrote: 20.06.2018 18:27, Vladimir Sementsov-Ogievskiy wrote: Libvirt part is ready, let's drop x- prefix. libvirt patches will be sent soon I hope. OK, can you ping this patch with a link to the series

Re: [Qemu-block] [PATCH v11 0/9] Take the image size into account when allocating the L2 cache

2018-09-26 Thread Alberto Garcia
On Wed 26 Sep 2018 12:48:18 PM CEST, Kevin Wolf wrote: > By the way, Berto: While reviewing this, I noticed that we don't make > use of the l2-cache-entry-size feature yet by default. I think you > said you wanted to do some more tests to find a good default > (intuitively, I'd expect it around

Re: [Qemu-block] [PATCH v11 7/9] qcow2: Resize the cache upon image resizing

2018-09-26 Thread Alberto Garcia
On Wed 26 Sep 2018 12:43:03 PM CEST, Kevin Wolf wrote: >> +/* Update cache sizes */ >> +options = qdict_clone_shallow(bs->options); >> +ret = qcow2_update_options(bs, options, s->flags, errp); >> +if (ret < 0) { >> +goto fail; >> +} > > Isn't options leaked, both in

Re: [Qemu-block] [PATCH v11 0/9] Take the image size into account when allocating the L2 cache

2018-09-26 Thread Kevin Wolf
Am 25.09.2018 um 00:53 hat Leonid Bloch geschrieben: > This series makes the qcow2 L2 cache assignment aware of the image size, > with the intention for it to cover the entire image. The importance of > this change is in noticeable performance improvement, especially with > heavy random I/O. The

Re: [Qemu-block] [PATCH v11 7/9] qcow2: Resize the cache upon image resizing

2018-09-26 Thread Kevin Wolf
Am 25.09.2018 um 00:53 hat Leonid Bloch geschrieben: > The caches are now recalculated upon image resizing. This is done > because the new default behavior of assigning L2 cache relatively to > the image size, implies that the cache will be adapted accordingly > after an image resize. > >

Re: [Qemu-block] [PATCH] file-posix: Include filename in locking error message

2018-09-26 Thread Kevin Wolf
Am 25.09.2018 um 07:05 hat Fam Zheng geschrieben: > Image locking errors happening at device initialization time doesn't say > which file cannot be locked, for instance, > > -device scsi-disk,drive=drive-1: Failed to get shared "write" lock > Is another process using the image? > > could