[PATCH 1/2] qcow2: Handle QCowL2Meta on error in preallocate_co()

2020-09-08 Thread Alberto Garcia
off-by: Alberto Garcia --- block/qcow2.c | 40 +--- 1 file changed, 17 insertions(+), 23 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c index da56b1a4df..eeb125c697 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -2102,7 +2102,6 @@ static coroutine_fn

Re: [PATCH v2 3/3] qcow2: Rewrite the documentation of qcow2_alloc_cluster_offset()

2020-09-04 Thread Alberto Garcia
On Fri 04 Sep 2020 11:28:18 AM CEST, Max Reitz wrote: >> + * If any clusters or subclusters were allocated then @m contains a >> + * list with the information of all the affected regions. Note that >> + * this can happen regardless of whether this function succeeds or >> + * not. The caller is

[PATCH v2 1/3] qcow2: Fix removal of list members from BDRVQcow2State.cluster_allocs

2020-09-03 Thread Alberto Garcia
be reproduced with the included test case. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 3 -- tests/qemu-iotests/305 | 74 ++ tests/qemu-iotests/305.out | 16 + tests/qemu-iotests/group | 1 + 4 files changed, 91 insertions(+), 3

[PATCH v2 2/3] qcow2: Don't check nb_clusters when removing l2meta from the list

2020-09-03 Thread Alberto Garcia
the case, l2meta (nowadays a pointer to a list) is only allocated when needed and nb_clusters is guaranteed to be > 0 so this check is unnecessary. Signed-off-by: Alberto Garcia --- block/qcow2.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/block/qcow2.c b/block/qcow2.c in

[PATCH v2 3/3] qcow2: Rewrite the documentation of qcow2_alloc_cluster_offset()

2020-09-03 Thread Alberto Garcia
The current text corresponds to an earlier, simpler version of this function and it does not explain how it works now. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 24 ++-- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/block/qcow2-cluster.c b

[PATCH v2 0/3] qcow2: Fix removal of list members from BDRVQcow2State.cluster_allocs

2020-09-03 Thread Alberto Garcia
]. Berto Alberto Garcia (3): qcow2: Fix removal of list members from BDRVQcow2State.cluster_allocs qcow2: Don't check nb_clusters when removing l2meta from the list qcow2: Rewrite the documentation of qcow2_alloc_cluster_offset() block/qcow2-cluster.c | 27 +++--- block/qcow2.c

Re: [PATCH] qcow2: Fix removal of list members from BDRVQcow2State.cluster_allocs

2020-09-03 Thread Alberto Garcia
On Thu 03 Sep 2020 03:12:22 PM CEST, Max Reitz wrote: > On my search for /(\.|->)nb_clusters/ to verify this, I noticed this > comment for qcow2_alloc_cluster_offset(): > > * If the cluster was already allocated, m->nb_clusters is set to 0 and > * other fields in m are meaningless. > > Should

[PATCH] qcow2: Fix removal of list members from BDRVQcow2State.cluster_allocs

2020-09-02 Thread Alberto Garcia
ged in commit f50f88b9fe and nowadays nb_clusters is guaranteed to be > 0. Signed-off-by: Alberto Garcia --- block/qcow2-cluster.c | 3 -- block/qcow2.c | 4 +- tests/qemu-iotests/305 | 75 ++ tests/qemu-iotests/305.out | 16 test

Re: [PATCH 2/2] block: file-posix: Replace posix_fallocate with fallocate

2020-09-01 Thread Alberto Garcia
/* If fallocate() is not supported, fallback to full preallocation. > */ With the missing braces in this if statement, Reviewed-by: Alberto Garcia Berto

Re: [PATCH] qcow2: cleanup created file when qcow2_co_create

2020-09-01 Thread Alberto Garcia
On Thu 16 Jul 2020 01:33:59 PM CEST, Maxim Levitsky wrote: > if (ret < 0) { > + > +Error *local_delete_err = NULL; > +int r_del = bdrv_co_delete_file(bs, _delete_err); > +/* > + * ENOTSUP will happen if the block driver doesn't support > + * the

Re: [PATCH 1/2] block: file-posix: Extract preallocate helpers

2020-09-01 Thread Alberto Garcia
On Mon 31 Aug 2020 04:01:26 PM CEST, Nir Soffer wrote: > +static int preallocate_falloc(int fd, int64_t current_length, int64_t offset, > + Error **errp) > +{ > +#ifdef CONFIG_POSIX_FALLOCATE > +int result; > + > +if (offset == current_length) > +return

Re: [PATCH 1/2] block: file-posix: Extract preallocate helpers

2020-09-01 Thread Alberto Garcia
prone > with the current code. > > Extract a helper for each of the preallocation modes (falloc, full, off) > and leave only the common preparation and cleanup code in > handle_aiocb_truncate(). > > Signed-off-by: Nir Soffer Reviewed-by: Alberto Garcia Berto

[PATCH] qcow2: Use macros for the L1, refcount and bitmap table entry sizes

2020-08-28 Thread Alberto Garcia
This patch replaces instances of sizeof(uint64_t) in the qcow2 driver with macros that indicate what those sizes are actually referring to. Signed-off-by: Alberto Garcia --- block/qcow2.h | 6 +++ block/qcow2-bitmap.c | 11 -- block/qcow2-cluster.c | 24 ++-- block

[PATCH v2] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-27 Thread Alberto Garcia
fine so there is no reason why we cannot use that information. After testing 4KB writes on an image that only contains zero clusters this patch results in almost five times more IOPS. Signed-off-by: Alberto Garcia --- v2: - Add new, simpler API: bdrv_is_unallocated_or_zero_above() include/block

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-26 Thread Alberto Garcia
On Tue 25 Aug 2020 09:47:24 PM CEST, Brian Foster wrote: > My fio fallocates the entire file by default with this command. Is that > the intent of this particular test? I added --fallocate=none to my test > runs to incorporate the allocation cost in the I/Os. That wasn't intentional, you're

Re: qcow2 overlay performance

2020-08-26 Thread Alberto Garcia
On Wed 26 Aug 2020 03:18:32 PM CEST, Kevin Wolf wrote: >> My understanding is that writing 4K blocks requires a >> read-modify-write because you must fetch a complete cluster from >> deeper in the overlay chain before writing to the active >> overlay. However, this does not explain the drop in

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-25 Thread Alberto Garcia
On Tue 25 Aug 2020 06:54:15 PM CEST, Brian Foster wrote: > If I compare this 5m fio test between XFS and ext4 on a couple of my > systems (with either no prealloc or full file prealloc), I end up seeing > ext4 run slightly faster on my vm and XFS slightly faster on bare metal. > Either way, I

Re: [PULL 00/34] Block patches

2020-08-25 Thread Alberto Garcia
On Tue 25 Aug 2020 04:01:14 PM CEST, Peter Maydell wrote: > On Tue, 25 Aug 2020 at 09:33, Max Reitz wrote: >> >> The following changes since commit 30aa19446d82358a30eac3b556b4d6641e00b7c1: >> >> Merge remote-tracking branch 'remotes/cschoenebeck/tags/pull-9p-20200812' >> into staging

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-25 Thread Alberto Garcia
On Fri 21 Aug 2020 07:02:32 PM CEST, Brian Foster wrote: >> I was running fio with --ramp_time=5 which ignores the first 5 seconds >> of data in order to let performance settle, but if I remove that I can >> see the effect more clearly. I can observe it with raw files (in 'off' >> and 'prealloc'

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-24 Thread Alberto Garcia
On Sun 23 Aug 2020 11:59:07 PM CEST, Dave Chinner wrote: >> >> Option 4 is described above as initial file preallocation whereas >> >> option 1 is per 64k cluster prealloc. Prealloc mode mixup aside, Berto >> >> is reporting that the initial file preallocation mode is slower than >> >> the per

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-21 Thread Alberto Garcia
On Thu 20 Aug 2020 11:58:11 PM CEST, Dave Chinner wrote: >> The virtual drive (/dev/vdb) is a freshly created qcow2 file stored on >> the host (on an xfs or ext4 filesystem as the table above shows), and >> it is attached to QEMU using a virtio-blk-pci device: >> >>-drive

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-21 Thread Alberto Garcia
On Fri 21 Aug 2020 02:59:44 PM CEST, Brian Foster wrote: >> > Option 4 is described above as initial file preallocation whereas >> > option 1 is per 64k cluster prealloc. Prealloc mode mixup aside, Berto >> > is reporting that the initial file preallocation mode is slower than >> > the per cluster

Re: qcow2 merge_cow() question

2020-08-21 Thread Alberto Garcia
On Fri 21 Aug 2020 02:32:00 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > Hi! > > I'm sorry, I feel I already asked about it, but can't now find the answer. > > What are these ifs for? > >/* The data (middle) region must be immediately after the > * start region */ >

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-21 Thread Alberto Garcia
On Fri 21 Aug 2020 01:42:52 PM CEST, Alberto Garcia wrote: > On Fri 21 Aug 2020 01:05:06 PM CEST, Brian Foster wrote: >>> > 1) off: for every write request QEMU initializes the cluster (64KB) >>> > with fallocate(ZERO_RANGE) and then writes the 4KB of da

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-21 Thread Alberto Garcia
On Fri 21 Aug 2020 01:05:06 PM CEST, Brian Foster wrote: >> > 1) off: for every write request QEMU initializes the cluster (64KB) >> > with fallocate(ZERO_RANGE) and then writes the 4KB of data. >> > >> > 2) off w/o ZERO_RANGE: QEMU writes the 4KB of data and fills the rest >> >

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-20 Thread Alberto Garcia
Cc: linux-xfs On Wed 19 Aug 2020 07:53:00 PM CEST, Brian Foster wrote: > In any event, if you're seeing unclear or unexpected performance > deltas between certain XFS configurations or other fs', I think the > best thing to do is post a more complete description of the workload, >

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-19 Thread Alberto Garcia
On Wed 19 Aug 2020 05:37:12 PM CEST, Alberto Garcia wrote: > I ran the test again on a newly created filesystem just to make sure, > here are the full results (numbers are IOPS): > > |--+---+---| > | preallocation

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-19 Thread Alberto Garcia
On Wed 19 Aug 2020 05:07:11 PM CEST, Kevin Wolf wrote: >> I checked with xfs on my computer. I'm not very familiar with that >> filesystem so I was using the default options and I didn't tune >> anything. >> >> What I got with my tests (using fio): >> >> - Using extent_size_hint didn't make any

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-19 Thread Alberto Garcia
On Mon 17 Aug 2020 05:53:07 PM CEST, Kevin Wolf wrote: >> > Or are you saying that ZERO_RANGE + pwrite on a sparse file (= >> > cluster allocation) is faster for you than just the pwrite alone (= >> > writing to already allocated cluster)? >> >> Yes, 20% faster in my tests (4KB random writes),

Re: [PATCH v5 1/5] block/io: fix bdrv_co_block_status_above

2020-08-19 Thread Alberto Garcia
On Wed 19 Aug 2020 11:48:25 AM CEST, Vladimir Sementsov-Ogievskiy wrote: >>> + * The top layer deferred to this layer, and because this >>> layer is >>> + * short, any zeroes that we synthesize beyond EOF behave as >>> if they >>> + * were allocated at this

Re: [PATCH v2] qemu-img: Explicit number replaced by a constant

2020-08-19 Thread Alberto Garcia
On Wed 19 Aug 2020 03:36:07 AM CEST, Yi Li wrote: > Signed-off-by: Yi Li Reviewed-by: Alberto Garcia Berto

Re: [PATCH v5 5/5] iotests: add commit top->base cases to 274

2020-08-18 Thread Alberto Garcia
On Wed 10 Jun 2020 02:04:26 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > These cases are fixed by previous patches around block_status and > is_allocated. Reviewed-by: Alberto Garcia Berto

Re: [PATCH v5 4/5] block/io: fix bdrv_is_allocated_above

2020-08-18 Thread Alberto Garcia
rlays has > unallocated area at that place). > > Reusing bdrv_common_block_status_above fixes the issue and unifies code > path. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > Reviewed-by: Eric Blake Reviewed-by: Alberto Garcia Berto

Re: [PATCH v5 3/5] block/io: bdrv_common_block_status_above: support bs == base

2020-08-18 Thread Alberto Garcia
gt; So, support this corner case. Reviewed-by: Alberto Garcia Berto

Re: [PATCH v5 2/5] block/io: bdrv_common_block_status_above: support include_base

2020-08-18 Thread Alberto Garcia
On Wed 10 Jun 2020 02:04:23 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > In order to reuse bdrv_common_block_status_above in > bdrv_is_allocated_above, let's support include_base parameter. > > Signed-off-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Alberto Garcia Berto

Re: [PATCH v5 1/5] block/io: fix bdrv_co_block_status_above

2020-08-18 Thread Alberto Garcia
On Wed 10 Jun 2020 02:04:22 PM CEST, Vladimir Sementsov-Ogievskiy wrote: > + * The top layer deferred to this layer, and because this layer > is > + * short, any zeroes that we synthesize beyond EOF behave as if > they > + * were allocated at this layer >

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-17 Thread Alberto Garcia
On Mon 17 Aug 2020 05:53:07 PM CEST, Kevin Wolf wrote: > Maybe the difference is in allocating 64k at once instead of doing a > separate allocation for every 4k block? But with the extent size hint > patches to file-posix, we should allocate 1 MB at once by default now > (if your test image was

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-17 Thread Alberto Garcia
On Mon 17 Aug 2020 12:10:19 PM CEST, Kevin Wolf wrote: >> Since commit c8bb23cbdbe / QEMU 4.1.0 (and if the storage backend >> allows it) writing to an image created with preallocation=metadata >> can be slower (20% in my tests) than writing to an image with no >> preallocation at all. > > A while

Re: [PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-17 Thread Alberto Garcia
On Mon 17 Aug 2020 05:53:07 PM CEST, Kevin Wolf wrote: >> > Or are you saying that ZERO_RANGE + pwrite on a sparse file (= >> > cluster allocation) is faster for you than just the pwrite alone (= >> > writing to already allocated cluster)? >> >> Yes, 20% faster in my tests (4KB random writes),

[PATCH 0/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-14 Thread Alberto Garcia
ed, i.e. no backing file). This would make reading from them (and writing to them, after this patch) faster. Berto Alberto Garcia (1): qcow2: Skip copy-on-write when allocating a zero cluster include/block/block.h | 2 +- block/commit.c| 2 +- block/io.c

[PATCH 1/1] qcow2: Skip copy-on-write when allocating a zero cluster

2020-08-14 Thread Alberto Garcia
fine so there is no reason why we cannot use that information. After testing 4KB writes on an image that only contains zero clusters this patch results in almost five times more IOPS. Signed-off-by: Alberto Garcia --- include/block/block.h | 2 +- block/commit.c| 2 +- block/io.c

Re: [PATCH] qcow2-cluster: Fix integer left shift error in qcow2_alloc_cluster_link_l2()

2020-08-05 Thread Alberto Garcia
On Wed 05 Aug 2020 04:16:57 PM CEST, Kevin Wolf wrote: >> nb_clusters is an int and there are more cases of >> >> nb_clusters << s->cluster_bits >> >> I can see at least these: handle_alloc(), qcow2_free_any_clusters(), >> qcow2_alloc_cluster_abort(). > > Actuallyx, handle_alloc() and

Re: [PATCH] qcow2-cluster: Fix integer left shift error in qcow2_alloc_cluster_link_l2()

2020-08-05 Thread Alberto Garcia
On Wed 05 Aug 2020 03:44:08 PM CEST, Alberto Garcia wrote: > On Wed 05 Aug 2020 11:22:58 AM CEST, Tuguoyi wrote: >> This patch fix it by casting @i to uint64_t before doing left shift >> operation > > The patch seems fine and I also think that it's perhaps worth a test >

Re: [PATCH] qcow2-cluster: Fix integer left shift error in qcow2_alloc_cluster_link_l2()

2020-08-05 Thread Alberto Garcia
On Wed 05 Aug 2020 11:22:58 AM CEST, Tuguoyi wrote: > This patch fix it by casting @i to uint64_t before doing left shift > operation The patch seems fine and I also think that it's perhaps worth a test case (although it only seems to happen with preallocation=falloc or full so the test would

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-15 Thread Alberto Garcia
On Tue, Jul 14, 2020 at 02:56:57PM +0200, Max Reitz wrote: > Thanks, fixed up patch 21 (by initializing expected_type to just > anything, I chose QCOW2_SUBCLUSTER_NORMAL, which I preferred over > just using 0), resolved some iotest conflicts (I’m sure, more are > to come before the release, but

Re: [PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-10 Thread Alberto Garcia
On Fri 10 Jul 2020 06:43:59 PM CEST, no-re...@patchew.org wrote: > /tmp/qemu-test/src/block/qcow2-cluster.c:473:19: error: 'expected_type' may > be used uninitialized in this function [-Werror=maybe-uninitialized] > } else if (type != expected_type) { >^ >

[PATCH v11 31/34] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

2020-07-10 Thread Alberto Garcia
Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- qapi/block

[PATCH v11 21/34] qcow2: Add subcluster support to qcow2_get_host_offset()

2020-07-10 Thread Alberto Garcia
as a separate function anymore so it's inlined into its caller. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.h | 38 --- block/qcow2-cluster.c | 150 ++ 2 files changed, 92 insertions(+), 96 deletions(-) diff --git

[PATCH v11 34/34] iotests: Add tests for qcow2 images with extended L2 entries

2020-07-10 Thread Alberto Garcia
Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- tests/qemu-iotests/271 | 901 + tests/qemu-iotests/271.out | 726 ++ tests/qemu-iotests/group | 1 + 3 files changed, 1628 insertions(+) create mode 100755 tests

[PATCH v11 28/34] qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()

2020-07-10 Thread Alberto Garcia
. 2) if the tail area was compressed we are writing zeroes to the head and the body areas, which are already zeroized. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.h | 4 +-- block/qcow2-cluster.c | 81 +++ block

[PATCH v11 26/34] qcow2: Clear the L2 bitmap when allocating a compressed cluster

2020-07-10 Thread Alberto Garcia
Compressed clusters always have the bitmap part of the extended L2 entry set to 0. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 5bdbb65e7b..6b5d0698d0

[PATCH v11 30/34] qcow2: Add prealloc field to QCowL2Meta

2020-07-10 Thread Alberto Garcia
then we have to hide any possible stale data and zero out the new clusters (see commit 955c7d6687 for more details). In this case the subclusters cannot be left as unallocated so the L2 bitmap must be updated. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block

[PATCH v11 33/34] qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters

2020-07-10 Thread Alberto Garcia
This function is only used by qcow2_expand_zero_clusters() to downgrade a qcow2 image to a previous version. This would require transforming all extended L2 entries into normal L2 entries but this is not a simple task and there are no plans to implement this at the moment. Signed-off-by: Alberto

[PATCH v11 17/34] qcow2: Add cluster type parameter to qcow2_get_host_offset()

2020-07-10 Thread Alberto Garcia
and in order to detect errors more easily. This patch makes qcow2_get_host_offset() return 0 on success and puts the returned cluster type in a separate parameter. There are no semantic changes. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block

[PATCH v11 16/34] qcow2: Add qcow2_cluster_is_allocated()

2020-07-10 Thread Alberto Garcia
This helper function tells us if a cluster is allocated (that is, there is an associated host offset for it). Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h

[PATCH v11 29/34] qcow2: Add subcluster support to qcow2_measure()

2020-07-10 Thread Alberto Garcia
Extended L2 entries are bigger than normal L2 entries so this has an impact on the amount of metadata needed for a qcow2 file. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.c | 20 +--- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/block

[PATCH v11 27/34] qcow2: Add subcluster support to handle_alloc_space()

2020-07-10 Thread Alberto Garcia
with zeroes the other subclusters if we can guarantee that we are not overwriting existing data. However this would waste more disk space, so we should first evaluate if it's really worth doing. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block

[PATCH v11 23/34] qcow2: Add subcluster support to discard_in_l2_slice()

2020-07-10 Thread Alberto Garcia
that the image will read back as zeroes. If this is important for the caller it should forbid it as qcow2_co_pdiscard() does (see 80f5c01183 for more details). Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 52

[PATCH v11 13/34] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

2020-07-10 Thread Alberto Garcia
the get/set_l2_bitmap() functions that are used to access the bitmaps. For convenience we allow calling get_l2_bitmap() on images without subclusters. In this case the returned value is always 0 and has no meaning. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz

[PATCH v11 25/34] qcow2: Update L2 bitmap in qcow2_alloc_cluster_link_l2()

2020-07-10 Thread Alberto Garcia
affected subclusters were already allocated). This is detected in calculate_l2_meta(), and qcow2_alloc_cluster_link_l2() is never called in those cases. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 18 ++ 1 file changed, 18

[PATCH v11 19/34] qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC

2020-07-10 Thread Alberto Garcia
When dealing with subcluster types there is a new value called QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in QCow2ClusterType. This patch handles that value in all places where subcluster types are processed. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed

[PATCH v11 04/34] qcow2: Split cluster_needs_cow() out of count_cow_clusters()

2020-07-10 Thread Alberto Garcia
We are going to need it in other places. Signed-off-by: Alberto Garcia Reviewed-by: Vladimir Sementsov-Ogievskiy Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 34 +++--- 1 file changed, 19 insertions(+), 15 deletions(-) diff --git a/block/qcow2-cluster.c b

[PATCH v11 12/34] qcow2: Add l2_entry_size()

2020-07-10 Thread Alberto Garcia
. This function returns the proper value for the image. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.h | 9 + block/qcow2-cluster.c | 12 ++-- block/qcow2-refcount.c | 14 -- block/qcow2.c | 8 4 files changed, 27 insertions

[PATCH v11 24/34] qcow2: Add subcluster support to check_refcounts_l2()

2020-07-10 Thread Alberto Garcia
: if none of the subclusters are allocated then we can clear the offset field and leave the bitmap untouched. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2-refcount.c | 16 +++- tests/qemu-iotests/060.out | 2 +- 2 files changed, 12 insertions(+), 6 deletions

[PATCH v11 22/34] qcow2: Add subcluster support to zero_in_l2_slice()

2020-07-10 Thread Alberto Garcia
not need to be updated. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 36 +++- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 492296a90f

[PATCH v11 32/34] qcow2: Allow preallocation and backing files if extended_l2 is set

2020-07-10 Thread Alberto Garcia
it is perfectly possible to have an allocated cluster with all its subclusters unallocated. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.c | 7 --- tests/qemu-iotests/206.out | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git

[PATCH v11 09/34] qcow2: Add subcluster-related fields to BDRVQcow2State

2020-07-10 Thread Alberto Garcia
reated as if they had exactly one subcluster per cluster (i.e. subcluster_size = cluster_size). Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 5 + block/qcow2.c | 5 + 2 files changed, 10 insertions(+) diff --git a/block/q

[PATCH v11 06/34] qcow2: Add get_l2_entry() and set_l2_entry()

2020-07-10 Thread Alberto Garcia
to get_l2_entry() and set_l2_entry(). Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 12 block/qcow2-cluster.c | 63 ++ block/qcow2-refcount.c | 17

[PATCH v11 05/34] qcow2: Process QCOW2_CLUSTER_ZERO_ALLOC clusters in handle_copied()

2020-07-10 Thread Alberto Garcia
for subclusters. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 256 +++--- 1 file changed, 141 insertions(+), 115 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 543f515c81

[PATCH v11 07/34] qcow2: Document the Extended L2 Entries feature

2020-07-10 Thread Alberto Garcia
-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Eric Blake --- docs/interop/qcow2.txt | 68 -- docs/qcow2-cache.txt | 19 +++- 2 files changed, 83 insertions(+), 4 deletions(-) diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2

[PATCH v11 18/34] qcow2: Replace QCOW2_CLUSTER_* with QCOW2_SUBCLUSTER_*

2020-07-10 Thread Alberto Garcia
of QCow2ClusterType with their QCow2SubclusterType equivalents. This patch only changes the data types, there are no semantic changes. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 2 +- block/qcow2-cluster.c | 10 +++ block

[PATCH v11 11/34] qcow2: Add offset_into_subcluster() and size_to_subclusters()

2020-07-10 Thread Alberto Garcia
Like offset_into_cluster() and size_to_clusters(), but for subclusters. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 10 ++ 1 file changed, 10 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 2503374677..4fe31adfd3

[PATCH v11 00/34] Add subcluster allocation to qcow2

2020-07-10 Thread Alberto Garcia
and backing files if extended_l2 is set' 033/34:[] [--] 'qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters' 034/34:[0006] [FC] 'iotests: Add tests for qcow2 images with extended L2 entries' Alberto Garcia (34): qcow2: Make Qcow2AioTask store the full host offset qcow2

[PATCH v11 10/34] qcow2: Add offset_to_sc_index()

2020-07-10 Thread Alberto Garcia
For a given offset, return the subcluster number within its cluster (i.e. with 32 subclusters per cluster it returns a number between 0 and 31). Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 5 + 1 file changed, 5

[PATCH v11 14/34] qcow2: Add QCow2SubclusterType and qcow2_get_subcluster_type()

2020-07-10 Thread Alberto Garcia
patch. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 126 +- 1 file changed, 125 insertions(+), 1 deletion(-) diff --git a/block/qcow2.h b/block/qcow2.h index 82b86f6cec..3aec6f452a 100644

[PATCH v11 20/34] qcow2: Add subcluster support to calculate_l2_meta()

2020-07-10 Thread Alberto Garcia
() and l2meta_cow_end() are not necessarily cluster-aligned anymore. We need to update the calculation of old_start and old_end in handle_dependencies() to guarantee that no two requests try to write on the same cluster. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz

[PATCH v11 03/34] qcow2: Add calculate_l2_meta()

2020-07-10 Thread Alberto Garcia
handle_alloc() creates a QCowL2Meta structure in order to update the image metadata and perform the necessary copy-on-write operations. This patch moves that code to a separate function so it can be used from other places. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2

[PATCH v11 15/34] qcow2: Add qcow2_get_subcluster_range_type()

2020-07-10 Thread Alberto Garcia
of a subcluster we can check the rest efficiently by counting the number of adjacent ones (or zeroes) in the bitmap. This is what this function does. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 51

[PATCH v11 01/34] qcow2: Make Qcow2AioTask store the full host offset

2020-07-10 Thread Alberto Garcia
with this patch but it is documented now. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.c | 69 ++ block/trace-events | 2 +- 2 files changed, 34 insertions

[PATCH v11 08/34] qcow2: Add dummy has_subclusters() function

2020-07-10 Thread Alberto Garcia
value. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index eecbadc4cb..2064dd3d85 100644 --- a/block/qcow2.h +++ b

[PATCH v11 02/34] qcow2: Convert qcow2_get_cluster_offset() into qcow2_get_host_offset()

2020-07-10 Thread Alberto Garcia
-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.h | 4 ++-- block/qcow2-cluster.c | 41 +++-- block/qcow2.c | 24 +++- 3 files changed, 32 insertions(+), 37 deletions(-) diff --git a/block/qcow2.h b/block/qcow2

Re: [PATCH v10 31/34] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

2020-07-09 Thread Alberto Garcia
On Thu 09 Jul 2020 03:07:29 PM CEST, Max Reitz wrote: > On 03.07.20 17:58, Alberto Garcia wrote: >> Now that the implementation of subclusters is complete we can finally >> add the necessary options to create and read images with this feature, >> which we call "extended

[PATCH v10 29/34] qcow2: Add subcluster support to qcow2_measure()

2020-07-03 Thread Alberto Garcia
Extended L2 entries are bigger than normal L2 entries so this has an impact on the amount of metadata needed for a qcow2 file. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz --- block/qcow2.c | 20 +--- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/block

[PATCH v10 32/34] qcow2: Allow preallocation and backing files if extended_l2 is set

2020-07-03 Thread Alberto Garcia
it is perfectly possible to have an allocated cluster with all its subclusters unallocated. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.c | 7 --- tests/qemu-iotests/206.out | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git

[PATCH v10 23/34] qcow2: Add subcluster support to discard_in_l2_slice()

2020-07-03 Thread Alberto Garcia
that the image will read back as zeroes. If this is important for the caller it should forbid it as qcow2_co_pdiscard() does (see 80f5c01183 for more details). Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 52

[PATCH v10 20/34] qcow2: Add subcluster support to calculate_l2_meta()

2020-07-03 Thread Alberto Garcia
() and l2meta_cow_end() are not necessarily cluster-aligned anymore. We need to update the calculation of old_start and old_end in handle_dependencies() to guarantee that no two requests try to write on the same cluster. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz

[PATCH v10 33/34] qcow2: Assert that expand_zero_clusters_in_l1() does not support subclusters

2020-07-03 Thread Alberto Garcia
This function is only used by qcow2_expand_zero_clusters() to downgrade a qcow2 image to a previous version. This would require transforming all extended L2 entries into normal L2 entries but this is not a simple task and there are no plans to implement this at the moment. Signed-off-by: Alberto

[PATCH v10 17/34] qcow2: Add cluster type parameter to qcow2_get_host_offset()

2020-07-03 Thread Alberto Garcia
and in order to detect errors more easily. This patch makes qcow2_get_host_offset() return 0 on success and puts the returned cluster type in a separate parameter. There are no semantic changes. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block

[PATCH v10 13/34] qcow2: Update get/set_l2_entry() and add get/set_l2_bitmap()

2020-07-03 Thread Alberto Garcia
the get/set_l2_bitmap() functions that are used to access the bitmaps. For convenience we allow calling get_l2_bitmap() on images without subclusters. In this case the returned value is always 0 and has no meaning. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz

[PATCH v10 34/34] iotests: Add tests for qcow2 images with extended L2 entries

2020-07-03 Thread Alberto Garcia
Signed-off-by: Alberto Garcia --- tests/qemu-iotests/271 | 901 + tests/qemu-iotests/271.out | 724 + tests/qemu-iotests/group | 1 + 3 files changed, 1626 insertions(+) create mode 100755 tests/qemu-iotests/271 create

[PATCH v10 31/34] qcow2: Add the 'extended_l2' option and the QCOW2_INCOMPAT_EXTL2 bit

2020-07-03 Thread Alberto Garcia
Now that the implementation of subclusters is complete we can finally add the necessary options to create and read images with this feature, which we call "extended L2 entries". Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- qapi/block

[PATCH v10 28/34] qcow2: Add subcluster support to qcow2_co_pwrite_zeroes()

2020-07-03 Thread Alberto Garcia
. 2) if the tail area was compressed we are writing zeroes to the head and the body areas, which are already zeroized. Signed-off-by: Alberto Garcia --- block/qcow2.h | 4 +-- block/qcow2-cluster.c | 81 +++ block/qcow2.c | 33

[PATCH v10 21/34] qcow2: Add subcluster support to qcow2_get_host_offset()

2020-07-03 Thread Alberto Garcia
as a separate function anymore so it's inlined into its caller. Signed-off-by: Alberto Garcia --- block/qcow2.h | 38 --- block/qcow2-cluster.c | 150 ++ 2 files changed, 92 insertions(+), 96 deletions(-) diff --git a/block/qcow2.h b/block

[PATCH v10 30/34] qcow2: Add prealloc field to QCowL2Meta

2020-07-03 Thread Alberto Garcia
then we have to hide any possible stale data and zero out the new clusters (see commit 955c7d6687 for more details). In this case the subclusters cannot be left as unallocated so the L2 bitmap must be updated. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block

[PATCH v10 19/34] qcow2: Handle QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC

2020-07-03 Thread Alberto Garcia
When dealing with subcluster types there is a new value called QCOW2_SUBCLUSTER_UNALLOCATED_ALLOC that has no equivalent in QCow2ClusterType. This patch handles that value in all places where subcluster types are processed. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed

[PATCH v10 16/34] qcow2: Add qcow2_cluster_is_allocated()

2020-07-03 Thread Alberto Garcia
This helper function tells us if a cluster is allocated (that is, there is an associated host offset for it). Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 6 ++ 1 file changed, 6 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h

[PATCH v10 11/34] qcow2: Add offset_into_subcluster() and size_to_subclusters()

2020-07-03 Thread Alberto Garcia
Like offset_into_cluster() and size_to_clusters(), but for subclusters. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2.h | 10 ++ 1 file changed, 10 insertions(+) diff --git a/block/qcow2.h b/block/qcow2.h index 2503374677..4fe31adfd3

[PATCH v10 15/34] qcow2: Add qcow2_get_subcluster_range_type()

2020-07-03 Thread Alberto Garcia
of a subcluster we can check the rest efficiently by counting the number of adjacent ones (or zeroes) in the bitmap. This is what this function does. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 51

[PATCH v10 27/34] qcow2: Add subcluster support to handle_alloc_space()

2020-07-03 Thread Alberto Garcia
with zeroes the other subclusters if we can guarantee that we are not overwriting existing data. However this would waste more disk space, so we should first evaluate if it's really worth doing. Signed-off-by: Alberto Garcia Reviewed-by: Max Reitz Reviewed-by: Vladimir Sementsov-Ogievskiy --- block

[PATCH v10 22/34] qcow2: Add subcluster support to zero_in_l2_slice()

2020-07-03 Thread Alberto Garcia
not need to be updated. Signed-off-by: Alberto Garcia Reviewed-by: Eric Blake Reviewed-by: Max Reitz --- block/qcow2-cluster.c | 36 +++- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/block/qcow2-cluster.c b/block/qcow2-cluster.c index 492296a90f

<    1   2   3   4   5   6   7   8   9   10   >