Re: [Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

On Thu, 07/05 14:44, Kevin Wolf wrote: > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation > > here is that discard requests don't affect bs->wr_highest_offset. > > > > Signed-off-by: Fam Zheng > > --- > > block/io.c | 21

Re: [Qemu-block] [PATCH v2 1/9] block: Add copy offloading trace points

On Thu, 07/05 13:08, Kevin Wolf wrote: > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > A few trace points that can help reveal what is happening in a copy > > offloading I/O path. > > > > Signed-off-by: Fam Zheng > > --- > > block/file-posix.c | 2 ++ > > block/io.c | 2 ++ > > b

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] qapi: add transaction support for x-block-dirty-bitmap-merge

On 07/03/2018 06:53 AM, Vladimir Sementsov-Ogievskiy wrote: > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > qapi/transaction.json | 2 ++ > blockdev.c| 53 > ++- > 2 files changed, 54 insertions(+), 1 deletion(-) > > diff --g

Re: [Qemu-block] [PATCH V3] qemu-img: align result of is_allocated_sectors

> Am 05.07.2018 um 17:15 schrieb Kevin Wolf : > > Am 05.07.2018 um 12:52 hat Peter Lieven geschrieben: >> We currently don't enforce that the sparse segments we detect during convert >> are >> aligned. This leads to unnecessary and costly read-modify-write cycles either >> internally in Qemu or

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] drity-bitmap: refactor merge: separte can_merge

On 07/05/2018 02:55 PM, Eric Blake wrote: > On 07/05/2018 01:51 PM, John Snow wrote: > >> >>> -    assert(!bdrv_dirty_bitmap_readonly(dest)); >>> - >>> -    if (!hbitmap_merge(dest->bitmap, src->bitmap)) { >>> -    error_setg(errp, "Bitmaps are incompatible and can't be >>> merged"); >>> + 

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] drity-bitmap: refactor merge: separte can_merge

On 07/05/2018 01:51 PM, John Snow wrote: -assert(!bdrv_dirty_bitmap_readonly(dest)); - -if (!hbitmap_merge(dest->bitmap, src->bitmap)) { -error_setg(errp, "Bitmaps are incompatible and can't be merged"); +if (bdrv_can_merge_dirty_bitmap(dest, src, errp)) { +bool ret

Re: [Qemu-block] [Qemu-devel] [PATCH 1/2] drity-bitmap: refactor merge: separte can_merge

On 07/03/2018 06:53 AM, Vladimir Sementsov-Ogievskiy wrote: > Separate can_merge, to reuse it for transaction support for merge > command. > > Also, switch some asserts to errors. > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > include/block/dirty-bitmap.h | 3 +++ > include/qemu/hb

Re: [Qemu-block] [PATCH 2/2] nbd/server: send more than one extent of base:allocation context

05.07.2018 18:59, Eric Blake wrote: On 07/04/2018 06:23 AM, Vladimir Sementsov-Ogievskiy wrote: This is necessary for efficient block-status export, for clients which support it. qemu as client doesn't currently process additional information when querying block status.  Should it? Are ther

Re: [Qemu-block] [PATCH] block/crypto: Simplify block_crypto_co_create_opts_luks to avoid a memory leak

On 07/05/2018 07:20 AM, Kevin Wolf wrote: > Am 04.07.2018 um 17:02 hat Philippe Mathieu-Daudé geschrieben: >> After 1ec4f4160a1 Coverity reported: >> >> Variable cryptoopts going out of scope leaks the storage it points to. >> >> Fixes: Coverity CID 1393782 (Resource leak) >> Signed-off-by: Phili

Re: [Qemu-block] [PATCH] block/crypto: Simplify block_crypto_co_create_opts_luks to avoid a memory leak

Am 05.07.2018 um 18:04 hat Philippe Mathieu-Daudé geschrieben: > On 07/05/2018 07:20 AM, Kevin Wolf wrote: > > Am 04.07.2018 um 17:02 hat Philippe Mathieu-Daudé geschrieben: > >> After 1ec4f4160a1 Coverity reported: > >> > >> Variable cryptoopts going out of scope leaks the storage it points to.

Re: [Qemu-block] [Qemu-devel] [PULL v2 09/12] block/blklogwrites: Add an option for appending to an old log

On 07/05/2018 05:26 AM, Kevin Wolf wrote: From: Ari Sundholm Suggested by Kevin Wolf. May be useful when testing multiple batches of writes or doing long-term testing involving restarts of the VM. Signed-off-by: Ari Sundholm Signed-off-by: Kevin Wolf --- qapi/block-core.json | 3 +- blo

Re: [Qemu-block] [PATCH 2/2] nbd/server: send more than one extent of base:allocation context

On 07/04/2018 06:23 AM, Vladimir Sementsov-Ogievskiy wrote: This is necessary for efficient block-status export, for clients which support it. qemu as client doesn't currently process additional information when querying block status. Should it? Are there other clients which DO make use of

[Qemu-block] [PATCH v2 1/7] qobject: Catch another straggler for use of qdict_put_str()

Patch created mechanically by rerunning: $ spatch --sp-file scripts/coccinelle/qobject.cocci \ --macro-file scripts/cocci-macro-file.h \ --dir . --in-place Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Markus Armbruster Acked-by: Michael S. Tsirkin --- qobject/

Re: [Qemu-block] [PATCH v2 0/9] block: Fix dst reading after tail copy offloading

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > Qcow2 allocates new clusters after the end of the file. If it is the > destinaton > of copy offloading, we must adjust dst->bs->total_sectors. Otherwise, further > reads will drop to the "beyond EOF" code path and return zeroes, which problem > i

Re: [Qemu-block] [PULL v2 00/12] Block layer patches

On 5 July 2018 at 11:25, Kevin Wolf wrote: > The following changes since commit fe8d2d5737ab20ed0118863f5eb888cae37122ab: > > Merge remote-tracking branch > 'remotes/vivier2/tags/linux-user-for-3.0-pull-request' into staging > (2018-07-04 22:38:10 +0100) > > are available in the git repository

Re: [Qemu-block] [PATCH 1/2] nbd/server: fix nbd_co_send_block_status

On 07/04/2018 06:23 AM, Vladimir Sementsov-Ogievskiy wrote: Call nbd_co_send_extents() with correct length parameter (extent.length may be smaller than original length). Only matters for tracing purposes, but indeed worth an accurate trace. Also, switch length parameter type to uint32_t, to

Re: [Qemu-block] [PATCH V3] qemu-img: align result of is_allocated_sectors

Am 05.07.2018 um 12:52 hat Peter Lieven geschrieben: > We currently don't enforce that the sparse segments we detect during convert > are > aligned. This leads to unnecessary and costly read-modify-write cycles either > internally in Qemu or in the background on the storage device as nearly all >

[Qemu-block] [PATCH v4] qcow2: add overlap check for bitmap directory

Signed-off-by: Vladimir Sementsov-Ogievskiy --- If it appropriate for 3.0, let's push it. If not - then for 3.1 with fixed "since". Should I cc stable? v4: - fix indentation v3: - update Qcow2OverlapCheckFlags in qapi/block-core.json [Max] v2: - squash 02 (indentation fix) to 01 - drop com

Re: [Qemu-block] [PATCH v3] qcow2: add overlap check for bitmap directory

05.07.2018 13:46, Kevin Wolf wrote: Am 05.07.2018 um 11:21 hat Vladimir Sementsov-Ogievskiy geschrieben: If it appropriate for 3.0, let's push it. If not - then for 3.1 with fixed "since". Should I cc stable? v3: - update Qcow2OverlapCheckFlags in qapi/block-core.json [Max] v2: - squash 02 (in

Re: [Qemu-block] [PATCH 0/8] Coccinelle cleanups

On Wed, Jul 04, 2018 at 12:39:11PM -0300, Philippe Mathieu-Daudé wrote: > Nothing exciting here, patches created mechanically > (common after soft freeze). Acked-by: Michael S. Tsirkin > Philippe Mathieu-Daudé (8): > qobject: Catch another straggler for use of qdict_put_str() > error: Remove

Re: [Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation > here is that discard requests don't affect bs->wr_highest_offset. > > Signed-off-by: Fam Zheng > --- > block/io.c | 21 ++--- > 1 file changed, 14 insertion

Re: [Qemu-block] [PATCH v2 5/9] block: Fix handling of image enlarging write

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > Two problems exist when a write request that enlarges the image (i.e. > write beyond EOF) finishes: > > 1) parent is not notified about size change; > 2) dirty bitmap is not resized although we try to set the dirty bits; > > Fix them just like h

Re: [Qemu-block] [PATCH v2 4/9] block: Extract common write req handling

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > As a mechanical refactoring patch, this is the first step towards > unified and more correct write code paths. This is helpful because > multiple BlockDriverState fields need to be updated after modifying > image data, and it's hard to maintain in

Re: [Qemu-block] [PATCH v2 3/9] block: Use uint64_t for BdrvTrackedRequest byte fields

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > This matches the types used for bytes in the rest parts of block layer. > In the case of bdrv_co_truncate, new_bytes can be the image size which > probably doesn't fit in a 32 bit int. > > Signed-off-by: Fam Zheng mark_request_serialising() has

Re: [Qemu-block] [PATCH v2 1/9] block: Add copy offloading trace points

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > A few trace points that can help reveal what is happening in a copy > offloading I/O path. > > Signed-off-by: Fam Zheng > --- > block/file-posix.c | 2 ++ > block/io.c | 2 ++ > block/iscsi.c | 3 +++ > block/trace-events | 6 +

Re: [Qemu-block] [PATCH v2 2/9] block: Use BdrvChild to discard

Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > Other I/O functions are already using a BdrvChild pointer in the API, so > make discard do the same. It makes it possible to initiate the same > permission checks before doing I/O, and much easier to share the > helper functions for this, which wi

[Qemu-block] [PATCH V3] qemu-img: align result of is_allocated_sectors

We currently don't enforce that the sparse segments we detect during convert are aligned. This leads to unnecessary and costly read-modify-write cycles either internally in Qemu or in the background on the storage device as nearly all modern filesystems or hardware have a 4k alignment internally.

Re: [Qemu-block] [PATCH v3] qcow2: add overlap check for bitmap directory

Am 05.07.2018 um 11:21 hat Vladimir Sementsov-Ogievskiy geschrieben: > If it appropriate for 3.0, let's push it. If not - then for 3.1 > with fixed "since". Should I cc stable? > > v3: - update Qcow2OverlapCheckFlags in qapi/block-core.json [Max] > > v2: - squash 02 (indentation fix) to 01 >

[Qemu-block] [PULL v2 11/12] file-posix: Fix creation locking

From: Max Reitz raw_apply_lock_bytes() takes a bit mask of "permissions that are NOT shared". Also, make the "perm" and "shared" variables uint64_t, because I do not particularly like using ~ on signed integers (and other permission masks are usually uint64_t, too). Reported-by: Kevin Wolf Sig

[Qemu-block] [PULL v2 08/12] block/blklogwrites: Change log_sector_size from int64_t to uint64_t

From: Ari Sundholm This was a simple oversight when working on intermediate versions of the original patch which introduced blklogwrites. Signed-off-by: Ari Sundholm Signed-off-by: Kevin Wolf --- block/blklogwrites.c | 8 +++- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/

[Qemu-block] [PULL v2 06/12] block: Don't silently truncate node names

If the user passes a too long node name string, we silently truncate it to fit into BlockDriverState.node_name, i.e. to 31 characters. Apart from surprising the user when the node has a different name than requested, this also bypasses the check for duplicate names, so that the same name can be ass

[Qemu-block] [PULL v2 10/12] block/blklogwrites: Add an option for the update interval of the log superblock

From: Ari Sundholm This is a way to ensure that the log superblock is periodically updated. Before, this was only done on flush requests, which may not be enough if the VM exits abnormally, omitting the final flush. The default interval is 4096 write requests. Signed-off-by: Ari Sundholm Signe

[Qemu-block] [PULL v2 05/12] block: Add blklogwrites

From: Aapo Vienamo Implements a block device write logging system, similar to Linux kernel device mapper dm-log-writes. The write operations that are performed on a block device are logged to a file or another block device. The write log format is identical to the dm-log-writes format. Currently,

[Qemu-block] [PULL v2 09/12] block/blklogwrites: Add an option for appending to an old log

From: Ari Sundholm Suggested by Kevin Wolf. May be useful when testing multiple batches of writes or doing long-term testing involving restarts of the VM. Signed-off-by: Ari Sundholm Signed-off-by: Kevin Wolf --- qapi/block-core.json | 3 +- block/blklogwrites.c | 147 ++

[Qemu-block] [PULL v2 07/12] block/crypto: Fix memory leak in create error path

Fixes: Coverity CID 1393782 Signed-off-by: Kevin Wolf Reviewed-by: Daniel P. Berrangé --- block/crypto.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/block/crypto.c b/block/crypto.c index 994172a3de..146d81c90a 100644 --- a/block/crypto.c +++ b/block/crypto.c @@ -551,7 +55

[Qemu-block] [PULL v2 12/12] file-posix: Unlock FD after creation

From: Max Reitz Closing the FD does not necessarily mean that it is unlocked. Fix this by relinquishing all permission locks before qemu_close(). Reported-by: Kevin Wolf Signed-off-by: Max Reitz Signed-off-by: Kevin Wolf --- block/file-posix.c | 17 ++--- 1 file changed, 14 inse

[Qemu-block] [PULL v2 03/12] qcow2: add compress threads

From: Vladimir Sementsov-Ogievskiy Do data compression in separate threads. This significantly improve performance for qemu-img convert with -W (allow async writes) and -c (compressed) options. Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Kevin Wolf --- block/qcow2.h | 3 +++ b

[Qemu-block] [PULL v2 04/12] block: Move two block permission constants to the relevant enum

From: Ari Sundholm This allows using the two constants outside of block.c, which will happen in a subsequent patch. Signed-off-by: Ari Sundholm Signed-off-by: Kevin Wolf --- include/block/block.h | 7 +++ block.c | 6 -- 2 files changed, 7 insertions(+), 6 deletions(-)

[Qemu-block] [PULL v2 02/12] qcow2: refactor data compression

From: Vladimir Sementsov-Ogievskiy Make a separate function for compression to be parallelized later. - use .avail_out field instead of .next_out to calculate size of compressed data. It looks more natural and it allows to keep dest to be void pointer - set avail_out to be at least one by

[Qemu-block] [PULL v2 01/12] qemu-img: allow compressed not-in-order writes

From: Vladimir Sementsov-Ogievskiy No reason to forbid them, and they are needed to improve performance with compress-threads in further patches. Signed-off-by: Vladimir Sementsov-Ogievskiy Signed-off-by: Kevin Wolf --- qemu-img.c | 5 - 1 file changed, 5 deletions(-) diff --git a/qemu-i

[Qemu-block] [PULL v2 00/12] Block layer patches

The following changes since commit fe8d2d5737ab20ed0118863f5eb888cae37122ab: Merge remote-tracking branch 'remotes/vivier2/tags/linux-user-for-3.0-pull-request' into staging (2018-07-04 22:38:10 +0100) are available in the git repository at: git://repo.or.cz/qemu/kevin.git tags/for-upstrea

Re: [Qemu-block] [PATCH] block/crypto: Simplify block_crypto_co_create_opts_luks to avoid a memory leak

Am 04.07.2018 um 17:02 hat Philippe Mathieu-Daudé geschrieben: > After 1ec4f4160a1 Coverity reported: > > Variable cryptoopts going out of scope leaks the storage it points to. > > Fixes: Coverity CID 1393782 (Resource leak) > Signed-off-by: Philippe Mathieu-Daudé I already sent a much simple

[Qemu-block] [PATCH v3] qcow2: add overlap check for bitmap directory

If it appropriate for 3.0, let's push it. If not - then for 3.1 with fixed "since". Should I cc stable? v3: - update Qcow2OverlapCheckFlags in qapi/block-core.json [Max] v2: - squash 02 (indentation fix) to 01 - drop comment from qcow2_check_metadata_overlap() - set @ign to QCOW2_OL_BITMA

Re: [Qemu-block] [PATCH 0/2] file-posix: Fix creation locking

Am 04.07.2018 um 16:47 hat Max Reitz geschrieben: > This series fixes two bugs Kevin spotted in file-posix's creation > locking. Thanks, applied to the block branch. Kevin

Re: [Qemu-block] [PATCH v2 0/3] blklogwrites improvements

Am 04.07.2018 um 16:59 hat Ari Sundholm geschrieben: > This patch set contains a trivial printf format fix, followed by two > new features for the blklogwrites driver. > > The first feature is an option to append to an existing log, adopting > its sector size and other metadata in its superblock.

Re: [Qemu-block] [PATCH 1/2] block: add BDRV_REQ_SERIALISING flag

Am 04.07.2018 um 19:06 hat Vladimir Sementsov-Ogievskiy geschrieben: > 04.07.2018 19:36, Vladimir Sementsov-Ogievskiy wrote: > > 04.07.2018 19:20, Kevin Wolf wrote: > > > Am 04.07.2018 um 18:11 hat Vladimir Sementsov-Ogievskiy geschrieben: > > > > 04.07.2018 18:08, Kevin Wolf wrote: > > > > > Am 04

Re: [Qemu-block] [Qemu-devel] [PATCH 0/8] Coccinelle cleanups

Philippe Mathieu-Daudé writes: > Nothing exciting here, patches created mechanically > (common after soft freeze). Cc: qemu-trivial

Re: [Qemu-block] [PULL 0/5] Block layer patches

Am 04.07.2018 um 20:30 hat Peter Maydell geschrieben: > On 3 July 2018 at 15:59, Kevin Wolf wrote: > > The following changes since commit a395717cbd26e7593d3c3fe81faca121ec6d13e8: > > > > Merge remote-tracking branch 'remotes/cody/tags/block-pull-request' into > > staging (2018-07-03 11:49:51 +

[Qemu-block] [PATCH v3 4/4] block/backup: fix fleecing scheme: use serialized writes

Fleecing scheme works as follows: we want a kind of temporary snapshot of active drive A. We create temporary image B, with B->backing = A. Then we start backup(sync=none) from A to B. From this point, B reads as point-in-time snapshot of A (A continues to be active drive, accepting guest IO). Thi

[Qemu-block] [PATCH v3 3/4] block: add BDRV_REQ_SERIALISING flag

Serialized writes should be used in copy-on-write of backup(sync=none) for image fleecing scheme. We need to change an assert in bdrv_aligned_pwritev, added in 28de2dcd88de. The assert may fail now, because call to wait_serialising_requests here may become first call to it for this request with se

[Qemu-block] [PATCH v3 1/4] block: disallow BDRV_REQ_NO_SERIALISING for write

Before commit 9ded4a01149 "backup: Use copy offloading", BDRV_REQ_NO_SERIALISING was used for only one case: read in copy-on-write operation during backup. Also, the flag was handled only on read path (in bdrv_co_preadv and bdrv_aligned_preadv). After 9ded4a01149, flag is used for not waiting seri

[Qemu-block] [PATCH v3 2/4] block: split flags in copy_range

Pass read flags and write flags separately. This is needed to handle coming BDRV_REQ_NO_SERIALISING clearly in following patches. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block.h | 3 ++- include/block/block_int.h | 14 ++ include/sysemu/block-back

[Qemu-block] [PATCH v3 0/4] fix image fleecing

Hi all. This fixes image fleecing scheme for 3.0, details are in 04 patch. v3: 02: fix typo in assert, to fix build v2: 01,02: new patches 03: - improve comment - fix assert in bdrv_aligned_pwritev - add asserts to not use the flag on read requests - support copy_range 04: - expand

[Qemu-block] [PATCH v2 8/9] block: Fix bdrv_co_truncate overlap check

If we are growing the image and potentially using preallocation for the new area, we need to make sure that no write requests are made to the "preallocated" area which [@old_size, @offset), not [@offset, offset * 2 - @old_size). Signed-off-by: Fam Zheng --- block/io.c | 3 ++- 1 file changed, 2

[Qemu-block] [PATCH v2 9/9] block: Use common write req handling in truncate

Truncation is the last to convert from open coded req handling to reusing helpers. This time the permission check in prepare has to adapt to the new caller: it checks a different permission bit, and don't trigger the before write notifier. Also, truncation should always trigger a bs->total_sectors

[Qemu-block] [PATCH v2 5/9] block: Fix handling of image enlarging write

Two problems exist when a write request that enlarges the image (i.e. write beyond EOF) finishes: 1) parent is not notified about size change; 2) dirty bitmap is not resized although we try to set the dirty bits; Fix them just like how bdrv_co_truncate works. Reported-by: Kevin Wolf Signed-off-

[Qemu-block] [PATCH v2 3/9] block: Use uint64_t for BdrvTrackedRequest byte fields

This matches the types used for bytes in the rest parts of block layer. In the case of bdrv_co_truncate, new_bytes can be the image size which probably doesn't fit in a 32 bit int. Signed-off-by: Fam Zheng --- block/io.c| 2 +- include/block/block_int.h | 4 ++-- 2 files changed,

[Qemu-block] [PATCH v2 7/9] block: Use common req handling in copy offloading

This brings the request handling logic inline with write and discard, fixing write_gen, resize_cb, dirty bitmaps and image size refreshing. The last of these issues broke iotest case 222, which is now fixed. Signed-off-by: Fam Zheng --- block/io.c | 8 +++- 1 file changed, 7 insertions(+), 1

[Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation here is that discard requests don't affect bs->wr_highest_offset. Signed-off-by: Fam Zheng --- block/io.c | 21 ++--- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/block/io.c b/block/io.c index

[Qemu-block] [PATCH v2 1/9] block: Add copy offloading trace points

A few trace points that can help reveal what is happening in a copy offloading I/O path. Signed-off-by: Fam Zheng --- block/file-posix.c | 2 ++ block/io.c | 2 ++ block/iscsi.c | 3 +++ block/trace-events | 6 ++ 4 files changed, 13 insertions(+) diff --git a/block/file-posix.

[Qemu-block] [PATCH v2 2/9] block: Use BdrvChild to discard

Other I/O functions are already using a BdrvChild pointer in the API, so make discard do the same. It makes it possible to initiate the same permission checks before doing I/O, and much easier to share the helper functions for this, which will be added and used by write, truncate and copy range pat

[Qemu-block] [PATCH v2 4/9] block: Extract common write req handling

As a mechanical refactoring patch, this is the first step towards unified and more correct write code paths. This is helpful because multiple BlockDriverState fields need to be updated after modifying image data, and it's hard to maintain in multiple places such as copy offload, discard and truncat

[Qemu-block] [PATCH v2 0/9] block: Fix dst reading after tail copy offloading

Qcow2 allocates new clusters after the end of the file. If it is the destinaton of copy offloading, we must adjust dst->bs->total_sectors. Otherwise, further reads will drop to the "beyond EOF" code path and return zeroes, which problem is caught by iotests 222. Follow the logic in the normal writ