Re: [Qemu-block] [PATCH v2 9/9] block: Use common write req handling in truncate

2018-07-06 Thread Eric Blake
On 07/05/2018 02:37 AM, Fam Zheng wrote: Truncation is the last to convert from open coded req handling to reusing helpers. This time the permission check in prepare has to adapt to the new caller: it checks a different permission bit, and don't did you mean "won't" or "doesn't"? trigger the

Re: [Qemu-block] [PATCH v2 8/9] block: Fix bdrv_co_truncate overlap check

2018-07-06 Thread Eric Blake
On 07/05/2018 02:37 AM, Fam Zheng wrote: If we are growing the image and potentially using preallocation for the new area, we need to make sure that no write requests are made to the "preallocated" area which [@old_size, @offset), not [@offset, offset * 2 s/which/which is/ - @old_size).

Re: [Qemu-block] [Qemu-devel] [PATCH v4 0/4] fix image fleecing

2018-07-06 Thread Eric Blake
On 07/06/2018 01:30 PM, Vladimir Sementsov-Ogievskiy wrote: Hi all. This fixes image fleecing scheme for 3.0, details are in 04 patch. 01 is a significant fix too. v4: 01: fix copy_range architecture here too 02: rebase on 01 03: rebase Hmm - I commented on v3 before seeing you'd already

Re: [Qemu-block] [Qemu-devel] [PATCH v3 3/4] block: add BDRV_REQ_SERIALISING flag

2018-07-06 Thread Eric Blake
On 07/05/2018 02:46 AM, Vladimir Sementsov-Ogievskiy wrote: Serialized writes should be used in copy-on-write of backup(sync=none) for image fleecing scheme. We need to change an assert in bdrv_aligned_pwritev, added in 28de2dcd88de. The assert may fail now, because call to

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/4] block: disallow BDRV_REQ_NO_SERIALISING for write

2018-07-06 Thread Eric Blake
On 07/06/2018 04:32 PM, Eric Blake wrote: On 07/05/2018 02:46 AM, Vladimir Sementsov-Ogievskiy wrote: Before commit 9ded4a01149 "backup: Use copy offloading", BDRV_REQ_NO_SERIALISING was used for only one case: read in copy-on-write operation during backup. Also, the flag was handled only on

Re: [Qemu-block] [Qemu-devel] [PATCH v3 1/4] block: disallow BDRV_REQ_NO_SERIALISING for write

2018-07-06 Thread Eric Blake
On 07/05/2018 02:46 AM, Vladimir Sementsov-Ogievskiy wrote: Before commit 9ded4a01149 "backup: Use copy offloading", BDRV_REQ_NO_SERIALISING was used for only one case: read in copy-on-write operation during backup. Also, the flag was handled only on read path (in bdrv_co_preadv and

Re: [Qemu-block] [Qemu-devel] [PATCH] block: Fix copy-on-read crash with partial final cluster

2018-07-06 Thread Eric Blake
On 07/06/2018 11:45 AM, Kevin Wolf wrote: If the virtual disk size isn't aligned to full clusters, bdrv_co_do_copy_on_readv() may get pnum == 0 before having the full cluster completed, which will let it run into an assertion failure: qemu-io: block/io.c:1203: bdrv_co_do_copy_on_readv:

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] qapi: add transaction support for x-block-dirty-bitmap-merge

2018-07-06 Thread Eric Blake
On 07/06/2018 10:38 AM, John Snow wrote: I think the "discard stashed state, making undo impossible" interpretation is good because .commit() is not allowed to fail. That function should only do things that never fail. I think this is probably the correct way to proceed, and we ought to

Re: [Qemu-block] [PATCH] qapi/block-core.json: Add missing documentation for blklogwrites log-append option

2018-07-06 Thread Eric Blake
On 07/06/2018 07:21 AM, Ari Sundholm wrote: This was accidentally omitted. Thanks to Eric Blake for spotting this. Signed-off-by: Ari Sundholm --- qapi/block-core.json | 2 ++ 1 file changed, 2 insertions(+) Reviewed-by: Eric Blake diff --git a/qapi/block-core.json

[Qemu-block] [PATCH v4 2/4] block: split flags in copy_range

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Pass read flags and write flags separately. This is needed to handle coming BDRV_REQ_NO_SERIALISING clearly in following patches. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block.h | 3 ++- include/block/block_int.h | 14 ++

[Qemu-block] [PATCH v4 3/4] block: add BDRV_REQ_SERIALISING flag

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Serialized writes should be used in copy-on-write of backup(sync=none) for image fleecing scheme. We need to change an assert in bdrv_aligned_pwritev, added in 28de2dcd88de. The assert may fail now, because call to wait_serialising_requests here may become first call to it for this request with

[Qemu-block] [PATCH v4 1/4] block/io: fix copy_range

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Here two things are fixed: 1. Architecture On each recursion step, we go to the child of src or dst, only for one of them. So, it's wrong to create tracked requests for both on each step. It leads to tracked requests duplication. 2. Wait for serializing requests on write path independently of

[Qemu-block] [PATCH v4 4/4] block/backup: fix fleecing scheme: use serialized writes

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Fleecing scheme works as follows: we want a kind of temporary snapshot of active drive A. We create temporary image B, with B->backing = A. Then we start backup(sync=none) from A to B. From this point, B reads as point-in-time snapshot of A (A continues to be active drive, accepting guest IO).

[Qemu-block] [PATCH v4 0/4] fix image fleecing

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Hi all. This fixes image fleecing scheme for 3.0, details are in 04 patch. 01 is a significant fix too. v4: 01: fix copy_range architecture here too 02: rebase on 01 03: rebase v3: 02: fix typo in assert, to fix build v2: 01,02: new patches 03: - improve comment - fix assert in

[Qemu-block] [PATCH] block: Fix copy-on-read crash with partial final cluster

2018-07-06 Thread Kevin Wolf
If the virtual disk size isn't aligned to full clusters, bdrv_co_do_copy_on_readv() may get pnum == 0 before having the full cluster completed, which will let it run into an assertion failure: qemu-io: block/io.c:1203: bdrv_co_do_copy_on_readv: Assertion `skip_bytes < pnum' failed. Check for

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] qapi: add transaction support for x-block-dirty-bitmap-merge

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
06.07.2018 18:38, John Snow wrote: On 07/06/2018 06:12 AM, Vladimir Sementsov-Ogievskiy wrote: Ok, let's go this way for now, I'll rewrite it. Last requirement looks a bit strange for me in transaction context. We should not assume that action is done before commit. What is main idea of

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] qapi: add transaction support for x-block-dirty-bitmap-merge

2018-07-06 Thread John Snow
On 07/06/2018 06:12 AM, Vladimir Sementsov-Ogievskiy wrote: > > Ok, let's go this way for now, I'll rewrite it. > > Last requirement looks a bit strange for me in transaction context. We > should not assume that action is done before commit. > What is main idea of transaction action, do

Re: [Qemu-block] [Qemu-devel] [libvirt] [PULL 25/26] block: Remove deprecated -drive option serial

2018-07-06 Thread Peter Maydell
On 6 July 2018 at 15:56, Kevin Wolf wrote: > Am 06.07.2018 um 13:11 hat Cornelia Huck geschrieben: >> That way, we can still easily remove old cruft (case (a)), but still >> accommodate cases like this (case (c)). The obvious drawback is that >> we'd need someone to curate the deprecation

Re: [Qemu-block] [Qemu-devel] [libvirt] [PULL 25/26] block: Remove deprecated -drive option serial

2018-07-06 Thread Daniel P . Berrangé
On Fri, Jul 06, 2018 at 04:56:46PM +0200, Kevin Wolf wrote: > Am 06.07.2018 um 13:11 hat Cornelia Huck geschrieben: > > On Wed, 4 Jul 2018 17:14:02 +0100 > > Peter Maydell wrote: > > > > > On 4 July 2018 at 14:34, Kevin Wolf wrote: > > > > Essentially, what is important to me isn't getting

Re: [Qemu-block] [Qemu-devel] [libvirt] [PULL 25/26] block: Remove deprecated -drive option serial

2018-07-06 Thread Kevin Wolf
Am 06.07.2018 um 13:11 hat Cornelia Huck geschrieben: > On Wed, 4 Jul 2018 17:14:02 +0100 > Peter Maydell wrote: > > > On 4 July 2018 at 14:34, Kevin Wolf wrote: > > > Essentially, what is important to me isn't getting these options dropped > > > exactly in 3.0, but not setting a bad precedence

Re: [Qemu-block] [PATCH v3 0/4] fix image fleecing

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
06.07.2018 16:41, Vladimir Sementsov-Ogievskiy wrote: 06.07.2018 10:17, Fam Zheng wrote: On Thu, 07/05 10:46, Vladimir Sementsov-Ogievskiy wrote: Hi all. This fixes image fleecing scheme for 3.0, details are in 04 patch. Looks like this breaks 'test-replication':

[Qemu-block] [PATCH] qapi/block-core.json: Add missing documentation for blklogwrites log-append option

2018-07-06 Thread Ari Sundholm
This was accidentally omitted. Thanks to Eric Blake for spotting this. Signed-off-by: Ari Sundholm --- qapi/block-core.json | 2 ++ 1 file changed, 2 insertions(+) diff --git a/qapi/block-core.json b/qapi/block-core.json index 38b3125..62a92fa 100644 --- a/qapi/block-core.json +++

Re: [Qemu-block] [PATCH v3 0/4] fix image fleecing

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
06.07.2018 10:17, Fam Zheng wrote: On Thu, 07/05 10:46, Vladimir Sementsov-Ogievskiy wrote: Hi all. This fixes image fleecing scheme for 3.0, details are in 04 patch. Looks like this breaks 'test-replication': http://patchew.org/QEMU/20180705074638.770905-1-vsement...@virtuozzo.com/

[Qemu-block] [PATCH for-3.0 4/4] Revert "block: Remove deprecated -drive geometry options"

2018-07-06 Thread Cornelia Huck
This reverts commit a7aff6dd10b16b67e8b142d0c94c5d92c3fe88f6. Hold off removing this for one more QEMU release (current libvirt release still uses it.) Signed-off-by: Cornelia Huck --- blockdev.c| 75 ++- hmp-commands.hx |

[Qemu-block] [PATCH for-3.0 2/4] Revert "block: Remove deprecated -drive option serial"

2018-07-06 Thread Cornelia Huck
This reverts commit b0083267444a5e0f28391f6c2831a539f878d424. Hold off removing this for one more QEMU release (current libvirt release still uses it.) Signed-off-by: Cornelia Huck --- block/block-backend.c | 1 + blockdev.c| 10 ++ hw/block/block.c | 13

[Qemu-block] [PATCH for-3.0 3/4] Revert "block: Remove deprecated -drive option addr"

2018-07-06 Thread Cornelia Huck
This reverts commit eae3bd1eb7c6b105d30ec06008b3bc3dfc5f45bb. Reverted to avoid conflicts for geometry options revert. Signed-off-by: Cornelia Huck --- blockdev.c| 17 - device-hotplug.c | 4 include/sysemu/blockdev.h | 1 + qemu-doc.texi

[Qemu-block] [PATCH for-3.0 1/4] Revert "block: Remove dead deprecation warning code"

2018-07-06 Thread Cornelia Huck
This reverts commit 6266e900b8083945cb766b45c124fb3c42932cb3. Some deprecated -drive options were still in use by libvirt, only fixed with libvirt commit b340c6c614 ("qemu: format serial and geometry on frontend disk device"), which is not yet in any released version of libvirt. So let's hold

[Qemu-block] [PATCH for-3.0 0/4] block: revert removal of deprecated options

2018-07-06 Thread Cornelia Huck
Several -drive options had been marked as deprecated for two releases and had therefore been removed; unfortunately, libvirt still used at least two of them (serial number and disk geometry) and the patch switching away from it is not yet in any released libvirt version. As agreed, let's revert

[Qemu-block] [PATCH v2 1/4] drity-bitmap: switch assert-fails to errors in bdrv_merge_dirty_bitmap

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Move checks from qmp_x_block_dirty_bitmap_merge() to bdrv_merge_dirty_bitmap(), to share them with dirty bitmap merge transaction action in future commit. Note: for now, only qmp_x_block_dirty_bitmap_merge() calls bdrv_merge_dirty_bitmap(). Signed-off-by: Vladimir Sementsov-Ogievskiy ---

[Qemu-block] [PATCH v2 3/4] blockdev: rename block-dirty-bitmap-clear transaction handlers

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Rename block-dirty-bitmap-clear transaction handlers to reuse them for x-block-dirty-bitmap-merge transaction in the following patch. Signed-off-by: Vladimir Sementsov-Ogievskiy --- blockdev.c | 8 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/blockdev.c b/blockdev.c

[Qemu-block] [PATCH v2 2/4] dirty-bitmap: restore bitmap after merge

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
Add backup parameter to bdrv_merge_dirty_bitmap() and refactor bdrv_undo_clear_dirty_bitmap() to use it both for undo clean and merge. Signed-off-by: Vladimir Sementsov-Ogievskiy --- include/block/block_int.h| 2 +- include/block/dirty-bitmap.h | 2 +- include/qemu/hbitmap.h | 25

[Qemu-block] [PATCH v2 0/4] transaction support for bitmap merge

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
This is a last brick, necessary to play with nbd bitmap export in conjunction with image fleecing. v2: don't compare with v1, it is changed a lot, to do the whole thing in .prepare instead of .commit. It is needed to be compatible with backup block job transaction actions [John] Vladimir

[Qemu-block] [PATCH v2 4/4] qapi: add transaction support for x-block-dirty-bitmap-merge

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
New action is like clean action: do the whole thing in .prepare and undo in .abort. This behavior for bitmap-changing actions is needed because backup job actions use bitmap in .prepare. Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/transaction.json | 2 ++ blockdev.c| 38

Re: [Qemu-block] [Qemu-devel] [libvirt] [PULL 25/26] block: Remove deprecated -drive option serial

2018-07-06 Thread Cornelia Huck
On Wed, 4 Jul 2018 17:14:02 +0100 Peter Maydell wrote: > On 4 July 2018 at 14:34, Kevin Wolf wrote: > > Essentially, what is important to me isn't getting these options dropped > > exactly in 3.0, but not setting a bad precedence that deprecation isn't > > actually worth anything. We may easily

Re: [Qemu-block] [Qemu-devel] [PULL v2 09/12] block/blklogwrites: Add an option for appending to an old log

2018-07-06 Thread Ari Sundholm
On 07/05/2018 07:06 PM, Eric Blake wrote: On 07/05/2018 05:26 AM, Kevin Wolf wrote: From: Ari Sundholm Suggested by Kevin Wolf. May be useful when testing multiple batches of writes or doing long-term testing involving restarts of the VM. Signed-off-by: Ari Sundholm Signed-off-by: Kevin

[Qemu-block] [PATCH] block/blklogwrites: Make sure the log sector size is not too small

2018-07-06 Thread Ari Sundholm
The sector size needs to be large enough to accommodate the data structures for the log super block and log write entries. This was previously not properly checked, which made it possible to cause QEMU to badly misbehave. Signed-off-by: Ari Sundholm --- block/blklogwrites.c | 5 - 1 file

Re: [Qemu-block] [Qemu-devel] [PATCH 2/2] qapi: add transaction support for x-block-dirty-bitmap-merge

2018-07-06 Thread Vladimir Sementsov-Ogievskiy
05.07.2018 23:40, John Snow wrote: On 07/03/2018 06:53 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/transaction.json | 2 ++ blockdev.c| 53 ++- 2 files changed, 54 insertions(+),

Re: [Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

2018-07-06 Thread Kevin Wolf
Am 06.07.2018 um 08:51 hat Fam Zheng geschrieben: > On Thu, 07/05 14:44, Kevin Wolf wrote: > > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > > Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation > > > here is that discard requests don't affect bs->wr_highest_offset. > > >

Re: [Qemu-block] [PATCH v3 0/4] fix image fleecing

2018-07-06 Thread Fam Zheng
On Thu, 07/05 10:46, Vladimir Sementsov-Ogievskiy wrote: > Hi all. > > This fixes image fleecing scheme for 3.0, details are in 04 patch. Looks like this breaks 'test-replication': http://patchew.org/QEMU/20180705074638.770905-1-vsement...@virtuozzo.com/ test-replication:

Re: [Qemu-block] [PATCH v2 6/9] block: Use common req handling for discard

2018-07-06 Thread Fam Zheng
On Thu, 07/05 14:44, Kevin Wolf wrote: > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > Reuse the new bdrv_co_write_req_prepare/finish helpers. The variation > > here is that discard requests don't affect bs->wr_highest_offset. > > > > Signed-off-by: Fam Zheng > > --- > > block/io.c | 21

Re: [Qemu-block] [PATCH v2 1/9] block: Add copy offloading trace points

2018-07-06 Thread Fam Zheng
On Thu, 07/05 13:08, Kevin Wolf wrote: > Am 05.07.2018 um 09:36 hat Fam Zheng geschrieben: > > A few trace points that can help reveal what is happening in a copy > > offloading I/O path. > > > > Signed-off-by: Fam Zheng > > --- > > block/file-posix.c | 2 ++ > > block/io.c | 2 ++ > >