Re: [Qemu-block] [Qemu-devel] [PATCH v9 13/13] iotests: add dirty bitmap postcopy test

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
20.01.2018 03:37, John Snow wrote: On 01/19/2018 01:08 PM, Vladimir Sementsov-Ogievskiy wrote: 18.01.2018 12:57, Vladimir Sementsov-Ogievskiy wrote: 17.01.2018 21:30, John Snow wrote: On 12/28/2017 06:49 AM, Vladimir Sementsov-Ogievskiy wrote: Thank you for reviewing my code! Time for the

Re: [Qemu-block] [Qemu-devel] [PATCH v2 1/6] block: maintain persistent disabled bitmaps

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
20.01.2018 02:43, John Snow wrote: On 01/16/2018 07:54 AM, Vladimir Sementsov-Ogievskiy wrote: To maintain load/store disabled bitmap there is new approach: - deprecate @autoload flag of block-dirty-bitmap-add, make it ignored - store enabled bitmaps as "auto" to qcow2 - store disabled b

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/6] qapi: add block-dirty-bitmap-enable/disable

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
20.01.2018 02:50, John Snow wrote: On 01/16/2018 07:54 AM, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy --- qapi/block-core.json | 42 ++ blockdev.c | 42 ++ 2 files

Re: [Qemu-block] [Qemu-devel] [PATCH v2 4/6] qmp: transaction support for block-dirty-bitmap-enable/disable

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
20.01.2018 03:28, John Snow wrote: On 01/17/2018 10:06 AM, Vladimir Sementsov-Ogievskiy wrote: 16.01.2018 15:54, Vladimir Sementsov-Ogievskiy wrote: Signed-off-by: Vladimir Sementsov-Ogievskiy ---   qapi/transaction.json |  4 +++   blockdev.c    | 79

Re: [Qemu-block] [Qemu-devel] [PATCH v2 0/6] qmp dirty bitmap API

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
20.01.2018 02:30, John Snow wrote: On 01/16/2018 07:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi all. There are three qmp commands, needed to implement external backup API. Using these three commands, client may do all needed bitmap management by hand: on backup start we need to do a transa

[Qemu-block] [PATCH v2] block: maintain persistent disabled bitmaps

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
To maintain load/store disabled bitmap there is new approach: - deprecate @autoload flag of block-dirty-bitmap-add, make it ignored - store enabled bitmaps as "auto" to qcow2 - store disabled bitmaps without "auto" flag to qcow2 - on qcow2 open load "auto" bitmaps as enabled and others as d

Re: [Qemu-block] [Qemu-devel] [PATCH v9 03/13] block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
19.01.2018 17:12, Vladimir Sementsov-Ogievskiy wrote: 18.01.2018 13:09, Paolo Bonzini wrote: On 18/01/2018 10:55, Vladimir Sementsov-Ogievskiy wrote: Most functions that looks at the list are "called with BQL taken". Functions that write to the list are "called with BQL taken" and call bdrv_dir

Re: [Qemu-block] [Qemu-devel] [PATCH v2 0/6] qmp dirty bitmap API

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
22.01.2018 12:20, Vladimir Sementsov-Ogievskiy wrote: 20.01.2018 02:30, John Snow wrote: On 01/16/2018 07:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi all. There are three qmp commands, needed to implement external backup API. Using these three commands, client may do all needed bitmap man

Re: [Qemu-block] [PATCH v6 0/9] block: Add VFIO based driver for NVMe device

2018-01-22 Thread Stefan Hajnoczi
On Tue, Jan 16, 2018 at 02:08:52PM +0800, Fam Zheng wrote: > v6: Fix the cleaning up around nvme_init() and write cache setup in > nvme_file_open(). [Stefan] > Add Stefan's rev-by to the doc patch. > Rebase on to Paolo's coroutine series for q->free_req_queue: > > Based-on: 20180115220

[Qemu-block] [Qemu-devel] [PATCH] raw: Disable probing if image format is given by driver-specific options

2018-01-22 Thread Lin Ma
If the user specifies image format through driver-specific options, The format probing should be prohibited and the warning message should not be printed. e.g.: $ qemu-system-x86_64 ... -drive file.file.filename=disk0.raw,file.driver=raw ... WARNING: Image format was not specified for 'disk0.raw'

Re: [Qemu-block] [PATCH 0/5 v3] preparation for Parallels Disk xml driver

2018-01-22 Thread Stefan Hajnoczi
On Fri, Jan 12, 2018 at 12:01:17PM +0300, Klim Kireev wrote: > Parallels Desktop and Parallels Cloud Server uses images glued with the > bundle description in XML format. This series contains very basic > description of this XML files and makes preparations for actual > implementation to be followe

[Qemu-block] [RFC] block-backend: fix double inc/dec inflight requests number

2018-01-22 Thread Vladimir Sementsov-Ogievskiy
blk_co_preadv and blk_co_pwritev calls bdrv_inc_in_flight and bdrv_dec_in_flight. But they calls correspondingly bdrv_co_preadv and bdrv_co_pwritev, which inc/dec in-flight count by themselves. Counting in-flight requests in blk_ layer is redundant, drop it. Signed-off-by: Vladimir Sementsov-Ogiev

[Qemu-block] [PATCH] virtio-blk: check for NULL BlockDriverState

2018-01-22 Thread Mark Kanda
Add a BlockDriverState NULL check to virtio_blk_handle_request() to prevent a segfault if the drive is forcibly removed using HMP 'drive_del' (without performing a hotplug 'device_del' first). Signed-off-by: Mark Kanda Reviewed-by: Karl Heubaum Reviewed-by: Ameya More --- hw/block/virtio-blk.c

Re: [Qemu-block] [PATCH v2 1/6] qapi: Add qobject_to()

2018-01-22 Thread Eric Blake
On 01/20/2018 09:44 AM, Max Reitz wrote: > This is a dynamic casting macro that, given a QObject type, returns an > object as that type or NULL if the object is of a different type (or > NULL itself). > > The macro uses lower-case letters because: > 1. There does not seem to be a hard rule on whet

Re: [Qemu-block] [Qemu-devel] [PATCH v2 31/32] qcow2: Allow configuring the L2 slice size

2018-01-22 Thread Alberto Garcia
On Tue 16 Jan 2018 05:57:22 PM CET, Anton Nefedov wrote: >> -r->l2_slice_size = s->cluster_size / sizeof(uint64_t); >> -r->l2_table_cache = qcow2_cache_create(bs, l2_cache_size); >> -r->refcount_block_cache = qcow2_cache_create(bs, refcount_cache_size); >> +l2_cache_size *= s->clust

Re: [Qemu-block] [Qemu-devel] [PATCH] raw: Disable probing if image format is given by driver-specific options

2018-01-22 Thread Max Reitz
On 2018-01-22 08:21, Lin Ma wrote: > If the user specifies image format through driver-specific options, The > format probing should be prohibited and the warning message should not > be printed. > > e.g.: > $ qemu-system-x86_64 ... -drive file.file.filename=disk0.raw,file.driver=raw > ... > WARN

Re: [Qemu-block] [PATCH v2 1/6] qapi: Add qobject_to()

2018-01-22 Thread Max Reitz
On 2018-01-22 16:33, Eric Blake wrote: > On 01/20/2018 09:44 AM, Max Reitz wrote: >> This is a dynamic casting macro that, given a QObject type, returns an >> object as that type or NULL if the object is of a different type (or >> NULL itself). >> >> The macro uses lower-case letters because: >> 1.

Re: [Qemu-block] [Qemu-devel] [PATCH v2 0/6] qmp dirty bitmap API

2018-01-22 Thread John Snow
On 01/22/2018 07:22 AM, Vladimir Sementsov-Ogievskiy wrote: > 22.01.2018 12:20, Vladimir Sementsov-Ogievskiy wrote: >> 20.01.2018 02:30, John Snow wrote: >>> >>> On 01/16/2018 07:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi all. There are three qmp commands, needed to implement ex

Re: [Qemu-block] [Qemu-devel] [PATCH v9 03/13] block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap

2018-01-22 Thread John Snow
On 01/22/2018 07:14 AM, Vladimir Sementsov-Ogievskiy wrote: > 19.01.2018 17:12, Vladimir Sementsov-Ogievskiy wrote: >> 18.01.2018 13:09, Paolo Bonzini wrote: >>> On 18/01/2018 10:55, Vladimir Sementsov-Ogievskiy wrote: > Most functions that looks at the list are "called with BQL taken". >

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/6] qapi: add block-dirty-bitmap-enable/disable

2018-01-22 Thread Eric Blake
On 01/22/2018 03:09 AM, Vladimir Sementsov-Ogievskiy wrote: >>> >> I have to admit exposing this interface still makes me nervous, but :) >> >> Mechanically correct, and with suggesting phrasing changes: >> >> Reviewed-by: John Snow Should we go with an x- prefix for now, and let things settle f

Re: [Qemu-block] [Qemu-devel] [PATCH v2 3/6] qapi: add block-dirty-bitmap-enable/disable

2018-01-22 Thread John Snow
On 01/22/2018 02:51 PM, Eric Blake wrote: > On 01/22/2018 03:09 AM, Vladimir Sementsov-Ogievskiy wrote: > >>> I have to admit exposing this interface still makes me nervous, but :) >>> >>> Mechanically correct, and with suggesting phrasing changes: >>> >>> Reviewed-by: John Snow > > Shoul

Re: [Qemu-block] [Qemu-devel] [PATCH 1/5] docs/interop/prl-xml: description of Parallels Disk format

2018-01-22 Thread Eric Blake
On 01/12/2018 03:01 AM, Klim Kireev wrote: > This patch adds main information about Parallels Disk > format, which consists of DiskDescriptor.xml and other files. > > Signed-off-by: Edgar Kaziakhmedov > Signed-off-by: Klim Kireev > Signed-off-by: Vladimir Sementsov-Ogievskiy > Signed-off-by: De

Re: [Qemu-block] [Qemu-devel] [PATCH v2 29/32] qcow2: Rename l2_table in count_contiguous_clusters_unallocated()

2018-01-22 Thread Eric Blake
On 12/15/2017 06:53 AM, Alberto Garcia wrote: > This function doesn't need any changes to support L2 slices, but since > it's now dealing with slices intead of full tables, the l2_table > variable is renamed for clarity. > > Signed-off-by: Alberto Garcia > --- > block/qcow2-cluster.c | 6 +++---

Re: [Qemu-block] [Qemu-devel] [PATCH 1/5] docs/interop/prl-xml: description of Parallels Disk format

2018-01-22 Thread Eric Blake
On 01/22/2018 02:32 PM, Eric Blake wrote: > On 01/12/2018 03:01 AM, Klim Kireev wrote: >> This patch adds main information about Parallels Disk >> format, which consists of DiskDescriptor.xml and other files. >> >> Signed-off-by: Edgar Kaziakhmedov >> Signed-off-by: Klim Kireev >> Signed-off-by:

Re: [Qemu-block] [Qemu-devel] [PATCH v2 30/32] qcow2: Rename l2_table in count_cow_clusters()

2018-01-22 Thread Eric Blake
On 12/15/2017 06:53 AM, Alberto Garcia wrote: > This function doesn't need any changes to support L2 slices, but since > it's now dealing with slices intead of full tables, the l2_table > variable is renamed for clarity. > > Signed-off-by: Alberto Garcia > --- > block/qcow2-cluster.c | 4 ++-- >

Re: [Qemu-block] [PATCH v2 2/8] qapi: add unmap to BlockDeviceStats

2018-01-22 Thread Eric Blake
On 01/19/2018 06:50 AM, Anton Nefedov wrote: > Signed-off-by: Anton Nefedov > Reviewed-by: Vladimir Sementsov-Ogievskiy > Reviewed-by: Alberto Garcia > --- > qapi/block-core.json | 29 +++-- > include/block/accounting.h | 1 + > block/qapi.c | 6 +++

Re: [Qemu-block] [PATCH v2 3/8] ide: account UNMAP (TRIM) operations

2018-01-22 Thread Eric Blake
On 01/19/2018 06:50 AM, Anton Nefedov wrote: > Signed-off-by: Anton Nefedov > --- > hw/ide/core.c | 13 + > 1 file changed, 13 insertions(+) > > @@ -460,10 +468,15 @@ static void ide_issue_trim_cb(void *opaque, int ret) > } > > if (!ide_sect_range

Re: [Qemu-block] [PATCH v2 8/8] qapi: query-blockstat: add driver specific file-posix stats

2018-01-22 Thread Eric Blake
On 01/19/2018 06:50 AM, Anton Nefedov wrote: > A block driver can provide a callback to report driver-specific > statistics. > > file-posix driver now reports discard statistics > > Signed-off-by: Anton Nefedov > Reviewed-by: Vladimir Sementsov-Ogievskiy > --- > +++ b/qapi/block-core.json > @@

[Qemu-block] [PATCH v2 01/16] block: BDS deletion during bdrv_drain_recurse

2018-01-22 Thread Max Reitz
Draining a BDS child may lead to other children of the same parent being detached and/or deleted. We should prepare for the former case (by copying the children list before iterating through it) and prevent the latter (by bdrv_ref()'ing all nodes if we are in the main loop). Signed-off-by: Max Re

[Qemu-block] [PATCH v2 02/16] block: BDS deletion in bdrv_do_drained_begin()

2018-01-22 Thread Max Reitz
Draining a BDS (in the main loop) may cause it to go be deleted. That is rather suboptimal if we still plan to access it afterwards, so let us enclose the main body of the function with a bdrv_ref()/bdrv_unref() pair. Signed-off-by: Max Reitz --- block/io.c | 19 +++ 1 file chan

[Qemu-block] [PATCH v2 00/16] block/mirror: Add active-sync mirroring

2018-01-22 Thread Max Reitz
This series implements an active and synchronous mirroring mode. Currently, the mirror block job is passive an asynchronous: Depending on your start conditions, some part of the source disk starts as "dirty". Then, the block job will (as a background operation) continuously copy dirty parts to the

[Qemu-block] [PATCH v2 03/16] tests: Add bdrv-drain test for node deletion

2018-01-22 Thread Max Reitz
This patch adds two bdrv-drain tests for what happens if some BDS goes away during the drainage. The basic idea is that you have a parent BDS with some child nodes. Then, you drain one of the children. Because of that, the party who actually owns the parent decides to (A) delete it, or (B) detach

[Qemu-block] [PATCH v2 04/16] block/mirror: Pull out mirror_perform()

2018-01-22 Thread Max Reitz
When converting mirror's I/O to coroutines, we are going to need a point where these coroutines are created. mirror_perform() is going to be that point. Signed-off-by: Max Reitz Reviewed-by: Fam Zheng Reviewed-by: Vladimir Sementsov-Ogievskiy --- block/mirror.c | 51 ++

[Qemu-block] [PATCH v2 11/16] block/dirty-bitmap: Add bdrv_dirty_iter_next_area

2018-01-22 Thread Max Reitz
This new function allows to look for a consecutively dirty area in a dirty bitmap. Signed-off-by: Max Reitz --- include/block/dirty-bitmap.h | 2 ++ block/dirty-bitmap.c | 51 2 files changed, 53 insertions(+) diff --git a/include/block/dirt

[Qemu-block] [PATCH v2 06/16] block/mirror: Use CoQueue to wait on in-flight ops

2018-01-22 Thread Max Reitz
Attach a CoQueue to each in-flight operation so if we need to wait for any we can use it to wait instead of just blindly yielding and hoping for some operation to wake us. A later patch will use this infrastructure to allow requests accessing the same area of the virtual disk to specifically wait

[Qemu-block] [PATCH v2 07/16] block/mirror: Wait for in-flight op conflicts

2018-01-22 Thread Max Reitz
This patch makes the mirror code differentiate between simply waiting for any operation to complete (mirror_wait_for_free_in_flight_slot()) and specifically waiting for all operations touching a certain range of the virtual disk to complete (mirror_wait_on_conflicts()). Signed-off-by: Max Reitz -

[Qemu-block] [PATCH v2 08/16] block/mirror: Use source as a BdrvChild

2018-01-22 Thread Max Reitz
With this, the mirror_top_bs is no longer just a technically required node in the BDS graph but actually represents the block job operation. Also, drop MirrorBlockJob.source, as we can reach it through mirror_top_bs->backing. Signed-off-by: Max Reitz --- block/mirror.c | 14 ++ 1 fi

[Qemu-block] [PATCH v2 14/16] block/mirror: Add active mirroring

2018-01-22 Thread Max Reitz
This patch implements active synchronous mirroring. In active mode, the passive mechanism will still be in place and is used to copy all initially dirty clusters off the source disk; but every write request will write data both to the source and the target disk, so the source cannot be dirtied fas

[Qemu-block] [PATCH v2 09/16] block: Generalize should_update_child() rule

2018-01-22 Thread Max Reitz
Currently, bdrv_replace_node() refuses to create loops from one BDS to itself if the BDS to be replaced is the backing node of the BDS to replace it: Say there is a node A and a node B. Replacing B by A means making all references to B point to A. If B is a child of A (i.e. A has a reference to B

[Qemu-block] [PATCH v2 16/16] iotests: Add test for active mirroring

2018-01-22 Thread Max Reitz
Signed-off-by: Max Reitz --- tests/qemu-iotests/151 | 114 + tests/qemu-iotests/151.out | 5 ++ tests/qemu-iotests/group | 1 + 3 files changed, 120 insertions(+) create mode 100755 tests/qemu-iotests/151 create mode 100644 tests/qemu-iotest

[Qemu-block] [PATCH v2 13/16] block/mirror: Add MirrorBDSOpaque

2018-01-22 Thread Max Reitz
This will allow us to access the block job data when the mirror block driver becomes more complex. Signed-off-by: Max Reitz --- block/mirror.c | 12 1 file changed, 12 insertions(+) diff --git a/block/mirror.c b/block/mirror.c index bb46f3c4e9..c1987b0230 100644 --- a/block/mirror.

[Qemu-block] [PATCH v2 05/16] block/mirror: Convert to coroutines

2018-01-22 Thread Max Reitz
In order to talk to the source BDS (and maybe in the future to the target BDS as well) directly, we need to convert our existing AIO requests into coroutine I/O requests. Signed-off-by: Max Reitz --- block/mirror.c | 154 ++--- 1 file changed,

[Qemu-block] [PATCH v2 15/16] block/mirror: Add copy mode QAPI interface

2018-01-22 Thread Max Reitz
This patch allows the user to specify whether to use active or only passive mode for mirror block jobs. Currently, this setting will remain constant for the duration of the entire block job. Signed-off-by: Max Reitz --- qapi/block-core.json | 11 +-- include/block/block_int.h | 4

[Qemu-block] [PATCH v2 10/16] hbitmap: Add @advance param to hbitmap_iter_next()

2018-01-22 Thread Max Reitz
This new parameter allows the caller to just query the next dirty position without moving the iterator. Signed-off-by: Max Reitz --- include/qemu/hbitmap.h | 5 - block/backup.c | 2 +- block/dirty-bitmap.c | 2 +- tests/test-hbitmap.c | 26 +- util/hbi

[Qemu-block] [PATCH v2 12/16] block/mirror: Distinguish active from passive ops

2018-01-22 Thread Max Reitz
Currently, the mirror block job only knows passive operations. But once we introduce active writes, we need to distinguish between the two; for example, mirror_wait_for_free_in_flight_slot() should wait for a passive operation because active writes will not use the same in-flight slots. Signed-of

Re: [Qemu-block] [PATCH v3 2/5] qapi: add nbd-server-remove

2018-01-22 Thread Eric Blake
On 01/19/2018 07:57 AM, Vladimir Sementsov-Ogievskiy wrote: > Add command for removing an export. It is needed for cases when we > don't want to keep export after the operation on it was completed. > The other example is temporary node, created with blockdev-add. > If we want to delete it we should

Re: [Qemu-block] [Qemu-devel] [PATCH] raw: Disable probing if image format is given by driver-specific options

2018-01-22 Thread Lin Ma
On 01/23/2018 12:45 AM, Max Reitz wrote: On 2018-01-22 08:21, Lin Ma wrote: If the user specifies image format through driver-specific options, The format probing should be prohibited and the warning message should not be printed. e.g.: $ qemu-system-x86_64 ... -drive file.file.filename=disk0.