[Qemu-block] [PATCH v2 10/11] blockdev: Block device IO during blockdev-backup transaction

2015-05-13 Thread Fam Zheng
Signed-off-by: Fam Zheng f...@redhat.com --- blockdev.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/blockdev.c b/blockdev.c index ae52d27..bd28183 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1649,6 +1649,7 @@ typedef struct BlockdevBackupState {

Re: [Qemu-block] [PATCH 14/34] qcow2: Factor out qcow2_update_options()

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Eventually we want to be able to change options at runtime. As a first step towards that goal, separate some option handling code from the general initialisation code in qcow2_open(). Signed-off-by: Kevin Wolf kw...@redhat.com --- block/qcow2.c | 135

[Qemu-block] [PATCH v2 00/11] Fix transactional snapshot with virtio-blk dataplane and NBD export

2015-05-13 Thread Fam Zheng
Changes from RFC: - Add op blocker listener in nbd server. - Add other transaction types. - Only notify listeners when changing from/to empty. (Paolo) Reported by Paolo. Unlike the iohandler in main loop, iothreads currently process the event notifier used by virtio-blk ioeventfd in nested

[Qemu-block] [PATCH v2 04/11] virtio-blk: Move complete_request to 'ops' structure

2015-05-13 Thread Fam Zheng
Should more ops be added to differentiate code between dataplane and non-dataplane, the new saved_ops approach will be cleaner than messing with N pointers. Signed-off-by: Fam Zheng f...@redhat.com --- hw/block/dataplane/virtio-blk.c | 13 - hw/block/virtio-blk.c | 8

Re: [Qemu-block] [PATCH v2 05/11] virtio-blk: Don't handle output when there is device IO op blocker

2015-05-13 Thread Fam Zheng
On Wed, 05/13 12:26, Paolo Bonzini wrote: On 13/05/2015 19:28, Fam Zheng wrote: +static void virtio_blk_data_plane_pause(VirtIOBlock *vblk) +{ +VirtIOBlockDataPlane *s = vblk-dataplane; + +event_notifier_test_and_clear(s-host_notifier); +aio_set_event_notifier(s-ctx,

Re: [Qemu-block] [Qemu-devel] [PATCH 20/34] qcow2: Support updating driver-specific options in reopen

2015-05-13 Thread Kevin Wolf
Am 12.05.2015 um 23:47 hat Eric Blake geschrieben: On 05/08/2015 11:21 AM, Kevin Wolf wrote: For updating the cache sizes or disabling lazy refcounts there is a bit more to do than just changing the variables, but otherwise we're all set for changing options during bdrv_reopen(). Just

Re: [Qemu-block] [PATCH v2 05/11] virtio-blk: Don't handle output when there is device IO op blocker

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 19:28, Fam Zheng wrote: +static void virtio_blk_data_plane_pause(VirtIOBlock *vblk) +{ +VirtIOBlockDataPlane *s = vblk-dataplane; + +event_notifier_test_and_clear(s-host_notifier); +aio_set_event_notifier(s-ctx, s-host_notifier, NULL); +} + +static void

Re: [Qemu-block] [PATCH v2 11/11] block: Block device IO during bdrv_drain and bdrv_drain_all

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 19:28, Fam Zheng wrote: We don't want new requests from guest, so block the operation around the nested poll. Signed-off-by: Fam Zheng f...@redhat.com --- block/io.c | 12 1 file changed, 12 insertions(+) diff --git a/block/io.c b/block/io.c index

Re: [Qemu-block] [PATCH 14/34] qcow2: Factor out qcow2_update_options()

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Eventually we want to be able to change options at runtime. As a first step towards that goal, separate some option handling code from the general initialisation code in qcow2_open(). Signed-off-by: Kevin Wolf kw...@redhat.com --- block/qcow2.c | 135

Re: [Qemu-block] [Qemu-devel] [PATCH v2 01/11] block: Add op blocker type device IO

2015-05-13 Thread Wen Congyang
On 05/14/2015 01:28 AM, Fam Zheng wrote: Preventing device from submitting IO is useful around various nested aio_poll. Op blocker is a good place to put this flag. Devices would submit IO requests through blk_* block backend interface, which calls blk_check_request to check the validity.

[Qemu-block] [PATCH v2 08/11] blockdev: Block device IO during external snapshot transaction

2015-05-13 Thread Fam Zheng
Signed-off-by: Fam Zheng f...@redhat.com --- blockdev.c | 18 -- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/blockdev.c b/blockdev.c index 7f763d9..923fc90 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1404,6 +1404,7 @@ typedef struct ExternalSnapshotState {

[Qemu-block] [PATCH v2 06/11] nbd-server: Clear can_read when device io blocker is set

2015-05-13 Thread Fam Zheng
So that NBD export cannot submit IO during bdrv_drain_all(). Signed-off-by: Fam Zheng f...@redhat.com --- nbd.c | 18 ++ 1 file changed, 18 insertions(+) diff --git a/nbd.c b/nbd.c index 06b501b..7d9d3e4 100644 --- a/nbd.c +++ b/nbd.c @@ -160,6 +160,8 @@ struct NBDExport {

[Qemu-block] [PATCH v2 09/11] blockdev: Block device IO during drive-backup transaction

2015-05-13 Thread Fam Zheng
Signed-off-by: Fam Zheng f...@redhat.com --- blockdev.c | 9 - 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/blockdev.c b/blockdev.c index 923fc90..ae52d27 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1575,6 +1575,7 @@ typedef struct DriveBackupState { BlockDriverState

Re: [Qemu-block] [Qemu-devel] [PATCH 4/5] qemu-io: prompt for encryption keys when required

2015-05-13 Thread Markus Armbruster
Daniel P. Berrange berra...@redhat.com writes: On Tue, May 12, 2015 at 12:32:53PM -0600, Eric Blake wrote: On 05/12/2015 10:09 AM, Daniel P. Berrange wrote: The qemu-io tool does not check if the image is encrypted so historically would silently corrupt the sectors by writing plain text

Re: [Qemu-block] [Qemu-devel] [PATCH 14/34] qcow2: Factor out qcow2_update_options()

2015-05-13 Thread Kevin Wolf
Am 12.05.2015 um 22:04 hat Eric Blake geschrieben: On 05/08/2015 11:21 AM, Kevin Wolf wrote: Eventually we want to be able to change options at runtime. As a first step towards that goal, separate some option handling code from the general initialisation code in qcow2_open().

[Qemu-block] [PATCH v2 01/11] block: Add op blocker type device IO

2015-05-13 Thread Fam Zheng
Preventing device from submitting IO is useful around various nested aio_poll. Op blocker is a good place to put this flag. Devices would submit IO requests through blk_* block backend interface, which calls blk_check_request to check the validity. Return -EBUSY if the operation is blocked, which

Re: [Qemu-block] [PATCH] block/mirror: Sleep periodically during bitmap scanning

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 05:11, Fam Zheng wrote: Before, we only yield after initializing dirty bitmap, where the QMP command would return. That may take very long, and guest IO will be blocked. Add sleep points like the later mirror iterations. You were also planning to let

Re: [Qemu-block] [PATCH v2 11/11] block: Block device IO during bdrv_drain and bdrv_drain_all

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 13:08, Fam Zheng wrote: I think this isn't enough. It's the callers of bdrv_drain and bdrv_drain_all that need to block before drain and unblock before aio_context_release. Which callers do you mean? qmp_transaction is covered in this series. All of them. :( In some

[Qemu-block] [PATCH v2 07/11] blockdev: Block device IO during internal snapshot transaction

2015-05-13 Thread Fam Zheng
Signed-off-by: Fam Zheng f...@redhat.com --- blockdev.c | 12 +--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/blockdev.c b/blockdev.c index 5eaf77e..7f763d9 100644 --- a/blockdev.c +++ b/blockdev.c @@ -1262,6 +1262,7 @@ typedef struct InternalSnapshotState {

Re: [Qemu-block] [PATCH] qemu-io: Use getopt() correctly

2015-05-13 Thread Kevin Wolf
Am 12.05.2015 um 17:10 hat Eric Blake geschrieben: POSIX says getopt() returns -1 on completion. While Linux happens to define EOF as -1, this definition is not required by POSIX, and there may be platforms where checking for EOF instead of -1 would lead to an infinite loop. Signed-off-by:

Re: [Qemu-block] [PATCH 15/34] qcow2: Move qcow2_update_options() call up

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: qcow2_update_options() only updates some variables in BDRVQcowState and doesn't really depend on other parts of it being initialised yet, so it can be moved so that it immediately follows the other half of option handling code in qcow2_open().

Re: [Qemu-block] [PATCH 17/34] qcow2: Leave s unchanged on qcow2_update_options() failure

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: On return, either all new options should be applied to BDRVQcowState (on success), or all of the old setting should be preserved (on failure). Signed-off-by: Kevin Wolf kw...@redhat.com --- block/qcow2.c | 52

Re: [Qemu-block] [PATCH 18/34] qcow2: Fix memory leak in qcow2_update_options() error path

2015-05-13 Thread Max Reitz
On 13.05.2015 14:02, Kevin Wolf wrote: Am 13.05.2015 um 13:52 hat Max Reitz geschrieben: On 08.05.2015 19:21, Kevin Wolf wrote: Signed-off-by: Kevin Wolf kw...@redhat.com --- block/qcow2.c | 12 ++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/block/qcow2.c

[Qemu-block] [PATCH 01/11] block: keep a list of block jobs

2015-05-13 Thread Alberto Garcia
The current way to obtain the list of existing block jobs is to iterate over all root nodes and check which ones own a job. Since we want to be able to support block jobs in other nodes as well, this patch keeps a list of jobs that is updated every time one is created or destroyed. This also

[Qemu-block] [PATCH 03/11] block: never cancel a streaming job without running stream_complete()

2015-05-13 Thread Alberto Garcia
We need to call stream_complete() in order to do all the necessary clean-ups, even if there's an early failure. At the moment it's only useful to make sure that s-backing_file_str is not leaked, but it will become more important as we introduce support for streaming to any intermediate node.

[Qemu-block] [PATCH 09/11] qemu-iotests: test streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
This adds test_stream_intermediate(), similar to test_stream() but streams to the intermediate image instead. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 18 +- tests/qemu-iotests/030.out | 4 ++-- 2

Re: [Qemu-block] [PATCH 19/34] qcow2: Make qcow2_update_options() suitable for transactions

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Before we can allow updating options at runtime with bdrv_reopen(), we need to split the function into prepare/commit/abort parts. Signed-off-by: Kevin Wolf kw...@redhat.com --- block/qcow2.c | 101

[Qemu-block] [PATCH 07/11] qemu-iotests: fix test_stream_partial()

2015-05-13 Thread Alberto Garcia
This test is streaming to the top layer using the intermediate image as the base. This is a mistake since block-stream never copies data from the base image and its backing chain, so this is effectively a no-op. In addition to fixing the base parameter, this patch also writes some data to the

[Qemu-block] [PATCH v7 00/11] Support streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
v7: - Rebased against the current master - Updated bdrv_drain_all() to use the new block_job_next() API. v6: https://lists.gnu.org/archive/html/qemu-devel/2015-04/msg03046.html - fix the no-op test following Max's suggestions v5:

Re: [Qemu-block] [PATCH 23/34] block: Pass driver-specific options to .bdrv_refresh_filename()

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: In order to decide whether a blkdebug: filename can be produced or a json: one is necessary, blkdebug checked whether bs-options had more options than just config, x-image or image (the latter including nested options). That doesn't work well when generic

[Qemu-block] [PATCH 05/11] block: Add QMP support for streaming to an intermediate layer

2015-05-13 Thread Alberto Garcia
This patch makes the 'device' parameter of the 'block-stream' command accept a node name as well as a device name. In addition to that, operation blockers will be checked in all intermediate nodes between the top and the base node. Since qmp_block_stream() now uses the error from

[Qemu-block] [PATCH 10/11] qemu-iotests: test block-stream operations in parallel

2015-05-13 Thread Alberto Garcia
This test case checks that it's possible to launch several stream operations in parallel in the same snapshot chain, each one involving a different set of nodes. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 80

[Qemu-block] [PATCH 06/11] docs: Document how to stream to an intermediate layer

2015-05-13 Thread Alberto Garcia
Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com --- docs/live-block-ops.txt | 31 --- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/docs/live-block-ops.txt

[Qemu-block] [PATCH 11/11] qemu-iotests: test overlapping block-stream operations

2015-05-13 Thread Alberto Garcia
This test case checks that it's not possible to perform two block-stream operations if there are nodes involved in both. Signed-off-by: Alberto Garcia be...@igalia.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/030 | 27 +++

Re: [Qemu-block] [PATCH 22/34] block: Exclude nested options only for children in append_open_options()

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: Some drivers have nested options (e.g. blkdebug rule arrays), which don't belong to a child node and shouldn't be removed. Don't remove all options with . in their name, but check for the complete prefixes of actually existing child nodes. Signed-off-by:

Re: [Qemu-block] [PATCH 25/34] block: Allow specifying child options in reopen

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: If the child was defined in the same context (-drive argument or blockdev-add QMP command) as its parent, a reopen of the parent should work the same and allow changing options of the child. Signed-off-by: Kevin Wolf kw...@redhat.com --- block.c | 12

Re: [Qemu-block] [PATCH v2 10/11] blockdev: Block device IO during blockdev-backup transaction

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 19:28, Fam Zheng wrote: +state-bs = bs; +error_setg(state-blocker, blockdev-backup in progress); +bdrv_op_block(bs, BLOCK_OP_TYPE_DEVICE_IO, state-blocker); + qmp_blockdev_backup(backup-device, backup-target, backup-sync,

Re: [Qemu-block] [Qemu-devel] [PATCH v2 10/11] blockdev: Block device IO during blockdev-backup transaction

2015-05-13 Thread Fam Zheng
On Wed, 05/13 19:22, Wen Congyang wrote: On 05/14/2015 01:28 AM, Fam Zheng wrote: Signed-off-by: Fam Zheng f...@redhat.com --- blockdev.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/blockdev.c b/blockdev.c index ae52d27..bd28183 100644 ---

Re: [Qemu-block] [PATCH 21/34] block: Consider all block layer options in append_open_options

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: The code already special-cased node-name, which is currently the only option passed in the QDict that isn't driver-specific. Generalise the code to take all general block layer options into consideration. Signed-off-by: Kevin Wolf kw...@redhat.com ---

Re: [Qemu-block] [PATCH v8 0/2] block: enforce minimal 4096 alignment in qemu_blockalign

2015-05-13 Thread Kevin Wolf
Am 12.05.2015 um 16:30 hat Denis V. Lunev geschrieben: I have used the following program to test [...] and the amount of requests sent to disk (could be calculated counting number of lines in the output of blktrace) is reduced about 2 times. [...] Signed-off-by: Denis V. Lunev

Re: [Qemu-block] [PATCH 26/34] block: reopen: Document option precedence and refactor accordingly

2015-05-13 Thread Max Reitz
On 08.05.2015 19:21, Kevin Wolf wrote: The interesting part of reopening an image is from which sources the effective options should be taken, i.e. which options take precedence over which other options. This patch documents the precedence that will be implemented in the following patches. It

Re: [Qemu-block] [PATCH v2 01/11] block: Add op blocker type device IO

2015-05-13 Thread Fam Zheng
On Wed, 05/13 14:04, Paolo Bonzini wrote: On 13/05/2015 19:28, Fam Zheng wrote: @@ -478,6 +478,10 @@ static int blk_check_request(BlockBackend *blk, int64_t sector_num, return -EIO; } +if (bdrv_op_is_blocked(blk-bs, BLOCK_OP_TYPE_DEVICE_IO, NULL)) { +

Re: [Qemu-block] [PATCH v2 01/11] block: Add op blocker type device IO

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 17:02, Fam Zheng wrote: For example, SCSI requests can result in many consecutive I/Os: (1) FUA requests are split in write+flush (2) adapters that do not use QEMUSGList-based I/O only read 128K at a time (3) WRITE SAME operations are also split in chunks (4)

Re: [Qemu-block] [PATCH v2 10/11] blockdev: Block device IO during blockdev-backup transaction

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 16:55, Fam Zheng wrote: So if we consider this idea worthwhile, and decide that pausing device I/O on the target should pause the block job, the backup job actually has to prevent *adding a DEVICE_IO blocker* on the target. When do we need to pause device IO on the target?

Re: [Qemu-block] [PATCH 27/34] block: Add infrastructure for option inheritance

2015-05-13 Thread Kevin Wolf
Am 13.05.2015 um 17:10 hat Max Reitz geschrieben: On 08.05.2015 19:21, Kevin Wolf wrote: Options are not actually inherited from the parent node yet, but this commit lays the grounds for doing so. Signed-off-by: Kevin Wolf kw...@redhat.com --- block.c | 51

[Qemu-block] [PATCH] block: Let bdrv_drain_all() to call aio_poll() for each AioContext

2015-05-13 Thread Alexander Yarygin
After the commit 9b536adc (block: acquire AioContext in bdrv_drain_all()) the aio_poll() function got called for every BlockDriverState, in assumption that every device may have its own AioContext. The bdrv_drain_all() function is called in each virtio_reset() call, which in turn is called for

Re: [Qemu-block] [PATCH] block: Let bdrv_drain_all() to call aio_poll() for each AioContext

2015-05-13 Thread Alexander Yarygin
Paolo Bonzini pbonz...@redhat.com writes: On 13/05/2015 17:18, Alexander Yarygin wrote: After the commit 9b536adc (block: acquire AioContext in bdrv_drain_all()) the aio_poll() function got called for every BlockDriverState, in assumption that every device may have its own AioContext. The

Re: [Qemu-block] [PATCH v2 11/11] block: Block device IO during bdrv_drain and bdrv_drain_all

2015-05-13 Thread Paolo Bonzini
On 13/05/2015 17:17, Fam Zheng wrote: It can be the topic of a separate series. But this patch brings a false sense of security (either the blocker is unnecessary, or it needs to last after bdrv_drain returns), so I think it should be dropped. Doesn't this let bdrv_drain_all return

Re: [Qemu-block] [Qemu-devel] [PATCH v2 10/11] blockdev: Block device IO during blockdev-backup transaction

2015-05-13 Thread Wen Congyang
On 05/13/2015 08:55 PM, Fam Zheng wrote: On Wed, 05/13 19:22, Wen Congyang wrote: On 05/14/2015 01:28 AM, Fam Zheng wrote: Signed-off-by: Fam Zheng f...@redhat.com --- blockdev.c | 10 +- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/blockdev.c b/blockdev.c index

Re: [Qemu-block] [Qemu-devel] [PATCH v2 10/11] blockdev: Block device IO during blockdev-backup transaction

2015-05-13 Thread Fam Zheng
On Thu, 05/14 09:12, Wen Congyang wrote: We will use bdrv_op_block_all() in the job, and don't unblock BLOCK_OP_TYPE_DEVICE_IO. Is it OK? Good question and you're right, it's broken in this series. I will fix it. Fam

Re: [Qemu-block] [Qemu-devel] [PATCH 14/34] qcow2: Factor out qcow2_update_options()

2015-05-13 Thread Eric Blake
On 05/13/2015 03:11 AM, Kevin Wolf wrote: Am 12.05.2015 um 22:04 hat Eric Blake geschrieben: On 05/08/2015 11:21 AM, Kevin Wolf wrote: Eventually we want to be able to change options at runtime. As a first step towards that goal, separate some option handling code from the general

Re: [Qemu-block] [PATCH v5 0/2] block: enforce minimal 4096 alignment in qemu_blockalign

2015-05-13 Thread Denis V. Lunev
On 13/05/15 18:43, Stefan Hajnoczi wrote: On Tue, May 12, 2015 at 12:46:57PM +0200, Paolo Bonzini wrote: On 12/05/2015 12:19, Denis V. Lunev wrote: hades /vol $ strace -f -e pwrite -e raw=write,pwrite qemu-io -n -c write -P 0x11 0 64M ./1.img Process 19326 attached [pid 19326] pwrite(0x6,

Re: [Qemu-block] [PATCH] block: Let bdrv_drain_all() to call aio_poll() for each AioContext

2015-05-13 Thread Alexander Yarygin
Alberto Garcia be...@igalia.com writes: On Wed 13 May 2015 05:18:31 PM CEST, Alexander Yarygin yary...@linux.vnet.ibm.com wrote: +if (!aio_ctxs || !g_list_find(aio_ctxs, aio_context)) { +busy |= aio_poll(aio_context, busy); +aio_ctxs =

Re: [Qemu-block] [Qemu-devel] [PATCH] block/mirror: Sleep periodically during bitmap scanning

2015-05-13 Thread Wen Congyang
On 05/13/2015 11:11 AM, Fam Zheng wrote: Before, we only yield after initializing dirty bitmap, where the QMP command would return. That may take very long, and guest IO will be blocked. Add sleep points like the later mirror iterations. Signed-off-by: Fam Zheng f...@redhat.com

Re: [Qemu-block] [Qemu-devel] [PATCH] block/mirror: Sleep periodically during bitmap scanning

2015-05-13 Thread Fam Zheng
On Wed, 05/13 13:17, Wen Congyang wrote: On 05/13/2015 11:11 AM, Fam Zheng wrote: Before, we only yield after initializing dirty bitmap, where the QMP command would return. That may take very long, and guest IO will be blocked. Do you have such case to reproduce it? If the disk image is

Re: [Qemu-block] [Qemu-devel] [PATCH 4/5] qemu-io: prompt for encryption keys when required

2015-05-13 Thread Daniel P. Berrange
On Tue, May 12, 2015 at 12:32:53PM -0600, Eric Blake wrote: On 05/12/2015 10:09 AM, Daniel P. Berrange wrote: The qemu-io tool does not check if the image is encrypted so historically would silently corrupt the sectors by writing plain text data into them instead of cipher text. The earlier