Re: [Qemu-block] [PATCH 4/6] block: Support streaming to an intermediate layer

2015-04-17 Thread Max Reitz
On 16.04.2015 16:30, Alberto Garcia wrote: On Wed 15 Apr 2015 06:09:18 PM CEST, Max Reitz mre...@redhat.com wrote: +orig_bs_flags = bdrv_get_flags(bs); +if (!(orig_bs_flags BDRV_O_RDWR)) { I feel like we don't want to do this if we're not streaming to an intermediate layer but to the

Re: [Qemu-block] [PATCH v2 01/11] qapi: Add transaction support to block-dirty-bitmap operations

2015-04-17 Thread Max Reitz
On 27.03.2015 20:19, John Snow wrote: This adds two qmp commands to transactions. block-dirty-bitmap-add allows you to create a bitmap simultaneously alongside a new full backup to accomplish a clean synchronization point. block-dirty-bitmap-clear allows you to reset a bitmap back to as-if it

Re: [Qemu-block] [PATCH v5 20/21] iotests: add incremental backup failure recovery test

2015-04-17 Thread Max Reitz
On 09.04.2015 00:20, John Snow wrote: Test the failure case for incremental backups. Signed-off-by: John Snow js...@redhat.com --- tests/qemu-iotests/124 | 57 ++ tests/qemu-iotests/124.out | 4 ++-- 2 files changed, 59 insertions(+), 2

Re: [Qemu-block] [PATCH v5 21/21] iotests: add incremental backup granularity tests

2015-04-17 Thread Max Reitz
On 09.04.2015 00:20, John Snow wrote: Test what happens if you fiddle with the granularity. Signed-off-by: John Snow js...@redhat.com --- tests/qemu-iotests/124 | 58 +- tests/qemu-iotests/124.out | 4 ++-- 2 files changed, 49 insertions(+),

Re: [Qemu-block] [PATCH v2 02/11] iotests: add transactional incremental backup test

2015-04-17 Thread Max Reitz
On 27.03.2015 20:19, John Snow wrote: Test simple usage cases for using transactions to create and synchronize incremental backups. Signed-off-by: John Snow js...@redhat.com --- tests/qemu-iotests/124 | 51 ++ tests/qemu-iotests/124.out | 4

Re: [Qemu-block] [PATCH v2 01/11] qapi: Add transaction support to block-dirty-bitmap operations

2015-04-17 Thread Eric Blake
On 03/27/2015 01:19 PM, John Snow wrote: This adds two qmp commands to transactions. block-dirty-bitmap-add allows you to create a bitmap simultaneously alongside a new full backup to accomplish a clean synchronization point. block-dirty-bitmap-clear allows you to reset a bitmap back to

Re: [Qemu-block] [PATCH v5 10/21] qmp: Add support of dirty-bitmap sync mode for drive-backup

2015-04-17 Thread Max Reitz
On 09.04.2015 00:19, John Snow wrote: For dirty-bitmap sync mode, the block job will iterate through the given dirty bitmap to decide if a sector needs backup (backup all the dirty clusters and skip clean ones), just as allocation conditions of top sync mode. Signed-off-by: Fam Zheng

Re: [Qemu-block] [PATCH v5 18/21] iotests: add QMP event waiting queue

2015-04-17 Thread Max Reitz
On 09.04.2015 00:20, John Snow wrote: A filter is added to allow callers to request very specific events to be pulled from the event queue, while leaving undesired events still in the stream. This allows to poll for completion data for multiple asynchronous events in any arbitrary order. A new

Re: [Qemu-block] [PATCH v5 15/21] block: Resize bitmaps on bdrv_truncate

2015-04-17 Thread Max Reitz
On 09.04.2015 00:19, John Snow wrote: Signed-off-by: John Snow js...@redhat.com --- block.c| 18 ++ include/qemu/hbitmap.h | 10 ++ util/hbitmap.c | 48 3 files changed, 76 insertions(+) diff

Re: [Qemu-block] [PATCH v5 04/21] qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove

2015-04-17 Thread Eric Blake
On 04/08/2015 04:19 PM, John Snow wrote: The new command pair is added to manage a user created dirty bitmap. The dirty bitmap's name is mandatory and must be unique for the same device, but different devices can have bitmaps with the same names. The granularity is an optional field. If it

Re: [Qemu-block] [PATCH v5 01/21] docs: incremental backup documentation

2015-04-17 Thread Eric Blake
On 04/08/2015 04:19 PM, John Snow wrote: Reviewed-by: Max Reitz mre...@redhat.com Signed-off-by: John Snow js...@redhat.com --- docs/bitmaps.md | 311 1 file changed, 311 insertions(+) create mode 100644 docs/bitmaps.md diff

Re: [Qemu-block] [PATCH v2 04/11] block: re-add BlkTransactionState

2015-04-17 Thread Max Reitz
On 27.03.2015 20:19, John Snow wrote: Now that the structure formerly known as BlkTransactionState has been renamed to something sensible (BlkActionState), re-introduce an actual BlkTransactionState that actually manages state for the entire Transaction. In the process, convert the old QSIMPLEQ

Re: [Qemu-block] [Qemu-devel] [PATCH v5 10/21] qmp: Add support of dirty-bitmap sync mode for drive-backup

2015-04-17 Thread John Snow
On 04/17/2015 09:17 AM, Max Reitz wrote: On 09.04.2015 00:19, John Snow wrote: For dirty-bitmap sync mode, the block job will iterate through the given dirty bitmap to decide if a sector needs backup (backup all the dirty clusters and skip clean ones), just as allocation conditions of top

Re: [Qemu-block] [PATCH v5 01/21] docs: incremental backup documentation

2015-04-17 Thread John Snow
On 04/17/2015 11:06 AM, Eric Blake wrote: On 04/08/2015 04:19 PM, John Snow wrote: Reviewed-by: Max Reitz mre...@redhat.com Signed-off-by: John Snow js...@redhat.com --- docs/bitmaps.md | 311 1 file changed, 311 insertions(+)

Re: [Qemu-block] [PATCH v2 05/11] block: add transactional callbacks feature

2015-04-17 Thread Max Reitz
On 27.03.2015 20:19, John Snow wrote: The goal here is to add a new method to transactions that allows developers to specify a callback that will get invoked only once all jobs spawned by a transaction are completed, allowing developers the chance to perform actions conditionally pending

Re: [Qemu-block] [PATCH v2 07/11] block: add delayed bitmap successor cleanup

2015-04-17 Thread Max Reitz
On 27.03.2015 20:20, John Snow wrote: Allow bitmap successors to carry reference counts. We can in a later patch use this ability to clean up the dirty bitmap according to both the individual job's success and the success of all jobs in the transaction group. The code for cleaning up a bitmap

Re: [Qemu-block] [PATCH v2 09/11] qmp: Add an implementation wrapper for qmp_drive_backup

2015-04-17 Thread Max Reitz
On 27.03.2015 20:20, John Snow wrote: We'd like to be able to specify the callback given to backup_start manually in the case of transactions, so split apart qmp_drive_backup into an implementation and a wrapper. Switch drive_backup_prepare to use the new wrapper, but don't overload the

Re: [Qemu-block] [PATCH v5 01/21] docs: incremental backup documentation

2015-04-17 Thread Eric Blake
On 04/17/2015 09:50 AM, John Snow wrote: On 04/17/2015 11:06 AM, Eric Blake wrote: On 04/08/2015 04:19 PM, John Snow wrote: Reviewed-by: Max Reitz mre...@redhat.com Signed-off-by: John Snow js...@redhat.com --- docs/bitmaps.md | 311

Re: [Qemu-block] [PATCH v2 06/11] block: add refcount to Job object

2015-04-17 Thread Max Reitz
On 27.03.2015 20:20, John Snow wrote: If we want to get at the job after the life of the job, we'll need a refcount for this object. This may occur for example if we wish to inspect the actions taken by a particular job after a transactional group of jobs runs, and further actions are required.

[Qemu-block] [PATCH v6 06/21] hbitmap: cache array lengths

2015-04-17 Thread John Snow
As a convenience: between incremental backups, bitmap migrations and bitmap persistence we seem to need to recalculate these a lot. Because the lengths are a little bit-twiddly, let's just solidly cache them and be done with it. Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake

[Qemu-block] [PATCH v6 05/21] block: Introduce bdrv_dirty_bitmap_granularity()

2015-04-17 Thread John Snow
This returns the granularity (in bytes) of dirty bitmap, which matches the QMP interface and the existing query interface. Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Eric Blake ebl...@redhat.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com

[Qemu-block] [PATCH v6 11/21] qmp: add block-dirty-bitmap-clear

2015-04-17 Thread John Snow
Add bdrv_clear_dirty_bitmap and a matching QMP command, qmp_block_dirty_bitmap_clear that enables a user to reset the bitmap attached to a drive. This allows us to reset a bitmap in the event of a full drive backup. Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz

[Qemu-block] [PATCH v6 07/21] hbitmap: add hbitmap_merge

2015-04-17 Thread John Snow
We add a bitmap merge operation to assist in error cases where we wish to combine two bitmaps together. This is algorithmically O(bits) provided HBITMAP_LEVELS remains constant. For a full bitmap on a 64bit machine: sum(bits/64^k, k, 0, HBITMAP_LEVELS) ~= 1.01587 * bits We may be able to improve

[Qemu-block] [PATCH v6 16/21] hbitmap: truncate tests

2015-04-17 Thread John Snow
The general approach is to set bits close to the boundaries of where we are truncating and ensure that everything appears to have gone OK. We test growing and shrinking by different amounts: - Less than the granularity - Less than the granularity, but across a boundary - Less than sizeof(unsigned

[Qemu-block] [PATCH v6 19/21] iotests: add simple incremental backup case

2015-04-17 Thread John Snow
Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com --- tests/qemu-iotests/124 | 174 +++-- tests/qemu-iotests/124.out | 4 +- 2 files changed, 172 insertions(+), 6 deletions(-) diff --git a/tests/qemu-iotests/124

[Qemu-block] [PATCH v6 00/21] block: transactionless incremental backup series

2015-04-17 Thread John Snow
It's spring! The winter snow has thawed and so here is a new patch series to enter your life and warm your heart. This patchset enables the in-memory part of the incremental backup feature, without transactional support. Support for transactions was separated into a separate series which is also

[Qemu-block] [PATCH v6 15/21] block: Resize bitmaps on bdrv_truncate

2015-04-17 Thread John Snow
Signed-off-by: John Snow js...@redhat.com --- block.c| 18 ++ include/qemu/hbitmap.h | 10 ++ util/hbitmap.c | 48 3 files changed, 76 insertions(+) diff --git a/block.c b/block.c index

[Qemu-block] [PATCH v6 08/21] block: Add bitmap disabled status

2015-04-17 Thread John Snow
Add a status indicating the enabled/disabled state of the bitmap. A bitmap is by default enabled, but you can lock the bitmap into a read-only state by setting disabled = true. A previous version of this patch added a QMP interface for changing the state of the bitmap, but it has since been

[Qemu-block] [PATCH v6 17/21] iotests: add invalid input incremental backup tests

2015-04-17 Thread John Snow
Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com --- tests/qemu-iotests/124 | 104 + tests/qemu-iotests/124.out | 5 +++ tests/qemu-iotests/group | 1 + 3 files

[Qemu-block] [PATCH v6 21/21] iotests: add incremental backup granularity tests

2015-04-17 Thread John Snow
Test what happens if you fiddle with the granularity. Reviewed-by: Max Reitz mre...@redhat.com Signed-off-by: John Snow js...@redhat.com --- tests/qemu-iotests/124 | 58 +- tests/qemu-iotests/124.out | 4 ++-- 2 files changed, 49 insertions(+), 13

[Qemu-block] [PATCH v6 13/21] block: add BdrvDirtyBitmap documentation

2015-04-17 Thread John Snow
Signed-off-by: John Snow js...@redhat.com Reviewed-by: Max Reitz mre...@redhat.com Reviewed-by: Stefan Hajnoczi stefa...@redhat.com --- block.c | 10 +- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/block.c b/block.c index ca73a6a..30c8568 100644 --- a/block.c +++

[Qemu-block] [PATCH v6 10/21] qmp: Add support of dirty-bitmap sync mode for drive-backup

2015-04-17 Thread John Snow
For dirty-bitmap sync mode, the block job will iterate through the given dirty bitmap to decide if a sector needs backup (backup all the dirty clusters and skip clean ones), just as allocation conditions of top sync mode. Signed-off-by: Fam Zheng f...@redhat.com Signed-off-by: John Snow