I've run out of cheeky jokes for my cover letters.
This patchset enables the in-memory part of the incremental backup
feature, without transactional support.
Support for transactions was separated into a separate series which
is also now available on-list. Getting this portion of the series
committed will help stabilize work on bitmap persistence and bitmap
migration.
Thanks to Fam Zheng for the original versions of this patchset,
And thanks to Max for reviewing 2,395 versions of it since.
===
v5:
===
10: Code has been moved into backup_run_incremental()
'polyrhythm' check is removed,
clusters_per_iter variable is introduced instead.
If the bitmap granularity is larger than the backup granularity,
loop over the backup_do_cow call multiple times.
If the bitmap granularity is smaller, skip the iterator ahead as
we had been doing previously.
14: Only whitespace changes caused by patch 10.
15: Changed my approach for clearing out data for the hbitmap
truncate shrink case, as suggested by Stefan
18: Added a proper timeout mechanism to qmp.pull_event():
wait=False or wait=0.0 implies non-blocking.
wait=True implies blocking.
wait=60.0 implies a 60 second timeout.
VM.event_wait() now uses a 60 second timeout by default.
19: Many things:
The big picture is to add a set of full backups alongside the
incremental backups created during the test to be able to test
the validity of each incremental at the conclusion of the test
when we can shut the VM down.
To do this, there are two basic changes:
(1) Keep a list of pairs of backup filenames (incremental, reference);
create a full reference backup for every incremental created.
(2) Refactor the backup helper functions a bit.
20: Naming fallout from 19
Added calls to vm.shutdown() and check_backups().
21: NEW, adds granularity tests that cover the changes in patch 10.
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/21:[----] [--] 'docs: incremental backup documentation'
002/21:[----] [--] 'qapi: Add optional field "name" to block dirty bitmap'
003/21:[----] [--] 'qmp: Ensure consistent granularity type'
004/21:[----] [--] 'qmp: Add block-dirty-bitmap-add and
block-dirty-bitmap-remove'
005/21:[----] [--] 'block: Introduce bdrv_dirty_bitmap_granularity()'
006/21:[----] [--] 'hbitmap: cache array lengths'
007/21:[----] [--] 'hbitmap: add hbitmap_merge'
008/21:[----] [--] 'block: Add bitmap disabled status'
009/21:[----] [--] 'block: Add bitmap successors'
010/21:[0092] [FC] 'qmp: Add support of "dirty-bitmap" sync mode for
drive-backup'
011/21:[----] [--] 'qmp: add block-dirty-bitmap-clear'
012/21:[----] [--] 'qmp: Add dirty bitmap status field in query-block'
013/21:[----] [--] 'block: add BdrvDirtyBitmap documentation'
014/21:[0004] [FC] 'block: Ensure consistent bitmap function prototypes'
015/21:[0012] [FC] 'block: Resize bitmaps on bdrv_truncate'
016/21:[----] [--] 'hbitmap: truncate tests'
017/21:[----] [--] 'iotests: add invalid input incremental backup tests'
018/21:[0014] [FC] 'iotests: add QMP event waiting queue'
019/21:[0141] [FC] 'iotests: add simple incremental backup case'
020/21:[0005] [FC] 'iotests: add incremental backup failure recovery test'
021/21:[down] 'iotests: add incremental backup granularity tests'
===
v4:
===
04: Some in-line documentation for block_dirty_bitmap_lookup
Changed behavior with respect to aio_context
(always acquire, release if pbs == null, give to user otherwise)
10: Removed vestigial (currently nop) bdrv_enable_dirty_bitmap call
Kept R-B.
16: Added some comments to test_check_boundary_bits.
Kept R-B.
17: Folded in refactor from "incremental transactions v1" (Poor Kitty)
18: Pulled forward from "incremental transactions v1"
Kept R-B from that series.
19: Folded in refactor from "incremental transactions v1"
Added offset assertions into wait_incremental
20: Removed error tolerance from wait_until_completed, as
these patches no longer make use of that feature.
===
v3:
===
01: Removed enabled/disabled modes information.
Elaborated on events that can occur during error cases.
04: Added an AioContext out parameter to block_dirty_bitmap_lookup.
06: NEW:
Cache the array lengths for hbitmap.
07: hbitmap_merge now uses the cached array lengths.
11: block-dirty-bitmap-clear is edited for the new block_dirty_bitmap_lookup.
12: Removed the "disabled" status, leaving just "Frozen."
15: Moved bdrv_truncate_dirty_bitmap to be static local
Inlined dirty_bitmap_truncate function.
Removed size[] caching into new patch (06, above)
hbitmap_truncate now keeps correct bit population count
hbitmap_truncate now uses hbitmap_reset BEFORE the truncate,
to avoid tricky out-of-bounds usages.
Remove g_realloc_n call that is not available in glib 2.12 (or 2.22)
Renamed "truncate" to "shrink" to make that more clear
to people who aren't me (at last count: 7+ billion)
16 NEW:
hbitmap_truncate tests.
===
v2:
===
01: Added a new opening blurb.
Adjusted codeblock indentations to be 4 spaces instead of 3,
so it works as MD or GFMD.
Adjusted errors explanation.
Make visual separations between json data and shell commands
Eliminate any ligering single quotes
07: Remember that memset takes bytes, not n_items ...
===
v1:
===
Deletions:
- Removed Transactions, to be added later.
- Removed Transaction tests, as above.
Changes:
01: Indentation fixes.
Removed enable/disable documentation.
Added a note that transactions aren't implemented yet.
Removed my needless commas
Added error case documentation.
07: QMP enable/disable commands are deleted.
14: Some comments concerning assertions.
Scrub re-alloc memory if we expand the array.
Do not attempt to scrub memory if fix_count is 0
Changes made with Reviews kept:
02: Since 2.4
04: Since 2.4
Demingled the QMP command documentation.
08: Additions to what was qmp_block_dirty_enable/disable
are no longer present as those function no longer exist.
09: Since 2.4
10: Since 2.4
Demingled QMP command documentation.
11: Since 2.4
15: Test 112 --> 124
17: Number of tests altered. (Only 4, now.)
Fam Zheng (1):
qapi: Add optional field "name" to block dirty bitmap
John Snow (20):
docs: incremental backup documentation
qmp: Ensure consistent granularity type
qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
block: Introduce bdrv_dirty_bitmap_granularity()
hbitmap: cache array lengths
hbitmap: add hbitmap_merge
block: Add bitmap disabled status
block: Add bitmap successors
qmp: Add support of "dirty-bitmap" sync mode for drive-backup
qmp: add block-dirty-bitmap-clear
qmp: Add dirty bitmap status field in query-block
block: add BdrvDirtyBitmap documentation
block: Ensure consistent bitmap function prototypes
block: Resize bitmaps on bdrv_truncate
hbitmap: truncate tests
iotests: add invalid input incremental backup tests
iotests: add QMP event waiting queue
iotests: add simple incremental backup case
iotests: add incremental backup failure recovery test
iotests: add incremental backup granularity tests
block.c | 243 ++++++++++++++++++++++++++--
block/backup.c | 156 +++++++++++++++---
block/mirror.c | 46 +++---
blockdev.c | 176 +++++++++++++++++++-
docs/bitmaps.md | 311 ++++++++++++++++++++++++++++++++++++
hmp.c | 3 +-
include/block/block.h | 33 +++-
include/block/block_int.h | 4 +-
include/qemu/hbitmap.h | 21 +++
migration/block.c | 9 +-
qapi/block-core.json | 90 ++++++++++-
qmp-commands.hx | 92 ++++++++++-
scripts/qmp/qmp.py | 6 +-
tests/qemu-iotests/124 | 363 ++++++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/124.out | 5 +
tests/qemu-iotests/group | 1 +
tests/qemu-iotests/iotests.py | 38 +++++
tests/test-hbitmap.c | 255 +++++++++++++++++++++++++++++
util/hbitmap.c | 81 ++++++++++
19 files changed, 1845 insertions(+), 88 deletions(-)
create mode 100644 docs/bitmaps.md
create mode 100644 tests/qemu-iotests/124
create mode 100644 tests/qemu-iotests/124.out
--
2.1.0
