Hi all, This series introduces a simple format to enable support of persistence of block dirty bitmaps. Block dirty bitmap is the tool to achieve incremental backup, and persistence of block dirty bitmap makes incrememtal backup possible across VM shutdowns, where existing in-memory dirty bitmaps cannot survive.
When user creates a "persisted" dirty bitmap, the QBM driver will create a binary file and synchronize it with the existing in-memory block dirty bitmap (BdrvDirtyBitmap). When the VM is powered down, the binary file has all the bits saved on disk, which will be loaded and used to initialize the in-memory block dirty bitmap next time the guest is started. The idea of the format is to reuse as much existing infrastructure as possible and avoid introducing complex data structures - it works with any image format, by gluing it together plain bitmap files with a json descriptor file. The advantage of this approach over extending existing formats, such as qcow2, is that the new feature is implemented by an orthogonal driver, in a format agnostic way. This way, even raw images can have their persistent dirty bitmaps. (And you will notice in this series, with a little forging to the spec, raw images can also have backing files through a QBM overlay!) Rather than superseding it, this intends to be coexistent in parallel with the qcow2 bitmap extension that Vladimir is working on. The block driver interface changes in this series also try to be generic and compatible for both drivers. The format's specification is added to docs/specs/, see patch 1. Patches 2-7 are necessary block layer changes in order be friendly to persistent dirty bitmap drivers. Patches 8, 9 and 11 extends the QMP interface to expose the added feature. Patch 10 implements the driver. (todo: checksum extension for image/bitmap integrity check) Patch 12 - 16 are the tests I have for QBM so far. I'm sure more can be added as questions emerge. :) The series applies on top of my "qemu-img map" series and "meta bitmap" series: https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg04866.html https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg03656.html If you feel like to play with it, git branch is also available at: https://github.com/famz/qemu qbm Comments are welcome! Fam Fam Zheng (16): doc: Add QBM format specification block: Set dirty before doing write block: Allow .bdrv_close callback to release dirty bitmaps block: Move filename_decompose to block.c block: Make bdrv_get_cluster_size public block: Introduce bdrv_dirty_bitmap_set_persistent block: Only swap non-persistent dirty bitmaps qmp: Add optional parameter "persistent" in block-dirty-bitmap-add qmp: Add block-dirty-bitmap-set-persistent qbm: Implement format driver qapi: Add "qbm" as a generic cow format driver iotests: Add qbm format to 041 iotests: Add qbm to case 097 iotests: Add qbm to applicable test cases iotests: Add qbm specific test case 140 iotests: Add persistent bitmap test case 141 block.c | 54 +- block/Makefile.objs | 1 + block/dirty-bitmap.c | 63 ++ block/io.c | 6 +- block/qbm.c | 1315 ++++++++++++++++++++++++++++++++++++++++++ block/vmdk.c | 40 -- blockdev.c | 28 +- docs/specs/qbm.md | 118 ++++ include/block/block.h | 4 +- include/block/block_int.h | 8 + include/block/dirty-bitmap.h | 5 + qapi/block-core.json | 31 +- qmp-commands.hx | 34 +- tests/qemu-iotests/004 | 2 +- tests/qemu-iotests/017 | 2 +- tests/qemu-iotests/018 | 2 +- tests/qemu-iotests/019 | 2 +- tests/qemu-iotests/020 | 2 +- tests/qemu-iotests/024 | 2 +- tests/qemu-iotests/025 | 2 +- tests/qemu-iotests/027 | 2 +- tests/qemu-iotests/028 | 2 +- tests/qemu-iotests/030 | 2 +- tests/qemu-iotests/034 | 2 +- tests/qemu-iotests/037 | 2 +- tests/qemu-iotests/038 | 2 +- tests/qemu-iotests/040 | 2 +- tests/qemu-iotests/041 | 18 +- tests/qemu-iotests/050 | 2 +- tests/qemu-iotests/055 | 2 +- tests/qemu-iotests/056 | 2 +- tests/qemu-iotests/069 | 2 +- tests/qemu-iotests/072 | 2 +- tests/qemu-iotests/086 | 2 +- tests/qemu-iotests/095 | 2 +- tests/qemu-iotests/096 | 2 +- tests/qemu-iotests/097 | 4 +- tests/qemu-iotests/099 | 2 +- tests/qemu-iotests/110 | 2 +- tests/qemu-iotests/129 | 2 +- tests/qemu-iotests/132 | 2 +- tests/qemu-iotests/139 | 2 +- tests/qemu-iotests/140 | 80 +++ tests/qemu-iotests/140.out | 145 +++++ tests/qemu-iotests/141 | 62 ++ tests/qemu-iotests/141.out | 5 + tests/qemu-iotests/common | 6 + tests/qemu-iotests/group | 2 + 48 files changed, 1998 insertions(+), 85 deletions(-) create mode 100644 block/qbm.c create mode 100644 docs/specs/qbm.md create mode 100755 tests/qemu-iotests/140 create mode 100644 tests/qemu-iotests/140.out create mode 100644 tests/qemu-iotests/141 create mode 100644 tests/qemu-iotests/141.out -- 2.4.3
