Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series
(2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? - The name of the bitmap and also the size of this name (5) Partial Persistence We did not discuss only saving higher levels of the bitmap. What's the primary benefit you're seeking? Hmm. It may be used for faster sync. Maybe, save some of bitmap levels on timer while vm is running and save the last level on shutdown? CC qemu-devel - ok. Best regards, Vladimir On 18.11.2014 02:46, John Snow wrote: On 11/13/2014 08:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi I'd just like to start working on persistent dirty bitmap. My thoughts about it are the following: - qemu -drive file=file,dirty_bitmap=file so, bitmap will be loaded with drive open and saved with drive close. - save only meaningful (the last) level of the bitmap, restore all levels on bitmap loading - bool parameter persistent for bdrv_create_dirty_bitmap and BdrvDirtyBitmap - internal dirty_bitmaps, saved in qcow2 file Best regards, Vladimir I am thinking: (1) Command Lines If you enable dirty bitmaps and give it a file that doesn't exist, it should error out on you. If you enable dirty bitmaps and give it a file that's blank, it understands that it is to create a persistent bitmap file in this location and it should enable persistence. If a bitmap file is given and it has valid magic, this should imply persistence. I am hesitant to have it auto-create files that don't already exist in case the files become large in size and a misconfiguration leads to repeated creation of these files that get orphaned in random folders. Perhaps we can add a create=auto flag or similar to allow this behavior if wanted. (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At some point, based on a sync policy, re-commit the dirty information to the file and mark the file as clean once more and re-take the persistence lock. (4) Synchronization Policy - Sync after so many bits become dirty in the bitmap, either as an absolute threshold or a density percentage? - Sync periodically on a fixed timer? - Sync periodically opportunistically when I/O utilization becomes relatively low? (With some sort of starvation prevention timer?) - Sync only at shutdown? In discussing with Stefan, I think we rather liked the idea of a timer that tries to re-commit the block data during lulls in the I/O. (5) Partial Persistence We did not discuss only saving higher levels of the bitmap. What's the primary benefit you're seeking? (6) Inclusion as qcow2 Metadata And lastly, we did discuss the inclusion of the bitmap as qcow2 metadata, but decided it wasn't our principle target for the format to allow generality to other file formats. We didn't really discuss the idea of having it as an option or an extension, but I don't (off the top of my head) have any reasonings against it, but I will likely not work on it myself. You didn't CC qemu-devel on this (so I won't!), but perhaps we should re-send out our ideas to the wider list for feedback before we proceed any further. Maybe we can split the work if we agree upon a design. Thanks! --js P.S.: I'm still cleaning up Fam's first patchset based on Max's and your feedback. Hope to have it out by the end of this week. On 11.11.2014 18:59, John Snow wrote: On 11/10/2014 03:15 AM, Vladimir Sementsov-Ogievskiy wrote: Hi Fam, hi Jorn. Jagane's project - http://wiki.qemu.org/Features/Livebackup In two words: Normal delta - like in qemu, while backuping, we save all new writes to separate virtual disk - delta. When backup is done, we can merge delta back to original image. Reverse delta - while backuping, we don't stop writing to original image (and qemu works with it, not with delta), but before every write
Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series
(3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At some point, based on a sync policy, re-commit the dirty information to the file and mark the file as clean once more and re-take the persistence lock. Correct me if I'm wrong. #Read bitmap: read in blockdev_init, before any write to device, so no lock is needed. #Set bits in bitmap: if bitmap.dirty_flag: set bits else: LOCK set bits set bitmap.dirty_flag set dirty_flag in bitmap file UNLOCK #Sync: if not bitmap.dirty_flag: skip sync else: LOCK save one of bitmap levels (saving the last one is too long and not very good idea, because it is fast-updateing) unset dirty_flag in bitmap file unset bitmap.dirty_flag UNLOCK #Last sync in bdrv_close: Just save the last bitmap level and unset dirty_flag in bitmap file Also.. I'm not quite sure about locking.. As I understand, co-routines in qemu are not running in parallel, is locking required? Or sync timer will not be co-routine based? Best regards, Vladimir On 18.11.2014 13:54, Vladimir Sementsov-Ogievskiy wrote: (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? - The name of the bitmap and also the size of this name (5) Partial Persistence We did not discuss only saving higher levels of the bitmap. What's the primary benefit you're seeking? Hmm. It may be used for faster sync. Maybe, save some of bitmap levels on timer while vm is running and save the last level on shutdown? CC qemu-devel - ok. Best regards, Vladimir On 18.11.2014 02:46, John Snow wrote: On 11/13/2014 08:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi I'd just like to start working on persistent dirty bitmap. My thoughts about it are the following: - qemu -drive file=file,dirty_bitmap=file so, bitmap will be loaded with drive open and saved with drive close. - save only meaningful (the last) level of the bitmap, restore all levels on bitmap loading - bool parameter persistent for bdrv_create_dirty_bitmap and BdrvDirtyBitmap - internal dirty_bitmaps, saved in qcow2 file Best regards, Vladimir I am thinking: (1) Command Lines If you enable dirty bitmaps and give it a file that doesn't exist, it should error out on you. If you enable dirty bitmaps and give it a file that's blank, it understands that it is to create a persistent bitmap file in this location and it should enable persistence. If a bitmap file is given and it has valid magic, this should imply persistence. I am hesitant to have it auto-create files that don't already exist in case the files become large in size and a misconfiguration leads to repeated creation of these files that get orphaned in random folders. Perhaps we can add a create=auto flag or similar to allow this behavior if wanted. (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At some point, based on a sync policy, re-commit the dirty information to the file and mark the file as clean once more and re-take the persistence lock. (4) Synchronization Policy - Sync after so many bits become dirty in the bitmap, either as an absolute threshold or a density percentage? - Sync periodically on a fixed timer? - Sync periodically opportunistically when I/O utilization becomes
Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series
Also, if we sync not the last level, bitmap.dirty_flag should be related to syncing level, not to the whole bitmap, to reduce sync overhead. Or, if we implement difficult sync policy, there should be dirty flags for each bitmap level. Despite this, only one level is needed to be saved in the bitmap file. PS: more ideas about file format - thanks to Denis V. Lunev d...@parallels.com 1) Shouldn't we consider a possibility of storing several bitmaps in one file? Or one bitmap = one file? 2) Implement header extensions like in qcow2. Best regards, Vladimir On 18.11.2014 16:09, Vladimir Sementsov-Ogievskiy wrote: (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At some point, based on a sync policy, re-commit the dirty information to the file and mark the file as clean once more and re-take the persistence lock. Correct me if I'm wrong. #Read bitmap: read in blockdev_init, before any write to device, so no lock is needed. #Set bits in bitmap: if bitmap.dirty_flag: set bits else: LOCK set bits set bitmap.dirty_flag set dirty_flag in bitmap file UNLOCK #Sync: if not bitmap.dirty_flag: skip sync else: LOCK save one of bitmap levels (saving the last one is too long and not very good idea, because it is fast-updateing) unset dirty_flag in bitmap file unset bitmap.dirty_flag UNLOCK #Last sync in bdrv_close: Just save the last bitmap level and unset dirty_flag in bitmap file Also.. I'm not quite sure about locking.. As I understand, co-routines in qemu are not running in parallel, is locking required? Or sync timer will not be co-routine based? Best regards, Vladimir On 18.11.2014 13:54, Vladimir Sementsov-Ogievskiy wrote: (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? - The name of the bitmap and also the size of this name (5) Partial Persistence We did not discuss only saving higher levels of the bitmap. What's the primary benefit you're seeking? Hmm. It may be used for faster sync. Maybe, save some of bitmap levels on timer while vm is running and save the last level on shutdown? CC qemu-devel - ok. Best regards, Vladimir On 18.11.2014 02:46, John Snow wrote: On 11/13/2014 08:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi I'd just like to start working on persistent dirty bitmap. My thoughts about it are the following: - qemu -drive file=file,dirty_bitmap=file so, bitmap will be loaded with drive open and saved with drive close. - save only meaningful (the last) level of the bitmap, restore all levels on bitmap loading - bool parameter persistent for bdrv_create_dirty_bitmap and BdrvDirtyBitmap - internal dirty_bitmaps, saved in qcow2 file Best regards, Vladimir I am thinking: (1) Command Lines If you enable dirty bitmaps and give it a file that doesn't exist, it should error out on you. If you enable dirty bitmaps and give it a file that's blank, it understands that it is to create a persistent bitmap file in this location and it should enable persistence. If a bitmap file is given and it has valid magic, this should imply persistence. I am hesitant to have it auto-create files that don't already exist in case the files become large in size and a misconfiguration leads to repeated creation of these files that get orphaned in random folders. Perhaps we can add a create=auto flag or similar to allow this behavior if wanted. (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent
Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series
On 11/18/2014 08:09 AM, Vladimir Sementsov-Ogievskiy wrote: (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At some point, based on a sync policy, re-commit the dirty information to the file and mark the file as clean once more and re-take the persistence lock. Correct me if I'm wrong. #Read bitmap: read in blockdev_init, before any write to device, so no lock is needed. #Set bits in bitmap: if bitmap.dirty_flag: set bits else: LOCK set bits set bitmap.dirty_flag set dirty_flag in bitmap file UNLOCK #Sync: if not bitmap.dirty_flag: skip sync else: LOCK save one of bitmap levels (saving the last one is too long and not very good idea, because it is fast-updateing) unset dirty_flag in bitmap file unset bitmap.dirty_flag UNLOCK #Last sync in bdrv_close: Just save the last bitmap level and unset dirty_flag in bitmap file Also.. I'm not quite sure about locking.. As I understand, co-routines in qemu are not running in parallel, is locking required? Or sync timer will not be co-routine based? Best regards, Vladimir Might be being too informal. I just meant a lock or barrier to prevent actual IO throughput until we can confirm the dirty flag has been adjusted to indicate that the persistent bitmap is now officially out of date. Nothing fancy. Wasn't trying to imply that we needed threading protection, just locking the IO until we can configure the bitmap as we need it to be. On 18.11.2014 13:54, Vladimir Sementsov-Ogievskiy wrote: (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? - The name of the bitmap and also the size of this name (5) Partial Persistence We did not discuss only saving higher levels of the bitmap. What's the primary benefit you're seeking? Hmm. It may be used for faster sync. Maybe, save some of bitmap levels on timer while vm is running and save the last level on shutdown? CC qemu-devel - ok. Best regards, Vladimir On 18.11.2014 02:46, John Snow wrote: On 11/13/2014 08:54 AM, Vladimir Sementsov-Ogievskiy wrote: Hi I'd just like to start working on persistent dirty bitmap. My thoughts about it are the following: - qemu -drive file=file,dirty_bitmap=file so, bitmap will be loaded with drive open and saved with drive close. - save only meaningful (the last) level of the bitmap, restore all levels on bitmap loading - bool parameter persistent for bdrv_create_dirty_bitmap and BdrvDirtyBitmap - internal dirty_bitmaps, saved in qcow2 file Best regards, Vladimir I am thinking: (1) Command Lines If you enable dirty bitmaps and give it a file that doesn't exist, it should error out on you. If you enable dirty bitmaps and give it a file that's blank, it understands that it is to create a persistent bitmap file in this location and it should enable persistence. If a bitmap file is given and it has valid magic, this should imply persistence. I am hesitant to have it auto-create files that don't already exist in case the files become large in size and a misconfiguration leads to repeated creation of these files that get orphaned in random folders. Perhaps we can add a create=auto flag or similar to allow this behavior if wanted. (2) File Format Some standard file magic, which includes: - Some magic byte(s) - Dirty flag. Needed to tell if we can trust this data or not. - The size of the bitmap - The granularity of the bitmap - The offset to the first sector of bitmap data (Maybe? It can't hurt if we give ourselves a sector's worth to write metadata within.) - Data starting at... PAGESIZE? (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At
Re: [Qemu-devel] [PATCH v6 00/10] block: Incremental backup series
On 18/11/14 19:08, John Snow wrote: On 11/18/2014 08:09 AM, Vladimir Sementsov-Ogievskiy wrote: (3) Data Integrity The dirty flag could work something like: - If, on first open, the file has the dirty flag set, we need to discard the bitmap data because we can no longer trust it. - If the bitmap file is clean, proceed as normal, but take a lock against any of the bitmap functions to prevent them from marking any bits dirty. - On first write to a clean persistent bitmap, delay the write until we can mark the bitmap as dirty first. This incurs a write penalty when we try to use the bitmap at first... - Unlock the bitmap functions and allow them to mark blocks as needed. - At some point, based on a sync policy, re-commit the dirty information to the file and mark the file as clean once more and re-take the persistence lock. Correct me if I'm wrong. - first of all what we are protecting against? Any QEMU or kernel crash leads to the disaster. You can not guarantee at all the consistency between data written to the main file (disk) and data written to bitmap except if you wait that dirty bitmap is really updated. In this case you will have performance halved in comparison with the host (each guest write means 2 IOPS instead of 1) - this effectively means that we can not be protected against crash - if we don't we could not care at all about bitmap updates when QEMU is running. We should write it only on stop/suspend/etc. This simplifies things a lot. So, the procedure is simple - stop main VM by using vm_pause - open/create bitmap file - set dirty - unpause That's all. If QEMU is running, bitmap is always dirty. The only exception is backup creation. New backup means that old bitmap is synced with dirty clear, new one is created with dirty set and backup starts working with old bitmap. Once again. In all other cases we can not guarantee that we will report _all_ changed blocks to backup software if crash happens in the middle. One missed block in the bitmap and the entire backup is blown up. This also means that (4) aka sync policy is simple. Do this on close. #Read bitmap: read in blockdev_init, before any write to device, so no lock is needed. #Set bits in bitmap: if bitmap.dirty_flag: set bits else: LOCK set bits set bitmap.dirty_flag set dirty_flag in bitmap file UNLOCK #Sync: if not bitmap.dirty_flag: skip sync else: LOCK save one of bitmap levels (saving the last one is too long and not very good idea, because it is fast-updateing) unset dirty_flag in bitmap file unset bitmap.dirty_flag UNLOCK #Last sync in bdrv_close: Just save the last bitmap level and unset dirty_flag in bitmap file Also.. I'm not quite sure about locking.. As I understand, co-routines in qemu are not running in parallel, is locking required? Or sync timer will not be co-routine based? Best regards, Vladimir Might be being too informal. I just meant a lock or barrier to prevent actual IO throughput until we can confirm the dirty flag has been adjusted to indicate that the persistent bitmap is now officially out of date. Nothing fancy. Wasn't trying to imply that we needed threading protection, just locking the IO until we can configure the bitmap as we need it to be.
[Qemu-devel] [PATCH v6 00/10] block: Incremental backup series
v6: Rebased v4 of the series on top of qemu.git. (skipping v5 since it was used
by me as a private sending, for those who received it, the code is the same
:)
This is the in memory part of the incremental backup feature.
With the added commands, we can create a bitmap on a block backend, from which
point of time all the writes are tracked by the bitmap, marking sectors as
dirty. Later, we call drive-backup and pass the bitmap to it, to do an
incremental backup.
See the last patch which adds some tests for this use case.
Fam
Fam Zheng (10):
qapi: Add optional field name to block dirty bitmap
qmp: Add block-dirty-bitmap-add and block-dirty-bitmap-remove
block: Introduce bdrv_dirty_bitmap_granularity()
hbitmap: Add hbitmap_copy
block: Add bdrv_copy_dirty_bitmap and bdrv_reset_dirty_bitmap
qmp: Add block-dirty-bitmap-enable and block-dirty-bitmap-disable
qmp: Add support of dirty-bitmap sync mode for drive-backup
qapi: Add transaction support to
block-dirty-bitmap-{add,enable,disable}
qmp: Add dirty bitmap 'enabled' field in query-block
qemu-iotests: Add tests for drive-backup sync=dirty-bitmap
block-migration.c | 2 +-
block.c | 82 -
block/backup.c| 54 +++-
block/mirror.c| 6 +-
blockdev.c| 198 +-
hmp.c | 4 +-
include/block/block.h | 15 +++-
include/block/block_int.h | 4 +
include/qemu/hbitmap.h| 8 ++
qapi-schema.json | 5 +-
qapi/block-core.json | 120 +++--
qmp-commands.hx | 66 +-
tests/qemu-iotests/056| 33 ++-
tests/qemu-iotests/056.out| 4 +-
tests/qemu-iotests/iotests.py | 8 ++
util/hbitmap.c| 16
16 files changed, 603 insertions(+), 22 deletions(-)
--
1.9.3
