Re: [libvirt] [PATCH v4 00/20] Incremental Backup API additions
On 2/18/19 10:34 AM, Peter Krempa wrote: >>> Among other things, I still haven't determined how we want to >>> integrate checkpoints with external snapshots; we could either have: >>> >>> virDomainSnapshotCreateXML("...") # existing >>> virDomainBackupBegin("...", "...") # this >>> series >>> virDomainSnapshotCheckpointCreateXML("...", >>> "...") # new >> >> A slightly related question: when these new APIs (thanks for working on >> them!) are merged, am I right in assuming that they should be able to >> "replace" the existing (and provide additional): >> >> virDomainBlockRebase(); and >> virDomainBlockCopy() >> >> ... _provided_ that an application is adjusted to using libvirt that is >> new enough to drive QEMU's '-blockdev', QMP `blockdev-add` et al? >> >> Or is that (the new APIs being backward-compatible with blockRebase() >> and blockCopy()) an explicit non-goal? >> >> I'm only this out of curiosity. > > The checkpoints are really orthogonal to the backing chain/shapshot > managemet. Indeed. Think of virDomainBlockRebase() as syntactic sugar (it really is a wrapper around either virDomainBlockPull() streaming operation, or around virDomainBlockRebase() for a mirror operation). Given your question grouping it with virDomainBlockCopy(), I'm assuming you are only asking about the latter. The biggest difference between virDomainBlockCopy() and virDomainBackupBegin() is point-in-time: with mirroring, you start the job up front, but you do not have a valid backup image until you cancel the job; but since both files have the same content (once the job has hit the sync phase), cancellation gives you a choice of whether to stay with the old file (the mirror image is the backup) or to pivot to the new file (the original file is now the backup). On the other hand, with virDomainBackupBegin(), you select the point-in-time at the point where you start the job, and the backup copy is created independently of the running image so there is no pivoting possible. virDomainBackupBegin() will work even without checkpoints (in which case, it is a full image backup); but their main power is that when used WITH checkpoints, the backup operation can be done with much less effort than a full copy. Ideally, we want external snapshots to also be points where checkpoints can be created; John Snow and I have had some ideas about what is needed, but our focus is first getting the API working without worrying about external snapshots, while ensuring that the XML has enough flexibility to add in those improvements later without needing more API. Another difference: virDomainBlockCopy() operates on only one block device at a time (you have to issue multiple calls if your guest has multiple disks, although the calls can at least be run in parallel). But virDomainBackupBegin() operates on the entire domain at once, with your choice of granularity on which disk(s) to have involved (similar to how virDomainSnapshotCreateXML() lets you choose which disks to snapshot). > > Checkpoints don't really store any data but rather provide a way to > determine which blocks were changed and thus need backup. Also one point > is that checkpoints don't allow (or I did not notice it in the proposal) > to capture memory state along with the disk state. Correct - checkpoints ONLY track which portions of the disk have changed. In fact, when taking a differential backup, you can really only request all changes occurring between one point in the past and the present. In particular, if you have Check1 and Check2, you can request the backup of all sectors touched between Check1 and the present, or all sectors touched between Check2 and the present, but you cannot request the backup of only the sectors touched between Check1 and Check2 (at least not through the libvirt API), because those sectors may have changed again between Check2 and the present, and the bitmaps only requires which sectors have changed, and NOT what the data was at the time of Check2's creation. > In the ideal world of snapshots when deletion/revertion was implemented > we'd never expose the virDomainBlockCommit and virDomainBlockPull APIs > including the multi-use backdoor virDomainBlockRebase() which should > have never existed and users would do equivalent operations using the > snapshot APIs. > > virDomainBlockCopy is useful on it's own though but badly combines with > snapshots. This will need some fixing. Indeed, and that's true regardless of whether the backup API goes in (although the backup API probably compounds the issue on the number of corner cases we have to think about; the conservative approach is that at least in the beginning, you won't be able to run a BlockCopy and a BackupBegin job at the same time). > > With the new checkpoint APIs the situation is even more "fun" as > modification of the backing chain involves in some cases changes to the > bitmaps. Ideally these would do "the right thing (TM)" during snapshot > deletion/reversion.
Re: [libvirt] [PATCH v4 00/20] Incremental Backup API additions
On Mon, Feb 18, 2019 at 17:16:41 +0100, Kashyap Chamarthy wrote: > On Wed, Feb 06, 2019 at 01:17:58PM -0600, Eric Blake wrote: > > The following is the latest version of my API proposal for > > incremental backups, and follows along from the demo I presented > > as part of my KVM Forum 2018 talk: > > https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat > > > > The patches are also available via https://repo.or.cz/libvirt/ericb.git > > at the tag backup-v4. > > > > Integration with external snapshots is still not included in these > > patches, although I have been playing more with that locally, and > > I still haven't gotten a working demonstration of a push-mode > > incremental backup. > > > > At this point, I'm posting mainly because there have been enough > > changes in qemu (the backup APIs are now stable! and in qemu.git > > in time for 4.0) and libvirt (several releases and code cleanups > > in between, such that the v3 no longer applies easily), while I > > continue to work locally on more features, addressing the review > > comments I got on v3, and remove the 'wip' tag on several of the > > later patches. > > > > Among other things, I still haven't determined how we want to > > integrate checkpoints with external snapshots; we could either have: > > > > virDomainSnapshotCreateXML("...") # existing > > virDomainBackupBegin("...", "...") # this > > series > > virDomainSnapshotCheckpointCreateXML("...", > > "...") # new > > A slightly related question: when these new APIs (thanks for working on > them!) are merged, am I right in assuming that they should be able to > "replace" the existing (and provide additional): > > virDomainBlockRebase(); and > virDomainBlockCopy() > > ... _provided_ that an application is adjusted to using libvirt that is > new enough to drive QEMU's '-blockdev', QMP `blockdev-add` et al? > > Or is that (the new APIs being backward-compatible with blockRebase() > and blockCopy()) an explicit non-goal? > > I'm only this out of curiosity. The checkpoints are really orthogonal to the backing chain/shapshot managemet. Checkpoints don't really store any data but rather provide a way to determine which blocks were changed and thus need backup. Also one point is that checkpoints don't allow (or I did not notice it in the proposal) to capture memory state along with the disk state. In the ideal world of snapshots when deletion/revertion was implemented we'd never expose the virDomainBlockCommit and virDomainBlockPull APIs including the multi-use backdoor virDomainBlockRebase() which should have never existed and users would do equivalent operations using the snapshot APIs. virDomainBlockCopy is useful on it's own though but badly combines with snapshots. This will need some fixing. With the new checkpoint APIs the situation is even more "fun" as modification of the backing chain involves in some cases changes to the bitmaps. Ideally these would do "the right thing (TM)" during snapshot deletion/reversion. Given that at this time we don't support snapshot deletion/reversion for external snapshots we can use the excuse that snapshot management is not implemented so that checkpoints don't need to be modified. Since virDomainBlockCommit/virDomainBlockPull are basically a backdoor to do snapshot merging which is not actually recorded in the snapshot XML the same damage can happen to the checkpoints. I'm not sure how we want to deal with that at this point though. signature.asc Description: PGP signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v4 00/20] Incremental Backup API additions
On 2/19/19 1:30 AM, Peter Krempa wrote: >> Somewhat correct - but we DO have to think about how we plan for the API >> to grow in the future when we eventually DO fix snapshot >> deletion/reversion. Hence my question - would we rather have the >> creation of a checkpoint at the same time as the creation of an external >> snapshot (which we DO know we will want) to occur via the existing API >> (by extending the snapshot XML to include the checkpoint XML as a >> sub-element), or via a new API (by passing the checkpoint XML as a >> second parameter)? Once we've answered that question, it then >> determines what signature we want for virDomainBackupBegin() (either two >> separate XML parameters, one for the backup job and one for the >> checkpoint creation, as presented in v4 of the series, OR as one single >> XML call where the checkpoint XML is a sub-element of the backup XML). > > This is an interesting point. Given that the snapshot creates new files > (speaking of external snapshots obviously) you get an implicit > "checkpoint" at that moment as basically all blocks changed since that > moment are recorded in the overlay file. For any other snapshot you get > the same if you apply the overlay file on top of it. > > Now the question is whether we need to be able to track that checkpoint > explicitly or whether it needs an explicit bitmap, but I'm not that > familiar with qemu implementation. The easiest way to perform a differential backup is to bitwise-or all the bitmaps from the requested checkpoint to the present (where I've implemented things that only one bitmap is active at a time, to avoid qemu having to write to an ever-increasing number of bitmaps as more checkpoints are created). But unless I create a bitmap at the same time an external snapshot is created, there is no easy way to make current qemu expose that all clusters in the current active layer are dirty, to OR that with the clusters that were marked dirty prior to creating the snapshot. So yes, it would be best if the creation of a snapshot can ALSO be used as the creation of a bitmap, which in turn implies that creating an explicit checkpoint associated with the snapshot is the way to go. Fortunately, with blockdev-add, it is possible to create the qcow2 file, then create the bitmap, and only then perform the snapshot operation, so we don't need any new qemu commands to create an explicit bitmap, but we DO need to figure out where in the libvirt API to wire in this extra step when external snapshots and checkpoints are both used in the same image. > > If we don't require any explicit bitmap or marking in qemu to be able to > track a checkpoint from a snapshot point we can e.g. allow the backup > API to take an external checkpoint as a reference for the backup > itself. > > Creating both a snapshot and a checkpoint is certainly possible but I > fear that making the snapshot more complex than it is will be an > unexpected can of worms. I'm afraid that when I do integrate bitmaps with external snapshots, that I'll have to modify the creation of snapshots to create a bitmap, whether or not I also expose the ability in API to create an explicit Checkpoint object at the same time. So as long as we have to manage bitmaps, we might as well also manage the explicit creation of a checkpoint for both a snapshot as well as for a virDomainBackupBegin call. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v4 00/20] Incremental Backup API additions
On Mon, Feb 18, 2019 at 15:50:52 -0600, Eric Blake wrote: > On 2/18/19 10:34 AM, Peter Krempa wrote: [...] > > In the ideal world of snapshots when deletion/revertion was implemented > > we'd never expose the virDomainBlockCommit and virDomainBlockPull APIs > > including the multi-use backdoor virDomainBlockRebase() which should > > have never existed and users would do equivalent operations using the > > snapshot APIs. > > > > virDomainBlockCopy is useful on it's own though but badly combines with > > snapshots. This will need some fixing. > > Indeed, and that's true regardless of whether the backup API goes in > (although the backup API probably compounds the issue on the number of > corner cases we have to think about; the conservative approach is that > at least in the beginning, you won't be able to run a BlockCopy and a > BackupBegin job at the same time). We currently do this kind of interlocking between blockjobs and snapshots as well, so that will not be a problem. We don't even allow two blockjobs on non-conflicting parts of the backing chain while qemu allows that now. Allowing that will require some API design considerations/compromises as our job tracking and manipulation APIs and events identify the job by the disk. Given that a guest can detach a disk frontend without qemu interaction we will somehow need to be able to display jobs which don't have a disk assigned any more though which will have the same implications basically. > > With the new checkpoint APIs the situation is even more "fun" as > > modification of the backing chain involves in some cases changes to the > > bitmaps. Ideally these would do "the right thing (TM)" during snapshot > > deletion/reversion. Given that at this time we don't support snapshot > > deletion/reversion for external snapshots we can use the excuse that > > snapshot management is not implemented so that checkpoints don't need > > to be modified. > > Somewhat correct - but we DO have to think about how we plan for the API > to grow in the future when we eventually DO fix snapshot > deletion/reversion. Hence my question - would we rather have the > creation of a checkpoint at the same time as the creation of an external > snapshot (which we DO know we will want) to occur via the existing API > (by extending the snapshot XML to include the checkpoint XML as a > sub-element), or via a new API (by passing the checkpoint XML as a > second parameter)? Once we've answered that question, it then > determines what signature we want for virDomainBackupBegin() (either two > separate XML parameters, one for the backup job and one for the > checkpoint creation, as presented in v4 of the series, OR as one single > XML call where the checkpoint XML is a sub-element of the backup XML). This is an interesting point. Given that the snapshot creates new files (speaking of external snapshots obviously) you get an implicit "checkpoint" at that moment as basically all blocks changed since that moment are recorded in the overlay file. For any other snapshot you get the same if you apply the overlay file on top of it. Now the question is whether we need to be able to track that checkpoint explicitly or whether it needs an explicit bitmap, but I'm not that familiar with qemu implementation. If we don't require any explicit bitmap or marking in qemu to be able to track a checkpoint from a snapshot point we can e.g. allow the backup API to take an external checkpoint as a reference for the backup itself. Creating both a snapshot and a checkpoint is certainly possible but I fear that making the snapshot more complex than it is will be an unexpected can of worms. signature.asc Description: PGP signature -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [PATCH v4 00/20] Incremental Backup API additions
On Wed, Feb 06, 2019 at 01:17:58PM -0600, Eric Blake wrote: > The following is the latest version of my API proposal for > incremental backups, and follows along from the demo I presented > as part of my KVM Forum 2018 talk: > https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat > > The patches are also available via https://repo.or.cz/libvirt/ericb.git > at the tag backup-v4. > > Integration with external snapshots is still not included in these > patches, although I have been playing more with that locally, and > I still haven't gotten a working demonstration of a push-mode > incremental backup. > > At this point, I'm posting mainly because there have been enough > changes in qemu (the backup APIs are now stable! and in qemu.git > in time for 4.0) and libvirt (several releases and code cleanups > in between, such that the v3 no longer applies easily), while I > continue to work locally on more features, addressing the review > comments I got on v3, and remove the 'wip' tag on several of the > later patches. > > Among other things, I still haven't determined how we want to > integrate checkpoints with external snapshots; we could either have: > > virDomainSnapshotCreateXML("...") # existing > virDomainBackupBegin("...", "...") # this > series > virDomainSnapshotCheckpointCreateXML("...", > "...") # new A slightly related question: when these new APIs (thanks for working on them!) are merged, am I right in assuming that they should be able to "replace" the existing (and provide additional): virDomainBlockRebase(); and virDomainBlockCopy() ... _provided_ that an application is adjusted to using libvirt that is new enough to drive QEMU's '-blockdev', QMP `blockdev-add` et al? Or is that (the new APIs being backward-compatible with blockRebase() and blockCopy()) an explicit non-goal? I'm only this out of curiosity. > or to make checkpoint creation part of the snapshot and backup XML, as in: > > virDomainSnapshotCreateXML("...") > # extension of existing > virDomainBackupBegin("...") # > tweak to this series > > I'm also planning to add an API to query which job ids are currently > running (right now, the code only supports one job at a time, and > always calls it job 1, but that is not future-friendly) and an update > to the redefine XML command to make it easier to redefine multiple > checkpoints in one API call. [...] -- /kashyap -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] [PATCH v4 00/20] Incremental Backup API additions
The following is the latest version of my API proposal for incremental backups, and follows along from the demo I presented as part of my KVM Forum 2018 talk: https://kvmforum2018.sched.com/event/FzuB/facilitating-incremental-backup-eric-blake-red-hat The patches are also available via https://repo.or.cz/libvirt/ericb.git at the tag backup-v4. Integration with external snapshots is still not included in these patches, although I have been playing more with that locally, and I still haven't gotten a working demonstration of a push-mode incremental backup. At this point, I'm posting mainly because there have been enough changes in qemu (the backup APIs are now stable! and in qemu.git in time for 4.0) and libvirt (several releases and code cleanups in between, such that the v3 no longer applies easily), while I continue to work locally on more features, addressing the review comments I got on v3, and remove the 'wip' tag on several of the later patches. Among other things, I still haven't determined how we want to integrate checkpoints with external snapshots; we could either have: virDomainSnapshotCreateXML("...") # existing virDomainBackupBegin("...", "...") # this series virDomainSnapshotCheckpointCreateXML("...", "...") # new or to make checkpoint creation part of the snapshot and backup XML, as in: virDomainSnapshotCreateXML("...") # extension of existing virDomainBackupBegin("...") # tweak to this series I'm also planning to add an API to query which job ids are currently running (right now, the code only supports one job at a time, and always calls it job 1, but that is not future-friendly) and an update to the redefine XML command to make it easier to redefine multiple checkpoints in one API call. Since v3: 001/20:[] [--] 'snapshots: Avoid term 'checkpoint' for full system snapshot' 002/20:[] [--] 'domain_conf: Expose virDomainStorageNetworkParseHost' 003/20:[] [--] 'qemu: Allow optional export name during NBD export' 004/20:[] [--] 'backup: Document nuances between different state capture APIs' 005/20:[0032] [FC] 'backup: Introduce virDomainCheckpointPtr' 006/20:[] [--] 'backup: Document new XML for backups' 007/20:[0013] [FC] 'backup: Introduce virDomainCheckpoint APIs' 008/20:[0017] [FC] 'backup: Introduce virDomainBackup APIs' 009/20:[] [--] 'backup: Add new domain:checkpoint access control' 010/20:[0104] [FC] 'backup: Implement backup APIs for remote driver' 011/20:[0034] [FC] 'wip: backup: Parse and output checkpoint XML' 012/20:[0013] [FC] 'wip: backup: Parse and output backup XML' 013/20:[0006] [FC] 'backup: Implement virsh support for checkpoints' 014/20:[] [--] 'wip: backup: virsh support for backup' 015/20:[0126] [FC] 'backup: Add new qemu monitor interactions' 016/20:[0072] [FC] 'backup: qemu: Implement metadata tracking for checkpoint APIs' 017/20:[0069] [FC] 'wip: backup: Wire up qemu checkpoint commands over QMP' 018/20:[0018] [FC] 'wip: backup: qemu: Implement framework for backup job APIs' 019/20:[0045] [FC] 'backup: Wire up qemu full pull backup commands over QMP' 020/20:[0051] [FC] 'backup: implement qemu incremental pull backup' Eric Blake (20): snapshots: Avoid term 'checkpoint' for full system snapshot domain_conf: Expose virDomainStorageNetworkParseHost qemu: Allow optional export name during NBD export backup: Document nuances between different state capture APIs backup: Introduce virDomainCheckpointPtr backup: Document new XML for backups backup: Introduce virDomainCheckpoint APIs backup: Introduce virDomainBackup APIs backup: Add new domain:checkpoint access control backup: Implement backup APIs for remote driver wip: backup: Parse and output checkpoint XML wip: backup: Parse and output backup XML backup: Implement virsh support for checkpoints wip: backup: virsh support for backup backup: Add new qemu monitor interactions backup: qemu: Implement metadata tracking for checkpoint APIs wip: backup: Wire up qemu checkpoint commands over QMP wip: backup: qemu: Implement framework for backup job APIs backup: Wire up qemu full pull backup commands over QMP backup: implement qemu incremental pull backup include/libvirt/virterror.h |7 +- src/util/virerror.c | 12 +- include/libvirt/libvirt-domain-checkpoint.h | 176 ++ include/libvirt/libvirt-domain-snapshot.h |2 +- include/libvirt/libvirt-domain.h | 17 +- include/libvirt/libvirt.h |3 +- src/access/viraccessperm.h|8 +- src/conf/checkpoint_conf.h| 215 +++ src/conf/domain_conf.h| 16 +- src/datatypes.h | 31 +- src/driver-hypervisor.h | 74 +- src/qemu/qemu_block.h |3 + src/qemu/qemu_blockjob.h |1 + src/qemu/qemu_capabilities.h |2 +