[libvirt PATCH v2 1/2] docs: kbase: Add a doc on live full disk backup

[Kashyap Chamarthy]

This is a rewrite of:

https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

Once this commit merges, the above wiki should point to this kbase
document.

NB: I've intentionally left out the example for pull-based full backups.
I'll tackle it once QMP `x-blockdev-reopen` comes out of experimental
mode in upstream QEMU.  Then pull-based can be described for both full
and and differntial backups.

Overall, future documents should cover:

   - full backups using both push- and pull-mode
   - differential backups using both push- and pull-mode

Signed-off-by: Kashyap Chamarthy 
---
 docs/kbase/live_full_disk_backup.rst | 186 +++
 docs/kbase/meson.build   |   1 +
 2 files changed, 187 insertions(+)
 create mode 100644 docs/kbase/live_full_disk_backup.rst

diff --git a/docs/kbase/live_full_disk_backup.rst 
b/docs/kbase/live_full_disk_backup.rst
new file mode 100644
index 00..19f027daac
--- /dev/null
+++ b/docs/kbase/live_full_disk_backup.rst
@@ -0,0 +1,186 @@
+===
+Efficient live full disk backup
+===
+
+.. contents::
+
+Overview
+
+
+Live full disk backups are preferred in many scenarios, *despite* their
+space requirements.  The following outlines an efficient method to do
+that using libvirt's APIs.  This method involves concepts: the notion of
+`backing chains `_,
+`QCOW2 overlays
+`_,
+and a special operation called "active block-commit", which allows
+live-merging an overlay disk image into its backing file.
+
+Two kinds of backup: "push" and "pull"
+==
+
+QEMU and libvirt combine the concept of `bitmaps
+`_ and network
+block device (NBD) to allow copying out modified data blocks.  There are
+two approaches to it:  In the first, "push mode", when a user requests
+for it, libvirt creates a full backup in an external location (i.e.
+libvirt "pushes" the data to the target).
+
+In the other, "pull mode", libvirt (in coordination with QEMU) exposes
+the data that needs to be written out and allows a third-party tool to
+copy them out reliably (i.e. the data is being "pulled" from libvirt).
+The pull-based backup provides more flexibility by letting an external
+tool fetch the modified bits as it sees fit, rather than waiting on
+libvirt to push out a full backup to a target location.
+
+The push- and pull-mode techniques also apply for differential backups
+(it also includes incremental backups), which track what has changed
+since *any* given backup.
+
+This document covers only the full backups using the the "push" mode.
+
+
+Full disk backup using "push" mode
+==
+
+The below approach uses the modern backup API, virDomainBackupBegin().
+This requires libvirt-7.2.0 and QEMU-4.2, or higher versions.
+
+#. Start the guest::
+
+$> virsh start vm1
+Domain 'vm1' started
+
+#. Enumerate the disk(s) in use::
+
+$> virsh domblklist vm1
+ Target   Source
+--
+ vda  /var/lib/libvirt/images/vm1.qcow2
+
+#. Begin the backup::
+
+$> virsh backup-begin vm1
+Backup started
+
+#. Check the job status ("None" means the job has likely completed)::
+
+$> virsh domjobinfo vm1
+Job type: None
+
+#. Check the completed job status::
+
+$> virsh domjobinfo vm1 --completed
+Job type: Completed
+Operation:Backup
+Time elapsed: 183  ms
+File processed:   39.250 MiB
+File remaining:   0.000 B
+File total:   39.250 MiB
+
+#. Now we see the copy of the backup::
+
+$> ls -lash /var/lib/libvirt/images/vm1.qcow2*
+15M -rw-r--r--. 1 qemu qemu 15M May 10 12:22 vm1.qcow2
+21M -rw---. 1 root root 21M May 10 12:23 vm1.qcow2.1620642185
+
+
+Full backup with older libvirt versions
+===
+
+This is the alternative in case you cannot use libvirt-7.2.0 and
+QEMU-4.2 for some reason.  But this assumes you're using *at least* QEMU
+2.1 and libvirt-1.2.9.
+
+This backup approach is slightly more involved, and predates the
+virDomainBackupBegin() API: Assuming a guest with a single disk image,
+create a temporary live QCOW2 overlay (commonly called as "external
+snapshot") to track the live guest writes.  Then backup the original
+disk image while the guest (live QEMU) keeps writing to the temporary
+overlay.  Finally, perform the "active block-commit" opertion to
+live-merge the temporary overlay disk contents into the original image —
+i.e. the backing file — and "pivot" the live QEMU process to point to
+it.
+
+
+#. Start with a guest with a single disk image, ``base.raw``, which is
+   where the live QEMU is pointing at, and recording the guest writes

Re: [libvirt PATCH v2 1/2] docs: kbase: Add a doc on live full disk backup

[Michal Prívozník]

On 5/10/21 6:39 PM, Kashyap Chamarthy wrote:
> This is a rewrite of:
> 
> https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
> 
> Once this commit merges, the above wiki should point to this kbase
> document.
> 
> NB: I've intentionally left out the example for pull-based full backups.
> I'll tackle it once QMP `x-blockdev-reopen` comes out of experimental
> mode in upstream QEMU.  Then pull-based can be described for both full
> and and differntial backups.
> 
> Overall, future documents should cover:
> 
>- full backups using both push- and pull-mode
>- differential backups using both push- and pull-mode
> 
> Signed-off-by: Kashyap Chamarthy 
> ---
>  docs/kbase/live_full_disk_backup.rst | 186 +++
>  docs/kbase/meson.build   |   1 +
>  2 files changed, 187 insertions(+)
>  create mode 100644 docs/kbase/live_full_disk_backup.rst
> 
> diff --git a/docs/kbase/live_full_disk_backup.rst 
> b/docs/kbase/live_full_disk_backup.rst
> new file mode 100644
> index 00..19f027daac
> --- /dev/null
> +++ b/docs/kbase/live_full_disk_backup.rst
> @@ -0,0 +1,186 @@
> +===
> +Efficient live full disk backup
> +===
> +
> +.. contents::
> +
> +Overview
> +
> +
> +Live full disk backups are preferred in many scenarios, *despite* their
> +space requirements.  The following outlines an efficient method to do
> +that using libvirt's APIs.  This method involves concepts: the notion of
> +`backing chains `_,
> +`QCOW2 overlays
> +`_,
> +and a special operation called "active block-commit", which allows
> +live-merging an overlay disk image into its backing file.
> +
> +Two kinds of backup: "push" and "pull"
> +==
> +
> +QEMU and libvirt combine the concept of `bitmaps
> +`_ and network
> +block device (NBD) to allow copying out modified data blocks.  There are
> +two approaches to it:  In the first, "push mode", when a user requests
> +for it, libvirt creates a full backup in an external location (i.e.
> +libvirt "pushes" the data to the target).
> +
> +In the other, "pull mode", libvirt (in coordination with QEMU) exposes
> +the data that needs to be written out and allows a third-party tool to
> +copy them out reliably (i.e. the data is being "pulled" from libvirt).
> +The pull-based backup provides more flexibility by letting an external
> +tool fetch the modified bits as it sees fit, rather than waiting on
> +libvirt to push out a full backup to a target location.
> +
> +The push- and pull-mode techniques also apply for differential backups
> +(it also includes incremental backups), which track what has changed
> +since *any* given backup.
> +
> +This document covers only the full backups using the the "push" mode.

s/the the/the/

I'm surprised that 'ninja test' did not catch it. And it can also be
seen in the pipeline output you link in cover letter - if you know where
to look because it's way too verbose.

Michal