I am reviving this thread about the migration of VMs with local storage. I have
worked on a solution to be able to migrate VMs that use QEMU as the backend
disk driver. I have adapted the migration flow and piggybacked on the
“drive-mirroring” capability already provided by QEMU.
Overview
1. The “xl migrate” command has an additional “-q” flag. When provided the
local storage of the VM is mirrored to the destination during the migration
process.
2. Internally, the modification consists on adding a new
libxl__stream_read_state struct to the libxl__domain_create_state structure and
libxl__stream_read_state structure to the libxl__domain_save_state struct.
3. Migration flow can now be divided into three phases:
a. Phase One: Copies the necessary state to start a QEMU process on the
destination. It is started with the “-incoming defer” option.
b. Phase Two: Disk is mirrored using the QEMU embedded NBD server.
c. Phase Three: Once the disk is completely mirrored, virtual RAM of the
domain is live migrated to the destination. This phase most closely resembles
to the current migration flow.
4. If the “-q” option is not provided the migration is equivalent to the
current migration flow.
The new migration flow has follows the following major sequence of steps:
1. 1st stream copies the QEMU devices RAM from source to destination.
2. QEMU process is started on the destination with the option “-incoming
defer”. (This creates the QEMU process but it doesn’t start running the main
loop until “migrate incoming” command is executed)
3. “drive mirror” QMP command is executed so that the disk is mirrored to the
destination node.
4. An event listener waits for the QMP BLOCK_JOB_READY event sent by QEMU which
signals that the "disk mirror job" is complete.
5. 2nd Stream copies the virtual RAM from source to destination including QEMU
state. At this point, the VM is suspended on source.
6. “migrate incoming” QMP command is executed on destination.
7. VM is restored in destination.
This is sample configuration file that I have used to test my branch:
name="tinycore"
disk=['/home/balvisio/tinycore.img,raw,xvda,w']
memory=128
builder='hvm'
vcpus=1
vfb = ['type=vnc']
vif= ['bridge=xenbr0']
boot='b'
acpi=1
device_model_version='qemu-xen'
serial='pty'
vnc=1
xen_platform_pci=0
Notes
1. Note that the configuration file uses "xen_platform_pci=0”. This is
necessary so that the block device is seen by QEMU. Further modification should
be made for the case "xen_platform_pci=1” if we still want to use NBD mirroring
capability provided by QEMU.
2. The current branch has still many hardcoded values. Many of the can be
easily removed:
a. Port used for disk mirroring (11000)
b. Name of the block devices. (ide0-hd0) Currently the branch only supports
VM with on disk drive.
c. Live migration memory transfer: The pages transferred by libxc is
hardcoded. Only a VM with 128 MB of memory is supported.
Here is a link to the branch in Github:
https://github.com/balvisio/xen/tree/feature/local_storage_migration
Any feedback/suggestion is appreciated.
Cheers,
Bruno
Signed-off-by: Bruno Alvisio <bruno.alvi...@gmail.com>
---
tools/libxc/include/xenguest.h | 6 +-
tools/libxc/xc_nomigrate.c | 6 +-
tools/libxc/xc_sr_common.h | 3 +
tools/libxc/xc_sr_restore.c | 14 +-
tools/libxc/xc_sr_save.c | 117 +++++++++++++++-
tools/libxc/xc_sr_save_x86_hvm.c | 7 +-
tools/libxc/xc_sr_stream_format.h | 4 +
tools/libxl/libxl.h | 11 +-
tools/libxl/libxl_create.c | 189 +++++++++++++++++++++++--
tools/libxl/libxl_dm.c | 20 ++-
tools/libxl/libxl_dom_save.c | 66 ++++++++-
tools/libxl/libxl_domain.c | 4 +-
tools/libxl/libxl_internal.h | 62 ++++++++-
tools/libxl/libxl_qmp.c | 261 +++++++++++++++++++++++++++++++++++
tools/libxl/libxl_save_callout.c | 30 ++--
tools/libxl/libxl_save_helper.c | 13 +-
tools/libxl/libxl_stream_read.c | 17 ++-
tools/libxl/libxl_stream_write.c | 28 +++-
tools/libxl/libxl_types.idl | 6 +
tools/ocaml/libs/xl/xenlight_stubs.c | 4 +-
tools/xl/xl.h | 1 +
tools/xl/xl_cmdtable.c | 3 +-
tools/xl/xl_migrate.c | 36 +++--
tools/xl/xl_saverestore.c | 2 +-
tools/xl/xl_vmcontrol.c | 5 +-
25 files changed, 845 insertions(+), 70 deletions(-)
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
