Re: [Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration
On 07/26/2013 05:41 PM, Paolo Bonzini wrote:
Il 25/07/2013 22:18, Lei Li ha scritto:
Hi,
This patch series tries to add localhost migration support to
Qemu.
When doing localhost migration, the host memory will balloon
up during the period, might consume double memories for some time.
So we want to add a new live migration mechanism localhost
migration.
Following I copied from last version that Anthony added for the
benefit of the other reviewers:
The goal here is to allow live upgrade of a running QEMU instance.
The work flow would look like this:
1) Guests are running QEMU release 1.6.1
2) Admin installs QEMU release 1.6.2 via RPM or deb
3) Admin does localhost migration with page flipping to use new version
of QEMU.
Page flipping is used in order to avoid requiring that there is enough
free memory to fit an additional copy of the largest guest which is the
requirement today with localhost migration.
You can also read from the link below:
http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
The plan is:
1) Add new command to do localhost migration.
The qmp interface introduced like:
{ 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
2) Use different mechanism than current live migration.
The very basic work flow like:
qemu on the source (the source and destination are both on localhost)
|
V
Stop VM
|
V
Create threads
|
V
Page flipping through vmspice
|
V
MADV_DONTNEED the ram pages which are already flipped
|
V
Migration completes
As stopping VM first, we expect/resume the page flipping through vmspice
is fast enough to meet *live migration (low downtime).
Notes:
Currently the work flow is not exactly the same as description
above. For the first step, the work flow we implemented is:
stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
ram pages that already copied. After that, will replace to vmsplice
mechanism instead of copying pages.
Now it's still a draft version, and as it is implemented separately
to the current migration code for a easy start, the next step will
be trying to integrate it into the current migration implementation
closely. To make sure we are on the right direction that should be
headed, please let me know your suggestions on this.
For the interface, as Anthony has suggested using a flag or a
capability, which one would you prefer or any ideas?
Using a capability on the source makes sense; I don't think anything
special is needed on the destination. The destination just sees a
special packet telling it that a pipe is available via SCM_RIGHTS; then
it fetches the file descriptor and uses it for the new protocol.
It looks like this could reuse a lot of the RAM copying hooks that we
introduced for RDMA. You shouldn't need to change anything in savevm.c
or arch_init.c
Hi Paolo,
Thanks for your suggestions!
And sorry for the late reply, I am looking into RDMA implementation and
trying to figure out the way to integrate with its approach as you suggested.
For the interface on the source, I am OK with using a capability.
For the destination, currently we are copying ram pages via unix
domain socket as first step, will replace to pipe later. How about
add a new URI prefix as Michael R.Hines suggested?
Paolo
Your comments are very welcome!
TODO:
- Integrate to the current implement of migration closely?
- Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
- benchmark/evaluation.
Lei Li (12):
migration: export MIG_STATE_xxx flags
savevm: export qemu_save_device_state()
rename is_active to is_block_active
arch_init: introduce ram_page_save()
arch_init: introduce ram_save_local()
arch_init: add save_local_setup to savevm_ram_handlers
savevm: introduce qemu_savevm_local()
savevm: adjust is_ram check in register_savevm_live()
migration-local: implementation of outgoing part
migration-local: implementation of incoming part
migration-local: add option to command line for local incoming
hmp:add hmp_localhost_migration interface
Makefile.objs |1 +
arch_init.c | 110 +++
block-migration.c |2 +-
hmp-commands.hx | 17
hmp.c | 13 +++
hmp.h |1 +
include/migration/migration.h | 32 +++
include/sysemu/sysemu.h |1 +
migration-local.c | 228 +
migration-unix.c | 60
migration.c |8 --
qapi-schema.json | 14 +++
qemu-options.hx |9 ++
qmp-commands.hx | 22 +
savevm.c | 100 +++--
vl.c
Re: [Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration
Il 25/07/2013 22:18, Lei Li ha scritto:
Hi,
This patch series tries to add localhost migration support to
Qemu.
When doing localhost migration, the host memory will balloon
up during the period, might consume double memories for some time.
So we want to add a new live migration mechanism localhost
migration.
Following I copied from last version that Anthony added for the
benefit of the other reviewers:
The goal here is to allow live upgrade of a running QEMU instance.
The work flow would look like this:
1) Guests are running QEMU release 1.6.1
2) Admin installs QEMU release 1.6.2 via RPM or deb
3) Admin does localhost migration with page flipping to use new version
of QEMU.
Page flipping is used in order to avoid requiring that there is enough
free memory to fit an additional copy of the largest guest which is the
requirement today with localhost migration.
You can also read from the link below:
http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
The plan is:
1) Add new command to do localhost migration.
The qmp interface introduced like:
{ 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
2) Use different mechanism than current live migration.
The very basic work flow like:
qemu on the source (the source and destination are both on localhost)
|
V
Stop VM
|
V
Create threads
|
V
Page flipping through vmspice
|
V
MADV_DONTNEED the ram pages which are already flipped
|
V
Migration completes
As stopping VM first, we expect/resume the page flipping through vmspice
is fast enough to meet *live migration (low downtime).
Notes:
Currently the work flow is not exactly the same as description
above. For the first step, the work flow we implemented is:
stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
ram pages that already copied. After that, will replace to vmsplice
mechanism instead of copying pages.
Now it's still a draft version, and as it is implemented separately
to the current migration code for a easy start, the next step will
be trying to integrate it into the current migration implementation
closely. To make sure we are on the right direction that should be
headed, please let me know your suggestions on this.
For the interface, as Anthony has suggested using a flag or a
capability, which one would you prefer or any ideas?
Using a capability on the source makes sense; I don't think anything
special is needed on the destination. The destination just sees a
special packet telling it that a pipe is available via SCM_RIGHTS; then
it fetches the file descriptor and uses it for the new protocol.
It looks like this could reuse a lot of the RAM copying hooks that we
introduced for RDMA. You shouldn't need to change anything in savevm.c
or arch_init.c
Paolo
Your comments are very welcome!
TODO:
- Integrate to the current implement of migration closely?
- Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
- benchmark/evaluation.
Lei Li (12):
migration: export MIG_STATE_xxx flags
savevm: export qemu_save_device_state()
rename is_active to is_block_active
arch_init: introduce ram_page_save()
arch_init: introduce ram_save_local()
arch_init: add save_local_setup to savevm_ram_handlers
savevm: introduce qemu_savevm_local()
savevm: adjust is_ram check in register_savevm_live()
migration-local: implementation of outgoing part
migration-local: implementation of incoming part
migration-local: add option to command line for local incoming
hmp:add hmp_localhost_migration interface
Makefile.objs |1 +
arch_init.c | 110 +++
block-migration.c |2 +-
hmp-commands.hx | 17
hmp.c | 13 +++
hmp.h |1 +
include/migration/migration.h | 32 +++
include/sysemu/sysemu.h |1 +
migration-local.c | 228
+
migration-unix.c | 60
migration.c |8 --
qapi-schema.json | 14 +++
qemu-options.hx |9 ++
qmp-commands.hx | 22 +
savevm.c | 100 +++--
vl.c | 14 +++
16 files changed, 613 insertions(+), 19 deletions(-)
create mode 100644 migration-local.c
[Qemu-devel] [PATCH 0/12 RFC v2] Localhost migration
Hi,
This patch series tries to add localhost migration support to
Qemu.
When doing localhost migration, the host memory will balloon
up during the period, might consume double memories for some time.
So we want to add a new live migration mechanism localhost
migration.
Following I copied from last version that Anthony added for the
benefit of the other reviewers:
The goal here is to allow live upgrade of a running QEMU instance.
The work flow would look like this:
1) Guests are running QEMU release 1.6.1
2) Admin installs QEMU release 1.6.2 via RPM or deb
3) Admin does localhost migration with page flipping to use new version
of QEMU.
Page flipping is used in order to avoid requiring that there is enough
free memory to fit an additional copy of the largest guest which is the
requirement today with localhost migration.
You can also read from the link below:
http://lists.gnu.org/archive/html/qemu-devel/2013-06/msg02577.html
The plan is:
1) Add new command to do localhost migration.
The qmp interface introduced like:
{ 'command': 'localhost-migrate', 'data': {'uri': 'str'} }
2) Use different mechanism than current live migration.
The very basic work flow like:
qemu on the source (the source and destination are both on localhost)
|
V
Stop VM
|
V
Create threads
|
V
Page flipping through vmspice
|
V
MADV_DONTNEED the ram pages which are already flipped
|
V
Migration completes
As stopping VM first, we expect/resume the page flipping through vmspice
is fast enough to meet *live migration (low downtime).
Notes:
Currently the work flow is not exactly the same as description
above. For the first step, the work flow we implemented is:
stop VM and copy ram pages via unix domain socket, MADV_DONTNEED
ram pages that already copied. After that, will replace to vmsplice
mechanism instead of copying pages.
Now it's still a draft version, and as it is implemented separately
to the current migration code for a easy start, the next step will
be trying to integrate it into the current migration implementation
closely. To make sure we are on the right direction that should be
headed, please let me know your suggestions on this.
For the interface, as Anthony has suggested using a flag or a
capability, which one would you prefer or any ideas?
Your comments are very welcome!
TODO:
- Integrate to the current implement of migration closely?
- Introduce a mechanism to exchange a PIPE via SCM_RIGHTS.
- benchmark/evaluation.
Lei Li (12):
migration: export MIG_STATE_xxx flags
savevm: export qemu_save_device_state()
rename is_active to is_block_active
arch_init: introduce ram_page_save()
arch_init: introduce ram_save_local()
arch_init: add save_local_setup to savevm_ram_handlers
savevm: introduce qemu_savevm_local()
savevm: adjust is_ram check in register_savevm_live()
migration-local: implementation of outgoing part
migration-local: implementation of incoming part
migration-local: add option to command line for local incoming
hmp:add hmp_localhost_migration interface
Makefile.objs |1 +
arch_init.c | 110 +++
block-migration.c |2 +-
hmp-commands.hx | 17
hmp.c | 13 +++
hmp.h |1 +
include/migration/migration.h | 32 +++
include/sysemu/sysemu.h |1 +
migration-local.c | 228 +
migration-unix.c | 60
migration.c |8 --
qapi-schema.json | 14 +++
qemu-options.hx |9 ++
qmp-commands.hx | 22 +
savevm.c | 100 +++--
vl.c | 14 +++
16 files changed, 613 insertions(+), 19 deletions(-)
create mode 100644 migration-local.c
