On 01/01/2012 03:52 AM, Dor Laor wrote:
On 12/30/2011 12:39 AM, Anthony Liguori wrote:
On 12/28/2011 07:25 PM, Isaku Yamahata wrote:
Intro
=====
This patch series implements postcopy live migration.[1]
As discussed at KVM forum 2011, dedicated character device is used for
distributed shared memory between migration source and destination.
Now we can discuss/benchmark/compare with precopy. I believe there are
much rooms for improvement.
[1] http://wiki.qemu.org/Features/PostCopyLiveMigration
Usage
=====
You need load umem character device on the host before starting
migration.
Postcopy can be used for tcg and kvm accelarator. The implementation
depend
on only linux umem character device. But the driver dependent code is
split
into a file.
I tested only host page size == guest page size case, but the
implementation
allows host page size != guest page size case.
The following options are added with this patch series.
- incoming part
command line options
-postcopy [-postcopy-flags<flags>]
where flags is for changing behavior for benchmark/debugging
Currently the following flags are available
0: default
1: enable touching page request
example:
qemu -postcopy -incoming tcp:0:4444 -monitor stdio -machine accel=kvm
- outging part
options for migrate command
migrate [-p [-n]] URI
-p: indicate postcopy migration
-n: disable background transferring pages: This is for
benchmark/debugging
example:
migrate -p -n tcp:<dest ip address>:4444
TODO
====
- benchmark/evaluation. Especially how async page fault affects the
result.
I'll review this series next week (Mike/Juan, please also review when
you can).
But we really need to think hard about whether this is the right thing
to take into the tree. I worry a lot about the fact that we don't test
pre-copy migration nearly enough and adding a second form just
introduces more things to test.
It is an issue but it can't be a merge criteria, Isaku is not blame of
pre copy live migration lack of testing.
I would say that 90% of issues of live migration problems are not
related to the pre|post stage but more of issues of device model save
state. So post-copy shouldn't add a significant regression here.
Probably it will be good to ask every migration patch writer to write an
additional unit test for migration.
It's also not clear to me why post-copy is better. If you were going to
sit down and explain to someone building a management tool when they
should use pre-copy and when they should use post-copy, what would you
tell them?
Today, we have a default of max-downtime of 100ms.
If either the guest work set size or the host networking throughput
can't match the downtime, migration won't end.
The mgmt user options are:
- increase the downtime more and more to an actual stop
- fail migrate
W/ post-copy there is another option.
Performance measurements will teach us (probably prior to commit) when
this stage is valuable. Most likely, we better try first with pre-copy
and if we can't meet the downtime we can optionally use post-copy.
Umesh's paper seems to already have strong indications that at least 1
iteration of pre-copy is optimal in terms of downtime, so I wonder if
we're starting off on the wrong track with the all or nothing approach
taken with this series?
I only have Umesh's paper to go off (which, granted, notes shadow paging
(which I guess we effectively have with these patches) as a potential
improvement to the pseudo swap device used there), but otherwise it
seems like we'd just get more downtime as the target starts choking on
network-based page faults.
It's probably not too useful to speculate on performance at this point,
but I think it'll be easier to get the data (and hit closer to the mark
suggested by the paper) if we started off assuming that pre-copy should
still be in play. Maybe something like:
migrate -d tcp:host:123 -p[ostcopy] <after x iterations>
x=0 for post-copy only, no -p for pre-copy-only
Also seems a bit cleaner. And if post-copy proves to be optimal we just
make -p 0 implied... minor details at this point, but the main thing is
that integrating better with the pre-copy code will make it easier to
determine what the sweet spot is and how much we stand to gain.
Here's a paper by Umesh (the migration thread writer):
http://osnet.cs.binghamton.edu/publications/hines09postcopy_osr.pdf
Regards,
Dor
Regards,
Anthony Liguori