On Tue, May 29, 2018 at 11:00:21PM +0800, Hailiang Zhang wrote: > On 2018/4/24 14:13, Wei Wang wrote: > > This is the deivce part implementation to add a new feature, > > VIRTIO_BALLOON_F_FREE_PAGE_HINT to the virtio-balloon device. The device > > receives the guest free page hints from the driver and clears the > > corresponding bits in the dirty bitmap, so that those free pages are > > not transferred by the migration thread to the destination. > > > > - Test Environment > > Host: Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz > > Guest: 8G RAM, 4 vCPU > > Migration setup: migrate_set_speed 100G, migrate_set_downtime 2 second > > > > - Test Results > > - Idle Guest Live Migration Time (results are averaged over 10 runs): > > - Optimization v.s. Legacy = 271ms vs 1769ms --> ~86% reduction > > - Guest with Linux Compilation Workload (make bzImage -j4): > > - Live Migration Time (average) > > Optimization v.s. Legacy = 1265ms v.s. 2634ms --> ~51% reduction > > - Linux Compilation Time > > Optimization v.s. Legacy = 4min56s v.s. 5min3s > > --> no obvious difference > > > > - Source Code > > - QEMU: https://github.com/wei-w-wang/qemu-free-page-lm.git > > - Linux: https://github.com/wei-w-wang/linux-free-page-lm.git > > > > ChangeLog: > > v6->v7: > > virtio-balloon/virtio_balloo_poll_free_page_hints: > > - add virtio_notify() at the end to notify the driver that > > the optimization is done, which indicates that the entries > > have all been put back to the vq and ready to detach them. > > v5->v6: > > virtio-balloon: use iothread to get free page hint > > v4->v5: > > 1) migration: > > - bitmap_clear_dirty: update the dirty bitmap and dirty page > > count under the bitmap mutex as what other functions are doing; > > - qemu_guest_free_page_hint: > > - add comments for this function; > > - check the !block case; > > - check "offset > block->used_length" before proceed; > > - assign used_len inside the for{} body; > > - update the dirty bitmap and dirty page counter under the > > bitmap mutex; > > - ram_state_reset: > > - rs->free_page_support: && with use "migrate_postcopy" > > instead of migration_in_postcopy; > > - clear the ram_bulk_stage flag if free_page_support is true; > > 2) balloon: > > - add the usage documentation of balloon_free_page_start and > > balloon_free_page_stop in code; > > - the optimization thread is named "balloon_fpo" to meet the > > requirement of "less than 14 characters"; > > - virtio_balloon_poll_free_page_hints: > > - run on condition when runstate_is_running() is true; > > - add a qemu spin lock to synchronize accesses to the free > > page reporting related fields shared among the migration > > thread and the optimization thread; > > - virtio_balloon_free_page_start: just return if > > runstate_is_running is false; > > - virtio_balloon_free_page_stop: access to the free page > > reporting related fields under a qemu spin lock; > > - virtio_balloon_device_unrealize/reset: call > > virtio_balloon_free_page_stop is the free page hint feature is > > used; > > - virtio_balloon_set_status: call irtio_balloon_free_page_stop > > in case the guest is stopped by qmp when the optimization is > > running; > > v3->v4: > > 1) bitmap: add a new API to count 1s starting from an offset of a > > bitmap > > 2) migration: > > - qemu_guest_free_page_hint: calculate > > ram_state->migration_dirty_pages by counting how many bits of > > free pages are truely cleared. If some of the bits were > > already 0, they shouldn't be deducted by > > ram_state->migration_dirty_pages. This wasn't needed for > > previous versions since we optimized bulk stage only, > > where all bits are guaranteed to be set. It's needed now > > because we extened the usage of this optimizaton to all stages > > except the last stop© stage. From 2nd stage onward, there > > are possibilities that some bits of free pages are already 0. > > 3) virtio-balloon: > > - virtio_balloon_free_page_report_status: introduce a new status, > > FREE_PAGE_REPORT_S_EXIT. This status indicates that the > > optimization thread has exited. FREE_PAGE_REPORT_S_STOP means > > the reporting is stopped, but the optimization thread still > > needs > > to be joined by the migration thread. > > v2->v3: > > 1) virtio-balloon > > - virtio_balloon_free_page_start: poll the hints using a new > > thread; > > - use cmd id between [0x80000000, UINT_MAX]; > > - virtio_balloon_poll_free_page_hints: > > - stop the optimization only when it has started; > > - don't skip free pages when !poison_val; > > - add poison_val to vmsd to migrate; > > - virtio_balloon_get_features: add the F_PAGE_POISON feature when > > host has F_FREE_PAGE_HINT; > > - remove the timer patch which is not needed now. > > 2) migration > > - new api, qemu_guest_free_page_hint; > > - rs->free_page_support set only in the precopy case; > > - use the new balloon APIs. > > v1->v2: > > 1) virtio-balloon > > - use subsections to save free_page_report_cmd_id; > > - poll the free page vq after sending a cmd id to the driver; > > - change the free page vq size to VIRTQUEUE_MAX_SIZE; > > - virtio_balloon_poll_free_page_hints: handle the corner case > > that the free page block reported from the driver may cross > > the RAMBlock boundary. > > 2) migration/ram.c > > - use balloon_free_page_poll to start the optimization > > > > > > Wei Wang (5): > > bitmap: bitmap_count_one_with_offset > > migration: use bitmap_mutex in migration_bitmap_clear_dirty > > migration: API to clear bits of guest free pages from the dirty bitmap > > virtio-balloon: VIRTIO_BALLOON_F_FREE_PAGE_HINT > > migration: use the free page hint feature from balloon > > > > balloon.c | 58 +++++- > > hw/virtio/virtio-balloon.c | 241 > > ++++++++++++++++++++++-- > > include/hw/virtio/virtio-balloon.h | 27 ++- > > include/migration/misc.h | 2 + > > include/qemu/bitmap.h | 13 ++ > > include/standard-headers/linux/virtio_balloon.h | 7 + > > include/sysemu/balloon.h | 15 +- > > migration/ram.c | 73 ++++++- > > 8 files changed, 406 insertions(+), 30 deletions(-) > > Nice optimization, for the first stage of current migration method, we need > to migrate all the pages of > VM to destination, with this capability, we can reduce lots of unnecessary > pages migrating. > > Just a small piece of advice, it is better to split the fourth patch into > small ones, to make it more easy > for reviewing. Besides, should we make this capability an optional one, just > like other migration capabilities do ?
That's already the case, one has to enable it in the balloon, and set the iothread. -- MST