On Fri, Jul 06, 2018 at 11:56:59AM +0100, Dr. David Alan Gilbert wrote: > * Dr. David Alan Gilbert (dgilb...@redhat.com) wrote: > > * Peter Xu (pet...@redhat.com) wrote: > > > Based-on: <20180627132246.5576-1-pet...@redhat.com> > > > > > > Based on the series to unbreak postcopy: > > > Subject: [PATCH v3 0/4] migation: unbreak postcopy recovery > > > Message-Id: <20180627132246.5576-1-pet...@redhat.com> > > > > > > This series introduce a new postcopy recovery test. The new test > > > actually helped me to identify two bugs there so fix them as well > > > before 3.0 release. > > > > > > Patch 1: a trivial cleanup for existing postcopy ram load, which I > > > found a bit confusing during debugging the problem. > > > > > > Patch 2-3: two bug fixes that address different issues. Please see > > > the commit log for more information. > > > > > > Patch 4-9: add the postcopy recovery unit test. > > > > > > Please review. Thanks, > > > > Queued > > Hi Peter, > There's a problem in there somewhere; I'm getting > an intermittent failure of the test if I run a make check -j 8 on my > laptop. Just running two copies of tests/migration-test in parallel > sometimes triggers it (but not if I turn on QTEST_LOG!). > But it's always failing with: > > > ERROR:/home/dgilbert/git/migpull/tests/migration-test.c:373:migrate_recover: > assertion failed: (qdict_haskey(rsp, "return"))
Hi Peter and Dave, I have tested postcopy migration pause/recover after applying this patchset on upstream Qemu, Observation 1: We loose the target after triggering migrate_pause, source # ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none \ -machine pseries -m 64G,slots=128,maxmem=128G -smp 16,maxcpus=32 \ -device virtio-blk-pci,drive=rootdisk -drive \ file=/home/bala/sharing/hostos-ppc64le.qcow2,if=none,cache=none,format=qcow2,id=rootdisk \ -monitor telnet:127.0.0.1:1235,server,nowait -net nic,model=virtio \ -net user -redir tcp:2001::22 qemu-system-ppc64: Detected IO failure for postcopy. Migration paused. source Monitor (qemu) migrate_set_capability postcopy-ram on (qemu) migrate_set_parameter max-postcopy-bandwidth 4096 (qemu) migrate -d tcp:127.0.0.1:4444 (qemu) migrate_start_postcopy (qemu) migrate_pause (qemu) info migrate globals: store-global-state: on only-migratable: off send-configuration: on send-section-footer: on decompress-error-check: on capabilities: xbzrle: off rdma-pin-all: off auto-converge: off zero-blocks: off compress: off events: off postcopy-ram: on x-colo: off release-ram: off block: off return-path: off pause-before-switchover: off x-multifd: off dirty-bitmaps: off postcopy-blocktime: off late-block-activate: off Migration status: postcopy-paused total time: 371289 milliseconds expected downtime: 656414 milliseconds setup: 93 milliseconds transferred ram: 690856 kbytes throughput: 46.65 mbps remaining ram: 3716864 kbytes total ram: 67109120 kbytes duplicate: 16631167 pages skipped: 0 pages normal: 135905 pages normal bytes: 543620 kbytes dirty sync count: 2 page size: 4 kbytes multifd bytes: 0 kbytes dirty pages rate: 626209 pages postcopy request count: 395 source remains to be in postcopy-paused state as the target is lost. target # ppc64-softmmu/qemu-system-ppc64 --enable-kvm --nographic -vga none \ -machine pseries -m 64G,slots=128,maxmem=128G -smp 16,maxcpus=32 \ -device virtio-blk-pci,drive=rootdisk -drive \ file=/home/bala/sharing/hostos-ppc64le.qcow2,if=none,cache=none,format=qcow2,id=rootdisk \ -monitor telnet:127.0.0.1:1235,server,nowait -net nic,model=virtio \ -net user -redir tcp:2001::22 -incoming tcp:127.0.0.1:4444 Error observed check_section_footer: Read section footer failed: -5 qemu-system-ppc64: postcopy_ram_listen_thread: loadvm failed: -22 [ 188.815436] Unable to handle kernel paging request for instruction fetch Target Monitor (qemu) migrate_set_capability postcopy-ram on Observation 2: Unlike error observed by Dave in Qtest it hangs for me waiting for the migration to complete, but the source remains to be in migration-paused state. # time QTEST_QEMU_BINARY=./ppc64-softmmu/qemu-system-ppc64 # ./tests/migration-test /ppc64/migration/deprecated: OK /ppc64/migration/bad_dest: OK /ppc64/migration/postcopy/unix: OK /ppc64/migration/postcopy/recovery: ^C real 21m55.176s user 2m28.800s sys 4m55.980s -- Bala > > Dave > > > > Peter Xu (9): > > > migration: simplify check to use qemu file buffer > > > migration: loosen recovery check when load vm > > > migration: fix incorrect bitmap size calculation > > > tests: introduce migrate_postcopy_* helpers > > > tests: allow migrate() to take extra flags > > > tests: introduce migrate_query*() helpers > > > tests: introduce wait_for_migration_status() > > > tests: add postcopy recovery test > > > tests: hide stderr for postcopy recovery test > > > > > > migration/ram.c | 21 +++-- > > > migration/savevm.c | 16 ++-- > > > tests/migration-test.c | 198 ++++++++++++++++++++++++++++++++--------- > > > 3 files changed, 176 insertions(+), 59 deletions(-) > > > > > > -- > > > 2.17.1 > > > > > > > > -- > > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK > -- > Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK >