Am Di., 13. Okt. 2026 um 22:16 Uhr schrieb Peter Xu <[email protected]>:
>
> On Fri, Oct 09, 2026 at 07:52:29PM +0200, Alexander Mikhalitsyn wrote:
> > Am Mi., 7. Okt. 2026 um 15:27 Uhr schrieb Peter Xu <[email protected]>:
> > >
> > > On Mon, Sep 21, 2026 at 08:03:45PM +0100, Alex Bennée wrote:
> > > > This is a hacky patch for comment but I've included the diagnostic
> > > > hacks to the migration test as well as my workaround to get itworking.
> > > >
> > > > This test was failing on my KVM system. On investigation I found the
> > > > nVME activity meant the migration never completed. I ended up
> > > > inserting some sleeps into the workload script until I was able to
> > > > get it to run.
> > >
> > > Indeed, I can reproduce it too:
> >
> > Hi Peter,
>
> Hi, Alexander,
Hi Peter,
>
> Thanks for loooking into this, and sorry for a late update. Before saying
> this is a right fix, I have some questions inline.
No problem at all! ;-)
>
> >
> > thank you checking on your system and sharing a stacktrace.
> > Unfortunately, I wasn't able to reproduce this failure on my system,
> > but I think I have a good theory what's going on there. I think it is
> > because we use `snapshot=on` for a disk device and at the same time
> > keep important stuff like `/tmp/test_nvme_mig_workload.pid` on a root
> > filesystem. Easy fix is to use tmpfs for /tmp, so all data is actually
> > migrated.
> >
> > Please, can you try the following patch and let me know if this fixes
> > the issue or not:
> >
> > --- a/tests/functional/x86_64/test_nvme_migration.py
> > +++ b/tests/functional/x86_64/test_nvme_migration.py
> > @@ -75,6 +75,11 @@ def configure_machine(self, vm):
> > def launch_source_vm(self, vm):
> > vm.launch()
> >
> > + vm.cmd('migrate-set-parameters', **{
> > + 'downtime-limit': 2000000,
>
> One thing to mention is downtime-limit is in milliseconds, this is 2000sec,
> so I would expect whatever guest runs, this converges in the 1st second..
yeah, probably I've got a bit too far with this parameter :-)
>
> > + 'max-bandwidth': 10 * 1024**3,
>
> Yes we want some sane bandwidth maybe, indeed, default is normally too
> small.
My experiments show, that with these parameters test passes like in 20
seconds, while
without it takes ~35 seconds.
Actually, this change is not a fix. It's a tweak I should make
initially when adding a test.
>
> > + })
> > +
> > self.wait_for_console_pattern('Entering emergency mode.', vm)
> > prompt = '# '
> > self.wait_for_console_pattern(prompt, vm)
> > @@ -94,6 +99,8 @@ def launch_source_vm(self, vm):
> > prompt, vm=vm)
> > exec_command_and_wait_for_pattern(self, 'mount -t sysfs sysfs
> > /sys',
> > prompt, vm=vm)
> > + exec_command_and_wait_for_pattern(self, 'mount -t tmpfs tmpfs
> > /tmp',
> > + prompt, vm=vm)
>
> Why this is needed? I was expecting /tmp normally is already mounted with
> tmpfs, no?
Unfortunately, not. Cause first of all, we boot in a rescue mode
*from* initrd (not from the disk!) and then we:
exec_command_and_wait_for_pattern(self, 'mount /dev/nvme0n1p1
/sysroot', // << manually mount root disk with the system
prompt, vm=vm)
exec_command_and_wait_for_pattern(self, 'chroot /sysroot',
// << chroot, so there is no mounts other than
"/" in "/" subtree
prompt, vm=vm)
exec_command_and_wait_for_pattern(self, 'mount -t proc proc /proc',
prompt, vm=vm)
exec_command_and_wait_for_pattern(self, 'mount -t sysfs sysfs /sys',
prompt, vm=vm)
exec_command_and_wait_for_pattern(self, 'mount -t tmpfs tmpfs /tmp',
prompt, vm=vm)
>
> One thing that might be relevant is the use of snapshot=on on the drives,
> which means, IIUC, there's a top layer qcow2 overlay on the image file
> specified.
yes, this is what I've mentioned earlier. My theory is the following:
- we run some workload inside the VM to produce some I/O. We put PID of it
in `/tmp/test_nvme_mig_workload.pid`
- later we expect to see PID in that file to kill it. And here we fail
(in accordance with your stacktrace)
My explanation is that it's because `/tmp` leaves on NVMe device,
which is backed by the drive with
`snapshot=on` which means that after migration all the changes
*written* to the disk disappear.
In this test we don't really care about anything written to the disk,
cause our test is R/O and it's job is
to make NVMe device I/O path busy with requests.
>
> Now thinking about it, if snapshot=on implies an anon invisible overlay, it
> should not support migration at all? It means, this QEMU instance contains
> some anonymous qcow2 overlay that can never (am I right?) be migrated,
> since it's neither part of device state, nor guest memory (unlike /tmp
> mount in guest).
yes, I think so.
>
> So I feel like there's still something is off, e.g. I wonder if we should:
>
> - add a migration blocker for snapshot=on whenever used.. because I can't
> see how it can work at all...
yeah, initially when I was writing this test I was heavily inspired by
`test_virtio_balloon.py` (`snapshot=on` is used there too),
so this was basically an accidental choice. I made sure that test
works just fine, creates I/O stress and forget. But now
we see that there is a bit of a problem.
>
> - for this test, should we remove snapshot=on but instead create an
> explicit overlay that will be used on both src/dst qemu?
This should solve the issue too, at the same time if test is R/O I'm
not sure if we really need this.
But I'm happy to pick any way you suggest ;-)
Kind regards,
Alex
>
> Thanks,
>
> >
> > # Run workload before migration to check if it continues
> > # to run properly after migration.
> >
> > Kind regards,
> > Alex
> >
> > >
> > > $ ./run ../tests/functional/x86_64/test_nvme_migration.py
> > > X8664NVMeMigrationTest.test_migration_with_unix
> > > TAP version 13
> > > Traceback (most recent call last):
> > > File
> > > "/home/zhexu/git/qemu/tests/functional/x86_64/test_nvme_migration.py",
> > > line 162, in test_migration_with_unix
> > > self.migration_with_unix()
> > > ~~~~~~~~~~~~~~~~~~~~~~~~^^
> > > File "/home/zhexu/git/qemu/tests/functional/migration.py", line 98, in
> > > migration_with_unix
> > > self.migrate(dst_uri)
> > > ~~~~~~~~~~~~^^^^^^^^^
> > > File "/home/zhexu/git/qemu/tests/functional/migration.py", line 78, in
> > > migrate
> > > self.migrate_vms(dst_uri, src_uri, dst_vm, src_vm)
> > > ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > File "/home/zhexu/git/qemu/tests/functional/migration.py", line 63, in
> > > migrate_vms
> > > self.assert_dest_vm(dst_vm)
> > > ~~~~~~~~~~~~~~~~~~~^^^^^^^^
> > > File
> > > "/home/zhexu/git/qemu/tests/functional/x86_64/test_nvme_migration.py",
> > > line 128, in assert_dest_vm
> > > self.exec_command_and_check(
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
> > > "kill -0 $(cat /tmp/test_nvme_mig_workload.pid)", vm)
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > File
> > > "/home/zhexu/git/qemu/tests/functional/x86_64/test_nvme_migration.py",
> > > line 48, in exec_command_and_check
> > > wait_for_console_pattern(self, 'OK', failure_message="FAIL", vm=vm)
> > > ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > File "/home/zhexu/git/qemu/tests/functional/qemu_test/cmd.py", line
> > > 191, in wait_for_console_pattern
> > > return _console_interaction(test, success_message, failure_message,
> > > None, vm=vm)
> > > File "/home/zhexu/git/qemu/tests/functional/qemu_test/cmd.py", line
> > > 134, in _console_interaction
> > > done, line = _console_read_line_until_match(test, vm,
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
> > > success_message_b,
> > > ^^^^^^^^^^^^^^^^^^
> > > failure_message_b)
> > > ^^^^^^^^^^^^^^^^^^
> > > File "/home/zhexu/git/qemu/tests/functional/qemu_test/cmd.py", line 70,
> > > in _console_read_line_until_match
> > > test.fail(
> > > ~~~~~~~~~^
> > > f"'{failure}' found in console, expected '{success}'")
> > > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > > AssertionError: 'b'FAIL'' found in console, expected 'b'OK''
> > >
> > > not ok 1
> > > test_nvme_migration.X8664NVMeMigrationTest.test_migration_with_unix
> > > 1..1
> > >
> > > Copy the author too (Alexander).
> > >
> > > >
> > > > *NOTE*: obviously this can't be merged as is but I think we could
> > > > certainly do with more tracking in the test to see when things aren't
> > > > working with possibly the sleep as another patch.
> > > >
> > > > Signed-off-by: Alex Bennée <[email protected]>
> > > > ---
> > > > tests/functional/migration.py | 37 +++++++++++++++----
> > > > .../functional/x86_64/test_nvme_migration.py | 1 +
> > > > 2 files changed, 31 insertions(+), 7 deletions(-)
> > > >
> > > > diff --git a/tests/functional/migration.py
> > > > b/tests/functional/migration.py
> > > > index 8d2428efc24..e51a82c22f4 100644
> > > > --- a/tests/functional/migration.py
> > > > +++ b/tests/functional/migration.py
> > > > @@ -22,18 +22,41 @@ class MigrationTest(QemuSystemTest):
> > > > timeout = 10
> > > >
> > > > @staticmethod
> > > > - def migration_finished(vm):
> > > > - return vm.cmd('query-migrate')['status'] in ('completed',
> > > > 'failed')
> > > > + def migration_finished(query):
> > > > + return query['status'] in ('completed', 'failed')
> > > >
> > > > - def assert_migration(self, src_vm, dst_vm):
> > > > + def wait_for_vm(self, src_or_dst, vm):
> > > > + "Track the state of one VM in the migration"
> > > > + current_state = last_state = vm.cmd('query-migrate')
> > > >
> > > > - end = time.monotonic() + self.timeout
> > > > - while time.monotonic() < end and not
> > > > self.migration_finished(src_vm):
> > > > - time.sleep(0.1)
> > > > + self.log.info(f"waiting for {src_or_dst} to complete")
> > > >
> > > > end = time.monotonic() + self.timeout
> > > > - while time.monotonic() < end and not
> > > > self.migration_finished(dst_vm):
> > > > + while time.monotonic() < end:
> > > > + if self.migration_finished(current_state):
> > > > + break
> > > > + else:
> > > > + last_remain = last_state['remaining']
> > > > + current_remain = current_state['remaining']
> > > > +
> > > > + if current_remain > last_remain:
> > > > + self.log.warning(f"slow migration {current_remain}
> > > > > {last_remain}")
> > > > +
> > > > time.sleep(0.1)
> > > > + last_state = current_state
> > > > + current_state = vm.cmd('query-migrate')
> > > > +
> > > > +
> > > > + def assert_migration(self, src_vm, dst_vm):
> > > > +
> > > > + self.wait_for_vm("src", src_vm)
> > > > +
> > > > + # if we hit this we never completed...
> > > > + self.assertEqual(src_vm.cmd('query-migrate')['status'],
> > > > 'completed')
> > > > +
> > > > + self.wait_for_vm("dst", dst_vm)
> > > > +
> > > > + self.log.info("checking everything is OK")
> > > >
> > > > self.assertEqual(src_vm.cmd('query-migrate')['status'],
> > > > 'completed')
> > > > self.assertEqual(dst_vm.cmd('query-migrate')['status'],
> > > > 'completed')
> > > > diff --git a/tests/functional/x86_64/test_nvme_migration.py
> > > > b/tests/functional/x86_64/test_nvme_migration.py
> > > > index 890f0aab6d6..58596f63cd3 100755
> > > > --- a/tests/functional/x86_64/test_nvme_migration.py
> > > > +++ b/tests/functional/x86_64/test_nvme_migration.py
> > > > @@ -109,6 +109,7 @@ def launch_source_vm(self, vm):
> > > > rm -f /tmp/test_nvme_mig_workload.iter_finished; \
> > > > echo 3 > /proc/sys/vm/drop_caches; \
> > > > find /usr/bin -type f -exec cksum {} \\;; \
> > > > + sleep 0.3; \
> > > > touch /tmp/test_nvme_mig_workload.iter_finished; \
> > > > done) > /dev/null 2> /tmp/test_nvme_mig_workload.errors &",
> > > > prompt, vm=vm)
> > > > --
> > > > 2.47.3
> > > >
> > >
> > > --
> > > Peter Xu
> > >
> >
>
> --
> Peter Xu
>
