From: Alexander Mikhalitsyn <[email protected]>
Dear friends,
This patchset adds basic live migration support for
QEMU emulated NVMe device.
Implementation has some limitations:
- only one NVMe namespace is supported
- SMART counters are not preserved
- CMB is not supported
- PMR is not supported
- SPDM is not supported
- SR-IOV is not supported
I believe this is something I can support in next patchset versions or
separately on-demand (when usecase appears).
Testing.
This patch series was manually tested on:
- Debian 13.3 VM (kernel 6.12.69+deb13-amd64) using fio on *non-root* NVMe disk
(root disk was virtio-scsi):
time fio --name=nvme-verify \
--filename=/dev/nvme0n1 \
--size=5G \
--rw=randwrite \
--bs=4k \
--iodepth=16 \
--numjobs=1 \
--direct=0 \
--ioengine=io_uring \
--verify=crc32c \
--verify_fatal=1
- Windows Server 2022 VM (NVMe drive was a *root* disk) with opened browser
playing video.
No defects were found.
Git tree:
https://github.com/mihalicyn/qemu/commits/nvme-live-migration
Changelog for version 7:
- rebased on top of recent main
- addressed review comments from Stefan Hajnoczi:
- better incoming migration stream validation (SQ/CQids correctness)
- endianness bugs are fixed in qtest (validated on s390x)
- added RWB tags from Klaus
Changelog for version 6:
- rebased on top of:
https://gitlab.com/peterx/qemu/-/tree/vmstate-array-null
(see also
https://lore.kernel.org/all/[email protected])
- addressed review comments from Stefan Hajnoczi:
- supported "full CQ" case by serializing NvmeRequest state
- added qtest for NVMe device migration with full CQ
Changelog for version 5:
- rebased on top of
https://lore.kernel.org/all/[email protected]/
(as Peter has requested)
Changelog for version 4:
- vmstate dynamic array support reworked as suggested by Peter Xu
VMS_ARRAY_OF_POINTER_ALLOW_NULL flag was introduced
qtests were added
- NVMe migration blockers were reworked as Klaus has requested earlier
Now, instead of having "deny list" approach, we have more strict pattern
of NVMe features filtering and it should be harded to break migration when
adding new NVMe features.
Changelog for version 3:
- rebased
- simple functional test was added (in accordance with Klaus Jensen's review
comment)
$ meson test 'func-x86_64-nvme_migration' --setup thorough -C build
Changelog for version 2:
- full support for AERs (in-flight requests and queued events too)
Kind regards,
Alex
Alexander Mikhalitsyn (8):
tests/functional/migration: add VM launch/configure hooks
hw/nvme: add migration blockers for non-supported cases
hw/nvme: split nvme_init_sq/nvme_init_cq into helpers
hw/nvme: set CQE.sq_id earlier in nvme_process_sq
hw/nvme: unmap req->sg earlier in nvme_enqueue_req_completion
hw/nvme: add basic live migration support
tests/functional/x86_64: add migration test for NVMe device
tests/qtest/nvme-test: add migration test with full CQ
hw/nvme/ctrl.c | 1007 ++++++++++++++++-
hw/nvme/ns.c | 160 +++
hw/nvme/nvme.h | 12 +
hw/nvme/trace-events | 10 +
include/block/nvme.h | 12 +
tests/functional/migration.py | 22 +-
tests/functional/x86_64/meson.build | 1 +
.../functional/x86_64/test_nvme_migration.py | 159 +++
tests/qtest/nvme-test.c | 395 +++++++
9 files changed, 1742 insertions(+), 36 deletions(-)
create mode 100755 tests/functional/x86_64/test_nvme_migration.py
--
2.47.3
