[PULL 2/2] MAINTAINERS: Adjust migration documentation files

2024-04-07 Thread peterx
From: Avihai Horon Commit 8cb2f8b172e7 ("docs/migration: Create migration/ directory") changed migration documentation file structure but forgot to update the entries in the MAINTAINERS file. Commit 4c6f8a79ae53 ("docs/migration: Split 'dirty limit'") extracted dirty limit documentation to a

[PULL 0/2] Migration 20240407 patches

2024-04-07 Thread peterx
From: Peter Xu The following changes since commit ce64e6224affb8b4e4b019f76d2950270b391af5: Merge tag 'qemu-sparc-20240404' of https://github.com/mcayland/qemu into staging (2024-04-04 15:28:06 +0100) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags

[PULL 1/2] migration/postcopy: ensure preempt channel is ready before loading states

2024-04-07 Thread peterx
ink: https://lore.kernel.org/r/20240405034056.23933-1-wei.w.w...@intel.com [peterx: add a todo section, add Fixes and copy stable for 8.0+] Signed-off-by: Peter Xu --- migration/savevm.c | 21 + 1 file changed, 21 insertions(+) diff --git a/migration/savevm.c b/migration/save

[PULL 0/2] Migration 20240331 patches

2024-03-31 Thread peterx
From: Peter Xu The following changes since commit b9dbf6f9bf533564f6a4277d03906fcd32bb0245: Merge tag 'pull-tcg-20240329' of https://gitlab.com/rth7680/qemu into staging (2024-03-30 14:54:57 +) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags/migration

[PULL 2/2] migration/postcopy: Ensure postcopy_start() sets errp if it fails

2024-03-31 Thread peterx
From: Avihai Horon There are several places where postcopy_start() fails without setting errp. This can cause a null pointer de-reference, as in case of error, the caller of postcopy_start() copies/prints the error set in errp. Fix it by setting errp in all of postcopy_start() error paths. Cc:

[PULL 1/2] migration: Set migration error in migration_completion()

2024-03-31 Thread peterx
From: Avihai Horon After commit 9425ef3f990a ("migration: Use migrate_has_error() in close_return_path_on_source()"), close_return_path_on_source() assumes that migration error is set if an error occurs during migration. This may not be true if migration errors in migration_completion(). For

[PULL 2/3] migration/postcopy: Fix high frequency sync

2024-03-22 Thread peterx
From: Peter Xu With current code base I can observe extremely high sync count during precopy, as long as one enables postcopy-ram=on before switchover to postcopy. To provide some context of when QEMU decides to do a full sync: it checks must_precopy (which implies "data must be sent during

[PULL 3/3] migration/multifd: Fix clearing of mapped-ram zero pages

2024-03-22 Thread peterx
From: Fabiano Rosas When the zero page detection is done in the multifd threads, we need to iterate the second part of the pages->offset array and clear the file bitmap for each zero page. The piece of code we merged to do that is wrong. The reason this has passed all the tests is because the

[PULL 0/3] Migration 20240322 patches

2024-03-22 Thread peterx
From: Peter Xu The following changes since commit 853546f8128476eefb701d4a55b2781bb3a46faa: Merge tag 'pull-loongarch-20240322' of https://gitlab.com/gaosong/qemu into staging (2024-03-22 10:59:57 +) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags

[PULL 1/3] migration: Revert mapped-ram multifd support to fd: URI

2024-03-22 Thread peterx
From: Fabiano Rosas This reverts commit decdc76772c453ff1444612e910caa0d45cd8eac in full and also the relevant migration-tests from 7a09f092834641b7a793d50a3a261073bbb404a6. After the addition of the new QAPI-based migration address API in 8.2 we've been converting an "fd:" URI into a

[PATCH] migration/postcopy: Fix high frequency sync

2024-03-20 Thread peterx
From: Peter Xu On current code base I can observe extremely high sync count during precopy, as long as one enables postcopy-ram=on before switchover to postcopy. To provide some context of when we decide to do a full sync: we check must_precopy (which implies "data must be sent during precopy

[PULL 05/10] physmem: Fix migration dirty bitmap coherency with TCG memory access

2024-03-17 Thread peterx
From: Nicholas Piggin The fastpath in cpu_physical_memory_sync_dirty_bitmap() to test large aligned ranges forgot to bring the TCG TLB up to date after clearing some of the dirty memory bitmap bits. This can result in stores though the TCG TLB not setting the dirty memory bitmap and ultimately

[PULL 10/10] migration/multifd: Duplicate the fd for the outgoing_args

2024-03-17 Thread peterx
From: Fabiano Rosas We currently store the file descriptor used during the main outgoing channel creation to use it again when creating the multifd channels. Since this fd is used for the first iochannel, there's risk that the QIOChannel gets freed and the fd closed while outgoing_args.fd still

[PULL 03/10] physmem: Expose tlb_reset_dirty_range_all()

2024-03-17 Thread peterx
From: Philippe Mathieu-Daudé In order to call tlb_reset_dirty_range_all() outside of system/physmem.c, expose its prototype. Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Link: https://lore.kernel.org/r/20240312201458.79532-2-phi...@linaro.org Signed-off-by: Peter Xu

[PULL 01/10] io: Introduce qio_channel_file_new_dupfd

2024-03-17 Thread peterx
From: Fabiano Rosas Add a new helper function for creating a QIOChannelFile channel with a duplicated file descriptor. This saves the calling code from having to do error checking on the dup() call. Suggested-by: "Daniel P. Berrangé" Signed-off-by: Fabiano Rosas Reviewed-by: "Daniel P.

[PULL 04/10] physmem: Factor cpu_physical_memory_dirty_bits_cleared() out

2024-03-17 Thread peterx
From: Nicholas Piggin Signed-off-by: Nicholas Piggin Tested-by: Thomas Huth Message-ID: <20240219061731.232570-1-npig...@gmail.com> [PMD: Split patch in 2: part 1/2] Signed-off-by: Philippe Mathieu-Daudé Reviewed-by: Richard Henderson Link:

[PULL 02/10] migration: Fix error handling after dup in file migration

2024-03-17 Thread peterx
From: Fabiano Rosas The file migration code was allowing a possible -1 from a failed call to dup() to propagate into the new QIOFileChannel::fd before checking for validity. Coverity doesn't like that, possibly due to the the lseek(-1, ...) call that would ensue before returning from the channel

[PULL 06/10] migration: Skip only empty block devices

2024-03-17 Thread peterx
n Hajnoczi Reviewed-by: Kevin Wolf Link: https://lore.kernel.org/r/20240312120431.550054-1-...@redhat.com [peterx: fix "Suggested-by:"] Signed-off-by: Peter Xu --- migration/block.c | 5 - 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/migration/block.c b/migration/bloc

[PULL 09/10] migration/multifd: Ensure we're not given a socket for file migration

2024-03-17 Thread peterx
From: Fabiano Rosas When doing migration using the fd: URI, QEMU will fetch the file descriptor passed in via the monitor at fd_start_outgoing|incoming_migration(), which means the checks at migration_channels_and_transport_compatible() happen too soon and we don't know at that point whether the

[PULL 07/10] migration: cpr-reboot documentation

2024-03-17 Thread peterx
From: Steve Sistare Signed-off-by: Steve Sistare Reviewed-by: Cédric Le Goater Reviewed-by: Fabiano Rosas Link: https://lore.kernel.org/r/1710338119-330923-1-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- docs/devel/migration/CPR.rst | 147

[PULL 08/10] migration: Fix iocs leaks during file and fd migration

2024-03-17 Thread peterx
From: Fabiano Rosas The memory for the io channels is being leaked in three different ways during file migration: 1) if the offset check fails we never drop the ioc reference; 2) we allocate an extra channel for no reason; 3) if multifd is enabled but channel creation fails when calling

[PULL 00/10] Migration 20240317 patches

2024-03-17 Thread peterx
From: Peter Xu The following changes since commit 35ac6831d98e18e2c78c85c93e3a6ca1f1ae3e58: Merge tag 'net-pull-request' of https://github.com/jasowang/qemu into staging (2024-03-12 13:42:57 +) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags/migration

[PULL 20/34] migration: export migration_is_running

2024-03-11 Thread peterx
From: Steve Sistare Delete the MigrationState parameter from migration_is_running and move it to the public API in misc.h. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-5-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu ---

[PULL 31/34] migration/multifd: Implement zero page transmission on the multifd thread.

2024-03-11 Thread peterx
From: Hao Xiang 1. Add zero_pages field in MultiFDPacket_t. 2. Implements the zero page detection and handling on the multifd threads for non-compression, zlib and zstd compression backends. 3. Added a new value 'multifd' in ZeroPageDetection enumeration. 4. Adds zero page counters and updates

[PULL 30/34] migration/multifd: Add new migration option zero-page-detection.

2024-03-11 Thread peterx
From: Hao Xiang This new parameter controls where the zero page checking is running. 1. If this parameter is set to 'legacy', zero page checking is done in the migration main thread. 2. If this parameter is set to 'none', zero page checking is disabled. Signed-off-by: Hao Xiang Reviewed-by:

[PULL 25/34] migration: privatize colo interfaces

2024-03-11 Thread peterx
From: Steve Sistare Remove private migration interfaces from net/colo-compare.c and push them to migration/colo.c. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-10-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- migration/colo.c | 17

[PULL 27/34] migration: purge MigrationState from public interface

2024-03-11 Thread peterx
From: Steve Sistare Move remaining MigrationState references from the public file misc.h to the private file migration.h. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-12-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu ---

[PULL 21/34] migration: export vcpu_dirty_limit_period

2024-03-11 Thread peterx
From: Steve Sistare Define and export vcpu_dirty_limit_period to eliminate a dependency on MigrationState. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-6-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- include/migration/client-options.h

[PULL 15/34] migration: Fix format in error message

2024-03-11 Thread peterx
From: Anthony PERARD In file_write_ramblock_iov(), "offset" is "uintptr_t" and not "ram_addr_t". While usually they are both equivalent, this is not the case with CONFIG_XEN_BACKEND. Use the right format. This will fix build on 32-bit. Fixes: f427d90b9898 ("migration/multifd: Support outgoing

[PULL 29/34] migration/multifd: Allow clearing of the file_bmap from multifd

2024-03-11 Thread peterx
From: Fabiano Rosas We currently only need to clear the mapped-ram file bitmap from the migration thread during save_zero_page. We're about to add support for zero page detection on the multifd thread, so allow ramblock_set_file_bmap_atomic() to also clear the bits. Signed-off-by: Fabiano

[PULL 05/34] migration: Report error when shutdown fails

2024-03-11 Thread peterx
From: Cédric Le Goater This will help detect issues regarding I/O channels usage. Reviewed-by: Philippe Mathieu-Daudé Reviewed-by: Peter Xu Signed-off-by: Cédric Le Goater Link: https://lore.kernel.org/r/20240304122844.1888308-7-...@redhat.com Signed-off-by: Peter Xu ---

[PULL 32/34] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-03-11 Thread peterx
From: Hao Xiang 1. Add a dedicated handler for MigrationOps::ram_save_target_page in multifd live migration. 2. Refactor ram_save_target_page_legacy so that the legacy and multifd handlers don't have internal functions calling into each other. Signed-off-by: Hao Xiang Reviewed-by: Fabiano

[PULL 07/34] migration: Add documentation for SaveVMHandlers

2024-03-11 Thread peterx
From: Cédric Le Goater The SaveVMHandlers structure is still in use for complex subsystems and devices. Document the handlers since we are going to modify a few later. Reviewed-by: Peter Xu Signed-off-by: Cédric Le Goater Link: https://lore.kernel.org/r/20240304122844.1888308-9-...@redhat.com

[PULL 18/34] migration: export migration_is_setup_or_active

2024-03-11 Thread peterx
From: Steve Sistare Delete the MigrationState parameter from migration_is_setup_or_active and move it to the public API in misc.h. Signed-off-by: Steve Sistare Reviewed-by: Philippe Mathieu-Daudé Link: https://lore.kernel.org/r/1710179338-294359-3-git-send-email-steven.sist...@oracle.com

[PULL 22/34] migration: migration_thread_is_self

2024-03-11 Thread peterx
From: Steve Sistare Define and export migration_thread_is_self to eliminate a dependency on MigrationState. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-7-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- include/migration/misc.h | 1 +

[PULL 10/34] migration/rdma: Fix a memory issue for migration

2024-03-11 Thread peterx
notrxq6c...@mail.gmail.com Signed-off-by: Yu Zhang [peterx: use g_strdup() instead of g_strdup_printf(), per Zhijian] Signed-off-by: Peter Xu --- migration/rdma.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/migration/rdma.c b/migration/rdma.c index a355dcea89..855753c671 100

[PULL 28/34] migration/multifd: Allow zero pages in file migration

2024-03-11 Thread peterx
From: Fabiano Rosas Currently, it's an error to have no data pages in the multifd file migration because zero page detection is done in the migration thread and zero pages don't reach multifd. This is enforced with the pages->num assert. We're about to add zero page detection on the multifd

[PULL 34/34] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-03-11 Thread peterx
From: Hao Xiang Now that zero page checking is done on the multifd sender threads by default, we still provide an option for backward compatibility. This change adds a qtest migration test case to set the zero-page-detection option to "legacy" and run multifd migration with zero page checking on

[PULL 26/34] migration: delete unused accessors

2024-03-11 Thread peterx
From: Steve Sistare Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-11-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- include/migration/misc.h | 3 --- migration/migration.c| 10 -- 2 files changed, 13 deletions(-) diff --git

[PULL 09/34] migration/multifd: Don't fsync when closing QIOChannelFile

2024-03-11 Thread peterx
-1-faro...@suse.de Link: https://lore.kernel.org/r/20240305174332.2553-1-faro...@suse.de [peterx: add more comment to the qio_channel_close()] Signed-off-by: Peter Xu --- docs/devel/migration/main.rst | 3 ++- io/channel-file.c | 5 - migration/multifd.c | 28 +++

[PULL 33/34] migration/multifd: Enable multifd zero page checking by default.

2024-03-11 Thread peterx
From: Hao Xiang 1. Set default "zero-page-detection" option to "multifd". Now zero page checking can be done in the multifd threads and this becomes the default configuration. 2. Handle migration QEMU9.0 -> QEMU8.2 compatibility. We provide backward compatibility where zero page checking is done

[PULL 06/34] migration: Remove SaveStateHandler and LoadStateHandler typedefs

2024-03-11 Thread peterx
From: Cédric Le Goater They are only used once. Reviewed-by: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Cédric Le Goater Link: https://lore.kernel.org/r/20240304122844.1888308-8-...@redhat.com Signed-off-by: Peter Xu --- include/migration/register.h | 4 ++--

[PULL 24/34] migration: migration_file_set_error

2024-03-11 Thread peterx
From: Steve Sistare Define and export migration_file_set_error to eliminate a dependency on MigrationState. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-9-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- include/migration/misc.h | 2 ++

[PULL 12/34] physmem: Reduce local variable scope in flatview_read/write_continue()

2024-03-11 Thread peterx
From: Jonathan Cameron Precursor to factoring out the inner loops for reuse. Reviewed-by: Peter Xu Signed-off-by: Jonathan Cameron Reviewed-by: David Hildenbrand Reviewed-by: Philippe Mathieu-Daudé Link: https://lore.kernel.org/r/20240307153710.30907-3-jonathan.came...@huawei.com

[PULL 01/34] migration: Don't serialize devices in qemu_savevm_state_iterate()

2024-03-11 Thread peterx
From: Avihai Horon Commit 90697be8896c ("live migration: Serialize vmstate saving in stage 2") introduced device serialization in qemu_savevm_state_iterate(). The rationale behind it was to first complete migration of slower changing block devices and only then migrate the RAM, to avoid sending

[PULL 23/34] migration: migration_is_device

2024-03-11 Thread peterx
From: Steve Sistare Define and export migration_is_device to eliminate a dependency on MigrationState. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-8-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- include/migration/misc.h | 1 +

[PULL 17/34] migration: remove migration.h references

2024-03-11 Thread peterx
From: Steve Sistare Remove migration.h from files that no longer need it due to previous commits. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-2-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- hw/vfio/container.c| 1 -

[PULL 00/34] Migration 20240311 patches

2024-03-11 Thread peterx
From: Peter Xu The following changes since commit 7489f7f3f81dcb776df8c1b9a9db281fc21bf05f: Merge tag 'hw-misc-20240309' of https://github.com/philmd/qemu into staging (2024-03-09 20:12:21 +) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags/migration

[PULL 08/34] migration: Do not call PRECOPY_NOTIFY_SETUP notifiers in case of error

2024-03-11 Thread peterx
From: Cédric Le Goater When commit bd2270608fa0 ("migration/ram.c: add a notifier chain for precopy") added PRECOPY_NOTIFY_SETUP notifiers at the end of qemu_savevm_state_setup(), it didn't take into account a possible error in the loop calling vmstate_save() or .save_setup() handlers. Check

[PULL 16/34] migration: export fewer options

2024-03-11 Thread peterx
From: Steve Sistare A small number of migration options are accessed by migration clients, but to see them clients must include all of options.h, which is mostly for migration core code. migrate_mode() in particular will be needed by multiple clients. Refactor the option declarations so

[PULL 03/34] vfio/migration: Add a note about migration rate limiting

2024-03-11 Thread peterx
From: Avihai Horon VFIO migration buffer size is currently limited to 1MB. Therefore, there is no need to check if migration rate exceeded, as in the worst case it will exceed by only 1MB. However, if the buffer size is later changed to a bigger value, vfio_save_iterate() should enforce

[PULL 04/34] migration/ram: add additional check

2024-03-11 Thread peterx
From: Maksim Davydov If a migration stream is broken, the address and flag reading can return zero. Thus, an irrelevant flag error will be returned instead of EIO. It can be fixed by additional check after the reading. Signed-off-by: Maksim Davydov Link:

[PULL 02/34] vfio/migration: Refactor vfio_save_state() return value

2024-03-11 Thread peterx
From: Avihai Horon Currently, vfio_save_state() returns 1 regardless of whether there is more data to send or not. This was done to prevent a fast changing VFIO device from potentially blocking other devices from sending their data, as qemu_savevm_state_iterate() serialized devices. Now that

[PULL 19/34] migration: export migration_is_active

2024-03-11 Thread peterx
From: Steve Sistare Delete the MigrationState parameter from migration_is_active so it can be exported and used without including migration.h. Signed-off-by: Steve Sistare Link: https://lore.kernel.org/r/1710179338-294359-4-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu ---

[PULL 14/34] physmem: Fix wrong address in large address_space_read/write_cached_slow()

2024-03-11 Thread peterx
From: Jonathan Cameron If the access is bigger than the MemoryRegion supports, flatview_read/write_continue() will attempt to update the Memory Region. but the address passed to flatview_translate() is relative to the cache, not to the FlatView. On arm/virt with interleaved CXL memory emulation

[PULL 13/34] physmem: Factor out body of flatview_read/write_continue() loop

2024-03-11 Thread peterx
From: Jonathan Cameron This code will be reused for the address_space_cached accessors shortly. Also reduce scope of result variable now we aren't directly calling this in the loop. Signed-off-by: Jonathan Cameron Reviewed-by: David Hildenbrand Link:

[PULL 11/34] physmem: Rename addr1 to more informative mr_addr in flatview_read/write() and similar

2024-03-11 Thread peterx
From: Jonathan Cameron The calls to flatview_read/write[_continue]() have parameters addr and addr1 but the names give no indication of what they are addresses of. Rename addr1 to mr_addr to reflect that it is the translated address offset within the MemoryRegion returned by

[PULL 24/27] migration/multifd: Support incoming mapped-ram stream format

2024-03-03 Thread peterx
From: Fabiano Rosas For the incoming mapped-ram migration we need to read the ramblock headers, get the pages bitmap and send the host address of each non-zero page to the multifd channel thread for writing. Usage on HMP is: (qemu) migrate_set_capability multifd on (qemu)

[PULL 18/27] migration/multifd: Allow receiving pages without packets

2024-03-03 Thread peterx
From: Fabiano Rosas Currently multifd does not need to have knowledge of pages on the receiving side because all the information needed is within the packets that come in the stream. We're about to add support to mapped-ram migration, which cannot use packets because it expects the ramblock

[PULL 07/27] io: implement io_pwritev/preadv for QIOChannelFile

2024-03-03 Thread peterx
From: Nikolay Borisov The upcoming 'mapped-ram' feature will require qemu to write data to (and restore from) specific offsets of the migration file. Add a minimal implementation of pwritev/preadv and expose them via the io_pwritev and io_preadv interfaces. Signed-off-by: Nikolay Borisov

[PULL 12/27] migration/ram: Add outgoing 'mapped-ram' migration

2024-03-03 Thread peterx
From: Fabiano Rosas Implement the outgoing migration side for the 'mapped-ram' capability. A bitmap is introduced to track which pages have been written in the migration file. Pages are written at a fixed location for every ramblock. Zero pages are ignored as they'd be zero in the destination

[PULL 03/27] tests/migration: Set compression level in migration tests

2024-03-03 Thread peterx
From: Bryan Zhang Adds calls to set compression level for `zstd` and `zlib` migration tests, just to make sure that the calls work. Signed-off-by: Bryan Zhang Link: https://lore.kernel.org/r/20240301035901.4006936-3-bryan.zh...@bytedance.com Signed-off-by: Peter Xu ---

[PULL 13/27] migration/ram: Add incoming 'mapped-ram' migration

2024-03-03 Thread peterx
From: Fabiano Rosas Add the necessary code to parse the format changes for the 'mapped-ram' capability. One of the more notable changes in behavior is that in the 'mapped-ram' case ram pages are restored in one go rather than constantly looping through the migration stream. Signed-off-by:

[PULL 21/27] migration/multifd: Add incoming QIOChannelFile support

2024-03-03 Thread peterx
From: Fabiano Rosas On the receiving side we don't need to differentiate between main channel and threads, so whichever channel is defined first gets to be the main one. And since there are no packets, use the atomic channel count to index into the params array. Reviewed-by: Peter Xu

[PULL 27/27] migration/multifd: Document two places for mapped-ram

2024-03-03 Thread peterx
From: Peter Xu Add two documentations for mapped-ram migration on two spots that may not be extremely clear. Reviewed-by: Fabiano Rosas Link: https://lore.kernel.org/r/20240301091524.39900-1-pet...@redhat.com Cc: Prasad Pandit [peterx: fix two English errors per Prasad] Signed-off-by: Peter

[PULL 22/27] migration/multifd: Prepare multifd sync for mapped-ram migration

2024-03-03 Thread peterx
From: Fabiano Rosas The mapped-ram migration can be performed live or non-live, but it is always asynchronous, i.e. the source machine and the destination machine are not migrating at the same time. We only need some pieces of the multifd sync operations. multifd_send_sync_main()

[PULL 23/27] migration/multifd: Support outgoing mapped-ram stream format

2024-03-03 Thread peterx
From: Fabiano Rosas The new mapped-ram stream format uses a file transport and puts ram pages in the migration file at their respective offsets and can be done in parallel by using the pwritev system call which takes iovecs and an offset. Add support to enabling the new format along with

[PULL 25/27] migration/multifd: Add mapped-ram support to fd: URI

2024-03-03 Thread peterx
From: Fabiano Rosas If we receive a file descriptor that points to a regular file, there's nothing stopping us from doing multifd migration with mapped-ram to that file. Enable the fd: URI to work with multifd + mapped-ram. Note that the fds passed into multifd are duplicated because we want

[PULL 10/27] migration/ram: Introduce 'mapped-ram' migration capability

2024-03-03 Thread peterx
From: Fabiano Rosas Add a new migration capability 'mapped-ram'. The core of the feature is to ensure that RAM pages are mapped directly to offsets in the resulting migration file instead of being streamed at arbitrary points. The reasons why we'd want such behavior are: - The resulting file

[PULL 17/27] migration/multifd: Allow multifd without packets

2024-03-03 Thread peterx
From: Fabiano Rosas For the upcoming support to the new 'mapped-ram' migration stream format, we cannot use multifd packets because each write into the ramblock section in the migration file is expected to contain only the guest pages. They are written at their respective offsets relative to the

[PULL 20/27] migration/multifd: Add outgoing QIOChannelFile support

2024-03-03 Thread peterx
From: Fabiano Rosas Allow multifd to open file-backed channels. This will be used when enabling the mapped-ram migration stream format which expects a seekable transport. The QIOChannel read and write methods will use the preadv/pwritev versions which don't update the file offset at each call

[PULL 26/27] tests/qtest/migration: Add a multifd + mapped-ram migration test

2024-03-03 Thread peterx
From: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas Link: https://lore.kernel.org/r/20240229153017.2221-24-faro...@suse.de Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 68 1 file changed, 68 insertions(+) diff --git

[PULL 19/27] migration/multifd: Add a wrapper for channels_created

2024-03-03 Thread peterx
From: Fabiano Rosas We'll need to access multifd_send_state->channels_created from outside multifd.c, so introduce a helper for that. Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas Link: https://lore.kernel.org/r/20240229153017.2221-17-faro...@suse.de Signed-off-by: Peter Xu ---

[PULL 14/27] tests/qtest/migration: Add tests for mapped-ram file-based migration

2024-03-03 Thread peterx
From: Fabiano Rosas Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas Link: https://lore.kernel.org/r/20240229153017.2221-12-faro...@suse.de Signed-off-by: Peter Xu --- tests/qtest/migration-test.c | 59 1 file changed, 59 insertions(+) diff --git

[PULL 04/27] migration/multifd: Cleanup multifd_recv_sync_main

2024-03-03 Thread peterx
From: Fabiano Rosas Some minor cleanups and documentation for multifd_recv_sync_main. Use thread_count as done in other parts of the code. Remove p->id from the multifd_recv_state sync, since that is global and not tied to a channel. Add documentation for the sync steps. Reviewed-by: Peter Xu

[PULL 15/27] migration/multifd: Rename MultiFDSend|RecvParams::data to compress_data

2024-03-03 Thread peterx
From: Fabiano Rosas Use a more specific name for the compression data so we can use the generic for the multifd core code. Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas Link: https://lore.kernel.org/r/20240229153017.2221-13-faro...@suse.de Signed-off-by: Peter Xu ---

[PULL 16/27] migration/multifd: Decouple recv method from pages

2024-03-03 Thread peterx
From: Fabiano Rosas Next patches will abstract the type of data being received by the channels, so do some cleanup now to remove references to pages and dependency on 'normal_num'. Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas Link:

[PULL 00/27] Migration next patches

2024-03-03 Thread peterx
From: Peter Xu The following changes since commit c0c6a0e3528b88aaad0b9d333e295707a195587b: Merge tag 'migration-next-pull-request' of https://gitlab.com/peterx/qemu into staging (2024-02-28 17:27:10 +) are available in the Git repository at: https://gitlab.com/peterx/qemu.git tags

[PULL 08/27] io: fsync before closing a file channel

2024-03-03 Thread peterx
From: Fabiano Rosas Make sure the data is flushed to disk before closing file channels. This is to ensure data is on disk and not lost in the event of a host crash. This is currently being implemented to affect the migration code when migrating to a file, but all QIOChannelFile users should

[PULL 02/27] migration: Properly apply migration compression level parameters

2024-03-03 Thread peterx
From: Bryan Zhang Some glue code was missing, so that using `qmp_migrate_set_parameters` to set `multifd-zstd-level` or `multifd-zlib-level` did not work. This commit adds the glue code to fix that. Signed-off-by: Bryan Zhang Link:

[PULL 06/27] io: Add generic pwritev/preadv interface

2024-03-03 Thread peterx
From: Nikolay Borisov Introduce basic pwritev/preadv support in the generic channel layer. Specific implementation will follow for the file channel as this is required in order to support migration streams with fixed location of each ram page. Signed-off-by: Nikolay Borisov Reviewed-by:

[PULL 11/27] migration: Add mapped-ram URI compatibility check

2024-03-03 Thread peterx
From: Fabiano Rosas The mapped-ram migration format needs a channel that supports seeking to be able to write each page to an arbitrary offset in the migration stream. Reviewed-by: "Daniel P. Berrangé" Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas Link:

[PULL 01/27] migration: massage cpr-reboot documentation

2024-03-03 Thread peterx
-steven.sist...@oracle.com [peterx: s/qemu/QEMU per Markus's suggestion] Reviewed-by: Markus Armbruster Signed-off-by: Peter Xu --- qapi/migration.json | 46 +++-- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/qapi/migration.json b/qapi

[PULL 05/27] io: add and implement QIO_CHANNEL_FEATURE_SEEKABLE for channel file

2024-03-03 Thread peterx
From: Nikolay Borisov Add a generic QIOChannel feature SEEKABLE which would be used by the qemu_file* apis. For the time being this will be only implemented for file channels. Signed-off-by: Nikolay Borisov Reviewed-by: "Daniel P. Berrangé" Reviewed-by: Peter Xu Signed-off-by: Fabiano Rosas

[PULL 09/27] migration/qemu-file: add utility methods for working with seekable channels

2024-03-03 Thread peterx
From: Fabiano Rosas Add utility methods that will be needed when implementing 'mapped-ram' migration capability. Signed-off-by: Fabiano Rosas Reviewed-by: "Daniel P. Berrangé" Link: https://lore.kernel.org/r/20240229153017.2221-7-faro...@suse.de Signed-off-by: Peter Xu ---

[PATCH] migration/multifd: Document two places for mapped-ram

2024-03-01 Thread peterx
From: Peter Xu Add two documentations for mapped-ram migration on two spots that may not be extremely clear. Signed-off-by: Peter Xu --- Based-on: <20240229153017.2221-1-faro...@suse.de> --- migration/multifd.c | 12 migration/ram.c | 8 +++- 2 files changed, 19

[PULL 21/25] migration: update cpr-reboot description

2024-02-27 Thread peterx
From: Steve Sistare Clarify qapi for cpr-reboot migration mode, and add vfio support. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Link: https://lore.kernel.org/r/1708622920-68779-14-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- qapi/migration.json | 35

[PULL 17/25] migration: per-mode notifiers

2024-02-27 Thread peterx
From: Steve Sistare Keep a separate list of migration notifiers for each migration mode. Suggested-by: Peter Xu Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Reviewed-by: David Hildenbrand Link: https://lore.kernel.org/r/1708622920-68779-8-git-send-email-steven.sist...@oracle.com

[PULL 13/25] migration: convert to NotifierWithReturn

2024-02-27 Thread peterx
-by: David Hildenbrand Link: https://lore.kernel.org/r/1708622920-68779-4-git-send-email-steven.sist...@oracle.com [peterx: dropped unexpected update to roms/seabios-hppa] Signed-off-by: Peter Xu --- include/hw/vfio/vfio-common.h | 2 +- include/hw/virtio/virtio-net.h | 2 +- include/migration

[PULL 25/25] migration: Use migrate_has_error() in close_return_path_on_source()

2024-02-27 Thread peterx
From: Cédric Le Goater close_return_path_on_source() retrieves the migration error from the the QEMUFile '->to_dst_file' to know if a shutdown is required. This shutdown is required to exit the return-path thread. Avoid relying on '->to_dst_file' and use migrate_has_error() instead. (using

[PULL 16/25] migration: MigrationNotifyFunc

2024-02-27 Thread peterx
From: Steve Sistare Define MigrationNotifyFunc to improve type safety and simplify migration notifiers. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Reviewed-by: David Hildenbrand Link: https://lore.kernel.org/r/1708622920-68779-7-git-send-email-steven.sist...@oracle.com

[PULL 22/25] migration: options incompatible with cpr

2024-02-27 Thread peterx
From: Steve Sistare Fail the migration request if options are set that are incompatible with cpr. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Link: https://lore.kernel.org/r/1708622920-68779-15-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu --- qapi/migration.json

[PULL 19/25] migration: notifier error checking

2024-02-27 Thread peterx
From: Steve Sistare Check the status returned by migration notifiers for event type MIG_EVENT_PRECOPY_SETUP, and report errors. None of the notifiers return an error status at this time. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Link:

[PULL 24/25] migration: Join the return path thread before releasing to_dst_file

2024-02-27 Thread peterx
From: Fabiano Rosas The return path thread might hang at a blocking system call. Before joining the thread we might need to issue a shutdown() on the socket file descriptor to release it. To determine whether the shutdown() is necessary we look at the QEMUFile error. Make sure we only clean up

[PULL 15/25] migration: remove postcopy_after_devices

2024-02-27 Thread peterx
From: Steve Sistare postcopy_after_devices and migration_in_postcopy_after_devices are no longer used, so delete them. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Link: https://lore.kernel.org/r/1708622920-68779-6-git-send-email-steven.sist...@oracle.com Signed-off-by: Peter Xu ---

[PULL 20/25] migration: stop vm for cpr

2024-02-27 Thread peterx
From: Steve Sistare When migration for cpr is initiated, stop the vm and set state RUN_STATE_FINISH_MIGRATE before ram is saved. This eliminates the possibility of ram and device state being out of sync, and guarantees that a guest in the suspended state remains suspended, because qmp_cont

[PULL 23/25] migration: Fix qmp_query_migrate mbps value

2024-02-27 Thread peterx
From: Fabiano Rosas The QMP command query_migrate might see incorrect throughput numbers if it runs after we've set the migration completion status but before migration_calculate_complete() has updated s->total_time and s->mbps. The migration status would show COMPLETED, but the throughput

[PULL 18/25] migration: refactor migrate_fd_connect failures

2024-02-27 Thread peterx
From: Steve Sistare Move common code for the error path in migrate_fd_connect to a shared fail label. No functional change. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Reviewed-by: David Hildenbrand Link:

[PULL 12/25] migration: remove error from notifier data

2024-02-27 Thread peterx
From: Steve Sistare Remove the error object from opaque data passed to notifiers. Use the new error parameter passed to the notifier instead. Signed-off-by: Steve Sistare Reviewed-by: Peter Xu Reviewed-by: David Hildenbrand Link:

[PULL 05/25] migration/multifd: Release recv sem_sync earlier

2024-02-27 Thread peterx
From: Fabiano Rosas Now that multifd_recv_terminate_threads() is called only once, release the recv side sem_sync earlier like we do for the send side. Signed-off-by: Fabiano Rosas Reviewed-by: Peter Xu Link: https://lore.kernel.org/r/20240220224138.24759-6-faro...@suse.de Signed-off-by:

  1   2   3   4   >