[PATCH v4 09/14] migration/multifd: Prepare to introduce DSA acceleration on the multifd path.

2024-04-24 Thread Hao Xiang
1. Refactor multifd_send_thread function. 2. Introduce the batch task structure in MultiFDSendParams. Signed-off-by: Hao Xiang --- include/qemu/dsa.h | 51 +++-- migration/multifd.c | 5 + migration/multifd.h | 2 ++ util/dsa.c | 51

[PATCH v4 07/14] util/dsa: Implement DSA task asynchronous submission and wait for completion.

2024-04-24 Thread Hao Xiang
* Add a DSA task completion callback. * DSA completion thread will call the tasks's completion callback on every task/batch task completion. * DSA submission path to wait for completion. * Implement CPU fallback if DSA is not able to complete the task. Signed-off-by: Hao Xiang Signed-o

[PATCH v4 10/14] migration/multifd: Enable DSA offloading in multifd sender path.

2024-04-24 Thread Hao Xiang
sender thread submits a batch of pages to DSA hardware and waits for the DSA completion thread to signal for work completion. Signed-off-by: Hao Xiang --- migration/multifd-zero-page.c | 99 +-- migration/multifd.c | 27 +- migration/multifd.h

[PATCH v4 04/14] util/dsa: Implement DSA task enqueue and dequeue.

2024-04-24 Thread Hao Xiang
* Use a safe thread queue for DSA task enqueue/dequeue. * Implement DSA task submission. * Implement DSA batch task submission. Signed-off-by: Hao Xiang --- include/qemu/dsa.h | 28 +++ util/dsa.c | 201 + 2 files changed, 229 insertions

[PATCH v4 03/14] util/dsa: Implement DSA device start and stop logic.

2024-04-24 Thread Hao Xiang
* DSA device open and close. * DSA group contains multiple DSA devices. * DSA group configure/start/stop/clean. Signed-off-by: Hao Xiang Signed-off-by: Bryan Zhang --- include/qemu/dsa.h | 72 +++ util/dsa.c | 316 + util/meson.build

[PATCH v4 00/14] Use Intel DSA accelerator to offload zero page checking in multifd live migration.

2024-04-24 Thread Hao Xiang
: 1. Added unit tests for cover the added code path in dsa.c 2. Added integration tests to cover multifd live migration using DSA offloading. * Patchset Apply this patchset on top of commit 85b597413d4370cb168f711192eaef2eb70535ac Hao Xiang (14): meson: Introduce new instruction set enqcmd to the bu

[PATCH v4 11/14] migration/multifd: Add migration option set packet size.

2024-04-24 Thread Hao Xiang
The current multifd packet size is 128 * 4kb. This change adds an option to set the packet size. Both sender and receiver needs to set the same packet size for things to work. Signed-off-by: Hao Xiang --- migration/options.c | 36 migration/options.h | 1

[PATCH v4 02/14] util/dsa: Add dependency idxd.

2024-04-24 Thread Hao Xiang
Idxd is the device driver for DSA (Intel Data Streaming Accelerator). The driver is fully functioning since Linux kernel 5.19. This change adds the driver's header file used for userspace development. Signed-off-by: Hao Xiang --- linux-headers/linux/idxd.h

[PATCH v4 08/14] migration/multifd: Add new migration option for multifd DSA offloading.

2024-04-24 Thread Hao Xiang
Intel DSA offloading is an optional feature that turns on if proper hardware and software stack is available. To turn on DSA offloading in multifd live migration: multifd-dsa-accel="[dsa_dev_path1] [dsa_dev_path2] ... [dsa_dev_pathX]" This feature is turned off by default. Signed-o

[PATCH v4 12/14] migration/multifd: Enable set packet size migration option.

2024-04-24 Thread Hao Xiang
packet size is useful for performance tuning. Set the option: migrate_set_parameter multifd-packet-size 4190208 Signed-off-by: Hao Xiang --- migration/migration-hmp-cmds.c | 7 +++ migration/multifd-zlib.c | 6 -- migration/multifd-zstd.c | 6 -- migration/multifd.c

[PATCH v4 14/14] migration/multifd: Add integration tests for multifd with Intel DSA offloading.

2024-04-24 Thread Hao Xiang
* Add test case to start and complete multifd live migration with DSA offloading enabled. * Add test case to start and cancel multifd live migration with DSA offloading enabled. Signed-off-by: Bryan Zhang Signed-off-by: Hao Xiang --- tests/qtest/migration-test.c | 77

[PATCH v4 05/14] util/dsa: Implement DSA task asynchronous completion thread model.

2024-04-24 Thread Hao Xiang
n checking. Signed-off-by: Hao Xiang --- include/qemu/dsa.h | 1 + util/dsa.c | 274 - 2 files changed, 274 insertions(+), 1 deletion(-) diff --git a/include/qemu/dsa.h b/include/qemu/dsa.h index 37cae8d9d2..2513192a2b 100644 --- a/include/qemu/d

[PATCH v4 13/14] util/dsa: Add unit test coverage for Intel DSA task submission and completion.

2024-04-24 Thread Hao Xiang
* Test DSA start and stop path. * Test DSA configure and cleanup path. * Test DSA task submission and completion path. Signed-off-by: Bryan Zhang Signed-off-by: Hao Xiang --- tests/unit/meson.build | 6 + tests/unit/test-dsa.c | 499 + 2 files changed

[PATCH v4 01/14] meson: Introduce new instruction set enqcmd to the build system.

2024-04-24 Thread Hao Xiang
Enable instruction set enqcmd in build. Signed-off-by: Hao Xiang --- meson.build | 14 ++ meson_options.txt | 2 ++ scripts/meson-buildoptions.sh | 3 +++ 3 files changed, 19 insertions(+) diff --git a/meson.build b/meson.build index 95cee7046e

[PATCH v4 06/14] util/dsa: Implement zero page checking in DSA task.

2024-04-24 Thread Hao Xiang
at least two individual tasks. There are times we need to submit a single task and hence a single DSA task submission is also required. Signed-off-by: Hao Xiang Signed-off-by: Bryan Zhang --- include/qemu/dsa.h | 18 util/dsa.c | 247 + 2

[PATCH v6 0/7] Introduce multifd zero page checking.

2024-03-11 Thread Hao Xiang
--+ |multifd-4-threads | 4143 | ++ Apply this patchset on top of commit 7489f7f3f81dcb776df8c1b9a9db281fc21bf05f Fabiano Rosas (2): migration/multifd: Allow zero pages in file migration migration/multifd: Allow clearing of the file_bmap from

Re: [PATCH v4 3/7] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-03-11 Thread hao . xiang
March 11, 2024 at 6:20 AM, "Peter Xu" wrote: > > On Sat, Mar 09, 2024 at 02:06:33AM +, hao.xi...@linux.dev wrote: > > > > > > @@ -1122,10 +1122,6 @@ static int save_zero_page(RAMState *rs, > > > PageSearchStatus *pss, > > > > > QEMUFile *file = pss->pss_channel; > > > > > int len =

[PATCH v6 1/7] migration/multifd: Allow zero pages in file migration

2024-03-11 Thread Hao Xiang
From: Fabiano Rosas Currently, it's an error to have no data pages in the multifd file migration because zero page detection is done in the migration thread and zero pages don't reach multifd. This is enforced with the pages->num assert. We're about to add zero page detection on the multifd thre

[PATCH v6 5/7] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-03-11 Thread Hao Xiang
From: Hao Xiang 1. Add a dedicated handler for MigrationOps::ram_save_target_page in multifd live migration. 2. Refactor ram_save_target_page_legacy so that the legacy and multifd handlers don't have internal functions calling into each other. Signed-off-by: Hao Xiang Reviewed-by: Fa

[PATCH v6 6/7] migration/multifd: Enable multifd zero page checking by default.

2024-03-11 Thread Hao Xiang
From: Hao Xiang 1. Set default "zero-page-detection" option to "multifd". Now zero page checking can be done in the multifd threads and this becomes the default configuration. 2. Handle migration QEMU9.0 -> QEMU8.2 compatibility. We provide backward compatibility where zer

[PATCH v6 7/7] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-03-11 Thread Hao Xiang
From: Hao Xiang Now that zero page checking is done on the multifd sender threads by default, we still provide an option for backward compatibility. This change adds a qtest migration test case to set the zero-page-detection option to "legacy" and run multifd migration with zero page c

[PATCH v6 3/7] migration/multifd: Add new migration option zero-page-detection.

2024-03-11 Thread Hao Xiang
From: Hao Xiang This new parameter controls where the zero page checking is running. 1. If this parameter is set to 'legacy', zero page checking is done in the migration main thread. 2. If this parameter is set to 'none', zero page checking is disabled. Signed-off-by: Ha

[PATCH v6 2/7] migration/multifd: Allow clearing of the file_bmap from multifd

2024-03-11 Thread Hao Xiang
From: Fabiano Rosas We currently only need to clear the mapped-ram file bitmap from the migration thread during save_zero_page. We're about to add support for zero page detection on the multifd thread, so allow ramblock_set_file_bmap_atomic() to also clear the bits. Signed-off-by: Fabiano Rosas

[PATCH v6 4/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-03-11 Thread Hao Xiang
From: Hao Xiang 1. Add zero_pages field in MultiFDPacket_t. 2. Implements the zero page detection and handling on the multifd threads for non-compression, zlib and zstd compression backends. 3. Added a new value 'multifd' in ZeroPageDetection enumeration. 4. Adds zero page counters a

Re: [PATCH v4 7/7] Update maintainer contact for migration multifd zero page checking acceleration.

2024-03-09 Thread hao . xiang
> > On Sun, Mar 3, 2024 at 11:34 PM Peter Xu wrote: > > > > > On Fri, Mar 01, 2024 at 02:28:29AM +, Hao Xiang wrote: > > > > Add myself to maintain multifd zero page checking acceleration function. > > > > Signed-off-by: Hao X

Re: [PATCH v4 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-03-09 Thread hao . xiang
> > On Mon, Mar 4, 2024 at 10:24 AM Fabiano Rosas wrote: > > > > > > Fabiano Rosas writes: > > > > Peter Xu writes: > > > > On Fri, Mar 01, 2024 at 02:28:24AM +, Hao Xiang wrote: > > > > -GlobalProperty hw_c

[PATCH v5 3/7] migration/multifd: Add new migration option zero-page-detection.

2024-03-09 Thread Hao Xiang
From: Hao Xiang This new parameter controls where the zero page checking is running. 1. If this parameter is set to 'legacy', zero page checking is done in the migration main thread. 2. If this parameter is set to 'none', zero page checking is disabled. Signed-off-by: Ha

[PATCH v5 6/7] migration/multifd: Enable multifd zero page checking by default.

2024-03-08 Thread Hao Xiang
From: Hao Xiang 1. Set default "zero-page-detection" option to "multifd". Now zero page checking can be done in the multifd threads and this becomes the default configuration. 2. Handle migration QEMU9.0 -> QEMU8.2 compatibility. We provide backward compatibility where zer

[PATCH v5 0/7] Introduce multifd zero page checking.

2024-03-08 Thread Hao Xiang
From: Hao Xiang v5 update: * Move QEMU9.0 -> QEMU8.2 migration backward compatibility handling into the patch where "multifd" zero page checking becomes the default option. * A few function renaming according to feedback. * Fix bug in multifd_send_zero_page_detect. * Rebase on the n

[PATCH v5 2/7] migration/multifd: Allow clearing of the file_bmap from multifd

2024-03-08 Thread Hao Xiang
From: Fabiano Rosas We currently only need to clear the mapped-ram file bitmap from the migration thread during save_zero_page. We're about to add support for zero page detection on the multifd thread, so allow ramblock_set_file_bmap_atomic() to also clear the bits. Signed-off-by: Fabiano Rosas

[PATCH v5 7/7] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-03-08 Thread Hao Xiang
From: Hao Xiang Now that zero page checking is done on the multifd sender threads by default, we still provide an option for backward compatibility. This change adds a qtest migration test case to set the zero-page-detection option to "legacy" and run multifd migration with zero page c

[PATCH v5 4/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-03-08 Thread Hao Xiang
From: Hao Xiang 1. Add zero_pages field in MultiFDPacket_t. 2. Implements the zero page detection and handling on the multifd threads for non-compression, zlib and zstd compression backends. 3. Added a new value 'multifd' in ZeroPageDetection enumeration. 4. Adds zero page counters a

[PATCH v5 5/7] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-03-08 Thread Hao Xiang
From: Hao Xiang 1. Add a dedicated handler for MigrationOps::ram_save_target_page in multifd live migration. 2. Refactor ram_save_target_page_legacy so that the legacy and multifd handlers don't have internal functions calling into each other. Signed-off-by: Hao Xiang Reviewed-by: Fa

[PATCH v5 1/7] migration/multifd: Allow zero pages in file migration

2024-03-08 Thread Hao Xiang
From: Fabiano Rosas Currently, it's an error to have no data pages in the multifd file migration because zero page detection is done in the migration thread and zero pages don't reach multifd. This is enforced with the pages->num assert. We're about to add zero page detection on the multifd thre

Re: [External] Re: [PATCH v4 6/7] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-03-08 Thread hao . xiang
> > On Thu, Feb 29, 2024 at 11:40 PM Markus Armbruster wrote: > > > > > Hao Xiang writes: > > > > This change extends the MigrationStatus interface to track zero pages > > > > and zero bytes counter. > > > > Signed-off-b

Re: [External] Re: [PATCH v4 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-03-08 Thread hao . xiang
> > On Sun, Mar 3, 2024 at 11:16 PM Peter Xu wrote: > > > > > On Fri, Mar 01, 2024 at 02:28:24AM +, Hao Xiang wrote: > > > > -GlobalProperty hw_compat_8_2[] = {}; > > > > +GlobalProperty hw_compat_8_2[] = { > > &

Re: [PATCH v4 3/7] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-03-08 Thread hao . xiang
> > On Sun, Mar 3, 2024 at 11:46 PM Peter Xu wrote: > > > > > On Fri, Mar 01, 2024 at 02:28:25AM +, Hao Xiang wrote: > > > > 1. Add a dedicated handler for MigrationOps::ram_save_target_page in > > > > multifd live migration. > > &

Re: [External] Re: [PATCH v3 11/20] util/dsa: Implement DSA task asynchronous submission and wait for completion.

2024-03-08 Thread hao . xiang
> > On Fri, Mar 8, 2024 at 2:11 AM Jonathan Cameron > > wrote: > > > > > On Thu, 4 Jan 2024 00:44:43 + > > > > Hao Xiang wrote: > > > > * Add a DSA task completion callback. > > > > * DSA completion thread will call the

Re: [External] Re: [PATCH v4 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-03-01 Thread Hao Xiang
On Thu, Feb 29, 2024 at 11:28 PM Markus Armbruster wrote: > > Hao Xiang writes: > > > 1. Add zero_pages field in MultiFDPacket_t. > > 2. Implements the zero page detection and handling on the multifd > > threads for non-compression, zlib and zstd compression backend

[PATCH v4 3/7] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-02-29 Thread Hao Xiang
1. Add a dedicated handler for MigrationOps::ram_save_target_page in multifd live migration. 2. Refactor ram_save_target_page_legacy so that the legacy and multifd handlers don't have internal functions calling into each other. Signed-off-by: Hao Xiang Reviewed-by: Fabiano Rosas Messa

[PATCH v4 5/7] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-02-29 Thread Hao Xiang
migration main thread. Signed-off-by: Hao Xiang --- tests/qtest/migration-test.c | 52 1 file changed, 52 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 83512bce85..8a966364b5 100644 --- a/tests/qtest/migration-te

[PATCH v4 0/7] Introduce multifd zero page checking.

2024-02-29 Thread Hao Xiang
+ |multifd-1-threads | 6182 | ++ |multifd-2-threads | 4643 | ++ |multifd-4-threads | 4143 | +--------+ Apply this patchset on top of commit c0c6a0e3528b88aaad0b9d333e295707a195587b Hao

[PATCH v4 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-29 Thread Hao Xiang
This new parameter controls where the zero page checking is running. 1. If this parameter is set to 'legacy', zero page checking is done in the migration main thread. 2. If this parameter is set to 'none', zero page checking is disabled. Signed-off-by: Hao Xiang --- hw/

[PATCH v4 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-02-29 Thread Hao Xiang
patibility. 5. Adds zero page counters and updates multifd send/receive tracing format to track the newly added counters. Signed-off-by: Hao Xiang --- hw/core/machine.c| 4 +- hw/core/qdev-properties-system.c | 2 +- migration/meson.build| 1 + migration/multifd-zero-p

[PATCH v4 6/7] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-29 Thread Hao Xiang
This change extends the MigrationStatus interface to track zero pages and zero bytes counter. Signed-off-by: Hao Xiang --- migration/migration-hmp-cmds.c | 4 migration/migration.c | 2 ++ qapi/migration.json | 15 ++- tests/migration

[PATCH v4 7/7] Update maintainer contact for migration multifd zero page checking acceleration.

2024-02-29 Thread Hao Xiang
Add myself to maintain multifd zero page checking acceleration function. Signed-off-by: Hao Xiang --- MAINTAINERS | 5 + 1 file changed, 5 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 65dfdc9677..b547918e4d 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3414,6 +3414,11 @@ F

[PATCH v4 4/7] migration/multifd: Enable multifd zero page checking by default.

2024-02-29 Thread Hao Xiang
Set default "zero-page-detection" option to "multifd". Now zero page checking can be done in the multifd threads and this becomes the default configuration. We still provide backward compatibility where zero page checking is done from the migration main thread. Sign

Re: [External] Re: [PATCH v3 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-02-29 Thread Hao Xiang
On Wed, Feb 28, 2024 at 11:46 AM Fabiano Rosas wrote: > > Hao Xiang writes: > > > 1. Add zero_pages field in MultiFDPacket_t. > > 2. Implements the zero page detection and handling on the multifd > > threads for non-compression, zlib and zstd compression backend

Re: [External] Re: [PATCH v3 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-02-29 Thread Hao Xiang
On Thu, Feb 29, 2024 at 7:47 AM Fabiano Rosas wrote: > > Markus Armbruster writes: > > > Hao Xiang writes: > > > >> On Wed, Feb 28, 2024 at 1:50 AM Markus Armbruster > >> wrote: > >>> > >>> Hao Xiang writes: > >>> &g

Re: [External] Re: [PATCH v3 6/7] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-29 Thread Hao Xiang
On Wed, Feb 28, 2024 at 10:01 PM Markus Armbruster wrote: > > Hao Xiang writes: > > > On Wed, Feb 28, 2024 at 1:52 AM Markus Armbruster wrote: > >> > >> Hao Xiang writes: > >> > >> > This change extends the MigrationStatus inter

Re: [External] Re: [PATCH v3 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-02-28 Thread Hao Xiang
On Wed, Feb 28, 2024 at 1:50 AM Markus Armbruster wrote: > > Hao Xiang writes: > > > 1. Add zero_pages field in MultiFDPacket_t. > > 2. Implements the zero page detection and handling on the multifd > > threads for non-compression, zlib and zstd compression backend

Re: [External] Re: [PATCH v3 6/7] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-28 Thread Hao Xiang
On Wed, Feb 28, 2024 at 1:52 AM Markus Armbruster wrote: > > Hao Xiang writes: > > > This change extends the MigrationStatus interface to track zero pages > > and zero bytes counter. > > > > Signed-off-by: Hao Xiang > > [...] > > > diff -

Re: [External] Re: [PATCH v3 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-28 Thread Hao Xiang
On Wed, Feb 28, 2024 at 1:43 AM Markus Armbruster wrote: > > Hao Xiang writes: > > > This new parameter controls where the zero page checking is running. > > 1. If this parameter is set to 'legacy', zero page checking is > > done in the migration main threa

[PATCH v3 7/7] Update maintainer contact for migration multifd zero page checking acceleration.

2024-02-26 Thread Hao Xiang
Add myself to maintain multifd zero page checking acceleration function. Signed-off-by: Hao Xiang --- MAINTAINERS | 5 + 1 file changed, 5 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index 992799171f..4a4f8f24e0 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3413,6 +3413,11 @@ F

[PATCH v3 2/7] migration/multifd: Implement zero page transmission on the multifd thread.

2024-02-26 Thread Hao Xiang
patibility. 5. Adds zero page counters and updates multifd send/receive tracing format to track the newly added counters. Signed-off-by: Hao Xiang --- hw/core/machine.c| 4 +- hw/core/qdev-properties-system.c | 2 +- migration/meson.build| 1 + migration/multifd-zero-p

[PATCH v3 0/7] Introduce multifd zero page checking.

2024-02-26 Thread Hao Xiang
+ |main-thread| 9629 | ++ |multifd-1-threads | 6182 | ++ |multifd-2-threads | 4643 | ++ |multifd-4-threads | 4143 | +-----

[PATCH v3 6/7] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-26 Thread Hao Xiang
This change extends the MigrationStatus interface to track zero pages and zero bytes counter. Signed-off-by: Hao Xiang --- migration/migration-hmp-cmds.c | 4 migration/migration.c | 2 ++ qapi/migration.json | 15 ++- tests/migration

[PATCH v3 3/7] migration/multifd: Implement ram_save_target_page_multifd to handle multifd version of MigrationOps::ram_save_target_page.

2024-02-26 Thread Hao Xiang
1. Add a dedicated handler for MigrationOps::ram_save_target_page in multifd live migration. 2. Refactor ram_save_target_page_legacy so that the legacy and multifd handlers don't have internal functions calling into each other. Signed-off-by: Hao Xiang --- migration/ram.c

[PATCH v3 5/7] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-02-26 Thread Hao Xiang
migration main thread. Signed-off-by: Hao Xiang --- tests/qtest/migration-test.c | 52 1 file changed, 52 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 8a5bb1752e..65b531d871 100644 --- a/tests/qtest/migration-te

[PATCH v3 4/7] migration/multifd: Enable multifd zero page checking by default.

2024-02-26 Thread Hao Xiang
Set default "zero-page-detection" option to "multifd". Now zero page checking can be done in the multifd threads and this becomes the default configuration. We still provide backward compatibility where zero page checking is done from the migration main thread. Sign

[PATCH v3 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-26 Thread Hao Xiang
This new parameter controls where the zero page checking is running. 1. If this parameter is set to 'legacy', zero page checking is done in the migration main thread. 2. If this parameter is set to 'none', zero page checking is disabled. Signed-off-by: Hao Xiang --- hw/

Re: [External] Re: [PATCH v2 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-26 Thread Hao Xiang
On Sun, Feb 25, 2024 at 11:19 PM Wang, Lei wrote: > > On 2/17/2024 6:39, Hao Xiang wrote: > > This new parameter controls where the zero page checking is running. > > 1. If this parameter is set to 'legacy', zero page checking is > > done in the migration main t

Re: [External] Re: [PATCH v2 4/7] migration/multifd: Enable zero page checking from multifd threads.

2024-02-24 Thread Hao Xiang
On Thu, Feb 22, 2024 at 10:02 PM Hao Xiang wrote: > > On Thu, Feb 22, 2024 at 6:33 PM Peter Xu wrote: > > > > On Wed, Feb 21, 2024 at 06:06:19PM -0300, Fabiano Rosas wrote: > > > Hao Xiang writes: > > > > > > > This change adds a dedicated handler

Re: [External] Re: [PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-24 Thread Hao Xiang
On Thu, Feb 22, 2024 at 9:15 PM Hao Xiang wrote: > > On Thu, Feb 22, 2024 at 6:21 PM Peter Xu wrote: > > > > On Wed, Feb 21, 2024 at 06:04:10PM -0300, Fabiano Rosas wrote: > > > Hao Xiang writes: > > > > > > > 1. Implements the zero page detection

Re: [External] Re: [PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-24 Thread Hao Xiang
On Thu, Feb 22, 2024 at 8:38 PM Hao Xiang wrote: > > On Fri, Feb 16, 2024 at 9:08 PM Richard Henderson > wrote: > > > > On 2/16/24 12:39, Hao Xiang wrote: > > > +void multifd_zero_page_check_recv(MultiFDRecvParams *p) > > > +{ > &

Re: [External] Re: [PATCH v2 4/7] migration/multifd: Enable zero page checking from multifd threads.

2024-02-22 Thread Hao Xiang
On Thu, Feb 22, 2024 at 6:33 PM Peter Xu wrote: > > On Wed, Feb 21, 2024 at 06:06:19PM -0300, Fabiano Rosas wrote: > > Hao Xiang writes: > > > > > This change adds a dedicated handler for > > > MigrationOps::ram_save_target_page in > > > > nit:

Re: [External] Re: [PATCH v2 4/7] migration/multifd: Enable zero page checking from multifd threads.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 1:06 PM Fabiano Rosas wrote: > > Hao Xiang writes: > > > This change adds a dedicated handler for MigrationOps::ram_save_target_page > > in > > nit: Add a dedicated handler... > > Usually "this patch/change" is used only wh

Re: [External] Re: [PATCH v2 4/7] migration/multifd: Enable zero page checking from multifd threads.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 8:11 AM Elena Ufimtseva wrote: > > > > On Fri, Feb 16, 2024 at 2:42 PM Hao Xiang wrote: >> >> This change adds a dedicated handler for MigrationOps::ram_save_target_page >> in >> multifd live migration. Now zero page checking can be do

Re: [External] Re: [PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 1:04 PM Fabiano Rosas wrote: > > Hao Xiang writes: > > > 1. Implements the zero page detection and handling on the multifd > > threads for non-compression, zlib and zstd compression backends. > > 2. Added a new value 'multifd' in Zero

Re: [External] Re: [PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-22 Thread Hao Xiang
On Thu, Feb 22, 2024 at 6:21 PM Peter Xu wrote: > > On Wed, Feb 21, 2024 at 06:04:10PM -0300, Fabiano Rosas wrote: > > Hao Xiang writes: > > > > > 1. Implements the zero page detection and handling on the multifd > > > threads for non-compression, zlib an

Re: [External] Re: [PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 8:00 AM Elena Ufimtseva wrote: > > > > On Fri, Feb 16, 2024 at 2:42 PM Hao Xiang wrote: >> >> 1. Implements the zero page detection and handling on the multifd >> threads for non-compression, zlib and zstd compression backends. >&g

Re: [External] Re: [PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-22 Thread Hao Xiang
On Fri, Feb 16, 2024 at 9:08 PM Richard Henderson wrote: > > On 2/16/24 12:39, Hao Xiang wrote: > > +void multifd_zero_page_check_recv(MultiFDRecvParams *p) > > +{ > > +for (int i = 0; i < p->zero_num; i++) { > > +void *page = p->host + p->

Re: [External] Re: [PATCH v2 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 5:58 AM Elena Ufimtseva wrote: > > > > On Fri, Feb 16, 2024 at 2:41 PM Hao Xiang wrote: >> >> This new parameter controls where the zero page checking is running. >> 1. If this parameter is set to 'legacy', zero page checking is &g

Re: [External] Re: [PATCH v2 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 4:03 AM Markus Armbruster wrote: > > Hao Xiang writes: > > > This new parameter controls where the zero page checking is running. > > 1. If this parameter is set to 'legacy', zero page checking is > > done in the migration main threa

Re: [External] Re: [PATCH v2 5/7] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 12:59 PM Fabiano Rosas wrote: > > Hao Xiang writes: > > > Now that zero page checking is done on the multifd sender threads by > > default, we still provide an option for backward compatibility. This > > change adds a qtest migration tes

Re: [External] Re: [PATCH v2 2/7] migration/multifd: Support for zero pages transmission in multifd format.

2024-02-22 Thread Hao Xiang
On Wed, Feb 21, 2024 at 7:37 AM Elena Ufimtseva wrote: > > > > On Fri, Feb 16, 2024 at 2:41 PM Hao Xiang wrote: >> >> This change adds zero page counters and updates multifd send/receive >> tracing format to track the newly added counters. >> >> Sign

[PATCH v2 7/7] Update maintainer contact for migration multifd zero page checking acceleration.

2024-02-16 Thread Hao Xiang
Add myself to maintain multifd zero page checking acceleration function. Signed-off-by: Hao Xiang --- MAINTAINERS | 5 + 1 file changed, 5 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index a24c2b51b6..3ca407cb58 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -3403,6 +3403,11 @@ F

[PATCH v2 6/7] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-16 Thread Hao Xiang
This change extends the MigrationStatus interface to track zero pages and zero bytes counter. Signed-off-by: Hao Xiang --- migration/migration-hmp-cmds.c | 4 migration/migration.c | 2 ++ qapi/migration.json | 15 ++- tests/migration

[PATCH v2 4/7] migration/multifd: Enable zero page checking from multifd threads.

2024-02-16 Thread Hao Xiang
main thread. Signed-off-by: Hao Xiang --- migration/multifd.c | 1 + migration/options.c | 2 +- migration/ram.c | 53 ++--- 3 files changed, 42 insertions(+), 14 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index fbb40ea10b

[PATCH v2 2/7] migration/multifd: Support for zero pages transmission in multifd format.

2024-02-16 Thread Hao Xiang
This change adds zero page counters and updates multifd send/receive tracing format to track the newly added counters. Signed-off-by: Hao Xiang --- migration/multifd.c| 43 ++ migration/multifd.h| 21 - migration/ram.c

[PATCH v2 0/7] Introduce multifd zero page checking.

2024-02-16 Thread Hao Xiang
---+ |multifd-2-threads | 4643 | ++ |multifd-4-threads | 4143 | ++ Apply this patchset on top of commit 5767815218efd3cbfd409505ed824d5f356044ae Hao Xiang (7): migration/multifd: Add new migration option zer

[PATCH v2 3/7] migration/multifd: Zero page transmission on the multifd thread.

2024-02-16 Thread Hao Xiang
. Signed-off-by: Hao Xiang --- migration/meson.build | 1 + migration/multifd-zero-page.c | 59 +++ migration/multifd-zlib.c | 26 --- migration/multifd-zstd.c | 25 --- migration/multifd.c

[PATCH v2 5/7] migration/multifd: Add new migration test cases for legacy zero page checking.

2024-02-16 Thread Hao Xiang
migration main thread. Signed-off-by: Hao Xiang --- tests/qtest/migration-test.c | 52 1 file changed, 52 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 8a5bb1752e..c27083110a 100644 --- a/tests/qtest/migration-te

[PATCH v2 1/7] migration/multifd: Add new migration option zero-page-detection.

2024-02-16 Thread Hao Xiang
This new parameter controls where the zero page checking is running. 1. If this parameter is set to 'legacy', zero page checking is done in the migration main thread. 2. If this parameter is set to 'none', zero page checking is disabled. Signed-off-by: Hao Xiang --- hw/

Re: [External] [PATCH v2 05/23] migration/multifd: Drop MultiFDSendParams.normal[] array

2024-02-13 Thread Hao Xiang
On Fri, Feb 9, 2024 at 4:20 AM Fabiano Rosas wrote: > > Hao Xiang writes: > > > On Fri, Feb 2, 2024 at 2:30 AM wrote: > >> > >> From: Peter Xu > >> > >> This array is redundant when p->pages exists. Now we extended the life of > >

Re: [External] [PATCH v2 05/23] migration/multifd: Drop MultiFDSendParams.normal[] array

2024-02-08 Thread Hao Xiang
On Fri, Feb 2, 2024 at 2:30 AM wrote: > > From: Peter Xu > > This array is redundant when p->pages exists. Now we extended the life of > p->pages to the whole period where pending_job is set, it should be safe to > always use p->pages->offset[] rather than p->normal[]. Drop the array. > > Along

Re: [External] Re: [PATCH 3/6] migration/multifd: Support for zero pages transmission in multifd format.

2024-02-08 Thread Hao Xiang
On Tue, Feb 6, 2024 at 8:25 PM Peter Xu wrote: > > On Tue, Feb 06, 2024 at 11:19:05PM +0000, Hao Xiang wrote: > > diff --git a/migration/multifd.c b/migration/multifd.c > > index 25cbc6dc6b..a20d0ed10e 100644 > > --- a/migration/multifd.c > > +++ b/migration/

Re: [External] RE: Regarding to the recent Intel IAA/DSA/QAT support on migration

2024-02-07 Thread Hao Xiang
riginal Message- > > From: Peter Xu > > Sent: Wednesday, February 7, 2024 4:10 PM > > To: Bryan Zhang ; Hao Xiang > > ; Liu, Yuan1 > > Cc: Fabiano Rosas ; QEMU Devel Mailing List > de...@nongnu.org> > > Subject: Regarding to the recent Intel IAA/DSA/QAT

Re: [External] Re: [PATCH 1/6] migration/multifd: Add new migration option multifd-zero-page.

2024-02-07 Thread Hao Xiang
On Tue, Feb 6, 2024 at 7:45 PM Peter Xu wrote: > > On Tue, Feb 06, 2024 at 11:19:03PM +0000, Hao Xiang wrote: > > diff --git a/qapi/migration.json b/qapi/migration.json > > index 819708321d..ff033a0344 100644 > > --- a/qapi/migration.json > > +++ b/qapi/migrat

Re: [External] Re: [PATCH 0/6] Introduce multifd zero page checking.

2024-02-07 Thread Hao Xiang
On Tue, Feb 6, 2024 at 7:39 PM Peter Xu wrote: > > On Tue, Feb 06, 2024 at 11:19:02PM +0000, Hao Xiang wrote: > > This patchset is based on Juan Quintela's old series here > > https://lore.kernel.org/all/20220802063907.18882-1-quint...@redhat.com/ > > > >

Re: [External] Re: Re: [PATCH 2/6] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-07 Thread Hao Xiang
On Wed, Feb 7, 2024 at 12:41 AM Jiri Denemark wrote: > > On Wed, Feb 07, 2024 at 12:37:15 +0800, Peter Xu wrote: > > On Wed, Feb 07, 2024 at 12:13:10PM +0800, Peter Xu wrote: > > > On Tue, Feb 06, 2024 at 11:19:04PM +, Hao Xiang wrote: > > > > This change exte

[PATCH 4/6] migration/multifd: Zero page transmission on the multifd thread.

2024-02-06 Thread Hao Xiang
This implements the zero page detection and handling on the multifd threads. Signed-off-by: Hao Xiang --- migration/multifd.c | 62 + migration/multifd.h | 5 2 files changed, 62 insertions(+), 5 deletions(-) diff --git a/migration/multifd.c b

[PATCH 3/6] migration/multifd: Support for zero pages transmission in multifd format.

2024-02-06 Thread Hao Xiang
This change adds zero page counters and updates multifd send/receive tracing format to track the newly added counters. Signed-off-by: Hao Xiang --- migration/migration-hmp-cmds.c | 4 migration/multifd.c| 43 ++ migration/multifd.h

[PATCH 6/6] migration/multifd: Add a new migration test case for legacy zero page checking.

2024-02-06 Thread Hao Xiang
thread. Signed-off-by: Hao Xiang --- tests/qtest/migration-test.c | 26 ++ 1 file changed, 26 insertions(+) diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 7675519cfa..2c13df04c3 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest

[PATCH 5/6] migration/multifd: Enable zero page checking from multifd threads.

2024-02-06 Thread Hao Xiang
main thread. Signed-off-by: Hao Xiang --- migration/multifd.c | 3 ++- migration/ram.c | 49 - 2 files changed, 42 insertions(+), 10 deletions(-) diff --git a/migration/multifd.c b/migration/multifd.c index c031f947c7..c6833ccb07 100644 --- a

[PATCH 2/6] migration/multifd: Add zero pages and zero bytes counter to migration status interface.

2024-02-06 Thread Hao Xiang
This change extends the MigrationStatus interface to track zero pages and zero bytes counter. Signed-off-by: Hao Xiang --- qapi/migration.json | 15 ++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/qapi/migration.json b/qapi/migration.json index ff033a0344

[PATCH 1/6] migration/multifd: Add new migration option multifd-zero-page.

2024-02-06 Thread Hao Xiang
This new parameter controls where the zero page checking is running. If this parameter is set to true, zero page checking is done in the multifd sender threads. If this parameter is set to false, zero page checking is done in the migration main thread. Signed-off-by: Hao Xiang --- migration

[PATCH 0/6] Introduce multifd zero page checking.

2024-02-06 Thread Hao Xiang
reads | 4643 | 21.53GB/s | +--+ |multifd-4-threads | 4143 | 24.13GB/s | +--+ Apply this patchset on top of commit 39a6e4f87e7b75a45b08d6dc8b8b7c2954c87440 Hao Xiang (6): migration/multifd: Ad

Re: [External] Re: [PATCH v3 03/20] multifd: Zero pages transmission

2024-02-01 Thread Hao Xiang
On Wed, Jan 31, 2024 at 9:22 PM Peter Xu wrote: > > On Thu, Jan 04, 2024 at 12:44:35AM +0000, Hao Xiang wrote: > > From: Juan Quintela > > > > This implements the zero page dection and handling. > > > > Signed-off-by: Juan Quintela &

Re: [External] Re: [PATCH v3 15/20] migration/multifd: Add test hook to set normal page ratio.

2024-02-01 Thread Hao Xiang
On Wed, Jan 31, 2024 at 9:24 PM Peter Xu wrote: > > On Thu, Jan 04, 2024 at 12:44:47AM +0000, Hao Xiang wrote: > > +# @multifd-normal-page-ratio: Test hook setting the normal page ratio. > > +# (Since 8.2) > > Please remember to touch all of them to 9.0 when re

  1   2   3   >