The 1MiB default dates back to the original RDMA implementation in 2013 (commit 2da776db48), and is too conservative for modern hardware.
64MiB captures most of the throughput gain (~10x over 1MiB) while keeping transferred data low. Larger chunks cause more data to be retransferred per dirty page, so the largest chunk size is not necessarily optimal (see 1024MiB row). The x-rdma-chunk-size parameter remains available for user tuning. Test config: BlueField-3 ConnectX-7, 8GB VM RAM, pin-all off, `stress-ng --vm 4 --vm-bytes 1G --vm-method rand-set` chunk_size total(ms) down(ms) Throughput(Mbps) transferred 1m 45,156 1,166 1,252.50 6.46 GiB 32m 15,034 1,864 3,401.26 5.57 GiB 64m 4,492 1,554 13,637.46 5.75 GiB 128m 3,940 1,662 16,860.59 6.06 GiB 1024m 3,665 2,238 24,676.59 8.04 GiB Signed-off-by: Samuel Zhang <[email protected]> --- migration/options.c | 2 +- qapi/migration.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/migration/options.c b/migration/options.c index 5cbfd29099..ea2137372c 100644 --- a/migration/options.c +++ b/migration/options.c @@ -91,7 +91,7 @@ const PropertyInfo qdev_prop_StrOrNull; #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT_PERIOD 1000 /* milliseconds */ #define DEFAULT_MIGRATE_VCPU_DIRTY_LIMIT 1 /* MB/s */ -#define DEFAULT_MIGRATE_X_RDMA_CHUNK_SIZE MiB +#define DEFAULT_MIGRATE_X_RDMA_CHUNK_SIZE (64 * MiB) const Property migration_properties[] = { DEFINE_PROP_BOOL("store-global-state", MigrationState, diff --git a/qapi/migration.json b/qapi/migration.json index 0db115ec5e..b3815f0594 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -1009,7 +1009,7 @@ # the remainder its arguments. (Since 10.2) # # @x-rdma-chunk-size: RDMA memory registration chunk size in bytes. -# Default is 1MiB. Must be a power of 2 in the range +# Default is 64MiB. Must be a power of 2 in the range # [1MiB, 1024MiB]. Only applies when migrating via RDMA. # Must be set to the same value on both source and destination # before migration starts. (Since 11.1) -- 2.43.7
