On Tue, Feb 20, 2024 at 07:41:15PM -0300, Fabiano Rosas wrote:
> Add a new migration capability 'fixed-ram'.
>
> The core of the feature is to ensure that each RAM page has a specific
> offset in the resulting migration stream. The reasons why we'd want
> such behavior are:
>
> - The resulting file will have a bounded size, since pages which are
> dirtied multiple times will always go to a fixed location in the
> file, rather than constantly being added to a sequential
> stream. This eliminates cases where a VM with, say, 1G of RAM can
> result in a migration file that's 10s of GBs, provided that the
> workload constantly redirties memory.
>
> - It paves the way to implement O_DIRECT-enabled save/restore of the
> migration stream as the pages are ensured to be written at aligned
> offsets.
>
> - It allows the usage of multifd so we can write RAM pages to the
> migration file in parallel.
>
> For now, enabling the capability has no effect. The next couple of
> patches implement the core functionality.
>
> Signed-off-by: Fabiano Rosas <faro...@suse.de>
> ---
> - update migration.json to 9.0 and improve wording
> - move docs to a separate file and add use cases information
> ---
> docs/devel/migration/features.rst | 1 +
> docs/devel/migration/fixed-ram.rst | 137 +++++++++++++++++++++++++++++
> migration/options.c | 34 +++++++
> migration/options.h | 1 +
> migration/savevm.c | 1 +
> qapi/migration.json | 6 +-
> 6 files changed, 179 insertions(+), 1 deletion(-)
> create mode 100644 docs/devel/migration/fixed-ram.rst
>
> diff --git a/docs/devel/migration/features.rst
> b/docs/devel/migration/features.rst
> index a9acaf618e..4c708b679a 100644
> --- a/docs/devel/migration/features.rst
> +++ b/docs/devel/migration/features.rst
> @@ -10,3 +10,4 @@ Migration has plenty of features to support different use
> cases.
> dirty-limit
> vfio
> virtio
> + fixed-ram
> diff --git a/docs/devel/migration/fixed-ram.rst
> b/docs/devel/migration/fixed-ram.rst
> new file mode 100644
> index 0000000000..a6c0e5a360
> --- /dev/null
> +++ b/docs/devel/migration/fixed-ram.rst
> @@ -0,0 +1,137 @@
> +Fixed-ram
> +=========
> +
> +Fixed-ram is a new stream format for the RAM section designed to
> +supplement the existing ``file:`` migration and make it compatible
> +with ``multifd``. This enables parallel migration of a guest's RAM to
> +a file.
> +
> +The core of the feature is to ensure that each RAM page has a specific
> +offset in the resulting migration file. This enables the ``multifd``
> +threads to write exclusively to those offsets even if the guest is
> +constantly dirtying pages (i.e. live migration). Another benefit is
> +that the resulting file will have a bounded size, since pages which
> +are dirtied multiple times will always go to a fixed location in the
> +file, rather than constantly being added to a sequential
> +stream. Having the pages at fixed offsets also allows the usage of
> +O_DIRECT for save/restore of the migration stream as the pages are
> +ensured to be written respecting O_DIRECT alignment restrictions.
> +
> +Usage
> +-----
> +
> +On both source and destination, enable the ``multifd`` and
> +``fixed-ram`` capabilities:
> +
> + ``migrate_set_capability multifd on``
> +
> + ``migrate_set_capability fixed-ram on``
> +
> +Use a ``file:`` URL for migration:
> +
> + ``migrate file:/path/to/migration/file``
> +
> +Fixed-ram migration is best done non-live, i.e. by stopping the VM on
> +the source side before migrating.
> +
> +For best performance enable the ``direct-io`` capability as well:
> +
> + ``migrate_set_capability direct-io on``
> +
> +Use-cases
> +---------
> +
> +The fixed-ram feature was designed for use cases where the migration
> +stream will be directed to a file in the filesystem and not
> +immediately restored on the destination VM [#]_. These could be
> +thought of as snapshots. We can further categorize them into live and
> +non-live.
> +
> +- Non-live snapshot
> +
> +If the use case requires a VM to be stopped before taking a snapshot,
> +that's the ideal scenario for fixed-ram migration. Not having to track
> +dirty pages, the migration will write the RAM pages to the disk as
> +fast as it can.
> +
> +Note: if a snapshot is taken of a running VM, but the VM will be
> +stopped after the snapshot by the admin, then consider stopping it
> +right before the snapshot to take benefit of the performance gains
> +mentioned above.
> +
> +- Live snapshot
> +
> +If the use case requires that the VM keeps running during and after
> +the snapshot operation, then fixed-ram migration can still be used,
> +but will be less performant. Other strategies such as
> +background-snapshot should be evaluated as well. One benefit of
> +fixed-ram in this scenario is portability since background-snapshot
> +depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not
Background snapshot uses userfaultfd-wp rather than KVM_GET_DIRTY_LOG. The
statement is still correct though, that userfault is only supported on
Linux in general (wp is one sub-feature, represents "write-protect mode")
so this should help portability, as it removes the dependency on the OS.
> +supported outside of Linux.
> +
> +.. [#] While this same effect could be obtained with the usage of
> + snapshots or the ``file:`` migration alone, fixed-ram provides
> + a performance increase for VMs with larger RAM sizes (10s to
> + 100s of GiBs), specially if the VM has been stopped beforehand.
> +
> +RAM section format
> +------------------
> +
> +Instead of having a sequential stream of pages that follow the
> +RAMBlock headers, the dirty pages for a RAMBlock follow its header
> +instead. This ensures that each RAM page has a fixed offset in the
> +resulting migration file.
> +
> +A bitmap is introduced to track which pages have been written in the
> +migration file. Pages are written at a fixed location for every
> +ramblock. Zero pages are ignored as they'd be zero in the destination
> +migration as well.
> +
> +::
> +
> + Without fixed-ram: With fixed-ram:
> +
> + --------------------- --------------------------------
> + | ramblock 1 header | | ramblock 1 header |
> + --------------------- --------------------------------
> + | ramblock 2 header | | ramblock 1 fixed-ram header |
> + --------------------- --------------------------------
> + | ... | | padding to next 1MB boundary |
> + --------------------- | ... |
> + | ramblock n header | --------------------------------
> + --------------------- | ramblock 1 pages |
> + | RAM_SAVE_FLAG_EOS | | ... |
> + --------------------- --------------------------------
> + | stream of pages | | ramblock 2 header |
> + | (iter 1) | --------------------------------
> + | ... | | ramblock 2 fixed-ram header |
> + --------------------- --------------------------------
> + | RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary |
> + --------------------- | ... |
> + | stream of pages | --------------------------------
> + | (iter 2) | | ramblock 2 pages |
> + | ... | | ... |
> + --------------------- --------------------------------
> + | ... | | ... |
> + --------------------- --------------------------------
> + | RAM_SAVE_FLAG_EOS |
> + --------------------------------
> + | ... |
> + --------------------------------
> +
> + where:
> + - ramblock header: the generic information for a ramblock, such as
> + idstr, used_len, etc.
> +
> + - ramblock fixed-ram header: the information added by this feature:
> + bitmap of pages written, bitmap size and offset of pages in the
> + migration file.
> +
> +Restrictions
> +------------
> +
> +Since pages are written to their relative offsets and out of order
> +(due to the memory dirtying patterns), streaming channels such as
> +sockets are not supported. A seekable channel such as a file is
> +required. This can be verified in the QIOChannel by the presence of
> +the QIO_CHANNEL_FEATURE_SEEKABLE.
> diff --git a/migration/options.c b/migration/options.c
> index 3e3e0b93b4..4909e5c72a 100644
> --- a/migration/options.c
> +++ b/migration/options.c
> @@ -204,6 +204,7 @@ Property migration_properties[] = {
> DEFINE_PROP_MIG_CAP("x-switchover-ack",
> MIGRATION_CAPABILITY_SWITCHOVER_ACK),
> DEFINE_PROP_MIG_CAP("x-dirty-limit", MIGRATION_CAPABILITY_DIRTY_LIMIT),
> + DEFINE_PROP_MIG_CAP("x-fixed-ram", MIGRATION_CAPABILITY_FIXED_RAM),
Let's directly use "fixed-ram" (or "mapped-ram", or whatever new name we
decide to use, as long as without "x-")?
migration_properties is not documented anywhere, mostly yet for debugging
purpose. We could have dropped all the "x-"s, IMHO.
> DEFINE_PROP_END_OF_LIST(),
> };
>
> @@ -263,6 +264,13 @@ bool migrate_events(void)
> return s->capabilities[MIGRATION_CAPABILITY_EVENTS];
> }
>
> +bool migrate_fixed_ram(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return s->capabilities[MIGRATION_CAPABILITY_FIXED_RAM];
> +}
> +
> bool migrate_ignore_shared(void)
> {
> MigrationState *s = migrate_get_current();
> @@ -645,6 +653,32 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps,
> Error **errp)
> }
> }
>
> + if (new_caps[MIGRATION_CAPABILITY_FIXED_RAM]) {
> + if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) {
> + error_setg(errp,
> + "Fixed-ram migration is incompatible with multifd");
> + return false;
> + }
> +
> + if (new_caps[MIGRATION_CAPABILITY_XBZRLE]) {
> + error_setg(errp,
> + "Fixed-ram migration is incompatible with xbzrle");
> + return false;
> + }
> +
> + if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) {
> + error_setg(errp,
> + "Fixed-ram migration is incompatible with
> compression");
> + return false;
> + }
> +
> + if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) {
> + error_setg(errp,
> + "Fixed-ram migration is incompatible with postcopy
> ram");
> + return false;
> + }
> + }
> +
> return true;
> }
>
> diff --git a/migration/options.h b/migration/options.h
> index 246c160aee..8680a10b79 100644
> --- a/migration/options.h
> +++ b/migration/options.h
> @@ -31,6 +31,7 @@ bool migrate_compress(void);
> bool migrate_dirty_bitmaps(void);
> bool migrate_dirty_limit(void);
> bool migrate_events(void);
> +bool migrate_fixed_ram(void);
> bool migrate_ignore_shared(void);
> bool migrate_late_block_activate(void);
> bool migrate_multifd(void);
> diff --git a/migration/savevm.c b/migration/savevm.c
> index d612c8a902..4b928dd6bb 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -245,6 +245,7 @@ static bool should_validate_capability(int capability)
> /* Validate only new capabilities to keep compatibility. */
> switch (capability) {
> case MIGRATION_CAPABILITY_X_IGNORE_SHARED:
> + case MIGRATION_CAPABILITY_FIXED_RAM:
> return true;
> default:
> return false;
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 5a565d9b8d..3fce5fe53e 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -531,6 +531,10 @@
> # and can result in more stable read performance. Requires KVM
> # with accelerator property "dirty-ring-size" set. (Since 8.1)
> #
> +# @fixed-ram: Migrate using fixed offsets in the migration file for
> +# each RAM page. Requires a migration URI that supports seeking,
> +# such as a file. (since 9.0)
> +#
> # Features:
> #
> # @deprecated: Member @block is deprecated. Use blockdev-mirror with
> @@ -555,7 +559,7 @@
> { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
> 'validate-uuid', 'background-snapshot',
> 'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
> - 'dirty-limit'] }
> + 'dirty-limit', 'fixed-ram'] }
>
> ##
> # @MigrationCapabilityStatus:
> --
> 2.35.3
>
--
Peter Xu
