Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability
On Tue, Feb 20, 2024 at 07:41:15PM -0300, Fabiano Rosas wrote: > + Without fixed-ram: With fixed-ram: > + > + - > + | ramblock 1 header | | ramblock 1 header| > + - > + | ramblock 2 header | | ramblock 1 fixed-ram header | > + - > + | ... | | padding to next 1MB boundary | > + - | ... | > + | ramblock n header | > + - | ramblock 1 pages | > + | RAM_SAVE_FLAG_EOS | | ... | > + - > + | stream of pages | | ramblock 2 header| > + | (iter 1) | > + | ... | | ramblock 2 fixed-ram header | > + - > + | RAM_SAVE_FLAG_EOS | | padding to next 1MB boundary | > + - | ... | > + | stream of pages | > + | (iter 2) | | ramblock 2 pages | > + | ... | | ... | > + - > + | ... | | ... | > + - > + | RAM_SAVE_FLAG_EOS| > + > + | ... | > + > + > + where: Super-nit: you can drop the " " otherwise it's put into the quote. > + - ramblock header: the generic information for a ramblock, such as > +idstr, used_len, etc. > + > + - ramblock fixed-ram header: the information added by this feature: > +bitmap of pages written, bitmap size and offset of pages in the > +migration file. > + > +Restrictions > + > + > +Since pages are written to their relative offsets and out of order > +(due to the memory dirtying patterns), streaming channels such as > +sockets are not supported. A seekable channel such as a file is > +required. This can be verified in the QIOChannel by the presence of > +the QIO_CHANNEL_FEATURE_SEEKABLE. Would it worth also mention that it only provides fixed offsets to "guest physical RAM" only? For example, GPU RAM won't apply as they're migrated as part of device states, even if also iterable. IOW, IIUC if there's a VFIO device (or a few), the fixed-ram migration file will still be unbound in size, because device can keep flushing stale vRAM to the image.. Maybe that's too specific, I'll leave that to you to decide whether to even mention it. -- Peter Xu
Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability
On Tue, Feb 20, 2024 at 07:41:15PM -0300, Fabiano Rosas wrote: > Add a new migration capability 'fixed-ram'. > > The core of the feature is to ensure that each RAM page has a specific > offset in the resulting migration stream. The reasons why we'd want > such behavior are: > > - The resulting file will have a bounded size, since pages which are >dirtied multiple times will always go to a fixed location in the >file, rather than constantly being added to a sequential >stream. This eliminates cases where a VM with, say, 1G of RAM can >result in a migration file that's 10s of GBs, provided that the >workload constantly redirties memory. > > - It paves the way to implement O_DIRECT-enabled save/restore of the >migration stream as the pages are ensured to be written at aligned >offsets. > > - It allows the usage of multifd so we can write RAM pages to the >migration file in parallel. > > For now, enabling the capability has no effect. The next couple of > patches implement the core functionality. > > Signed-off-by: Fabiano Rosas > --- > - update migration.json to 9.0 and improve wording > - move docs to a separate file and add use cases information > --- > docs/devel/migration/features.rst | 1 + > docs/devel/migration/fixed-ram.rst | 137 + > migration/options.c| 34 +++ > migration/options.h| 1 + > migration/savevm.c | 1 + > qapi/migration.json| 6 +- > 6 files changed, 179 insertions(+), 1 deletion(-) > create mode 100644 docs/devel/migration/fixed-ram.rst > > diff --git a/docs/devel/migration/features.rst > b/docs/devel/migration/features.rst > index a9acaf618e..4c708b679a 100644 > --- a/docs/devel/migration/features.rst > +++ b/docs/devel/migration/features.rst > @@ -10,3 +10,4 @@ Migration has plenty of features to support different use > cases. > dirty-limit > vfio > virtio > + fixed-ram > diff --git a/docs/devel/migration/fixed-ram.rst > b/docs/devel/migration/fixed-ram.rst > new file mode 100644 > index 00..a6c0e5a360 > --- /dev/null > +++ b/docs/devel/migration/fixed-ram.rst > @@ -0,0 +1,137 @@ > +Fixed-ram > += > + > +Fixed-ram is a new stream format for the RAM section designed to > +supplement the existing ``file:`` migration and make it compatible > +with ``multifd``. This enables parallel migration of a guest's RAM to > +a file. > + > +The core of the feature is to ensure that each RAM page has a specific > +offset in the resulting migration file. This enables the ``multifd`` > +threads to write exclusively to those offsets even if the guest is > +constantly dirtying pages (i.e. live migration). Another benefit is > +that the resulting file will have a bounded size, since pages which > +are dirtied multiple times will always go to a fixed location in the > +file, rather than constantly being added to a sequential > +stream. Having the pages at fixed offsets also allows the usage of > +O_DIRECT for save/restore of the migration stream as the pages are > +ensured to be written respecting O_DIRECT alignment restrictions. > + > +Usage > +- > + > +On both source and destination, enable the ``multifd`` and > +``fixed-ram`` capabilities: > + > +``migrate_set_capability multifd on`` > + > +``migrate_set_capability fixed-ram on`` > + > +Use a ``file:`` URL for migration: > + > +``migrate file:/path/to/migration/file`` > + > +Fixed-ram migration is best done non-live, i.e. by stopping the VM on > +the source side before migrating. > + > +For best performance enable the ``direct-io`` capability as well: > + > +``migrate_set_capability direct-io on`` > + > +Use-cases > +- > + > +The fixed-ram feature was designed for use cases where the migration > +stream will be directed to a file in the filesystem and not > +immediately restored on the destination VM [#]_. These could be > +thought of as snapshots. We can further categorize them into live and > +non-live. > + > +- Non-live snapshot > + > +If the use case requires a VM to be stopped before taking a snapshot, > +that's the ideal scenario for fixed-ram migration. Not having to track > +dirty pages, the migration will write the RAM pages to the disk as > +fast as it can. > + > +Note: if a snapshot is taken of a running VM, but the VM will be > +stopped after the snapshot by the admin, then consider stopping it > +right before the snapshot to take benefit of the performance gains > +mentioned above. > + > +- Live snapshot > + > +If the use case requires that the VM keeps running during and after > +the snapshot operation, then fixed-ram migration can still be used, > +but will be less performant. Other strategies such as > +background-snapshot should be evaluated as well. One benefit of > +fixed-ram in this scenario is portability since background-snapshot > +depends on async dirty tracking (KVM_GET_DIRTY_LOG) which is not Background snapshot uses
Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability
Fabiano Rosas writes:
> Add a new migration capability 'fixed-ram'.
>
> The core of the feature is to ensure that each RAM page has a specific
> offset in the resulting migration stream. The reasons why we'd want
> such behavior are:
>
> - The resulting file will have a bounded size, since pages which are
>dirtied multiple times will always go to a fixed location in the
>file, rather than constantly being added to a sequential
>stream. This eliminates cases where a VM with, say, 1G of RAM can
>result in a migration file that's 10s of GBs, provided that the
>workload constantly redirties memory.
>
> - It paves the way to implement O_DIRECT-enabled save/restore of the
>migration stream as the pages are ensured to be written at aligned
>offsets.
>
> - It allows the usage of multifd so we can write RAM pages to the
>migration file in parallel.
>
> For now, enabling the capability has no effect. The next couple of
> patches implement the core functionality.
>
> Signed-off-by: Fabiano Rosas
[...]
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 5a565d9b8d..3fce5fe53e 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -531,6 +531,10 @@
> # and can result in more stable read performance. Requires KVM
> # with accelerator property "dirty-ring-size" set. (Since 8.1)
> #
> +# @fixed-ram: Migrate using fixed offsets in the migration file for
> +# each RAM page. Requires a migration URI that supports seeking,
> +# such as a file. (since 9.0)
> +#
> # Features:
> #
> # @deprecated: Member @block is deprecated. Use blockdev-mirror with
> @@ -555,7 +559,7 @@
> { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
> 'validate-uuid', 'background-snapshot',
> 'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
> - 'dirty-limit'] }
> + 'dirty-limit', 'fixed-ram'] }
>
> ##
> # @MigrationCapabilityStatus:
Can we find a better name than @fixed-ram? @fixed-ram-offsets?
@use-seek?
Apart from that, QAPI schema
Acked-by: Markus Armbruster
Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability
Daniel P. Berrangé writes:
> On Wed, Feb 21, 2024 at 10:24:05AM -0300, Fabiano Rosas wrote:
>> Markus Armbruster writes:
>>
>> > Fabiano Rosas writes:
>> >
>> >> Add a new migration capability 'fixed-ram'.
>> >>
>> >> The core of the feature is to ensure that each RAM page has a specific
>> >> offset in the resulting migration stream. The reasons why we'd want
>> >> such behavior are:
>> >>
>> >> - The resulting file will have a bounded size, since pages which are
>> >>dirtied multiple times will always go to a fixed location in the
>> >>file, rather than constantly being added to a sequential
>> >>stream. This eliminates cases where a VM with, say, 1G of RAM can
>> >>result in a migration file that's 10s of GBs, provided that the
>> >>workload constantly redirties memory.
>> >>
>> >> - It paves the way to implement O_DIRECT-enabled save/restore of the
>> >>migration stream as the pages are ensured to be written at aligned
>> >>offsets.
>> >>
>> >> - It allows the usage of multifd so we can write RAM pages to the
>> >>migration file in parallel.
>> >>
>> >> For now, enabling the capability has no effect. The next couple of
>> >> patches implement the core functionality.
>> >>
>> >> Signed-off-by: Fabiano Rosas
>> >
>> > [...]
>> >
>> >> diff --git a/qapi/migration.json b/qapi/migration.json
>> >> index 5a565d9b8d..3fce5fe53e 100644
>> >> --- a/qapi/migration.json
>> >> +++ b/qapi/migration.json
>> >> @@ -531,6 +531,10 @@
>> >> # and can result in more stable read performance. Requires KVM
>> >> # with accelerator property "dirty-ring-size" set. (Since 8.1)
>> >> #
>> >> +# @fixed-ram: Migrate using fixed offsets in the migration file for
>> >> +# each RAM page. Requires a migration URI that supports seeking,
>> >> +# such as a file. (since 9.0)
>> >> +#
>> >> # Features:
>> >> #
>> >> # @deprecated: Member @block is deprecated. Use blockdev-mirror with
>> >> @@ -555,7 +559,7 @@
>> >> { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
>> >> 'validate-uuid', 'background-snapshot',
>> >> 'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
>> >> - 'dirty-limit'] }
>> >> + 'dirty-limit', 'fixed-ram'] }
>> >>
>> >> ##
>> >> # @MigrationCapabilityStatus:
>> >
>> > Can we find a better name than @fixed-ram? @fixed-ram-offsets?
>> > @use-seek?
>>
>> I have no idea how we came to fixed-ram. The archives don't provide any
>> clarification. I find it confusing at first glance as well.
>>
>> A little brainstorming on how fixed-ram is different from exiting
>> migration:
>>
>> Fixed-ram:
>> uses a file, like the 'file:' migration;
>>
>> needs a seeking medium, such as a file;
>>
>> migrates ram by placing a page always in the same offset in the
>> file, contrary to normal migration which streams the page changes
>> continuously;
>>
>> ensures a migration file of size bounded to VM RAM size, contrary to
>> normal 'file:' migration which creates a file with unbounded size;
>>
>> enables multi-threaded RAM migration, even though we only use it when
>> multifd is enabled;
>>
>> uses scatter-gatter APIs (pwritev, preadv);
>>
>> So a few options:
>>
>> (disconsidering use-seek, it might be even more generic/vague)
>>
>> - fixed-ram-offsets
>> - non-streaming (or streaming: false)
>> - ram-scatter-gather (ram-sg)
>> - parallel-ram (even with the slight inaccuracy that we sometimes do it
>> single-threaded)
>
> I could add 'mapped-ram', as an alternative to 'fixed-ram'.
>
> The key distinguishing & motivating feature here is that
> RAM regions are mapped directly to file regions, instead
> of just being streamed at arbitrary points.
"map" is certainly a good shorthand for the various "placed at relative
offsets" that I used throughout this series.
Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability
On Wed, Feb 21, 2024 at 10:24:05AM -0300, Fabiano Rosas wrote:
> Markus Armbruster writes:
>
> > Fabiano Rosas writes:
> >
> >> Add a new migration capability 'fixed-ram'.
> >>
> >> The core of the feature is to ensure that each RAM page has a specific
> >> offset in the resulting migration stream. The reasons why we'd want
> >> such behavior are:
> >>
> >> - The resulting file will have a bounded size, since pages which are
> >>dirtied multiple times will always go to a fixed location in the
> >>file, rather than constantly being added to a sequential
> >>stream. This eliminates cases where a VM with, say, 1G of RAM can
> >>result in a migration file that's 10s of GBs, provided that the
> >>workload constantly redirties memory.
> >>
> >> - It paves the way to implement O_DIRECT-enabled save/restore of the
> >>migration stream as the pages are ensured to be written at aligned
> >>offsets.
> >>
> >> - It allows the usage of multifd so we can write RAM pages to the
> >>migration file in parallel.
> >>
> >> For now, enabling the capability has no effect. The next couple of
> >> patches implement the core functionality.
> >>
> >> Signed-off-by: Fabiano Rosas
> >
> > [...]
> >
> >> diff --git a/qapi/migration.json b/qapi/migration.json
> >> index 5a565d9b8d..3fce5fe53e 100644
> >> --- a/qapi/migration.json
> >> +++ b/qapi/migration.json
> >> @@ -531,6 +531,10 @@
> >> # and can result in more stable read performance. Requires KVM
> >> # with accelerator property "dirty-ring-size" set. (Since 8.1)
> >> #
> >> +# @fixed-ram: Migrate using fixed offsets in the migration file for
> >> +# each RAM page. Requires a migration URI that supports seeking,
> >> +# such as a file. (since 9.0)
> >> +#
> >> # Features:
> >> #
> >> # @deprecated: Member @block is deprecated. Use blockdev-mirror with
> >> @@ -555,7 +559,7 @@
> >> { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
> >> 'validate-uuid', 'background-snapshot',
> >> 'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
> >> - 'dirty-limit'] }
> >> + 'dirty-limit', 'fixed-ram'] }
> >>
> >> ##
> >> # @MigrationCapabilityStatus:
> >
> > Can we find a better name than @fixed-ram? @fixed-ram-offsets?
> > @use-seek?
>
> I have no idea how we came to fixed-ram. The archives don't provide any
> clarification. I find it confusing at first glance as well.
>
> A little brainstorming on how fixed-ram is different from exiting
> migration:
>
> Fixed-ram:
> uses a file, like the 'file:' migration;
>
> needs a seeking medium, such as a file;
>
> migrates ram by placing a page always in the same offset in the
> file, contrary to normal migration which streams the page changes
> continuously;
>
> ensures a migration file of size bounded to VM RAM size, contrary to
> normal 'file:' migration which creates a file with unbounded size;
>
> enables multi-threaded RAM migration, even though we only use it when
> multifd is enabled;
>
> uses scatter-gatter APIs (pwritev, preadv);
>
> So a few options:
>
> (disconsidering use-seek, it might be even more generic/vague)
>
> - fixed-ram-offsets
> - non-streaming (or streaming: false)
> - ram-scatter-gather (ram-sg)
> - parallel-ram (even with the slight inaccuracy that we sometimes do it
> single-threaded)
I could add 'mapped-ram', as an alternative to 'fixed-ram'.
The key distinguishing & motivating feature here is that
RAM regions are mapped directly to file regions, instead
of just being streamed at arbitrary points.
With regards,
Daniel
--
|: https://berrange.com -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|
Re: [PATCH v4 11/34] migration/ram: Introduce 'fixed-ram' migration capability
Markus Armbruster writes:
> Fabiano Rosas writes:
>
>> Add a new migration capability 'fixed-ram'.
>>
>> The core of the feature is to ensure that each RAM page has a specific
>> offset in the resulting migration stream. The reasons why we'd want
>> such behavior are:
>>
>> - The resulting file will have a bounded size, since pages which are
>>dirtied multiple times will always go to a fixed location in the
>>file, rather than constantly being added to a sequential
>>stream. This eliminates cases where a VM with, say, 1G of RAM can
>>result in a migration file that's 10s of GBs, provided that the
>>workload constantly redirties memory.
>>
>> - It paves the way to implement O_DIRECT-enabled save/restore of the
>>migration stream as the pages are ensured to be written at aligned
>>offsets.
>>
>> - It allows the usage of multifd so we can write RAM pages to the
>>migration file in parallel.
>>
>> For now, enabling the capability has no effect. The next couple of
>> patches implement the core functionality.
>>
>> Signed-off-by: Fabiano Rosas
>
> [...]
>
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 5a565d9b8d..3fce5fe53e 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -531,6 +531,10 @@
>> # and can result in more stable read performance. Requires KVM
>> # with accelerator property "dirty-ring-size" set. (Since 8.1)
>> #
>> +# @fixed-ram: Migrate using fixed offsets in the migration file for
>> +# each RAM page. Requires a migration URI that supports seeking,
>> +# such as a file. (since 9.0)
>> +#
>> # Features:
>> #
>> # @deprecated: Member @block is deprecated. Use blockdev-mirror with
>> @@ -555,7 +559,7 @@
>> { 'name': 'x-ignore-shared', 'features': [ 'unstable' ] },
>> 'validate-uuid', 'background-snapshot',
>> 'zero-copy-send', 'postcopy-preempt', 'switchover-ack',
>> - 'dirty-limit'] }
>> + 'dirty-limit', 'fixed-ram'] }
>>
>> ##
>> # @MigrationCapabilityStatus:
>
> Can we find a better name than @fixed-ram? @fixed-ram-offsets?
> @use-seek?
I have no idea how we came to fixed-ram. The archives don't provide any
clarification. I find it confusing at first glance as well.
A little brainstorming on how fixed-ram is different from exiting
migration:
Fixed-ram:
uses a file, like the 'file:' migration;
needs a seeking medium, such as a file;
migrates ram by placing a page always in the same offset in the
file, contrary to normal migration which streams the page changes
continuously;
ensures a migration file of size bounded to VM RAM size, contrary to
normal 'file:' migration which creates a file with unbounded size;
enables multi-threaded RAM migration, even though we only use it when
multifd is enabled;
uses scatter-gatter APIs (pwritev, preadv);
So a few options:
(disconsidering use-seek, it might be even more generic/vague)
- fixed-ram-offsets
- non-streaming (or streaming: false)
- ram-scatter-gather (ram-sg)
- parallel-ram (even with the slight inaccuracy that we sometimes do it
single-threaded)
Remember we also use this name internally, so I think a broader
"feature" name is better that a super specific one.
Does anyone have a strong preference? Other suggestions?
> Apart from that, QAPI schema
> Acked-by: Markus Armbruster
Thanks!
