Dongli Zhang <[email protected]> writes: > On Fri, Aug 21, 2026 12:13:30AM -0700, Markus Armbruster wrote: >> Dongli Zhang <[email protected]> writes: >> >>> On Thu, Aug 20, 2026 12:09:19AM -0700, Markus Armbruster wrote: >>>> Dongli Zhang <[email protected]> writes: >>>> >>>>> Add a memfd migration transport to the MigrationAddress QAPI schema. The >>>>> new transport has no user-supplied arguments because QEMU creates and >>>>> passes the backing memfd internally. >>>>> >>>>> The initial user is cpr-exec, where the main VM/device migration stream >>>>> can >>>>> be stored in an inherited memfd instead of an external file. >>>>> >>>>> Signed-off-by: Dongli Zhang <[email protected]> >>>>> --- >>>>> qapi/migration.json | 32 +++++++++++++++++++++++++++----- >>>>> 1 file changed, 27 insertions(+), 5 deletions(-) >>>>> >>>>> diff --git a/qapi/migration.json b/qapi/migration.json >>>>> index b1eaf7b054..1505c201ca 100644 >>>>> --- a/qapi/migration.json >>>>> +++ b/qapi/migration.json >>>>> @@ -702,9 +702,11 @@ >>>>> # @cpr-exec: The migrate command stops the VM, saves state to the >>>>> # migration channel, directly exec's a new version of QEMU on the >>>>> # same host, replacing the original process while retaining its >>>>> -# PID, and loads state from the channel. Guest RAM is preserved >>>>> -# in place. Devices and their pinned pages are also preserved for >>>>> -# VFIO and IOMMUFD. >>>>> +# PID, and loads state from the channel. With the @memfd >>>>> +# migration transport, QEMU saves VM/device state to an internal >>>>> +# memfd instead of an external main migration channel. Guest RAM >>>>> +# is preserved in place. Devices and their pinned pages are also >>>>> +# preserved for VFIO and IOMMUFD. >>>>> # >>>>> # Old QEMU starts new QEMU by exec'ing the command specified by >>>>> # the @cpr-exec-command parameter. The command may be a direct >>>>> @@ -716,6 +718,13 @@ >>>>> # as a file, that accepts all data before old QEMU exits. >>>>> # Otherwise, old QEMU may quietly block writing to the channel. >>>>> # >>>>> +# Alternatively, use the @memfd migration transport to save >>>>> +# VM/device state to an internal memfd inherited by new QEMU. >>>>> +# >>>> >>>> Any guidance on when to use @memfd? >>> >>> My objective is to avoid using disks, or running QEMU with writable storage. >>> >>> Both guest I/O and networking can go through VFIO, and everything remains >>> in memory. >>> >>> When I was testing these CPR features, my favorite was cpr-exec because it >>> does >>> not require creating an additional QEMU instance. >>> >>> Based on my understanding, the best approach to live update a running >>> binary is >>> to replace only the code section while keeping everything else in memory, >>> i.e. >>> guest memory, device fds, and device state. >>> >>> Unfortunately, device state cannot simply remain in memory across exec(), >>> so we >>> need to migrate it from the old instance to the new instance. I was >>> thinking: >>> why not use memory instead of storage? >>> >>> Initially, I added the change only for cpr-exec. Later, I reworked the >>> change to >>> make memfd a new migration transport. As a side benefit for me, I also >>> practiced >>> how to add a new transport to the QEMU migration code :) >> >> This is why you wrote the feature. Please work it into the commit >> message. > > Sure. > >> >> We should also help users decide whether to use the feature. I gather >> it's useful with migration mode cpr-exec. Any other use cases? Does it >> even work with other migration modes? If yes, which ones, and what >> happens when you try with another one? > > So far I only added support for cpr-exec. I have not tried othe modes.
What happens when you try to use it with other migration modes? > In my opinion, cpr-transfer could also use a memfd-backed channel, but I am > not > sure it is necessary. > > For cpr-transfer, the source and destination are separate QEMU processes. If > we > want to pass a memfd between them, we still need an IPC mechanism that can > transfer file descriptors, such as a Unix domain socket with SCM_RIGHTS. That > reduces the value of using memfd as the migration channel, because we need at > least one Unix domain socket anyway to pass the memfd fd and coordinate the > handoff. > > If a Unix domain socket is already required, the main benefit of memfd would > likely be performance, especially when the device state is large. We'd want to measure the performance impact to see whether it's worth complicating configuration and code. > cpr-exec is different. The new QEMU is started via exec(), so the memfd fd can > be preserved across exec by clearing CLOEXEC and passing the fd number through > an environment variable. This allows us to add a memfd-backed migration > channel > without needing an external file or socket for the device state. What are the actual user-visible benefits? Performance? Provide measurements, please. Simpler configuration? Not sure that would be worth the special case. Anything else? > One potential future use case is LUO/KHO support for QEMU. In that case, > device > state could be preserved in memory as a memfd and potentially survive across a > KHO kexec. We won't need to preserve anything on storage. Indeed, I did some > internal tests using memfd to preserve device state across kexec although > using > different implementation, and it worked. > > For other live migration modes, or even local migration, I do not think this > is > useful. For example, saving to and loading from a file won't use memfd, as the > QEMU processes are terminated during the this process. User documentation should spell out when memfd can be used. I recommend to make it usable only when it's known to improve things in at least some scenario that matters. Code should cleanly reject attempts to use memfd when it's not usable. User documentation should help users decide whether to use it. The commit message(s) may need to provide additional detail on how things improve. > Thank you very much! > > Dongli Zhang You're welcome! [...]
