Hi Tobin,
On 03/02/21 21:48, Tobin Feldman-Fitzthum wrote:
This is a demonstration of fast migration for encrypted virtual machines
using a Migration Handler that lives in OVMF. This demo uses AMD SEV,
but the ideas may generalize to other confidential computing platforms.
With AMD SEV, guest memory is encrypted and the hypervisor cannot access
or move it. This makes migration tricky. In this demo, we show how the
HV can ask a Migration Handler (MH) in the firmware for an encrypted
page. The MH encrypts the page with a transport key prior to releasing
it to the HV. The target machine also runs an MH that decrypts the page
once it is passed in by the target HV. These patches are not ready for
production, but the are a full end-to-end solution that facilitates a
fast live migration between two SEV VMs.
Corresponding patches for QEMU have been posted my colleague Dov Murik
on qemu-devel. Our approach needs little kernel support, requiring only
one hypercall that the guest can use to mark a page as encrypted or
shared. This series includes updated patches from Ashish Kalra and
Brijesh Singh that allow OVMF to use this hypercall.
The MH runs continuously in the guest, waiting for communication from
the HV. The HV starts an additional vCPU for the MH but does not expose
it to the guest OS via ACPI. We use the MpService to start the MH. The
MpService is only available at runtime and processes that are started by
it are usually cleaned up on ExitBootServices. Since we need the MH to
run continuously, we had to make some modifications. Ideally a feature
could be added to the MpService to allow for the starting of
long-running processes. Besides migration, this could support other
background processes that need to operate within the encryption
boundary. For now, we have included a handful of patches that modify the
MpService to allow the MH to keep running after ExitBootServices. These
are temporary.
I plan to do a lightweight review for this series. (My understanding is
that it's an RFC and not actually being proposed for merging.)
Regarding the MH's availability at runtime -- does that necessarily
require the isolation of an AP? Because in the current approach,
allowing the MP Services to survive into OS runtime (in some form or
another) seems critical, and I don't think it's going to fly.
I agree that the UefiCpuPkg patches have been well separated from the
rest of the series, but I'm somewhat doubtful the "firmware-initiated
background process" idea will be accepted. Have you investigated
exposing a new "runtime service" (a function pointer) via the UEFI
Configuration table, and calling that (perhaps periodically?) from the
guest kernel? It would be a form of polling I guess. Or maybe, poll the
mailbox directly in the kernel, and call the new firmware runtime
service when there's an actual command to process.
Continuous runtime availability for the MH is almost certainly the most
controversial part of this proposal, which is why I put it in the cover
letter and why it's good to discuss.
(You do spell out "little kernel support", and I'm not sure if that's a
technical benefit, or a political / community benefit.)
As you allude to, minimal kernel support is really one of the main things
that shapes our approach. This is partly a political and practical benefit,
but there are also technical benefits. Having the MH in firmware likely
leads to higher availability. It can be accessed when the OS is unreachable,
perhaps during boot or when the OS is hung. There are also potential
portability advantages although we do currently require support for one
hypercall. The cost of implementing this hypercall is low.
Generally speaking, our task is to find a home for functionality that was
traditionally provided by the hypervisor, but that needs to be inside the
trust domain, but that isn't really part of a guest. A meta-goal of this
project is to figure out the best way to do this.
I'm quite uncomfortable with an attempt to hide a CPU from the OS via
ACPI. The OS has other ways to learn (for example, a boot loader could
use the MP services itself, stash the information, and hand it to the OS
kernel -- this would minimally allow for detecting an inconsistency in
the OS). What about "all-but-self" IPIs too -- the kernel might think
all the processors it's poking like that were under its control.
This might be the second most controversial piece. Here's a question: if we
could successfully hide the MH vCPU from the OS, would it still make you
uncomfortable? In other words, is the worry that there might be some
inconsistency or more generally that there is something hidden from the OS?
One thing to think about is that the guest owner should generally be aware
that there is a migration handler running. The way I see it, a guest owner
of an SEV VM would need to opt-in to migration and should then expect that
there is an MH running even if they aren't able to see it. Of course we need
to be certain that the MH isn't going to break the OS.
Also, as far as I can tell from patch #7, the AP seems to be
busy-looping (with a CpuPause() added in), for the entire lifetime of
the OS. Do I understand right? If so -- is it a temporary trait as well?
In our approach the MH continuously checks for commands from the hypervisor.
There are potentially ways to optimize this, such as having the hypervisor
de-schedule the MH vCPU while not migrating. You could potentially shut down
down the MH on the target after receiving the MH_RESET command (when the
migration finishes), but what if you want to migrate that VM somewhere else?
