On Sun, 20 Sep 2020 15:25:19 +0200 "Maciej S. Szmigiero" <m...@maciej.szmigiero.name> wrote:
> From: "Maciej S. Szmigiero" <maciej.szmigi...@oracle.com> >From description it sounds like an alternative of virtio-mem, CCing David. > This series adds a Hyper-V Dynamic Memory Protocol driver (hv-balloon) > and its protocol definitions. > Also included is a driver providing backing devices for memory hot-add > protocols ("haprots"). > > A haprot device works like a virtual DIMM stick: it allows inserting > extra RAM into the guest at run time. > > The main differences from the ACPI-based PC DIMM hotplug are: > * Notifying the guest about the new memory range is not done via ACPI but > via a protocol handler that registers with the haprot framework. > This means that the ACPI DIMM slot limit does not apply. > > * A protocol handler can prevent removal of a haprot device when it is > still in use by setting its "busy" field. > > * A protocol handler can also register an "unplug" callback so it gets > notified when an user decides to remove the haprot device. > This way the protocol handler can inform the guest about this fact and / or > do its own cleanup. > > The hv-balloon driver is like virtio-balloon on steroids: it allows both > changing the guest memory allocation via ballooning and inserting extra > RAM into it by adding haprot virtual DIMM sticks. > One of advantages of these over ACPI-based PC DIMM hotplug is that such > memory can be hotplugged in much smaller granularity because the ACPI DIMM > slot limit does not apply. > > In contrast with ACPI DIMM hotplug where one can only request to unplug a > whole DIMM stick this driver allows removing memory from guest in single > page (4k) units via ballooning. > Then, once the guest has released the whole memory backed by a haprot > virtual DIMM stick such device is marked "unused" and can be removed from > the VM, if one wants so. > A "HV_BALLOON_HAPROT_UNUSED" QMP event is emitted in this case so the > software controlling QEMU knows that this operation is now possible. > > The haprot devices are also marked unused after a VM reboot (with a > corresponding "HV_BALLOON_HAPROT_UNUSED" QMP event). > They are automatically reinserted (if still present) after the guest > reconnects to this protocol (a "HV_BALLOON_HAPROT_INUSE" QMP event is then > emitted). > > For performance reasons, the guest-released memory is tracked in few range > trees, as a series of (start, count) ranges. > Each time a new page range is inserted into such tree its neighbors are > checked as candidates for possible merging with it. > > Besides performance reasons, the Dynamic Memory protocol itself uses page > ranges as the data structure in its messages, so relevant pages need to be > merged into such ranges anyway. > > One has to be careful when tracking the guest-released pages, since the > guest can maliciously report returning pages outside its current address > space, which later clash with the address range of newly added memory. > Similarly, the guest can report freeing the same page twice. > > The above design results in much better ballooning performance than when > using virtio-balloon with the same guest: 230 GB / minute with this driver > versus 70 GB / minute with virtio-balloon. > > During a ballooning operation most of time is spent waiting for the guest > to come up with newly freed page ranges, processing the received ranges on > the host side (in QEMU / KVM) is nearly instantaneous. > > The unballoon operation is also pretty much instantaneous: > thanks to the merging of the ballooned out page ranges 200 GB of memory can > be returned to the guest in about 1 second. > With virtio-balloon this operation takes about 2.5 minutes. > > These tests were done against a Windows Server 2019 guest running on a > Xeon E5-2699, after dirtying the whole memory inside guest before each > balloon operation. > > Using a range tree instead of a bitmap to track the removed memory also > means that the solution scales well with the guest size: even a 1 TB range > takes just few bytes of memory. > > The required GTree operations are available at > https://gitlab.gnome.org/maciejsszmigiero/glib/-/tree/gtree-add-iterators > Since they are not yet in the upstream Glib a check for them was added to > "configure" script, together with new "--enable-hv-balloon" and > "--disable-hv-balloon" arguments. > If these GTree operations are missing in the system Glib this driver will > be skipped during QEMU build. > > An optional "status-report=on" device parameter requests memory status > events from the guest (typically sent every second), which allow the host > to learn both the guest memory available and the guest memory in use > counts. > They are emitted externally as "HV_BALLOON_STATUS_REPORT" QMP events. > > The driver is named hv-balloon since the Linux kernel client driver for > the Dynamic Memory Protocol is named as such and to follow the naming > pattern established by the virtio-balloon driver. > The whole protocol runs over Hyper-V VMBus that has its implementation > recently merged in. > > The driver was tested against Windows Server 2012 R2, Windows Server 2016 > and Windows Server 2016 guests and obeys the guest alignment requirements > reported to the host via DM_CAPABILITIES_REPORT message. > Extensive event tracing is available under 'hv_balloon_*' prefix. > > Example usage: > * Add "-device vmbus-bridge,id=vmbus-bridge -device hv-balloon,id=hvb" > to the QEMU command line and set "maxmem" value to something large, > like 1T. > > * Use QEMU monitor commands to add a haprot virtual DIMM stick, together > with its memory backend: > object_add memory-backend-ram,id=mem1,size=200G > device_add mem-haprot,id=ha1,memdev=mem1 > The first command is actually the same as for ACPI-based DIMM hotplug. > > * Use the ballooning interface monitor commands to force the guest to give > out as much memory as possible: > balloon 1 > The ballooning interface monitor commands can also be used to resize > the guest up and down appropriately. > > * One can check the current guest size by issuing a "info balloon" command. > This is useful to know what is happening, since large ballooning or > unballooning operations take some time to complete. > > * Once the guest releases the whole memory backed by a haprot device > (or is restarted) a "HV_BALLOON_HAPROT_UNUSED" QMP event will be > generated. > The haprot device then can be removed, together with its memory backend: > device_del ha1 > object_del mem1 > > Future directions: > * Allow sharing the ballooning QEMU interface between hv-balloon and > virtio-balloon drivers. > Currently, only one of them can be added to the VM at the same time. > > * Allow new haport devices to reuse the same address range as the ones > that were previously deleted via device_del monitor command without > having to restart the VM. > > * Add vmstate / live migration support to the hv-balloon driver. > > * Use haprot device to also add memory via virtio interface (this requires > defining a new operation in virtio-balloon protocol and appropriate > support from the client virtio-balloon driver in the Linux kernel). > > Kconfig.host | 3 + > configure | 35 + > hw/hyperv/Kconfig | 5 + > hw/hyperv/hv-balloon.c | 2172 ++++++++++++++++++++++++++++++ > hw/hyperv/meson.build | 1 + > hw/hyperv/trace-events | 17 + > hw/i386/Kconfig | 2 + > hw/i386/pc.c | 18 +- > hw/mem/Kconfig | 4 + > hw/mem/haprot.c | 263 ++++ > hw/mem/meson.build | 1 + > include/hw/hyperv/dynmem-proto.h | 425 ++++++ > include/hw/mem/haprot.h | 72 + > meson.build | 2 + > qapi/misc.json | 74 + > 15 files changed, 3093 insertions(+), 1 deletion(-) > create mode 100644 hw/hyperv/hv-balloon.c > create mode 100644 hw/mem/haprot.c > create mode 100644 include/hw/hyperv/dynmem-proto.h > create mode 100644 include/hw/mem/haprot.h > >