On 5/17/19 8:58 AM, Jens Freimann wrote:
This is another attempt at implementing the host side of the
net_failover concept
(https://www.kernel.org/doc/html/latest/networking/net_failover.html)
Changes since last RFC:
- work around circular dependency of commandline options. Just add
failover=on to the virtio-net standby options and reference it from
primary (vfio-pci) device with standby=<id>
- add patch 3/4 to allow migration of vfio-pci device when it is part of a
failover pair, still disallow for all other devices
- add patch 4/4 to allow unplug of device during migrationm, make an
exception for failover primary devices. I'd like feedback on how to
solve this more elegant. I added a boolean to DeviceState, have it
default to false for all devices except for primary devices.
- not tested yet with surprise removal
- I don't expect this to go in as it is, still needs more testing but
I'd like to get feedback on above mentioned changes.
The general idea is that we have a pair of devices, a vfio-pci and a
emulated device. Before migration the vfio device is unplugged and data
flows to the emulated device, on the target side another vfio-pci device
is plugged in to take over the data-path. In the guest the net_failover
module will pair net devices with the same MAC address.
* In the first patch the infrastructure for hiding the device is added
for the qbus and qdev APIs.
* In the second patch the virtio-net uses the API to defer adding the vfio
device until the VIRTIO_NET_F_STANDBY feature is acked.
Previous discussion:
RFC v1 https://patchwork.ozlabs.org/cover/989098/
RFC v2 https://www.mail-archive.com/qemu-devel@nongnu.org/msg606906.html
To summarize concerns/feedback from previous discussion:
1.- guest OS can reject or worse _delay_ unplug by any amount of time.
Migration might get stuck for unpredictable time with unclear reason.
This approach combines two tricky things, hot/unplug and migration.
-> We can surprise-remove the PCI device and in QEMU we can do all
necessary rollbacks transparent to management software. Will it be
easy, probably not.
2. PCI devices are a precious ressource. The primary device should never
be added to QEMU if it won't be used by guest instead of hiding it in
QEMU.
-> We only hotplug the device when the standby feature bit was
negotiated. We save the device cmdline options until we need it for
qdev_device_add()
Hiding a device can be a useful concept to model. For example a
pci device in a powered-off slot could be marked as hidden until the slot
is
powered on (mst).
3. Management layer software should handle this. Open Stack already has
components/code to handle unplug/replug VFIO devices and metadata to
provide to the guest for detecting which devices should be paired.
-> An approach that includes all software from firmware to
higher-level management software wasn't tried in the last years. This is
an attempt to keep it simple and contained in QEMU as much as possible.
4. Hotplugging a device and then making it part of a failover setup is
not possible
-> addressed by extending qdev hotplug functions to check for hidden
attribute, so e.g. device_add can be used to plug a device.
I have tested this with a mlx5 NIC and was able to migrate the VM with
above mentioned workarounds for open problems.
Command line example:
qemu-system-x86_64 -enable-kvm -m 3072 -smp 3 \
-machine q35,kernel-irqchip=split -cpu host \
-k fr \
-serial stdio \
-net none \
-qmp unix:/tmp/qmp.socket,server,nowait \
-monitor telnet:127.0.0.1:5555,server,nowait \
-device pcie-root-port,id=root0,multifunction=on,chassis=0,addr=0xa \
-device pcie-root-port,id=root1,bus=pcie.0,chassis=1 \
-device pcie-root-port,id=root2,bus=pcie.0,chassis=2 \
-netdev
tap,script=/root/bin/bridge.sh,downscript=no,id=hostnet1,vhost=on \
-device
virtio-net-pci,netdev=hostnet1,id=net1,mac=52:54:00:6f:55:cc,bus=root2,failover=on
\
/root/rhel-guest-image-8.0-1781.x86_64.qcow2
Then the primary device can be hotplugged via
(qemu) device_add vfio-pci,host=5e:00.2,id=hostdev0,bus=root1,standby=net1
I guess this is the commandline on the migration destination, and as far
as I understand from this example, on the destination we (meaning
libvirt or higher level management application) must *not* include the
assigned device on the qemu commandline, but must instead hotplug the
device later after the guest CPUs have been restarted on the destination.
So if I'm understanding correctly, the idea is that on the migration
source, the device may have been hotplugged, or may have been included
when qemu was originally started. Then qemu automatically handles the
unplug of the device on the source, but it seems qemu does nothing on
the destination, leaving that up to libvirt or a higher layer to implement.
Then in order for this to work, libvirt (or OpenStack or oVirt or
whoever) needs to understand that the device in the libvirt config (it
will still be in the libvirt config, since from libvirt's POV it hasn't
been unplugged):
1) shouldn't be included in the qemu commandline on the destination,
2) will almost surely need to be replaced with a different device on the
destination (since it's almost certain that the destination won't have
an available device at the same PCI address)
3) will probably need to be unbinded from the VF net driver (does this
need to happen before migration is finished? If we want to lower the
probability of a failure after we're already committed to the migration,
then I think we must, but libvirt isn't set up for that in any way).
4) will need to be hotplugged after the migration has finished *and*
after the guest CPUs have been restarted on the destination.
While it will be possible to assure that there is a destination device,
and to replace the old device with new in the config (and maybe, either
with some major reworking of device assignment code, or offloading the
responsibility to the management application(s), possible to re-bind the
device to the vfio-pci driver), prior to marking the migration as
"successful" (thus committing to running it on the destination), we
can't say as much for actually assigning the device. So if the
assignment fails, then what happens?
So a few issues I see that will need to be solved by [someone]
(apparently either libvirt or management):
a) there isn't anything in libvirt's XML grammar that allows us to
signify a device that is "present in the config but shouldn't be
included in the commandline"
b) someone will need to replace the device from the source with an
equivalent device on the destination in the libvirt XML. There are other
cases of management modifying the XML during migration (I think), but
this does point out that putting the "auto-unplug code into qemu isn't
turning this into a trivial
c) there is nothing in libvirt's migration logic that can cause a device
to be re-binded to vfio-pci prior to completion of a migration. Unless
this is added to libvirt (or the re-bind operation is passed off to the
management application), we will need to live with the possibility that
hotplugging the device will fail due to failed re-bind *after* we've
committed to the migration.
d) once the guest CPUs are restarted on the destination, [someone]
(libvirt or management) needs to hotplug the new device on the
destination. (I'm guessing that a hotplug can only be done while the
guest CPUs are running; correct me if this is wrong!)
This sounds like a lot of complexity for something that was supposed to
be handled completely/transparently by qemu :-P.
