On 12/11/2015 1:16 AM, Alexander Duyck wrote:
On Thu, Dec 10, 2015 at 6:38 AM, Lan, Tianyu <tianyu....@intel.com> wrote:
On 12/10/2015 7:41 PM, Dr. David Alan Gilbert wrote:
Ideally, it is able to leave guest driver unmodified but it requires the
hypervisor or qemu to aware the device which means we may need a driver
in
hypervisor or qemu to handle the device on behalf of guest driver.
Can you answer the question of when do you use your code -
at the start of migration or
just before the end?
Just before stopping VCPU in this version and inject VF mailbox irq to
notify the driver if the irq handler is installed.
Qemu side also will check this via the faked PCI migration capability
and driver will set the status during device open() or resume() callback.
The VF mailbox interrupt is a very bad idea. Really the device should
be in a reset state on the other side of a migration. It doesn't make
sense to have the interrupt firing if the device is not configured.
This is one of the things that is preventing you from being able to
migrate the device while the interface is administratively down or the
VF driver is not loaded.
From my opinion, if VF driver is not loaded and hardware doesn't start
to work, the device state doesn't need to be migrated.
We may add a flag for driver to check whether migration happened during
it's down and reinitialize the hardware and clear the flag when system
try to put it up.
We may add migration core in the Linux kernel and provide some helps
functions to facilitate to add migration support for drivers.
Migration core is in charge to sync status with Qemu.
Example.
migration_register()
Driver provides
- Callbacks to be called before and after migration or for bad path
- Its irq which it prefers to deal with migration event.
migration_event_check()
Driver calls it in the irq handler. Migration core code will check
migration status and call its callbacks when migration happens.
My thought on all this is that it might make sense to move this
functionality into a PCI-to-PCI bridge device and make it a
requirement that all direct-assigned devices have to exist behind that
device in order to support migration. That way you would be working
with a directly emulated device that would likely already be
supporting hot-plug anyway. Then it would just be a matter of coming
up with a few Qemu specific extensions that you would need to add to
the device itself. The same approach would likely be portable enough
that you could achieve it with PCIe as well via the same configuration
space being present on the upstream side of a PCIe port or maybe a
PCIe switch of some sort.
It would then be possible to signal via your vendor-specific PCI
capability on that device that all devices behind this bridge require
DMA page dirtying, you could use the configuration in addition to the
interrupt already provided for hot-plug to signal things like when you
are starting migration, and possibly even just extend the shpc
functionality so that if this capability is present you have the
option to pause/resume instead of remove/probe the device in the case
of certain hot-plug events. The fact is there may be some use for a
pause/resume type approach for PCIe hot-plug in the near future
anyway. From the sounds of it Apple has required it for all
Thunderbolt device drivers so that they can halt the device in order
to shuffle resources around, perhaps we should look at something
similar for Linux.
The other advantage behind grouping functions on one bridge is things
like reset domains. The PCI error handling logic will want to be able
to reset any devices that experienced an error in the event of
something such as a surprise removal. By grouping all of the devices
you could disable/reset/enable them as one logical group in the event
of something such as the "bad path" approach Michael has mentioned.
These sounds we need to add a faked bridge for migration and adding a
driver in the guest for it. It also needs to extend PCI bus/hotplug
driver to do pause/resume other devices, right?
My concern is still that whether we can change PCI bus/hotplug like that
without spec change.
IRQ should be general for any devices and we may extend it for
migration. Device driver also can make decision to support migration
or not.
It would be great if we could avoid changing the guest; but at least
your guest
driver changes don't actually seem to be that hardware specific;
could your
changes actually be moved to generic PCI level so they could be made
to work for lots of drivers?
It is impossible to use one common solution for all devices unless the
PCIE
spec documents it clearly and i think one day it will be there. But
before
that, we need some workarounds on guest driver to make it work even it
looks
ugly.
Yes, so far there is not hardware migration support and it's hard to modify
bus level code. It also will block implementation on the Windows.
Please don't assume things. Unless you have hard data from Microsoft
that says they want it this way lets just try to figure out what works
best for us for now and then we can start worrying about third party
implementations after we have figured out a solution that actually
works.
- Alex
