Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Wed, 25 May 2016 11:45:11 +0300
"Michael S. Tsirkin"  wrote:

> On Tue, May 24, 2016 at 08:54:06PM -0600, Alex Williamson wrote:
> > On Tue, 24 May 2016 13:49:12 +0300
> > "Michael S. Tsirkin"  wrote:
> >   
> > > On Tue, Apr 26, 2016 at 08:48:15AM -0600, Alex Williamson wrote:  
> > > > I think that means that if we want to switch from a
> > > > simple halt-on-error to a mechanism for the guest to handle recovery,
> > > > we need to disable access to the device between being notified that the
> > > > error occurred and being notified to resume.
> > > 
> > > But this isn't what happens on bare metal.
> > > Errors are reported asynchronously and host might access the device
> > > meanwhile.  These accesses might or might not trigger more errors, but
> > > fundamentally this should not matter too much as device is going to be
> > > reset.  
> > 
> > Bare metal also doesn't have a hypervisor underneath performing a PCI
> > bus reset,  
> 
> This is where I get lost. I assumed we do reset when guest
> requests it. Isn't that the case? Why not?

Unless we can somehow opt-out vfio-pci devices from AER handling on the
host, then the host is going to try to recover the device as part of
the core AER handling, which is not driver directed.  The host driver
can register various callbacks, which is where we get the
error_detected notification, but until we get the resume notification,
I think we assume the device is being operated on by the core.

At the same time we're using the error_detected to signal the AER event
to the guest, which begins a similar recovery process.  We don't
particularly want to rely on how the host has recovered the device and
the guest driver needs to be aware that the device state may need to be
recovered, so we let the recovery proceed in both the host and guest,
but it seems we need some synchronization point and guest accesses to
the device while we know the host is still in recovery may do more harm
than good.
 
> > there's only one OS trying to control the device at a time,
> > so we have some clear differences from bare metal that I don't know we
> > can avoid.  The thought here was that we need to notify the guest at the
> > earliest point we can, but let the host recovery run to completion
> > before allowing the user to interact with the device.  Perhaps there is
> > no need to block region access to the device (ie. config space & BAR
> > resources), but I think we do need to somehow synchronize the bus resets
> > or else we get situations like that observed previously where the bus is
> > still in reset while userspace trys to proceed with using it.
> >  
> 
> Why do we have to trigger reset upon an error?
> Why not wait for guest to request reset?

Is there a way to opt-out of host AER handling?  Do we want to create a
special case for vfio-pci to do this?  Does doing so allow the user to
exploit the host in anyway, such as the user failing to recover the
device, continuing to signal error events on the host, and perhaps
leaving the device in a lest trustworthy state when returned to the
host (not that we should ever be trusting the state of the device at
that point).

> > The next question then would be whether that's QEMU's job or something
> > that should be done in the host kernel.  It's been proposed to add yet
> > another eventfd for the kernel vfio-pci to signal QEMU when a resume
> > notification has occured, but perhaps the better approach would be for
> > the hot reset ioctl (and base reset ioctl) to handle this situation more
> > transparently.  We could immediately return -EAGAIN and allow QEMU to
> > delay itself for any reset ioctl received after the AER error detected
> > event, but before the resume event.  We could also allow some sort of
> > timeout, that the ioctl might enter an interruptible sleep, woken on
> > the resume notification or timeout.  That sounds a bit better to me as
> > the specification of what's allowed between the error detected
> > notification and the resume notification is otherwise pretty poorly
> > defined.  
> 
> So if guest started reset, it might take a while for
> device to come out of that state, and access during this
> time might trigger errors. But that's already possible
> for guest to trigger, right?  How is this different?

You can look back through the history of this series, there was an
arbitrary delay added after reset because the device was still in
reset (presumably host and guest reset racing).  Yes we do not
currently block access to the device during a reset, the issue is
mostly that we don't expect device resets to be occurring in the host
except when directed by the guest.  In this case we expect a host
directed reset is occurring and the guest directed reset seems to be a
synchronization point.

> >  Do you think we can run completely asynchronous, letting the
> > host and guest bus resets race?  Thanks,
> > 
> > Alex  
> 
> I have a feeling we need to put some code out,
> 

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Michael S. Tsirkin]

On Tue, May 24, 2016 at 08:54:06PM -0600, Alex Williamson wrote:
> On Tue, 24 May 2016 13:49:12 +0300
> "Michael S. Tsirkin"  wrote:
> 
> > On Tue, Apr 26, 2016 at 08:48:15AM -0600, Alex Williamson wrote:
> > > I think that means that if we want to switch from a
> > > simple halt-on-error to a mechanism for the guest to handle recovery,
> > > we need to disable access to the device between being notified that the
> > > error occurred and being notified to resume.  
> > 
> > But this isn't what happens on bare metal.
> > Errors are reported asynchronously and host might access the device
> > meanwhile.  These accesses might or might not trigger more errors, but
> > fundamentally this should not matter too much as device is going to be
> > reset.
> 
> Bare metal also doesn't have a hypervisor underneath performing a PCI
> bus reset,

This is where I get lost. I assumed we do reset when guest
requests it. Isn't that the case? Why not?

> there's only one OS trying to control the device at a time,
> so we have some clear differences from bare metal that I don't know we
> can avoid.  The thought here was that we need to notify the guest at the
> earliest point we can, but let the host recovery run to completion
> before allowing the user to interact with the device.  Perhaps there is
> no need to block region access to the device (ie. config space & BAR
> resources), but I think we do need to somehow synchronize the bus resets
> or else we get situations like that observed previously where the bus is
> still in reset while userspace trys to proceed with using it.
>

Why do we have to trigger reset upon an error?
Why not wait for guest to request reset?

> The next question then would be whether that's QEMU's job or something
> that should be done in the host kernel.  It's been proposed to add yet
> another eventfd for the kernel vfio-pci to signal QEMU when a resume
> notification has occured, but perhaps the better approach would be for
> the hot reset ioctl (and base reset ioctl) to handle this situation more
> transparently.  We could immediately return -EAGAIN and allow QEMU to
> delay itself for any reset ioctl received after the AER error detected
> event, but before the resume event.  We could also allow some sort of
> timeout, that the ioctl might enter an interruptible sleep, woken on
> the resume notification or timeout.  That sounds a bit better to me as
> the specification of what's allowed between the error detected
> notification and the resume notification is otherwise pretty poorly
> defined.

So if guest started reset, it might take a while for
device to come out of that state, and access during this
time might trigger errors. But that's already possible
for guest to trigger, right?  How is this different?


>  Do you think we can run completely asynchronous, letting the
> host and guest bus resets race?  Thanks,
> 
> Alex

I have a feeling we need to put some code out,
disabled by default, and see how it behaves in the field.
For example ability to trigger UR errors seems benign but
I think we are trying to prevent them now because of
something we saw in the field.

-- 
MST

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Tue, 24 May 2016 13:49:12 +0300
"Michael S. Tsirkin"  wrote:

> On Tue, Apr 26, 2016 at 08:48:15AM -0600, Alex Williamson wrote:
> > I think that means that if we want to switch from a
> > simple halt-on-error to a mechanism for the guest to handle recovery,
> > we need to disable access to the device between being notified that the
> > error occurred and being notified to resume.  
> 
> But this isn't what happens on bare metal.
> Errors are reported asynchronously and host might access the device
> meanwhile.  These accesses might or might not trigger more errors, but
> fundamentally this should not matter too much as device is going to be
> reset.

Bare metal also doesn't have a hypervisor underneath performing a PCI
bus reset, there's only one OS trying to control the device at a time,
so we have some clear differences from bare metal that I don't know we
can avoid.  The thought here was that we need to notify the guest at the
earliest point we can, but let the host recovery run to completion
before allowing the user to interact with the device.  Perhaps there is
no need to block region access to the device (ie. config space & BAR
resources), but I think we do need to somehow synchronize the bus resets
or else we get situations like that observed previously where the bus is
still in reset while userspace trys to proceed with using it.

The next question then would be whether that's QEMU's job or something
that should be done in the host kernel.  It's been proposed to add yet
another eventfd for the kernel vfio-pci to signal QEMU when a resume
notification has occured, but perhaps the better approach would be for
the hot reset ioctl (and base reset ioctl) to handle this situation more
transparently.  We could immediately return -EAGAIN and allow QEMU to
delay itself for any reset ioctl received after the AER error detected
event, but before the resume event.  We could also allow some sort of
timeout, that the ioctl might enter an interruptible sleep, woken on
the resume notification or timeout.  That sounds a bit better to me as
the specification of what's allowed between the error detected
notification and the resume notification is otherwise pretty poorly
defined.  Do you think we can run completely asynchronous, letting the
host and guest bus resets race?  Thanks,

Alex

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Zhou Jie]

So I needn't disable access to the device between being notified that 
the error occurred and being notified to resume.

This will make code simpler.

Am I right?

On 2016/5/24 18:49, Michael S. Tsirkin wrote:

On Tue, Apr 26, 2016 at 08:48:15AM -0600, Alex Williamson wrote:

I think that means that if we want to switch from a
simple halt-on-error to a mechanism for the guest to handle recovery,
we need to disable access to the device between being notified that the
error occurred and being notified to resume.


But this isn't what happens on bare metal.
Errors are reported asynchronously and host might access the device
meanwhile.  These accesses might or might not trigger more errors, but
fundamentally this should not matter too much as device is going to be
reset.

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Michael S. Tsirkin]

On Tue, Apr 26, 2016 at 08:48:15AM -0600, Alex Williamson wrote:
> I think that means that if we want to switch from a
> simple halt-on-error to a mechanism for the guest to handle recovery,
> we need to disable access to the device between being notified that the
> error occurred and being notified to resume.

But this isn't what happens on bare metal.
Errors are reported asynchronously and host might access the device
meanwhile.  These accesses might or might not trigger more errors, but
fundamentally this should not matter too much as device is going to be
reset.

-- 
MST

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Wed, 11 May 2016 11:11:39 +0800
Zhou Jie  wrote:

> Hi, Alex
>  What do you think about the following solution?
>  1. Detect support for resume notification.
> If host vfio driver does not have resume notifier flags,
> Directly fail to boot up VM as with aer enabled.

It's not a flag used to detect the resume notifier, but simply probing
IRQ_INFO for the index allocated for this notification.

>  2. Immediately notify the VM on error detected.
>  3. Stall any access to the device until resume is signaled.
> Disable mmaps, drop writes, return -1 for reads.
>  4. Delay the guest directed bus reset.
> Don't reset bus in vfio_pci_reset function.
>  5. Wait for resume notification.
> If we don't get the resume notification from the host after
> some timeout, we would abort the guest directed bus reset
> altogether and make the device disappear,
> Initiating an unplug of the device to prevent it from further
> interacting with the VM.
>  6. After get the resume notification.
> Reset bus.
> It the second bus reset. Because the host did bus reset already.
> But as you said we shouldn't necessarily design the API that
> strictly around the current behavior of the Linux AER handler.

Otherwise it sounds like what I had proposed.  Thanks,

Alex

> On 2016/5/7 0:39, Alex Williamson wrote:
> > On Fri, 6 May 2016 09:38:41 +0800
> > Chen Fan  wrote:
> >  
> >> On 04/26/2016 10:48 PM, Alex Williamson wrote:  
> >>> On Tue, 26 Apr 2016 11:39:02 +0800
> >>> Chen Fan  wrote:
> >>>  
>  On 04/14/2016 09:02 AM, Chen Fan wrote:  
> > On 04/12/2016 05:38 AM, Alex Williamson wrote:  
> >> On Tue, 5 Apr 2016 19:42:02 +0800
> >> Cao jin  wrote:
> >>  
> >>> From: Chen Fan
> >>>
> >>> for supporting aer recovery, host and guest would run the same aer
> >>> recovery code, that would do the secondary bus reset if the error
> >>> is fatal, the aer recovery process:
> >>> 1. error_detected
> >>> 2. reset_link (if fatal)
> >>> 3. slot_reset/mmio_enabled
> >>> 4. resume
> >>>
> >>> it indicates that host will do secondary bus reset to reset
> >>> the physical devices under bus in step 2, that would cause
> >>> devices in D3 status in a short time. but in qemu, we register
> >>> an error detected handler, that would be invoked as host broadcasts
> >>> the error-detected event in step 1, in order to avoid guest do
> >>> reset_link when host do reset_link simultaneously. it may cause
> >>> fatal error. we introduce a resmue notifier to assure host reset
> >>> completely. then do guest aer injection.  
> >> Why is it safe to continue running the VM between the error detected
> >> notification and the resume notification?  We're just pushing back the
> >> point at which we inject the AER into the guest, potentially negating
> >> any benefit by allowing the VM to consume bad data.  Shouldn't we
> >> instead be immediately notifying the VM on error detected, but stalling
> >> any access to the device until resume is signaled?  How do we know that
> >> resume will ever be signaled?  We have both the problem that we may be
> >> running on an older kernel that won't support a resume notification and
> >> the problem that seeing a resume notification depends on the host being
> >> able to successfully complete a link reset after fatal error. We can
> >> detect support for resume notification, but we still need a strategy
> >> for never receiving it.  Thanks,  
> > That's make sense, but I haven't came up with a good idea. do you have
> > any idea, Alex?  
> >>> I don't know that there are any good solutions here.  We need to
> >>> respond to the current error notifier interrupt and not regress from
> >>> our support there.  I think that means that if we want to switch from a
> >>> simple halt-on-error to a mechanism for the guest to handle recovery,
> >>> we need to disable access to the device between being notified that the
> >>> error occurred and being notified to resume.  We can do that by
> >>> disabling mmaps to the device and preventing access via the slow path
> >>> handlers.  I don't know what the best solution is for preventing access,
> >>> do we block and pause the VM or do we drop writes and return -1 for
> >>> reads, that's something that needs to be determined.  We also need to
> >>> inject the AER into the VM at the point we're notified of an error
> >>> because the VM needs to know as soon as possible to stop using the
> >>> device or trusting any data from it.  The next coordination point would
> >>> be something like the resume notifier that you've added and there are
> >>> numerous questions 

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Zhou Jie]

Hi, Alex
What do you think about the following solution?
1. Detect support for resume notification.
   If host vfio driver does not have resume notifier flags,
   Directly fail to boot up VM as with aer enabled.
2. Immediately notify the VM on error detected.
3. Stall any access to the device until resume is signaled.
   Disable mmaps, drop writes, return -1 for reads.
4. Delay the guest directed bus reset.
   Don't reset bus in vfio_pci_reset function.
5. Wait for resume notification.
   If we don't get the resume notification from the host after
   some timeout, we would abort the guest directed bus reset
   altogether and make the device disappear,
   Initiating an unplug of the device to prevent it from further
   interacting with the VM.
6. After get the resume notification.
   Reset bus.
   It the second bus reset. Because the host did bus reset already.
   But as you said we shouldn't necessarily design the API that
   strictly around the current behavior of the Linux AER handler.

Sincerely,
Zhou Jie

On 2016/5/7 0:39, Alex Williamson wrote:

On Fri, 6 May 2016 09:38:41 +0800
Chen Fan  wrote:


On 04/26/2016 10:48 PM, Alex Williamson wrote:

On Tue, 26 Apr 2016 11:39:02 +0800
Chen Fan  wrote:


On 04/14/2016 09:02 AM, Chen Fan wrote:

On 04/12/2016 05:38 AM, Alex Williamson wrote:

On Tue, 5 Apr 2016 19:42:02 +0800
Cao jin  wrote:


From: Chen Fan

for supporting aer recovery, host and guest would run the same aer
recovery code, that would do the secondary bus reset if the error
is fatal, the aer recovery process:
1. error_detected
2. reset_link (if fatal)
3. slot_reset/mmio_enabled
4. resume

it indicates that host will do secondary bus reset to reset
the physical devices under bus in step 2, that would cause
devices in D3 status in a short time. but in qemu, we register
an error detected handler, that would be invoked as host broadcasts
the error-detected event in step 1, in order to avoid guest do
reset_link when host do reset_link simultaneously. it may cause
fatal error. we introduce a resmue notifier to assure host reset
completely. then do guest aer injection.

Why is it safe to continue running the VM between the error detected
notification and the resume notification?  We're just pushing back the
point at which we inject the AER into the guest, potentially negating
any benefit by allowing the VM to consume bad data.  Shouldn't we
instead be immediately notifying the VM on error detected, but stalling
any access to the device until resume is signaled?  How do we know that
resume will ever be signaled?  We have both the problem that we may be
running on an older kernel that won't support a resume notification and
the problem that seeing a resume notification depends on the host being
able to successfully complete a link reset after fatal error. We can
detect support for resume notification, but we still need a strategy
for never receiving it.  Thanks,

That's make sense, but I haven't came up with a good idea. do you have
any idea, Alex?

I don't know that there are any good solutions here.  We need to
respond to the current error notifier interrupt and not regress from
our support there.  I think that means that if we want to switch from a
simple halt-on-error to a mechanism for the guest to handle recovery,
we need to disable access to the device between being notified that the
error occurred and being notified to resume.  We can do that by
disabling mmaps to the device and preventing access via the slow path
handlers.  I don't know what the best solution is for preventing access,
do we block and pause the VM or do we drop writes and return -1 for
reads, that's something that needs to be determined.  We also need to
inject the AER into the VM at the point we're notified of an error
because the VM needs to know as soon as possible to stop using the
device or trusting any data from it.  The next coordination point would
be something like the resume notifier that you've added and there are
numerous questions around the interaction of that with the guest
handling.  Clearly we can't do a guest directed bus reset until we get
the resume notifier, so do we block that execution path in QEMU until
the resume notification is received?  What happens if we don't get that
notification?  Is there any way that we can rely on the host having
done a bus reset to the point where we don't need to act on the guest
directed reset?  These are all things that need to be figured out.
Thanks,

Maybe we can simply pause the vcpu running and avoid the VM to
access the device. and add two flags in VFIO_DEVICE_GET_INFO to query
whether the vfio pci driver has a resume notifier,
if it does not have resume notifier flags, we can directly fail to boot
up VM
as with aer enabled.


We can already tell if a resume 

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Fri, 6 May 2016 09:38:41 +0800
Chen Fan  wrote:

> On 04/26/2016 10:48 PM, Alex Williamson wrote:
> > On Tue, 26 Apr 2016 11:39:02 +0800
> > Chen Fan  wrote:
> >  
> >> On 04/14/2016 09:02 AM, Chen Fan wrote:  
> >>> On 04/12/2016 05:38 AM, Alex Williamson wrote:  
>  On Tue, 5 Apr 2016 19:42:02 +0800
>  Cao jin  wrote:
>  
> > From: Chen Fan
> >
> > for supporting aer recovery, host and guest would run the same aer
> > recovery code, that would do the secondary bus reset if the error
> > is fatal, the aer recovery process:
> > 1. error_detected
> > 2. reset_link (if fatal)
> > 3. slot_reset/mmio_enabled
> > 4. resume
> >
> > it indicates that host will do secondary bus reset to reset
> > the physical devices under bus in step 2, that would cause
> > devices in D3 status in a short time. but in qemu, we register
> > an error detected handler, that would be invoked as host broadcasts
> > the error-detected event in step 1, in order to avoid guest do
> > reset_link when host do reset_link simultaneously. it may cause
> > fatal error. we introduce a resmue notifier to assure host reset
> > completely. then do guest aer injection.  
>  Why is it safe to continue running the VM between the error detected
>  notification and the resume notification?  We're just pushing back the
>  point at which we inject the AER into the guest, potentially negating
>  any benefit by allowing the VM to consume bad data.  Shouldn't we
>  instead be immediately notifying the VM on error detected, but stalling
>  any access to the device until resume is signaled?  How do we know that
>  resume will ever be signaled?  We have both the problem that we may be
>  running on an older kernel that won't support a resume notification and
>  the problem that seeing a resume notification depends on the host being
>  able to successfully complete a link reset after fatal error. We can
>  detect support for resume notification, but we still need a strategy
>  for never receiving it.  Thanks,  
> >>> That's make sense, but I haven't came up with a good idea. do you have
> >>> any idea, Alex?  
> > I don't know that there are any good solutions here.  We need to
> > respond to the current error notifier interrupt and not regress from
> > our support there.  I think that means that if we want to switch from a
> > simple halt-on-error to a mechanism for the guest to handle recovery,
> > we need to disable access to the device between being notified that the
> > error occurred and being notified to resume.  We can do that by
> > disabling mmaps to the device and preventing access via the slow path
> > handlers.  I don't know what the best solution is for preventing access,
> > do we block and pause the VM or do we drop writes and return -1 for
> > reads, that's something that needs to be determined.  We also need to
> > inject the AER into the VM at the point we're notified of an error
> > because the VM needs to know as soon as possible to stop using the
> > device or trusting any data from it.  The next coordination point would
> > be something like the resume notifier that you've added and there are
> > numerous questions around the interaction of that with the guest
> > handling.  Clearly we can't do a guest directed bus reset until we get
> > the resume notifier, so do we block that execution path in QEMU until
> > the resume notification is received?  What happens if we don't get that
> > notification?  Is there any way that we can rely on the host having
> > done a bus reset to the point where we don't need to act on the guest
> > directed reset?  These are all things that need to be figured out.
> > Thanks,  
> Maybe we can simply pause the vcpu running and avoid the VM to
> access the device. and add two flags in VFIO_DEVICE_GET_INFO to query
> whether the vfio pci driver has a resume notifier,
> if it does not have resume notifier flags, we can directly fail to boot 
> up VM
> as with aer enabled.

We can already tell if a resume interrupt is supported between the IRQ
count in vfio_device_info and a probe with vfio_irq_info, what would
additional flags in vfio_device_info tell us beyond a resume interrupt
being supported?  Is pausing the VM acceptable from a service guarantee
perspective to users?  A bus reset can take a full second and I imagine
deeper PCI hierarchies can push that out depending on what level the
error occurs.  A second of downtime may be enough to trigger failovers
to other systems.  If we were to disable mmaps when a fault occurs, we
could trap any further device access, drop writes, return -1 for
reads.  This seems reasonable since we've already notified the VM that
the device had a fault.  The synchronization point seems like when the
guest tries to 

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Chen Fan]

On 04/26/2016 10:48 PM, Alex Williamson wrote:

On Tue, 26 Apr 2016 11:39:02 +0800
Chen Fan  wrote:


On 04/14/2016 09:02 AM, Chen Fan wrote:

On 04/12/2016 05:38 AM, Alex Williamson wrote:

On Tue, 5 Apr 2016 19:42:02 +0800
Cao jin  wrote:
  

From: Chen Fan

for supporting aer recovery, host and guest would run the same aer
recovery code, that would do the secondary bus reset if the error
is fatal, the aer recovery process:
1. error_detected
2. reset_link (if fatal)
3. slot_reset/mmio_enabled
4. resume

it indicates that host will do secondary bus reset to reset
the physical devices under bus in step 2, that would cause
devices in D3 status in a short time. but in qemu, we register
an error detected handler, that would be invoked as host broadcasts
the error-detected event in step 1, in order to avoid guest do
reset_link when host do reset_link simultaneously. it may cause
fatal error. we introduce a resmue notifier to assure host reset
completely. then do guest aer injection.

Why is it safe to continue running the VM between the error detected
notification and the resume notification?  We're just pushing back the
point at which we inject the AER into the guest, potentially negating
any benefit by allowing the VM to consume bad data.  Shouldn't we
instead be immediately notifying the VM on error detected, but stalling
any access to the device until resume is signaled?  How do we know that
resume will ever be signaled?  We have both the problem that we may be
running on an older kernel that won't support a resume notification and
the problem that seeing a resume notification depends on the host being
able to successfully complete a link reset after fatal error. We can
detect support for resume notification, but we still need a strategy
for never receiving it.  Thanks,

That's make sense, but I haven't came up with a good idea. do you have
any idea, Alex?

I don't know that there are any good solutions here.  We need to
respond to the current error notifier interrupt and not regress from
our support there.  I think that means that if we want to switch from a
simple halt-on-error to a mechanism for the guest to handle recovery,
we need to disable access to the device between being notified that the
error occurred and being notified to resume.  We can do that by
disabling mmaps to the device and preventing access via the slow path
handlers.  I don't know what the best solution is for preventing access,
do we block and pause the VM or do we drop writes and return -1 for
reads, that's something that needs to be determined.  We also need to
inject the AER into the VM at the point we're notified of an error
because the VM needs to know as soon as possible to stop using the
device or trusting any data from it.  The next coordination point would
be something like the resume notifier that you've added and there are
numerous questions around the interaction of that with the guest
handling.  Clearly we can't do a guest directed bus reset until we get
the resume notifier, so do we block that execution path in QEMU until
the resume notification is received?  What happens if we don't get that
notification?  Is there any way that we can rely on the host having
done a bus reset to the point where we don't need to act on the guest
directed reset?  These are all things that need to be figured out.
Thanks,

Maybe we can simply pause the vcpu running and avoid the VM to
access the device. and add two flags in VFIO_DEVICE_GET_INFO to query
whether the vfio pci driver has a resume notifier,
if it does not have resume notifier flags, we can directly fail to boot 
up VM

as with aer enabled. otherwise, we should wait for resume notifier coming to
restart the cpu. about the problem of the reduplicated bus reset by host 
and guest,
I think qemu can according to the error is fatal or non-fatal to decide 
whether need
to do a bus reset on guest, I think it's not critical and could be 
resolved later.


Thanks,
Chen



Alex


.

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Tue, 26 Apr 2016 11:39:02 +0800
Chen Fan  wrote:

> On 04/14/2016 09:02 AM, Chen Fan wrote:
> >
> > On 04/12/2016 05:38 AM, Alex Williamson wrote:  
> >> On Tue, 5 Apr 2016 19:42:02 +0800
> >> Cao jin  wrote:
> >>  
> >>> From: Chen Fan 
> >>>
> >>> for supporting aer recovery, host and guest would run the same aer
> >>> recovery code, that would do the secondary bus reset if the error
> >>> is fatal, the aer recovery process:
> >>>1. error_detected
> >>>2. reset_link (if fatal)
> >>>3. slot_reset/mmio_enabled
> >>>4. resume
> >>>
> >>> it indicates that host will do secondary bus reset to reset
> >>> the physical devices under bus in step 2, that would cause
> >>> devices in D3 status in a short time. but in qemu, we register
> >>> an error detected handler, that would be invoked as host broadcasts
> >>> the error-detected event in step 1, in order to avoid guest do
> >>> reset_link when host do reset_link simultaneously. it may cause
> >>> fatal error. we introduce a resmue notifier to assure host reset
> >>> completely. then do guest aer injection.  
> >> Why is it safe to continue running the VM between the error detected
> >> notification and the resume notification?  We're just pushing back the
> >> point at which we inject the AER into the guest, potentially negating
> >> any benefit by allowing the VM to consume bad data.  Shouldn't we
> >> instead be immediately notifying the VM on error detected, but stalling
> >> any access to the device until resume is signaled?  How do we know that
> >> resume will ever be signaled?  We have both the problem that we may be
> >> running on an older kernel that won't support a resume notification and
> >> the problem that seeing a resume notification depends on the host being
> >> able to successfully complete a link reset after fatal error. We can
> >> detect support for resume notification, but we still need a strategy
> >> for never receiving it.  Thanks,  
> > That's make sense, but I haven't came up with a good idea. do you have
> > any idea, Alex?

I don't know that there are any good solutions here.  We need to
respond to the current error notifier interrupt and not regress from
our support there.  I think that means that if we want to switch from a
simple halt-on-error to a mechanism for the guest to handle recovery,
we need to disable access to the device between being notified that the
error occurred and being notified to resume.  We can do that by
disabling mmaps to the device and preventing access via the slow path
handlers.  I don't know what the best solution is for preventing access,
do we block and pause the VM or do we drop writes and return -1 for
reads, that's something that needs to be determined.  We also need to
inject the AER into the VM at the point we're notified of an error
because the VM needs to know as soon as possible to stop using the
device or trusting any data from it.  The next coordination point would
be something like the resume notifier that you've added and there are
numerous questions around the interaction of that with the guest
handling.  Clearly we can't do a guest directed bus reset until we get
the resume notifier, so do we block that execution path in QEMU until
the resume notification is received?  What happens if we don't get that
notification?  Is there any way that we can rely on the host having
done a bus reset to the point where we don't need to act on the guest
directed reset?  These are all things that need to be figured out.
Thanks,

Alex

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Chen Fan]

On 04/14/2016 09:02 AM, Chen Fan wrote:


On 04/12/2016 05:38 AM, Alex Williamson wrote:

On Tue, 5 Apr 2016 19:42:02 +0800
Cao jin  wrote:


From: Chen Fan 

for supporting aer recovery, host and guest would run the same aer
recovery code, that would do the secondary bus reset if the error
is fatal, the aer recovery process:
   1. error_detected
   2. reset_link (if fatal)
   3. slot_reset/mmio_enabled
   4. resume

it indicates that host will do secondary bus reset to reset
the physical devices under bus in step 2, that would cause
devices in D3 status in a short time. but in qemu, we register
an error detected handler, that would be invoked as host broadcasts
the error-detected event in step 1, in order to avoid guest do
reset_link when host do reset_link simultaneously. it may cause
fatal error. we introduce a resmue notifier to assure host reset
completely. then do guest aer injection.

Why is it safe to continue running the VM between the error detected
notification and the resume notification?  We're just pushing back the
point at which we inject the AER into the guest, potentially negating
any benefit by allowing the VM to consume bad data.  Shouldn't we
instead be immediately notifying the VM on error detected, but stalling
any access to the device until resume is signaled?  How do we know that
resume will ever be signaled?  We have both the problem that we may be
running on an older kernel that won't support a resume notification and
the problem that seeing a resume notification depends on the host being
able to successfully complete a link reset after fatal error. We can
detect support for resume notification, but we still need a strategy
for never receiving it.  Thanks,

That's make sense, but I haven't came up with a good idea. do you have
any idea, Alex?


ping...


Thanks,
Chen




Alex


Signed-off-by: Chen Fan 
---
  hw/vfio/pci.c  | 157 
+++--

  hw/vfio/pci.h  |   2 +
  linux-headers/linux/vfio.h |   1 +
  3 files changed, 126 insertions(+), 34 deletions(-)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 691ff5e..d79fb3d 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2610,12 +2610,7 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
  static void vfio_err_notifier_handler(void *opaque)
  {
  VFIOPCIDevice *vdev = opaque;
-PCIDevice *dev = >pdev;
  Error *local_err = NULL;
-PCIEAERMsg msg = {
-.severity = 0,
-.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
-};
if (!event_notifier_test_and_clear(>err_notifier)) {
  return;
@@ -2636,35 +2631,7 @@ static void vfio_err_notifier_handler(void 
*opaque)

  goto stop;
  }
  -/*
- * we should read the error details from the real hardware
- * configuration spaces, here we only need to do is signaling
- * to guest an uncorrectable error has occurred.
- */
-if (dev->exp.aer_cap) {
-uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
-uint32_t uncor_status;
-bool isfatal;
-
-uncor_status = vfio_pci_read_config(dev,
-   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 
4);

-
-/*
- * if the error is not emitted by this device, we can
- * just ignore it.
- */
-if (!(uncor_status & ~0UL)) {
-return;
-}
-
-isfatal = uncor_status & pci_get_long(aer_cap + 
PCI_ERR_UNCOR_SEVER);

-
-msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
- PCI_ERR_ROOT_CMD_NONFATAL_EN;
-
-pcie_aer_msg(dev, );
-return;
-}
+return;
stop:
  /*
@@ -2757,6 +2724,126 @@ static void 
vfio_unregister_err_notifier(VFIOPCIDevice *vdev)

  event_notifier_cleanup(>err_notifier);
  }
  +static void vfio_resume_notifier_handler(void *opaque)
+{
+VFIOPCIDevice *vdev = opaque;
+PCIDevice *dev = >pdev;
+PCIEAERMsg msg = {
+.severity = 0,
+.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
+};
+
+if (!event_notifier_test_and_clear(>resume_notifier)) {
+return;
+}
+
+/*
+ * we should read the error details from the real hardware
+ * configuration spaces, here we only need to do is signaling
+ * to guest an uncorrectable error has occurred.
+ */
+if (dev->exp.aer_cap) {
+uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
+uint32_t uncor_status;
+bool isfatal;
+
+uncor_status = vfio_pci_read_config(dev,
+   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 
4);

+
+/*
+ * if the error is not emitted by this device, we can
+ * just ignore it.
+ */
+if (!(uncor_status & ~0UL)) {
+return;
+}
+
+isfatal = uncor_status & pci_get_long(aer_cap + 
PCI_ERR_UNCOR_SEVER);

+
+msg.severity = isfatal ? 

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Chen Fan]

On 04/12/2016 05:38 AM, Alex Williamson wrote:

On Tue, 5 Apr 2016 19:42:02 +0800
Cao jin  wrote:


From: Chen Fan 

for supporting aer recovery, host and guest would run the same aer
recovery code, that would do the secondary bus reset if the error
is fatal, the aer recovery process:
   1. error_detected
   2. reset_link (if fatal)
   3. slot_reset/mmio_enabled
   4. resume

it indicates that host will do secondary bus reset to reset
the physical devices under bus in step 2, that would cause
devices in D3 status in a short time. but in qemu, we register
an error detected handler, that would be invoked as host broadcasts
the error-detected event in step 1, in order to avoid guest do
reset_link when host do reset_link simultaneously. it may cause
fatal error. we introduce a resmue notifier to assure host reset
completely. then do guest aer injection.

Why is it safe to continue running the VM between the error detected
notification and the resume notification?  We're just pushing back the
point at which we inject the AER into the guest, potentially negating
any benefit by allowing the VM to consume bad data.  Shouldn't we
instead be immediately notifying the VM on error detected, but stalling
any access to the device until resume is signaled?  How do we know that
resume will ever be signaled?  We have both the problem that we may be
running on an older kernel that won't support a resume notification and
the problem that seeing a resume notification depends on the host being
able to successfully complete a link reset after fatal error.  We can
detect support for resume notification, but we still need a strategy
for never receiving it.  Thanks,

That's make sense, but I haven't came up with a good idea. do you have
any idea, Alex?

Thanks,
Chen




Alex


Signed-off-by: Chen Fan 
---
  hw/vfio/pci.c  | 157 +++--
  hw/vfio/pci.h  |   2 +
  linux-headers/linux/vfio.h |   1 +
  3 files changed, 126 insertions(+), 34 deletions(-)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 691ff5e..d79fb3d 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2610,12 +2610,7 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
  static void vfio_err_notifier_handler(void *opaque)
  {
  VFIOPCIDevice *vdev = opaque;
-PCIDevice *dev = >pdev;
  Error *local_err = NULL;
-PCIEAERMsg msg = {
-.severity = 0,
-.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
-};
  
  if (!event_notifier_test_and_clear(>err_notifier)) {

  return;
@@ -2636,35 +2631,7 @@ static void vfio_err_notifier_handler(void *opaque)
  goto stop;
  }
  
-/*

- * we should read the error details from the real hardware
- * configuration spaces, here we only need to do is signaling
- * to guest an uncorrectable error has occurred.
- */
-if (dev->exp.aer_cap) {
-uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
-uint32_t uncor_status;
-bool isfatal;
-
-uncor_status = vfio_pci_read_config(dev,
-   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
-
-/*
- * if the error is not emitted by this device, we can
- * just ignore it.
- */
-if (!(uncor_status & ~0UL)) {
-return;
-}
-
-isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
-
-msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
- PCI_ERR_ROOT_CMD_NONFATAL_EN;
-
-pcie_aer_msg(dev, );
-return;
-}
+return;
  
  stop:

  /*
@@ -2757,6 +2724,126 @@ static void vfio_unregister_err_notifier(VFIOPCIDevice 
*vdev)
  event_notifier_cleanup(>err_notifier);
  }
  
+static void vfio_resume_notifier_handler(void *opaque)

+{
+VFIOPCIDevice *vdev = opaque;
+PCIDevice *dev = >pdev;
+PCIEAERMsg msg = {
+.severity = 0,
+.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
+};
+
+if (!event_notifier_test_and_clear(>resume_notifier)) {
+return;
+}
+
+/*
+ * we should read the error details from the real hardware
+ * configuration spaces, here we only need to do is signaling
+ * to guest an uncorrectable error has occurred.
+ */
+if (dev->exp.aer_cap) {
+uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
+uint32_t uncor_status;
+bool isfatal;
+
+uncor_status = vfio_pci_read_config(dev,
+   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
+
+/*
+ * if the error is not emitted by this device, we can
+ * just ignore it.
+ */
+if (!(uncor_status & ~0UL)) {
+return;
+}
+
+isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
+
+msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
+  

Re: [Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Alex Williamson]

On Tue, 5 Apr 2016 19:42:02 +0800
Cao jin  wrote:

> From: Chen Fan 
> 
> for supporting aer recovery, host and guest would run the same aer
> recovery code, that would do the secondary bus reset if the error
> is fatal, the aer recovery process:
>   1. error_detected
>   2. reset_link (if fatal)
>   3. slot_reset/mmio_enabled
>   4. resume
> 
> it indicates that host will do secondary bus reset to reset
> the physical devices under bus in step 2, that would cause
> devices in D3 status in a short time. but in qemu, we register
> an error detected handler, that would be invoked as host broadcasts
> the error-detected event in step 1, in order to avoid guest do
> reset_link when host do reset_link simultaneously. it may cause
> fatal error. we introduce a resmue notifier to assure host reset
> completely. then do guest aer injection.

Why is it safe to continue running the VM between the error detected
notification and the resume notification?  We're just pushing back the
point at which we inject the AER into the guest, potentially negating
any benefit by allowing the VM to consume bad data.  Shouldn't we
instead be immediately notifying the VM on error detected, but stalling
any access to the device until resume is signaled?  How do we know that
resume will ever be signaled?  We have both the problem that we may be
running on an older kernel that won't support a resume notification and
the problem that seeing a resume notification depends on the host being
able to successfully complete a link reset after fatal error.  We can
detect support for resume notification, but we still need a strategy
for never receiving it.  Thanks,

Alex

> Signed-off-by: Chen Fan 
> ---
>  hw/vfio/pci.c  | 157 
> +++--
>  hw/vfio/pci.h  |   2 +
>  linux-headers/linux/vfio.h |   1 +
>  3 files changed, 126 insertions(+), 34 deletions(-)
> 
> diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
> index 691ff5e..d79fb3d 100644
> --- a/hw/vfio/pci.c
> +++ b/hw/vfio/pci.c
> @@ -2610,12 +2610,7 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
>  static void vfio_err_notifier_handler(void *opaque)
>  {
>  VFIOPCIDevice *vdev = opaque;
> -PCIDevice *dev = >pdev;
>  Error *local_err = NULL;
> -PCIEAERMsg msg = {
> -.severity = 0,
> -.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
> -};
>  
>  if (!event_notifier_test_and_clear(>err_notifier)) {
>  return;
> @@ -2636,35 +2631,7 @@ static void vfio_err_notifier_handler(void *opaque)
>  goto stop;
>  }
>  
> -/*
> - * we should read the error details from the real hardware
> - * configuration spaces, here we only need to do is signaling
> - * to guest an uncorrectable error has occurred.
> - */
> -if (dev->exp.aer_cap) {
> -uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
> -uint32_t uncor_status;
> -bool isfatal;
> -
> -uncor_status = vfio_pci_read_config(dev,
> -   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
> -
> -/*
> - * if the error is not emitted by this device, we can
> - * just ignore it.
> - */
> -if (!(uncor_status & ~0UL)) {
> -return;
> -}
> -
> -isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
> -
> -msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
> - PCI_ERR_ROOT_CMD_NONFATAL_EN;
> -
> -pcie_aer_msg(dev, );
> -return;
> -}
> +return;
>  
>  stop:
>  /*
> @@ -2757,6 +2724,126 @@ static void 
> vfio_unregister_err_notifier(VFIOPCIDevice *vdev)
>  event_notifier_cleanup(>err_notifier);
>  }
>  
> +static void vfio_resume_notifier_handler(void *opaque)
> +{
> +VFIOPCIDevice *vdev = opaque;
> +PCIDevice *dev = >pdev;
> +PCIEAERMsg msg = {
> +.severity = 0,
> +.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
> +};
> +
> +if (!event_notifier_test_and_clear(>resume_notifier)) {
> +return;
> +}
> +
> +/*
> + * we should read the error details from the real hardware
> + * configuration spaces, here we only need to do is signaling
> + * to guest an uncorrectable error has occurred.
> + */
> +if (dev->exp.aer_cap) {
> +uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
> +uint32_t uncor_status;
> +bool isfatal;
> +
> +uncor_status = vfio_pci_read_config(dev,
> +   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
> +
> +/*
> + * if the error is not emitted by this device, we can
> + * just ignore it.
> + */
> +if (!(uncor_status & ~0UL)) {
> +return;
> +}
> +
> +isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
> +
> +

[Qemu-devel] [patch v6 11/12] vfio: register aer resume notification handler for aer resume

[Cao jin]

From: Chen Fan 

for supporting aer recovery, host and guest would run the same aer
recovery code, that would do the secondary bus reset if the error
is fatal, the aer recovery process:
  1. error_detected
  2. reset_link (if fatal)
  3. slot_reset/mmio_enabled
  4. resume

it indicates that host will do secondary bus reset to reset
the physical devices under bus in step 2, that would cause
devices in D3 status in a short time. but in qemu, we register
an error detected handler, that would be invoked as host broadcasts
the error-detected event in step 1, in order to avoid guest do
reset_link when host do reset_link simultaneously. it may cause
fatal error. we introduce a resmue notifier to assure host reset
completely. then do guest aer injection.

Signed-off-by: Chen Fan 
---
 hw/vfio/pci.c  | 157 +++--
 hw/vfio/pci.h  |   2 +
 linux-headers/linux/vfio.h |   1 +
 3 files changed, 126 insertions(+), 34 deletions(-)

diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c
index 691ff5e..d79fb3d 100644
--- a/hw/vfio/pci.c
+++ b/hw/vfio/pci.c
@@ -2610,12 +2610,7 @@ static void vfio_put_device(VFIOPCIDevice *vdev)
 static void vfio_err_notifier_handler(void *opaque)
 {
 VFIOPCIDevice *vdev = opaque;
-PCIDevice *dev = >pdev;
 Error *local_err = NULL;
-PCIEAERMsg msg = {
-.severity = 0,
-.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
-};
 
 if (!event_notifier_test_and_clear(>err_notifier)) {
 return;
@@ -2636,35 +2631,7 @@ static void vfio_err_notifier_handler(void *opaque)
 goto stop;
 }
 
-/*
- * we should read the error details from the real hardware
- * configuration spaces, here we only need to do is signaling
- * to guest an uncorrectable error has occurred.
- */
-if (dev->exp.aer_cap) {
-uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
-uint32_t uncor_status;
-bool isfatal;
-
-uncor_status = vfio_pci_read_config(dev,
-   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
-
-/*
- * if the error is not emitted by this device, we can
- * just ignore it.
- */
-if (!(uncor_status & ~0UL)) {
-return;
-}
-
-isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
-
-msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
- PCI_ERR_ROOT_CMD_NONFATAL_EN;
-
-pcie_aer_msg(dev, );
-return;
-}
+return;
 
 stop:
 /*
@@ -2757,6 +2724,126 @@ static void vfio_unregister_err_notifier(VFIOPCIDevice 
*vdev)
 event_notifier_cleanup(>err_notifier);
 }
 
+static void vfio_resume_notifier_handler(void *opaque)
+{
+VFIOPCIDevice *vdev = opaque;
+PCIDevice *dev = >pdev;
+PCIEAERMsg msg = {
+.severity = 0,
+.source_id = (pci_bus_num(dev->bus) << 8) | dev->devfn,
+};
+
+if (!event_notifier_test_and_clear(>resume_notifier)) {
+return;
+}
+
+/*
+ * we should read the error details from the real hardware
+ * configuration spaces, here we only need to do is signaling
+ * to guest an uncorrectable error has occurred.
+ */
+if (dev->exp.aer_cap) {
+uint8_t *aer_cap = dev->config + dev->exp.aer_cap;
+uint32_t uncor_status;
+bool isfatal;
+
+uncor_status = vfio_pci_read_config(dev,
+   dev->exp.aer_cap + PCI_ERR_UNCOR_STATUS, 4);
+
+/*
+ * if the error is not emitted by this device, we can
+ * just ignore it.
+ */
+if (!(uncor_status & ~0UL)) {
+return;
+}
+
+isfatal = uncor_status & pci_get_long(aer_cap + PCI_ERR_UNCOR_SEVER);
+
+msg.severity = isfatal ? PCI_ERR_ROOT_CMD_FATAL_EN :
+ PCI_ERR_ROOT_CMD_NONFATAL_EN;
+
+pcie_aer_msg(dev, );
+}
+}
+
+static void vfio_register_aer_resume_notifier(VFIOPCIDevice *vdev)
+{
+int ret;
+int argsz;
+struct vfio_irq_set *irq_set;
+int32_t *pfd;
+
+if (!(vdev->features & VFIO_FEATURE_ENABLE_AER)) {
+return;
+}
+
+if (event_notifier_init(>resume_notifier, 0)) {
+error_report("vfio: Unable to init event notifier for"
+ " resume notification");
+return;
+}
+
+argsz = sizeof(*irq_set) + sizeof(*pfd);
+
+irq_set = g_malloc0(argsz);
+irq_set->argsz = argsz;
+irq_set->flags = VFIO_IRQ_SET_DATA_EVENTFD |
+ VFIO_IRQ_SET_ACTION_TRIGGER;
+irq_set->index = VFIO_PCI_RESUME_IRQ_INDEX;
+irq_set->start = 0;
+irq_set->count = 1;
+pfd = (int32_t *)_set->data;
+
+*pfd = event_notifier_get_fd(>resume_notifier);
+qemu_set_fd_handler(*pfd, vfio_resume_notifier_handler, NULL, vdev);
+
+ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS,