VFIO_DEVICE_RESET is implemented in the kernel with
pci_try_reset_function(), which uses pci_dev_trylock() and returns
-EAGAIN when the PCI device lock is already held.
That lock is taken by other kernel paths, in particular sysfs
uevent_show(). After Linux restored device_lock() in uevent_show()
(revert of "driver core: Fix uevent_show() vs driver detach race"),
udev or any reader of /sys/.../uevent can race with DPDK VFIO
setup, and the ioctl returns errno 11 (EAGAIN).
Without a retry, pci_rte_vfio_setup_device() fails, EAL skips the
PCI device, and the application cannot use it:
EAL: Unable to reset device! Error: 11 (Resource temporarily unavailable)
EAL: 0000:f4:02.3 setup device failed
EAL: Requested device 0000:f4:02.3 cannot be used
The device lock is held only for the sysfs/uevent critical section,
typically well under 20 ms. Retry the ioctl up to 5 times with a
20 ms delay (100 ms in total). That was enough for the transient
failure we observed; a persistent EAGAIN still fails device setup.
Fixes: f25f8f367644 ("bus/pci: check VFIO reset ioctl error")
Cc: [email protected]
Signed-off-by: Thanushree Sreerama <[email protected]>
---
Cc: [email protected]
Cc: Anatoly Burakov <[email protected]>
Changes since v1:
- Rebased on main
- Drop Change-Id
- Document EAGAIN condition, functional impact, and retry values
- Fail setup if retries are exhausted
- Match pci_vfio.c declaration style
drivers/bus/pci/linux/pci_vfio.c | 20 ++++++++++++++++----
1 file changed, 16 insertions(+), 4 deletions(-)
diff --git a/drivers/bus/pci/linux/pci_vfio.c b/drivers/bus/pci/linux/pci_vfio.c
index bc5c5c249..3ec5af8ed 100644
--- a/drivers/bus/pci/linux/pci_vfio.c
+++ b/drivers/bus/pci/linux/pci_vfio.c
@@ -420,6 +420,10 @@ pci_vfio_is_ioport_bar(const struct rte_pci_device *dev,
int vfio_dev_fd,
static int
pci_rte_vfio_setup_device(struct rte_pci_device *dev, int vfio_dev_fd)
{
+ const int max_retries = 5;
+ const int retry_delay_ms = 20;
+ int i, ret;
+
if (pci_vfio_setup_interrupts(dev, vfio_dev_fd) != 0) {
PCI_LOG(ERR, "Error setting up interrupts!");
return -1;
@@ -439,12 +443,20 @@ pci_rte_vfio_setup_device(struct rte_pci_device *dev, int
vfio_dev_fd)
* Reset the device. If the device is not capable of resetting,
* then it updates errno as EINVAL.
*/
- if (ioctl(vfio_dev_fd, VFIO_DEVICE_RESET) && errno != EINVAL) {
- PCI_LOG(ERR, "Unable to reset device! Error: %d (%s)", errno,
strerror(errno));
- return -1;
+ for (i = 0; i < max_retries; i++) {
+ errno = 0;
+ ret = ioctl(vfio_dev_fd, VFIO_DEVICE_RESET);
+ if (ret == 0 || errno == EINVAL)
+ return 0;
+ if (errno != EAGAIN || i + 1 == max_retries)
+ break;
+ PCI_LOG(DEBUG, "Device reset EAGAIN, retry %d/%d after %d ms",
+ i + 1, max_retries, retry_delay_ms);
+ usleep(retry_delay_ms * 1000);
}
- return 0;
+ PCI_LOG(ERR, "Unable to reset device! Error: %d (%s)", errno,
strerror(errno));
+ return -1;
}
static int
--
2.45.0
