On 06/30/2015 06:26 AM, Yishai Hadas wrote: > Currently, if there is any user space application using an IB device, > it is impossible to unload the HW device driver for this device. > > Similarly, if the device is hot-unplugged or reset, the device driver > hardware removal flow blocks until all user contexts are destroyed. > > This patchset removes the above limitations from both uverbs and ucma. > > The IB-core and uverbs layers are still required to remain loaded as > long as there are user applications using the verbs API. However, the > hardware device drivers are not blocked any more by the user space > activity. > > To support this, the hardware device needs to expose a new kernel API > named 'disassociate_ucontext'. The device driver is given a ucontext > to detach from, and it should block this user context from any future > hardware access. In the IB-core level, we use this interface to > deactivate all ucontext that address a specific device when handling a > remove_one callback for it. > > In the RDMA-CM layer, a similar change is needed in the ucma module, > to prevent blocking of the remove_one operation. This allows for > hot-removal of devices with RDMA-CM users in the user space. > > The first two patches are preparation for this series. > The first patch fixes a reference counting issue pointed by Jason Gunthorpe. > The second patch is a preparation step for deploying RCU for the device > removal flow. > > The third patch introduces the new API between the HW device driver and > the IB core. For devices which implement the functionality, IB core > will use it in remove_one, disassociating any active ucontext from the > hardware device. Other drivers that didn't implement it will behave as > today, remove_one will block until all ucontexts referring the device > are destroyed before returning. > > The fourth patch provides implementation of this API for the mlx4 > driver. > > The last patch extends ucma to avoid blocking remove_one operation in > the cma module. When such device removal event is received, ucma is > turning all user contexts to zombie contexts. This is done by > releasing all underlying resources and preventing any further user > operations on the context. > > Changes from V5: > Addressed Jason's comments for below patches: > patch #1: Improve kref usage. > patch #3: Use 2 different krefs for complete and memory, improve some > comments. > > Changes from V4: > patch #1,#3 - addressed Jason's comments. > patch #2, #4 - rebased upon last stuff. > > Changes from V3: > Add 2 patches as a preparation for this series, details above. > patch #3: Change the locking schema based on Jason's comments. > > Changes from V2: > patch #1: Rebase over ODP patches. > > Changes from V1: > patch #1: Use uverbs flags instead of disassociate support, drop > fatal_event_raised flag. > patch #3: Add support in ucma for handling device removal. > > Changes from V0: > patch #1: ib_uverbs_close, reduced mutex scope to enable tasks run in > parallel. > Yishai Hadas (5): > IB/uverbs: Fix reference counting usage of event files > IB/uverbs: Explicitly pass ib_dev to uverbs commands > IB/uverbs: Enable device removal when there are active user space > applications > IB/mlx4_ib: Disassociate support > IB/ucma: HW Device hot-removal support > > drivers/infiniband/core/ucma.c | 130 +++++++++- > drivers/infiniband/core/uverbs.h | 16 +- > drivers/infiniband/core/uverbs_cmd.c | 114 ++++++---- > drivers/infiniband/core/uverbs_main.c | 438 > +++++++++++++++++++++++++++------ > drivers/infiniband/hw/mlx4/main.c | 139 ++++++++++- > drivers/infiniband/hw/mlx4/mlx4_ib.h | 13 + > include/rdma/ib_verbs.h | 1 + > 7 files changed, 714 insertions(+), 137 deletions(-) >
I've pulled this series in for 4.3. There were some additional items in
some of Jason's comments that ought to be looked into, but I think this
patch set has reached the point where it's no worse than existing in
terms of locking, there were just some existing issues that should be
addressed too.
--
Doug Ledford <dledf...@redhat.com>
GPG KeyID: 0E572FDD
signature.asc
Description: OpenPGP digital signature
