Doug and list Hi,
This patchset introduces Soft RoCE driver.
Some background on the driver: The original Soft-RoCE driver was implemented by
Bob Pearson from SFW. Bob started the submission process [3], but his work was
abandoned after v2.
Mellanox decided to pick it up and continue the submission. As part of the
process we detected some problems with the original implementation. Mainly, we
wanted to RoCEv2, also, there are too many locks and
context switches in the data path. Most of them are already removed.
We've located the driver in the staging subtree. This follows a requirement
to implement an IB transport library - Soft RoCE is in the same boat like the
hfi1
driver. We need to define and implement a lib to prevent those code
duplications.
We did address the feedback provided on the original submission.
Another issue is, that this code is based on RoCEv2 patchesets.
So why not wait and submit it when the RoCEv2 IB core bits are upstream?
The main reason we want to submit it now, and not to wait is: "Submit early".
We understand that 4 years after v2 is not exactly early. But we started few
months ago to work on it, and did some heavy modifications to the code and the
design, and we would like this work to be done under the eye of the community
and not in house (although this work was done @ github [4]).
Soft-RoCE is sitting on top of Matan's 3 patchsets for gid cache and RoCEv2.
The first [1] is in, the second [2] and third [5] were posted already.
RXE user space (librxe) is located at github [6] with instructions how to use
it [7]
Some notes on the architecture and design:
ib_rxe, implements the RDMA transport and registers with the RDMA core as a
kernel verbs provider. It also implements the packet IO layer. ib_rxe attaches
to the Linux netdev stack as a udp encapsulating protocol and can send and
receive packets over any Ethernet device. It uses the RoCEv2 protocol to handle
RDMA transport.
The modules are configured by entries in /sys. There is a configuration script
(rxe_cfg) that simplifies the use of this interface. rxe_cfg is part of the
rxe user space code, librxe.
The use of rxe verbs in user space requires the inclusion of librxe as a device
specific plug-in to libibverbs. librxe is packaged separately [6].
Copies of the user space library and tools for 'upstream' and a clone of Doug's
tree with
these patches applied are available at github [4]
Architecture:
~~
+---+
| Application |
+---+
+---+
| libibverbs|
User +---+
++ ++
| librxe | | HW RoCE lib|
++ ++
~~
+--+ ++
| Sockets | | RDMA ULP |
+--+ ++
+--+ +-+
| TCP/IP | | ib_core |
+--+ +-+
++ ++
Kernel | ib_rxe | | HW RoCE driver |
++ ++
++
| NIC driver |
++
~~
The driver components and a non asci chart of the module could be found at a
pdf [8] presented by Bob before the original submission.
The design is very similar, one thing that was changed, is the arbiter task
that was removed. This reduced the number of context switches and locks during
the data path.
Currently IPv4 based sessions aren't supported, this will be addressed for V1.
A TODO file is placed under the driver folder.
The patchset is applied and tested over Dougs to-be-rebased/for-4.3 branch
153b730 ("Merge branch 'hfi1-v4' into to-be-rebased/for-4.3").
Thanks,
Kamal, Liran and Amir
[1] - http://www.spinics.net/lists/netdev/msg337683.html
[2] - http://www.spinics.net/lists/linux-rdma/msg28031.html
[3] - http://www.spinics.net/lists/linux-rdma/msg08936.html
[4] - https://github.com/SoftRoCE
[5] - http://www.spinics.net/lists/linux-rdma/msg28120.html
[6] - https://github.com/SoftRoCE/librxe-dev
[7] - https://github.com/SoftRoCE/rxe-dev/wiki/rxe-dev:-Home
[8] -
