From: Jack Wang <jinpu.w...@profitbricks.com> This series introduces IBNBD/IBTRS kernel modules.
IBNBD (InfiniBand network block device) allows for an RDMA transfer of block IO over InfiniBand network. The driver presents itself as a block device on client side and transmits the block requests in a zero-copy fashion to the server-side via InfiniBand. The server part of the driver converts the incoming buffers back into BIOs and hands them down to the underlying block device. As soon as IO responses come back from the drive, they are being transmitted back to the client. We design and implement this solution based on our need for Cloud Computing, the key features are: - High throughput and low latency due to: 1) Only two rdma messages per IO 2) Simplified client side server memory management 3) Eliminated SCSI sublayer - Simple configuration and handling 1) Server side is completely passive: volumes do not need to be explicitly exported 2) Only IB port GID and device path needed on client side to map a block device 3) A device can be remapped automatically i.e. after storage reboot - Pinning of IO-related processing to the CPU of the producer For usage please refer to Documentation/IBNBD.txt in later patch. My colleague Danil Kpnis presents IBNBD in Vault-2017 about our design/feature/ tradeoff/performance: http://events.linuxfoundation.org/sites/events/files/slides/IBNBD-Vault-2017.pdf The patchset is based on Linux 4.11-rc3. I've done functional tests with our tests framework on AMD64 machines with Mellanox CX-2 and CX-3. TODOs: - move some helpers to core - use new cq api, drain_cq etc - support poll callback in MQ - big endian machine support - better files layout We've learned a lot from other opensource project, namely SRP/SCST/LIO, etc, thanks all the contributors. We hope our IBNBD bring more value to the opensource world. A git tree is also avaiable at: https://github.com/xjtuwjp/linux-2.6/commits/ibnbdv0 As usual, comments and reviews are welcome. Jack Wang (28): ibtrs: add header shared between ibtrs_client and ibtrs_server ibtrs: add header for log MICROs shared between ibtrs_client and ibtrs_server ibtrs_lib: add common functions shared by client and server ibtrs_clt: add header file for exported interface ibtrs_clt: main functionality of ibtrs_client ibtrs_clt: add header file shared only in ibtrs_client ibtrs_clt: add files for sysfs interface ibtrs_clt: add Makefile and Kconfig ibtrs_srv: add header file for exported interface ibtrs_srv: add main functionality for ibtrs_server ibtrs_srv: add header shared in ibtrs_server ibtrs_srv: add sysfs interface ibtrs_srv: add Makefile and Kconfig ibnbd: add headers shared by ibnbd_client and ibnbd_server ibnbd: add shared library functions ibnbd_clt: add main functionality of ibnbd_client ibnbd_clt: add header shared in ibnbd_client ibnbd_clt: add sysfs interface ibnbd_clt: add log helpers ibnbd_clt: add Makefile and Kconfig ibnbd_srv: add header shared in ibnbd_server ibnbd_srv: add main functionality ibnbd_srv: add abstraction for submit IO to file or block device ibnbd_srv: add log helpers ibnbd_srv: add sysfs interface ibnbd_srv: add Makefile and Kconfig ibnbd: add doc for how to use ibnbd and sysfs interface MAINTRAINERS: Add maintainer for IBNBD/IBTRS Documentation/IBNBD.txt | 284 ++ MAINTAINERS | 14 + drivers/block/Kconfig | 3 + drivers/block/Makefile | 2 + drivers/block/ibnbd_client/Kconfig | 16 + drivers/block/ibnbd_client/Makefile | 5 + drivers/block/ibnbd_client/ibnbd_clt.c | 2007 ++++++++ drivers/block/ibnbd_client/ibnbd_clt.h | 231 + drivers/block/ibnbd_client/ibnbd_clt_log.h | 79 + drivers/block/ibnbd_client/ibnbd_clt_sysfs.c | 863 ++++ drivers/block/ibnbd_client/ibnbd_clt_sysfs.h | 64 + drivers/block/ibnbd_inc/ibnbd-proto.h | 273 + drivers/block/ibnbd_inc/ibnbd.h | 55 + drivers/block/ibnbd_inc/log.h | 68 + drivers/block/ibnbd_lib/ibnbd-proto.c | 244 + drivers/block/ibnbd_lib/ibnbd.c | 108 + drivers/block/ibnbd_server/Kconfig | 16 + drivers/block/ibnbd_server/Makefile | 3 + drivers/block/ibnbd_server/ibnbd_dev.c | 436 ++ drivers/block/ibnbd_server/ibnbd_dev.h | 149 + drivers/block/ibnbd_server/ibnbd_srv.c | 1074 ++++ drivers/block/ibnbd_server/ibnbd_srv.h | 115 + drivers/block/ibnbd_server/ibnbd_srv_log.h | 69 + drivers/block/ibnbd_server/ibnbd_srv_sysfs.c | 317 ++ drivers/block/ibnbd_server/ibnbd_srv_sysfs.h | 64 + drivers/infiniband/Kconfig | 3 + drivers/infiniband/ulp/Makefile | 2 + drivers/infiniband/ulp/ibtrs_client/Kconfig | 8 + drivers/infiniband/ulp/ibtrs_client/Makefile | 6 + drivers/infiniband/ulp/ibtrs_client/ibtrs_clt.c | 5329 ++++++++++++++++++++ .../ulp/ibtrs_client/ibtrs_clt_internal.h | 244 + .../infiniband/ulp/ibtrs_client/ibtrs_clt_sysfs.c | 412 ++ .../infiniband/ulp/ibtrs_client/ibtrs_clt_sysfs.h | 62 + drivers/infiniband/ulp/ibtrs_lib/common.c | 104 + drivers/infiniband/ulp/ibtrs_lib/heartbeat.c | 112 + drivers/infiniband/ulp/ibtrs_lib/ibtrs-proto.c | 248 + drivers/infiniband/ulp/ibtrs_lib/ibtrs.c | 412 ++ drivers/infiniband/ulp/ibtrs_lib/iu.c | 113 + drivers/infiniband/ulp/ibtrs_server/Kconfig | 8 + drivers/infiniband/ulp/ibtrs_server/Makefile | 6 + drivers/infiniband/ulp/ibtrs_server/ibtrs_srv.c | 3744 ++++++++++++++ .../ulp/ibtrs_server/ibtrs_srv_internal.h | 201 + .../infiniband/ulp/ibtrs_server/ibtrs_srv_sysfs.c | 301 ++ .../infiniband/ulp/ibtrs_server/ibtrs_srv_sysfs.h | 59 + include/rdma/ibtrs.h | 514 ++ include/rdma/ibtrs_clt.h | 316 ++ include/rdma/ibtrs_log.h | 88 + include/rdma/ibtrs_srv.h | 206 + 48 files changed, 19057 insertions(+) create mode 100644 Documentation/IBNBD.txt create mode 100644 drivers/block/ibnbd_client/Kconfig create mode 100644 drivers/block/ibnbd_client/Makefile create mode 100644 drivers/block/ibnbd_client/ibnbd_clt.c create mode 100644 drivers/block/ibnbd_client/ibnbd_clt.h create mode 100644 drivers/block/ibnbd_client/ibnbd_clt_log.h create mode 100644 drivers/block/ibnbd_client/ibnbd_clt_sysfs.c create mode 100644 drivers/block/ibnbd_client/ibnbd_clt_sysfs.h create mode 100644 drivers/block/ibnbd_inc/ibnbd-proto.h create mode 100644 drivers/block/ibnbd_inc/ibnbd.h create mode 100644 drivers/block/ibnbd_inc/log.h create mode 100644 drivers/block/ibnbd_lib/ibnbd-proto.c create mode 100644 drivers/block/ibnbd_lib/ibnbd.c create mode 100644 drivers/block/ibnbd_server/Kconfig create mode 100644 drivers/block/ibnbd_server/Makefile create mode 100644 drivers/block/ibnbd_server/ibnbd_dev.c create mode 100644 drivers/block/ibnbd_server/ibnbd_dev.h create mode 100644 drivers/block/ibnbd_server/ibnbd_srv.c create mode 100644 drivers/block/ibnbd_server/ibnbd_srv.h create mode 100644 drivers/block/ibnbd_server/ibnbd_srv_log.h create mode 100644 drivers/block/ibnbd_server/ibnbd_srv_sysfs.c create mode 100644 drivers/block/ibnbd_server/ibnbd_srv_sysfs.h create mode 100644 drivers/infiniband/ulp/ibtrs_client/Kconfig create mode 100644 drivers/infiniband/ulp/ibtrs_client/Makefile create mode 100644 drivers/infiniband/ulp/ibtrs_client/ibtrs_clt.c create mode 100644 drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_internal.h create mode 100644 drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_sysfs.c create mode 100644 drivers/infiniband/ulp/ibtrs_client/ibtrs_clt_sysfs.h create mode 100644 drivers/infiniband/ulp/ibtrs_lib/common.c create mode 100644 drivers/infiniband/ulp/ibtrs_lib/heartbeat.c create mode 100644 drivers/infiniband/ulp/ibtrs_lib/ibtrs-proto.c create mode 100644 drivers/infiniband/ulp/ibtrs_lib/ibtrs.c create mode 100644 drivers/infiniband/ulp/ibtrs_lib/iu.c create mode 100644 drivers/infiniband/ulp/ibtrs_server/Kconfig create mode 100644 drivers/infiniband/ulp/ibtrs_server/Makefile create mode 100644 drivers/infiniband/ulp/ibtrs_server/ibtrs_srv.c create mode 100644 drivers/infiniband/ulp/ibtrs_server/ibtrs_srv_internal.h create mode 100644 drivers/infiniband/ulp/ibtrs_server/ibtrs_srv_sysfs.c create mode 100644 drivers/infiniband/ulp/ibtrs_server/ibtrs_srv_sysfs.h create mode 100644 include/rdma/ibtrs.h create mode 100644 include/rdma/ibtrs_clt.h create mode 100644 include/rdma/ibtrs_log.h create mode 100644 include/rdma/ibtrs_srv.h -- 2.7.4
