Re: [PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
On Wed, 2015-04-29 at 13:27 +0200, Greg Kroah-Hartman wrote: > On Tue, Apr 28, 2015 at 01:45:12PM -0700, Sudeep Dutt wrote: > > On Mon, 2015-03-30 at 18:36 -0700, Sudeep Dutt wrote: > > > ChangeLog: > > > = > > > > > > v3 => v4: > > > a) Delete unused IOCTL definitions and IOCTL interface fixes as per > > > feedback from Greg Kroah-Hartman > > > b) Couple of tiny bug fixes since v3 > > > > > > v2 => v3 @ https://lkml.org/lkml/2015/1/26/849 > > > a) Formatting fixes for SCIF header file documentation, data type fixes > > >for SCIF IOCTL interface and added SCIF user space documentation in > > >scif_overview.txt as per feedback from Greg Kroah-Hartman > > > > > > v1 => v2 @ https://lkml.org/lkml/2015/1/12/1029 > > > a) Use kernel-doc formatting for SCIF header file documentation, enhance > > >SCIF ring buffer documentation and formatting cleanup in patch 4 as > > >per feedback from Greg Kroah-Hartman > > > b) SCIF bug fixes and cleanups since initial post > > > > > > v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 > > > > > > Description: > > > > > > > > > The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a > > > low > > > level communications API across PCIe currently implemented for MIC. > > > Currently > > > SCIF provides inter-node communication within a single host platform, > > > where a > > > node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details > > > of > > > communicating over the PCIe bus while providing an API that is symmetric > > > across all the nodes in the PCIe network. An important design objective > > > for SCIF > > > is to deliver the maximum possible performance given the communication > > > abilities of the hardware. SCIF has been used to implement an offload > > > compiler > > > runtime and OFED support for MPI implementations for MIC coprocessors. > > > > > > > Hi Greg, > > > > Please take a look at this patch series. > > What patch series? I don't have this in my todo queue at all, what > happened to it? That is strange indeed since it does show up on the mailing lists :( I have resent the patch series a few minutes back. Please let me know if you did not receive it. Thanks, Sudeep Dutt -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
ChangeLog: = v3 => v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 => v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 => v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. SCIF API Components The SCIF API has the following parts: 1. Connection establishment using a client server model 2. Byte stream messaging intended for short messages 3. Node enumeration to determine online nodes 4. Poll semantics for detection of incoming connections and messages 5. Memory registration to pin down pages 6. Remote memory mapping for low latency CPU accesses via mmap 7. Remote DMA (RDMA) for high bandwidth DMA transfers 8. Fence APIs for RDMA synchronization SCIF exposes the notion of a connection which can be used by peer processes on nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A process in a SCIF node initiates a SCIF connection to a peer process on a different node via a SCIF "endpoint". SCIF endpoints support messaging APIs which are similar to connection oriented socket APIs. Connected SCIF endpoints can also register local memory which is followed by data transfer using either DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and kernel mode clients which are functionally equivalent. SCIF Performance for MIC DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus SCIF shows the performance advantages of SCIF for HPC applications and runtimes. Comparison of TCP and SCIF based BW Throughput (GB/sec) 8 + PCIe Bandwidth ** +TCP ## 7 +** SCIF %% | %%% 6 + | %% | %%% 5 + %% |%% 4 + %% | %% 3 + %% |% 2 + %% | %% |% 1 + +## 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- 1 10 100 1000 1 10 Transfer Size (KBytes) SCIF allows memory sharing via mmap(..) between processes on different PCIe nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. For more information on SCIF please refer to the Linux Con 2013 slides titled "Intel® MIC x100 Coprocessor Driver - on the Frontiers of Linux & HPC" at http://events.linuxfoundation.org/sites/events/files/slides/LinuxConMicDriver.pdf The talk can also be viewed at https://www.youtube.com/watch?v=dylbmmQs4W0 This initial patch series introduces the SCIF API and implements the SCIF connection, messaging and node enumeration APIs. SCIF poll, mmap, RDMA and fence APIs will be submitted in a future patch series once the initial base patches are accepted upstream. This patch series is divided into 13 patches as follows: 1) SCIF header file documenting the API along with the IOCTL interface 2) SCIF ring buffer is a single producer, single consumer byte stream ring buffer optimized for avoiding reads across the PCIe bus. The ring buffer is used to implement a receive queue for SCIF driver messaging between nodes and for byte stream messaging between SCIF endpoints. Each SCIF node has a receive queue for every other SCIF node, and each connected endpoint has a receive queue for messages from its peer. This pair of receive queues is referred to as a SCIF queue pair. 3) SCIF
Re: [PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
On Tue, Apr 28, 2015 at 01:45:12PM -0700, Sudeep Dutt wrote: > On Mon, 2015-03-30 at 18:36 -0700, Sudeep Dutt wrote: > > ChangeLog: > > = > > > > v3 => v4: > > a) Delete unused IOCTL definitions and IOCTL interface fixes as per > > feedback from Greg Kroah-Hartman > > b) Couple of tiny bug fixes since v3 > > > > v2 => v3 @ https://lkml.org/lkml/2015/1/26/849 > > a) Formatting fixes for SCIF header file documentation, data type fixes > >for SCIF IOCTL interface and added SCIF user space documentation in > >scif_overview.txt as per feedback from Greg Kroah-Hartman > > > > v1 => v2 @ https://lkml.org/lkml/2015/1/12/1029 > > a) Use kernel-doc formatting for SCIF header file documentation, enhance > >SCIF ring buffer documentation and formatting cleanup in patch 4 as > >per feedback from Greg Kroah-Hartman > > b) SCIF bug fixes and cleanups since initial post > > > > v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 > > > > Description: > > > > > > The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low > > level communications API across PCIe currently implemented for MIC. > > Currently > > SCIF provides inter-node communication within a single host platform, where > > a > > node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of > > communicating over the PCIe bus while providing an API that is symmetric > > across all the nodes in the PCIe network. An important design objective for > > SCIF > > is to deliver the maximum possible performance given the communication > > abilities of the hardware. SCIF has been used to implement an offload > > compiler > > runtime and OFED support for MPI implementations for MIC coprocessors. > > > > Hi Greg, > > Please take a look at this patch series. What patch series? I don't have this in my todo queue at all, what happened to it? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
On Tue, Apr 28, 2015 at 01:45:12PM -0700, Sudeep Dutt wrote: On Mon, 2015-03-30 at 18:36 -0700, Sudeep Dutt wrote: ChangeLog: = v3 = v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 = v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 = v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. Hi Greg, Please take a look at this patch series. What patch series? I don't have this in my todo queue at all, what happened to it? -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
ChangeLog: = v3 = v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 = v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 = v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. SCIF API Components The SCIF API has the following parts: 1. Connection establishment using a client server model 2. Byte stream messaging intended for short messages 3. Node enumeration to determine online nodes 4. Poll semantics for detection of incoming connections and messages 5. Memory registration to pin down pages 6. Remote memory mapping for low latency CPU accesses via mmap 7. Remote DMA (RDMA) for high bandwidth DMA transfers 8. Fence APIs for RDMA synchronization SCIF exposes the notion of a connection which can be used by peer processes on nodes in a SCIF PCIe network to share memory windows and to communicate. A process in a SCIF node initiates a SCIF connection to a peer process on a different node via a SCIF endpoint. SCIF endpoints support messaging APIs which are similar to connection oriented socket APIs. Connected SCIF endpoints can also register local memory which is followed by data transfer using either DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and kernel mode clients which are functionally equivalent. SCIF Performance for MIC DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus SCIF shows the performance advantages of SCIF for HPC applications and runtimes. Comparison of TCP and SCIF based BW Throughput (GB/sec) 8 + PCIe Bandwidth ** +TCP ## 7 +** SCIF %% | %%% 6 + | %% | %%% 5 + %% |%% 4 + %% | %% 3 + %% |% 2 + %% | %% |% 1 + +## 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- 1 10 100 1000 1 10 Transfer Size (KBytes) SCIF allows memory sharing via mmap(..) between processes on different PCIe nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. For more information on SCIF please refer to the Linux Con 2013 slides titled Intel® MIC x100 Coprocessor Driver - on the Frontiers of Linux HPC at http://events.linuxfoundation.org/sites/events/files/slides/LinuxConMicDriver.pdf The talk can also be viewed at https://www.youtube.com/watch?v=dylbmmQs4W0 This initial patch series introduces the SCIF API and implements the SCIF connection, messaging and node enumeration APIs. SCIF poll, mmap, RDMA and fence APIs will be submitted in a future patch series once the initial base patches are accepted upstream. This patch series is divided into 13 patches as follows: 1) SCIF header file documenting the API along with the IOCTL interface 2) SCIF ring buffer is a single producer, single consumer byte stream ring buffer optimized for avoiding reads across the PCIe bus. The ring buffer is used to implement a receive queue for SCIF driver messaging between nodes and for byte stream messaging between SCIF endpoints. Each SCIF node has a receive queue for every other SCIF node, and each connected endpoint has a receive queue for messages from its peer. This pair of receive queues is referred to as a SCIF queue pair. 3) SCIF hardware bus which
Re: [PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
On Wed, 2015-04-29 at 13:27 +0200, Greg Kroah-Hartman wrote: On Tue, Apr 28, 2015 at 01:45:12PM -0700, Sudeep Dutt wrote: On Mon, 2015-03-30 at 18:36 -0700, Sudeep Dutt wrote: ChangeLog: = v3 = v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 = v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 = v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. Hi Greg, Please take a look at this patch series. What patch series? I don't have this in my todo queue at all, what happened to it? That is strange indeed since it does show up on the mailing lists :( I have resent the patch series a few minutes back. Please let me know if you did not receive it. Thanks, Sudeep Dutt -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
On Mon, 2015-03-30 at 18:36 -0700, Sudeep Dutt wrote: > ChangeLog: > = > > v3 => v4: > a) Delete unused IOCTL definitions and IOCTL interface fixes as per > feedback from Greg Kroah-Hartman > b) Couple of tiny bug fixes since v3 > > v2 => v3 @ https://lkml.org/lkml/2015/1/26/849 > a) Formatting fixes for SCIF header file documentation, data type fixes >for SCIF IOCTL interface and added SCIF user space documentation in >scif_overview.txt as per feedback from Greg Kroah-Hartman > > v1 => v2 @ https://lkml.org/lkml/2015/1/12/1029 > a) Use kernel-doc formatting for SCIF header file documentation, enhance >SCIF ring buffer documentation and formatting cleanup in patch 4 as >per feedback from Greg Kroah-Hartman > b) SCIF bug fixes and cleanups since initial post > > v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 > > Description: > > > The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low > level communications API across PCIe currently implemented for MIC. Currently > SCIF provides inter-node communication within a single host platform, where a > node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of > communicating over the PCIe bus while providing an API that is symmetric > across all the nodes in the PCIe network. An important design objective for > SCIF > is to deliver the maximum possible performance given the communication > abilities of the hardware. SCIF has been used to implement an offload compiler > runtime and OFED support for MPI implementations for MIC coprocessors. > Hi Greg, Please take a look at this patch series. Thanks, Sudeep Dutt -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Re: [PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
On Mon, 2015-03-30 at 18:36 -0700, Sudeep Dutt wrote: ChangeLog: = v3 = v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 = v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 = v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. Hi Greg, Please take a look at this patch series. Thanks, Sudeep Dutt -- To unsubscribe from this list: send the line unsubscribe linux-kernel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
[PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
ChangeLog: = v3 => v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 => v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 => v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. SCIF API Components The SCIF API has the following parts: 1. Connection establishment using a client server model 2. Byte stream messaging intended for short messages 3. Node enumeration to determine online nodes 4. Poll semantics for detection of incoming connections and messages 5. Memory registration to pin down pages 6. Remote memory mapping for low latency CPU accesses via mmap 7. Remote DMA (RDMA) for high bandwidth DMA transfers 8. Fence APIs for RDMA synchronization SCIF exposes the notion of a connection which can be used by peer processes on nodes in a SCIF PCIe "network" to share memory "windows" and to communicate. A process in a SCIF node initiates a SCIF connection to a peer process on a different node via a SCIF "endpoint". SCIF endpoints support messaging APIs which are similar to connection oriented socket APIs. Connected SCIF endpoints can also register local memory which is followed by data transfer using either DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and kernel mode clients which are functionally equivalent. SCIF Performance for MIC DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus SCIF shows the performance advantages of SCIF for HPC applications and runtimes. Comparison of TCP and SCIF based BW Throughput (GB/sec) 8 + PCIe Bandwidth ** +TCP ## 7 +** SCIF %% | %%% 6 + | %% | %%% 5 + %% |%% 4 + %% | %% 3 + %% |% 2 + %% | %% |% 1 + +## 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- 1 10 100 1000 1 10 Transfer Size (KBytes) SCIF allows memory sharing via mmap(..) between processes on different PCIe nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. For more information on SCIF please refer to the Linux Con 2013 slides titled "Intel® MIC x100 Coprocessor Driver - on the Frontiers of Linux & HPC" at http://events.linuxfoundation.org/sites/events/files/slides/LinuxConMicDriver.pdf The talk can also be viewed at https://www.youtube.com/watch?v=dylbmmQs4W0 This initial patch series introduces the SCIF API and implements the SCIF connection, messaging and node enumeration APIs. SCIF poll, mmap, RDMA and fence APIs will be submitted in a future patch series once the initial base patches are accepted upstream. This patch series is divided into 13 patches as follows: 1) SCIF header file documenting the API along with the IOCTL interface 2) SCIF ring buffer is a single producer, single consumer byte stream ring buffer optimized for avoiding reads across the PCIe bus. The ring buffer is used to implement a receive queue for SCIF driver messaging between nodes and for byte stream messaging between SCIF endpoints. Each SCIF node has a receive queue for every other SCIF node, and each connected endpoint has a receive queue for messages from its peer. This pair of receive queues is referred to as a SCIF queue pair. 3) SCIF
[PATCH char-misc-next v4 00/13] misc: mic: SCIF driver
ChangeLog: = v3 = v4: a) Delete unused IOCTL definitions and IOCTL interface fixes as per feedback from Greg Kroah-Hartman b) Couple of tiny bug fixes since v3 v2 = v3 @ https://lkml.org/lkml/2015/1/26/849 a) Formatting fixes for SCIF header file documentation, data type fixes for SCIF IOCTL interface and added SCIF user space documentation in scif_overview.txt as per feedback from Greg Kroah-Hartman v1 = v2 @ https://lkml.org/lkml/2015/1/12/1029 a) Use kernel-doc formatting for SCIF header file documentation, enhance SCIF ring buffer documentation and formatting cleanup in patch 4 as per feedback from Greg Kroah-Hartman b) SCIF bug fixes and cleanups since initial post v1: Initial post @ https://lkml.org/lkml/2014/12/10/546 Description: The Symmetric Communication Interface (SCIF (pronounced as skiff)) is a low level communications API across PCIe currently implemented for MIC. Currently SCIF provides inter-node communication within a single host platform, where a node is a MIC Coprocessor or Xeon based host. SCIF abstracts the details of communicating over the PCIe bus while providing an API that is symmetric across all the nodes in the PCIe network. An important design objective for SCIF is to deliver the maximum possible performance given the communication abilities of the hardware. SCIF has been used to implement an offload compiler runtime and OFED support for MPI implementations for MIC coprocessors. SCIF API Components The SCIF API has the following parts: 1. Connection establishment using a client server model 2. Byte stream messaging intended for short messages 3. Node enumeration to determine online nodes 4. Poll semantics for detection of incoming connections and messages 5. Memory registration to pin down pages 6. Remote memory mapping for low latency CPU accesses via mmap 7. Remote DMA (RDMA) for high bandwidth DMA transfers 8. Fence APIs for RDMA synchronization SCIF exposes the notion of a connection which can be used by peer processes on nodes in a SCIF PCIe network to share memory windows and to communicate. A process in a SCIF node initiates a SCIF connection to a peer process on a different node via a SCIF endpoint. SCIF endpoints support messaging APIs which are similar to connection oriented socket APIs. Connected SCIF endpoints can also register local memory which is followed by data transfer using either DMA, CPU copies or remote memory mapping via mmap. SCIF supports both user and kernel mode clients which are functionally equivalent. SCIF Performance for MIC DMA bandwidth comparison between the TCP (over ethernet over PCIe) stack versus SCIF shows the performance advantages of SCIF for HPC applications and runtimes. Comparison of TCP and SCIF based BW Throughput (GB/sec) 8 + PCIe Bandwidth ** +TCP ## 7 +** SCIF %% | %%% 6 + | %% | %%% 5 + %% |%% 4 + %% | %% 3 + %% |% 2 + %% | %% |% 1 + +## 0 +++---+++--+--+-+--+--+-++-+--+-++-+--+-++-+- 1 10 100 1000 1 10 Transfer Size (KBytes) SCIF allows memory sharing via mmap(..) between processes on different PCIe nodes and thus provides bare-metal PCIe latency. The round trip SCIF mmap latency from the host to an x100 MIC for an 8 byte message is 0.44 usecs. For more information on SCIF please refer to the Linux Con 2013 slides titled Intel® MIC x100 Coprocessor Driver - on the Frontiers of Linux HPC at http://events.linuxfoundation.org/sites/events/files/slides/LinuxConMicDriver.pdf The talk can also be viewed at https://www.youtube.com/watch?v=dylbmmQs4W0 This initial patch series introduces the SCIF API and implements the SCIF connection, messaging and node enumeration APIs. SCIF poll, mmap, RDMA and fence APIs will be submitted in a future patch series once the initial base patches are accepted upstream. This patch series is divided into 13 patches as follows: 1) SCIF header file documenting the API along with the IOCTL interface 2) SCIF ring buffer is a single producer, single consumer byte stream ring buffer optimized for avoiding reads across the PCIe bus. The ring buffer is used to implement a receive queue for SCIF driver messaging between nodes and for byte stream messaging between SCIF endpoints. Each SCIF node has a receive queue for every other SCIF node, and each connected endpoint has a receive queue for messages from its peer. This pair of receive queues is referred to as a SCIF queue pair. 3) SCIF hardware bus which
