[PATCH char-misc-next v4 09/13] misc: mic: SCIF connections APIs i.e. accept and connect
From: Nikhil Rao
SCIF connection APIs which establish a SCIF connection between
a pair of SCIF endpoints. A SCIF connection consists of a
dedicated queue-pair between the endpoints. Client messages are
sent over the queue-pair whereas the signaling associated with the
message is multiplexed over the node queue-pair. Similarly other
control messages such as exposing registered memory are also sent
over the node queue-pair. The SCIF endpoints must be in connected
state to exchange messages, register memory, map remote memory and
trigger DMA transfers. SCIF connections can be set up
asynchronously or synchronously.
Thanks to Johnnie S Peters for authoring parts of this patch during
early bring up of the SCIF driver.
Reviewed-by: Ashutosh Dixit
Signed-off-by: Sudeep Dutt
Signed-off-by: Nikhil Rao
---
drivers/misc/mic/scif/scif_epd.h| 8 +
drivers/misc/mic/scif/scif_api.c| 457
drivers/misc/mic/scif/scif_epd.c| 229 ++
drivers/misc/mic/scif/scif_fd.c | 106 +
drivers/misc/mic/scif/scif_main.c | 1 +
drivers/misc/mic/scif/scif_nm.c | 36 +++
drivers/misc/mic/scif/scif_nodeqp.c | 16 +-
7 files changed, 852 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/mic/scif/scif_epd.h b/drivers/misc/mic/scif/scif_epd.h
index b973b99..65c628e 100644
--- a/drivers/misc/mic/scif/scif_epd.h
+++ b/drivers/misc/mic/scif/scif_epd.h
@@ -144,5 +144,13 @@ int scif_rsrv_port(u16 port);
void scif_get_port(u16 port);
int scif_get_new_port(void);
void scif_put_port(u16 port);
+void scif_cnctreq(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_nack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctrej(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnct(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+int __scif_connect(scif_epd_t epd, struct scif_port_id *dst, bool non_block);
int __scif_flush(scif_epd_t epd);
#endif /* SCIF_EPD_H */
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index cbdc27b..da64cee 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -415,3 +415,460 @@ int scif_listen(scif_epd_t epd, int backlog)
return 0;
}
EXPORT_SYMBOL_GPL(scif_listen);
+
+/*
+
+ * SCIF connection flow:
+ *
+ * 1) A SCIF listening endpoint can call scif_accept(..) to wait for SCIF
+ * connections via a SCIF_CNCT_REQ message
+ * 2) A SCIF endpoint can initiate a SCIF connection by calling
+ * scif_connect(..) which calls scif_setup_qp_connect(..) which
+ * allocates the local qp for the endpoint ring buffer and then sends
+ * a SCIF_CNCT_REQ to the remote node and waits for a SCIF_CNCT_GNT or
+ * a SCIF_CNCT_REJ message
+ * 3) The peer node handles a SCIF_CNCT_REQ via scif_cnctreq_resp(..) which
+ * wakes up any threads blocked in step 1 or sends a SCIF_CNCT_REJ
+ * message otherwise
+ * 4) A thread blocked waiting for incoming connections allocates its local
+ * endpoint QP and ring buffer following which it sends a SCIF_CNCT_GNT
+ * and waits for a SCIF_CNCT_GNT(N)ACK. If the allocation fails then
+ * the node sends a SCIF_CNCT_REJ message
+ * 5) Upon receipt of a SCIF_CNCT_GNT or a SCIF_CNCT_REJ message the
+ * connecting endpoint is woken up as part of handling
+ * scif_cnctgnt_resp(..) following which it maps the remote endpoints'
+ * QP, updates its outbound QP and sends a SCIF_CNCT_GNTACK message on
+ * success or a SCIF_CNCT_GNTNACK message on failure and completes
+ * the scif_connect(..) API
+ * 6) Upon receipt of a SCIF_CNCT_GNT(N)ACK the accepting endpoint blocked
+ * in step 4 is woken up and completes the scif_accept(..) API
+ * 7) The SCIF connection is now established between the two SCIF endpoints.
+ */
+static int scif_conn_func(struct scif_endpt *ep)
+{
+ int err = 0;
+ struct scifmsg msg;
+ struct device *spdev;
+
+ /* Initiate the first part of the endpoint QP setup */
+ err = scif_setup_qp_connect(ep->qp_info.qp, >qp_info.qp_offset,
+ SCIF_ENDPT_QP_SIZE, ep->remote_dev);
+ if (err) {
+ dev_err(>remote_dev->sdev->dev,
+ "%s err %d qp_offset 0x%llx\n",
+ __func__, err, ep->qp_info.qp_offset);
+ ep->state = SCIFEP_BOUND;
+ goto connect_error_simple;
+ }
+
+ spdev = scif_get_peer_dev(ep->remote_dev);
+ if (IS_ERR(spdev)) {
+ err = PTR_ERR(spdev);
+ goto cleanup_qp;
+ }
+ /* Format connect message and send it */
+ msg.src = ep->port;
+
[PATCH char-misc-next v4 09/13] misc: mic: SCIF connections APIs i.e. accept and connect
From: Nikhil Rao nikhil@intel.com
SCIF connection APIs which establish a SCIF connection between
a pair of SCIF endpoints. A SCIF connection consists of a
dedicated queue-pair between the endpoints. Client messages are
sent over the queue-pair whereas the signaling associated with the
message is multiplexed over the node queue-pair. Similarly other
control messages such as exposing registered memory are also sent
over the node queue-pair. The SCIF endpoints must be in connected
state to exchange messages, register memory, map remote memory and
trigger DMA transfers. SCIF connections can be set up
asynchronously or synchronously.
Thanks to Johnnie S Peters for authoring parts of this patch during
early bring up of the SCIF driver.
Reviewed-by: Ashutosh Dixit ashutosh.di...@intel.com
Signed-off-by: Sudeep Dutt sudeep.d...@intel.com
Signed-off-by: Nikhil Rao nikhil@intel.com
---
drivers/misc/mic/scif/scif_epd.h| 8 +
drivers/misc/mic/scif/scif_api.c| 457
drivers/misc/mic/scif/scif_epd.c| 229 ++
drivers/misc/mic/scif/scif_fd.c | 106 +
drivers/misc/mic/scif/scif_main.c | 1 +
drivers/misc/mic/scif/scif_nm.c | 36 +++
drivers/misc/mic/scif/scif_nodeqp.c | 16 +-
7 files changed, 852 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/mic/scif/scif_epd.h b/drivers/misc/mic/scif/scif_epd.h
index b973b99..65c628e 100644
--- a/drivers/misc/mic/scif/scif_epd.h
+++ b/drivers/misc/mic/scif/scif_epd.h
@@ -144,5 +144,13 @@ int scif_rsrv_port(u16 port);
void scif_get_port(u16 port);
int scif_get_new_port(void);
void scif_put_port(u16 port);
+void scif_cnctreq(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_nack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctrej(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnct(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+int __scif_connect(scif_epd_t epd, struct scif_port_id *dst, bool non_block);
int __scif_flush(scif_epd_t epd);
#endif /* SCIF_EPD_H */
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index cbdc27b..da64cee 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -415,3 +415,460 @@ int scif_listen(scif_epd_t epd, int backlog)
return 0;
}
EXPORT_SYMBOL_GPL(scif_listen);
+
+/*
+
+ * SCIF connection flow:
+ *
+ * 1) A SCIF listening endpoint can call scif_accept(..) to wait for SCIF
+ * connections via a SCIF_CNCT_REQ message
+ * 2) A SCIF endpoint can initiate a SCIF connection by calling
+ * scif_connect(..) which calls scif_setup_qp_connect(..) which
+ * allocates the local qp for the endpoint ring buffer and then sends
+ * a SCIF_CNCT_REQ to the remote node and waits for a SCIF_CNCT_GNT or
+ * a SCIF_CNCT_REJ message
+ * 3) The peer node handles a SCIF_CNCT_REQ via scif_cnctreq_resp(..) which
+ * wakes up any threads blocked in step 1 or sends a SCIF_CNCT_REJ
+ * message otherwise
+ * 4) A thread blocked waiting for incoming connections allocates its local
+ * endpoint QP and ring buffer following which it sends a SCIF_CNCT_GNT
+ * and waits for a SCIF_CNCT_GNT(N)ACK. If the allocation fails then
+ * the node sends a SCIF_CNCT_REJ message
+ * 5) Upon receipt of a SCIF_CNCT_GNT or a SCIF_CNCT_REJ message the
+ * connecting endpoint is woken up as part of handling
+ * scif_cnctgnt_resp(..) following which it maps the remote endpoints'
+ * QP, updates its outbound QP and sends a SCIF_CNCT_GNTACK message on
+ * success or a SCIF_CNCT_GNTNACK message on failure and completes
+ * the scif_connect(..) API
+ * 6) Upon receipt of a SCIF_CNCT_GNT(N)ACK the accepting endpoint blocked
+ * in step 4 is woken up and completes the scif_accept(..) API
+ * 7) The SCIF connection is now established between the two SCIF endpoints.
+ */
+static int scif_conn_func(struct scif_endpt *ep)
+{
+ int err = 0;
+ struct scifmsg msg;
+ struct device *spdev;
+
+ /* Initiate the first part of the endpoint QP setup */
+ err = scif_setup_qp_connect(ep-qp_info.qp, ep-qp_info.qp_offset,
+ SCIF_ENDPT_QP_SIZE, ep-remote_dev);
+ if (err) {
+ dev_err(ep-remote_dev-sdev-dev,
+ %s err %d qp_offset 0x%llx\n,
+ __func__, err, ep-qp_info.qp_offset);
+ ep-state = SCIFEP_BOUND;
+ goto connect_error_simple;
+ }
+
+ spdev = scif_get_peer_dev(ep-remote_dev);
+ if (IS_ERR(spdev)) {
+ err = PTR_ERR(spdev);
+ goto cleanup_qp;
+ }
+ /*
[PATCH char-misc-next v4 09/13] misc: mic: SCIF connections APIs i.e. accept and connect
From: Nikhil Rao
SCIF connection APIs which establish a SCIF connection between
a pair of SCIF endpoints. A SCIF connection consists of a
dedicated queue-pair between the endpoints. Client messages are
sent over the queue-pair whereas the signaling associated with the
message is multiplexed over the node queue-pair. Similarly other
control messages such as exposing registered memory are also sent
over the node queue-pair. The SCIF endpoints must be in connected
state to exchange messages, register memory, map remote memory and
trigger DMA transfers. SCIF connections can be set up
asynchronously or synchronously.
Thanks to Johnnie S Peters for authoring parts of this patch during
early bring up of the SCIF driver.
Reviewed-by: Ashutosh Dixit
Signed-off-by: Sudeep Dutt
Signed-off-by: Nikhil Rao
---
drivers/misc/mic/scif/scif_epd.h| 8 +
drivers/misc/mic/scif/scif_api.c| 457
drivers/misc/mic/scif/scif_epd.c| 229 ++
drivers/misc/mic/scif/scif_fd.c | 106 +
drivers/misc/mic/scif/scif_main.c | 1 +
drivers/misc/mic/scif/scif_nm.c | 36 +++
drivers/misc/mic/scif/scif_nodeqp.c | 16 +-
7 files changed, 852 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/mic/scif/scif_epd.h b/drivers/misc/mic/scif/scif_epd.h
index b973b99..65c628e 100644
--- a/drivers/misc/mic/scif/scif_epd.h
+++ b/drivers/misc/mic/scif/scif_epd.h
@@ -144,5 +144,13 @@ int scif_rsrv_port(u16 port);
void scif_get_port(u16 port);
int scif_get_new_port(void);
void scif_put_port(u16 port);
+void scif_cnctreq(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_nack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctrej(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnct(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+int __scif_connect(scif_epd_t epd, struct scif_port_id *dst, bool non_block);
int __scif_flush(scif_epd_t epd);
#endif /* SCIF_EPD_H */
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index cbdc27b..da64cee 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -415,3 +415,460 @@ int scif_listen(scif_epd_t epd, int backlog)
return 0;
}
EXPORT_SYMBOL_GPL(scif_listen);
+
+/*
+
+ * SCIF connection flow:
+ *
+ * 1) A SCIF listening endpoint can call scif_accept(..) to wait for SCIF
+ * connections via a SCIF_CNCT_REQ message
+ * 2) A SCIF endpoint can initiate a SCIF connection by calling
+ * scif_connect(..) which calls scif_setup_qp_connect(..) which
+ * allocates the local qp for the endpoint ring buffer and then sends
+ * a SCIF_CNCT_REQ to the remote node and waits for a SCIF_CNCT_GNT or
+ * a SCIF_CNCT_REJ message
+ * 3) The peer node handles a SCIF_CNCT_REQ via scif_cnctreq_resp(..) which
+ * wakes up any threads blocked in step 1 or sends a SCIF_CNCT_REJ
+ * message otherwise
+ * 4) A thread blocked waiting for incoming connections allocates its local
+ * endpoint QP and ring buffer following which it sends a SCIF_CNCT_GNT
+ * and waits for a SCIF_CNCT_GNT(N)ACK. If the allocation fails then
+ * the node sends a SCIF_CNCT_REJ message
+ * 5) Upon receipt of a SCIF_CNCT_GNT or a SCIF_CNCT_REJ message the
+ * connecting endpoint is woken up as part of handling
+ * scif_cnctgnt_resp(..) following which it maps the remote endpoints'
+ * QP, updates its outbound QP and sends a SCIF_CNCT_GNTACK message on
+ * success or a SCIF_CNCT_GNTNACK message on failure and completes
+ * the scif_connect(..) API
+ * 6) Upon receipt of a SCIF_CNCT_GNT(N)ACK the accepting endpoint blocked
+ * in step 4 is woken up and completes the scif_accept(..) API
+ * 7) The SCIF connection is now established between the two SCIF endpoints.
+ */
+static int scif_conn_func(struct scif_endpt *ep)
+{
+ int err = 0;
+ struct scifmsg msg;
+ struct device *spdev;
+
+ /* Initiate the first part of the endpoint QP setup */
+ err = scif_setup_qp_connect(ep->qp_info.qp, >qp_info.qp_offset,
+ SCIF_ENDPT_QP_SIZE, ep->remote_dev);
+ if (err) {
+ dev_err(>remote_dev->sdev->dev,
+ "%s err %d qp_offset 0x%llx\n",
+ __func__, err, ep->qp_info.qp_offset);
+ ep->state = SCIFEP_BOUND;
+ goto connect_error_simple;
+ }
+
+ spdev = scif_get_peer_dev(ep->remote_dev);
+ if (IS_ERR(spdev)) {
+ err = PTR_ERR(spdev);
+ goto cleanup_qp;
+ }
+ /* Format connect message and send it */
+ msg.src = ep->port;
+
[PATCH char-misc-next v4 09/13] misc: mic: SCIF connections APIs i.e. accept and connect
From: Nikhil Rao nikhil@intel.com
SCIF connection APIs which establish a SCIF connection between
a pair of SCIF endpoints. A SCIF connection consists of a
dedicated queue-pair between the endpoints. Client messages are
sent over the queue-pair whereas the signaling associated with the
message is multiplexed over the node queue-pair. Similarly other
control messages such as exposing registered memory are also sent
over the node queue-pair. The SCIF endpoints must be in connected
state to exchange messages, register memory, map remote memory and
trigger DMA transfers. SCIF connections can be set up
asynchronously or synchronously.
Thanks to Johnnie S Peters for authoring parts of this patch during
early bring up of the SCIF driver.
Reviewed-by: Ashutosh Dixit ashutosh.di...@intel.com
Signed-off-by: Sudeep Dutt sudeep.d...@intel.com
Signed-off-by: Nikhil Rao nikhil@intel.com
---
drivers/misc/mic/scif/scif_epd.h| 8 +
drivers/misc/mic/scif/scif_api.c| 457
drivers/misc/mic/scif/scif_epd.c| 229 ++
drivers/misc/mic/scif/scif_fd.c | 106 +
drivers/misc/mic/scif/scif_main.c | 1 +
drivers/misc/mic/scif/scif_nm.c | 36 +++
drivers/misc/mic/scif/scif_nodeqp.c | 16 +-
7 files changed, 852 insertions(+), 1 deletion(-)
diff --git a/drivers/misc/mic/scif/scif_epd.h b/drivers/misc/mic/scif/scif_epd.h
index b973b99..65c628e 100644
--- a/drivers/misc/mic/scif/scif_epd.h
+++ b/drivers/misc/mic/scif/scif_epd.h
@@ -144,5 +144,13 @@ int scif_rsrv_port(u16 port);
void scif_get_port(u16 port);
int scif_get_new_port(void);
void scif_put_port(u16 port);
+void scif_cnctreq(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctgnt_nack(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_cnctrej(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnct(struct scif_dev *scifdev, struct scifmsg *msg);
+void scif_discnt_ack(struct scif_dev *scifdev, struct scifmsg *msg);
+int __scif_connect(scif_epd_t epd, struct scif_port_id *dst, bool non_block);
int __scif_flush(scif_epd_t epd);
#endif /* SCIF_EPD_H */
diff --git a/drivers/misc/mic/scif/scif_api.c b/drivers/misc/mic/scif/scif_api.c
index cbdc27b..da64cee 100644
--- a/drivers/misc/mic/scif/scif_api.c
+++ b/drivers/misc/mic/scif/scif_api.c
@@ -415,3 +415,460 @@ int scif_listen(scif_epd_t epd, int backlog)
return 0;
}
EXPORT_SYMBOL_GPL(scif_listen);
+
+/*
+
+ * SCIF connection flow:
+ *
+ * 1) A SCIF listening endpoint can call scif_accept(..) to wait for SCIF
+ * connections via a SCIF_CNCT_REQ message
+ * 2) A SCIF endpoint can initiate a SCIF connection by calling
+ * scif_connect(..) which calls scif_setup_qp_connect(..) which
+ * allocates the local qp for the endpoint ring buffer and then sends
+ * a SCIF_CNCT_REQ to the remote node and waits for a SCIF_CNCT_GNT or
+ * a SCIF_CNCT_REJ message
+ * 3) The peer node handles a SCIF_CNCT_REQ via scif_cnctreq_resp(..) which
+ * wakes up any threads blocked in step 1 or sends a SCIF_CNCT_REJ
+ * message otherwise
+ * 4) A thread blocked waiting for incoming connections allocates its local
+ * endpoint QP and ring buffer following which it sends a SCIF_CNCT_GNT
+ * and waits for a SCIF_CNCT_GNT(N)ACK. If the allocation fails then
+ * the node sends a SCIF_CNCT_REJ message
+ * 5) Upon receipt of a SCIF_CNCT_GNT or a SCIF_CNCT_REJ message the
+ * connecting endpoint is woken up as part of handling
+ * scif_cnctgnt_resp(..) following which it maps the remote endpoints'
+ * QP, updates its outbound QP and sends a SCIF_CNCT_GNTACK message on
+ * success or a SCIF_CNCT_GNTNACK message on failure and completes
+ * the scif_connect(..) API
+ * 6) Upon receipt of a SCIF_CNCT_GNT(N)ACK the accepting endpoint blocked
+ * in step 4 is woken up and completes the scif_accept(..) API
+ * 7) The SCIF connection is now established between the two SCIF endpoints.
+ */
+static int scif_conn_func(struct scif_endpt *ep)
+{
+ int err = 0;
+ struct scifmsg msg;
+ struct device *spdev;
+
+ /* Initiate the first part of the endpoint QP setup */
+ err = scif_setup_qp_connect(ep-qp_info.qp, ep-qp_info.qp_offset,
+ SCIF_ENDPT_QP_SIZE, ep-remote_dev);
+ if (err) {
+ dev_err(ep-remote_dev-sdev-dev,
+ %s err %d qp_offset 0x%llx\n,
+ __func__, err, ep-qp_info.qp_offset);
+ ep-state = SCIFEP_BOUND;
+ goto connect_error_simple;
+ }
+
+ spdev = scif_get_peer_dev(ep-remote_dev);
+ if (IS_ERR(spdev)) {
+ err = PTR_ERR(spdev);
+ goto cleanup_qp;
+ }
+ /*
