[PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs
xprtrdma is currently throwing away queued completions during
a reconnect. RPC replies posted just before connection loss, or
successful completions that change the state of an FRMR, can be
missed.
Signed-off-by: Chuck Lever chuck.le...@oracle.com
---
net/sunrpc/xprtrdma/verbs.c | 14 +-
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 0d5187d..7fd457e 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -310,6 +310,13 @@ rpcrdma_recvcq_upcall(struct ib_cq *cq, void *cq_context)
rpcrdma_recvcq_poll(cq, ep);
}
+static void
+rpcrdma_flush_cqs(struct rpcrdma_ep *ep)
+{
+ rpcrdma_recvcq_upcall(ep-rep_attr.recv_cq, ep);
+ rpcrdma_sendcq_upcall(ep-rep_attr.send_cq, ep);
+}
+
#ifdef RPC_DEBUG
static const char * const conn[] = {
address resolved,
@@ -872,9 +879,7 @@ retry:
if (rc rc != -ENOTCONN)
dprintk(RPC: %s: rpcrdma_ep_disconnect
status %i\n, __func__, rc);
-
- rpcrdma_clean_cq(ep-rep_attr.recv_cq);
- rpcrdma_clean_cq(ep-rep_attr.send_cq);
+ rpcrdma_flush_cqs(ep);
xprt = container_of(ia, struct rpcrdma_xprt, rx_ia);
id = rpcrdma_create_id(xprt, ia,
@@ -985,8 +990,7 @@ rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct
rpcrdma_ia *ia)
{
int rc;
- rpcrdma_clean_cq(ep-rep_attr.recv_cq);
- rpcrdma_clean_cq(ep-rep_attr.send_cq);
+ rpcrdma_flush_cqs(ep);
rc = rdma_disconnect(ia-ri_id);
if (!rc) {
/* returns without wait if not connected */
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs
Should all rdma_clean_cq be replaced by flush_cqs? The outstanding CQEs should
be processed in any context.
Shirley
On 07/09/2014 09:57 AM, Chuck Lever wrote:
xprtrdma is currently throwing away queued completions during
a reconnect. RPC replies posted just before connection loss, or
successful completions that change the state of an FRMR, can be
missed.
Signed-off-by: Chuck Lever chuck.le...@oracle.com
---
net/sunrpc/xprtrdma/verbs.c | 14 +-
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 0d5187d..7fd457e 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -310,6 +310,13 @@ rpcrdma_recvcq_upcall(struct ib_cq *cq, void *cq_context)
rpcrdma_recvcq_poll(cq, ep);
}
+static void
+rpcrdma_flush_cqs(struct rpcrdma_ep *ep)
+{
+ rpcrdma_recvcq_upcall(ep-rep_attr.recv_cq, ep);
+ rpcrdma_sendcq_upcall(ep-rep_attr.send_cq, ep);
+}
+
#ifdef RPC_DEBUG
static const char * const conn[] = {
address resolved,
@@ -872,9 +879,7 @@ retry:
if (rc rc != -ENOTCONN)
dprintk(RPC: %s: rpcrdma_ep_disconnect
status %i\n, __func__, rc);
-
- rpcrdma_clean_cq(ep-rep_attr.recv_cq);
- rpcrdma_clean_cq(ep-rep_attr.send_cq);
+ rpcrdma_flush_cqs(ep);
xprt = container_of(ia, struct rpcrdma_xprt, rx_ia);
id = rpcrdma_create_id(xprt, ia,
@@ -985,8 +990,7 @@ rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct
rpcrdma_ia *ia)
{
int rc;
- rpcrdma_clean_cq(ep-rep_attr.recv_cq);
- rpcrdma_clean_cq(ep-rep_attr.send_cq);
+ rpcrdma_flush_cqs(ep);
rc = rdma_disconnect(ia-ri_id);
if (!rc) {
/* returns without wait if not connected */
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH v2 05/21] xprtrdma: On disconnect, don't ignore pending CQEs
Hi Shirley-
On Jul 9, 2014, at 5:28 PM, Shirley Ma shirley...@oracle.com wrote:
Should all rdma_clean_cq be replaced by flush_cqs? The outstanding CQEs
should be processed in any context.
The only other context is when the transport is being destroyed (ie,
at umount time).
Send CQEs don’t matter at that point: the FRMRs are going to get
deregistered no matter what state they are in.
The RPC client waits for all remaining RPCs to complete before it
attempts to destroy the transport, so there shouldn’t be any receive
CQEs at that point either.
Shirley
On 07/09/2014 09:57 AM, Chuck Lever wrote:
xprtrdma is currently throwing away queued completions during
a reconnect. RPC replies posted just before connection loss, or
successful completions that change the state of an FRMR, can be
missed.
Signed-off-by: Chuck Lever chuck.le...@oracle.com
---
net/sunrpc/xprtrdma/verbs.c | 14 +-
1 file changed, 9 insertions(+), 5 deletions(-)
diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c
index 0d5187d..7fd457e 100644
--- a/net/sunrpc/xprtrdma/verbs.c
+++ b/net/sunrpc/xprtrdma/verbs.c
@@ -310,6 +310,13 @@ rpcrdma_recvcq_upcall(struct ib_cq *cq, void
*cq_context)
rpcrdma_recvcq_poll(cq, ep);
}
+static void
+rpcrdma_flush_cqs(struct rpcrdma_ep *ep)
+{
+rpcrdma_recvcq_upcall(ep-rep_attr.recv_cq, ep);
+rpcrdma_sendcq_upcall(ep-rep_attr.send_cq, ep);
+}
+
#ifdef RPC_DEBUG
static const char * const conn[] = {
address resolved,
@@ -872,9 +879,7 @@ retry:
if (rc rc != -ENOTCONN)
dprintk(RPC: %s: rpcrdma_ep_disconnect
status %i\n, __func__, rc);
-
-rpcrdma_clean_cq(ep-rep_attr.recv_cq);
-rpcrdma_clean_cq(ep-rep_attr.send_cq);
+rpcrdma_flush_cqs(ep);
xprt = container_of(ia, struct rpcrdma_xprt, rx_ia);
id = rpcrdma_create_id(xprt, ia,
@@ -985,8 +990,7 @@ rpcrdma_ep_disconnect(struct rpcrdma_ep *ep, struct
rpcrdma_ia *ia)
{
int rc;
-rpcrdma_clean_cq(ep-rep_attr.recv_cq);
-rpcrdma_clean_cq(ep-rep_attr.send_cq);
+rpcrdma_flush_cqs(ep);
rc = rdma_disconnect(ia-ri_id);
if (!rc) {
/* returns without wait if not connected */
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
