Re: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA adapter

[frank zago]

On 03/20/2012 05:39 PM, parav.pan...@emulex.com wrote:
 +struct ib_mr *ocrdma_get_dma_mr(struct ib_pd *ibpd, int acc)
 +{
 + struct ocrdma_mr *mr;
 +
 + mr = ocrdma_alloc_lkey(ibpd, acc, 0, OCRDMA_ADDR_CHECK_DISABLE);
 + if (!mr)
 + return ERR_PTR(-ENOMEM);

ocrdma_alloc_lkey does not return NULL on error.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Fwd: Re: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA adapter

[frank zago]

(Resent becasue the first one got lost)

On 03/20/2012 05:39 PM, parav.pan...@emulex.com wrote:
 +
 +int ocrdma_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
 +  struct ib_send_wr **bad_wr)
 +{
 + int status = 0;
 + struct ocrdma_qp *qp = get_ocrdma_qp(ibqp);
 + struct ocrdma_hdr_wqe *hdr;
 + unsigned long flags;
 +
 + spin_lock_irqsave(qp-q_lock, flags);
 + if (qp-state != OCRDMA_QPS_RTS  qp-state != OCRDMA_QPS_SQD) {
 + spin_unlock_irqrestore(qp-q_lock, flags);
 + return -EINVAL;
 + }

There, and in several places in this function, you return an error
without setting bad_wr.

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 4/9] ocrdma: Driver for Emulex OneConnect RDMA adapter

[frank zago]

On 03/20/2012 05:39 PM, parav.pan...@emulex.com wrote:
 From: Parav Pandit parav.pan...@emulex.com

 +
 +int ocrdma_mbx_dealloc_lkey(struct ocrdma_dev *dev, int fr_mr, u32 lkey)
 +{
 + int status = -ENOMEM;
 + struct ocrdma_dealloc_lkey *cmd;
 +
 + cmd = ocrdma_init_emb_mqe(OCRDMA_CMD_DEALLOC_LKEY, sizeof(*cmd));
 + if (!cmd)
 + return -ENOMEM;
 + cmd-lkey = lkey;
 + cmd-rsvd_frmr = fr_mr ? 1 : 0;
 + status = ocrdma_mbx_cmd(dev, (struct ocrdma_mqe *)cmd);
 + if (status)
 + goto mbx_err;
 +mbx_err:

Missing return before the tag ? Or unnecessary goto.

 + kfree(cmd);
 + return status;
 +}

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 6/9] ocrdma: Driver for Emulex OneConnect RDMA adapter

[frank zago]

On 03/20/2012 05:39 PM, parav.pan...@emulex.com wrote:
 +
 +int ocrdma_post_send(struct ib_qp *ibqp, struct ib_send_wr *wr,
 +  struct ib_send_wr **bad_wr)
 +{
 + int status = 0;
 + struct ocrdma_qp *qp = get_ocrdma_qp(ibqp);
 + struct ocrdma_hdr_wqe *hdr;
 + unsigned long flags;
 +
 + spin_lock_irqsave(qp-q_lock, flags);
 + if (qp-state != OCRDMA_QPS_RTS  qp-state != OCRDMA_QPS_SQD) {
 + spin_unlock_irqrestore(qp-q_lock, flags);
 + return -EINVAL;
 + }

There, and in several places in this function, you return an error
without setting bad_wr.

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 05/14] SIWv3: User Interface: siw_ae.c, siw_verbs.c, siw_user.h, siw_verbs.h

[frank zago]

On 07/22/2011 01:47 PM, Bernard Metzler wrote:
 +int siw_post_receive(struct ib_qp *ofa_qp, struct ib_recv_wr *wr,
 +  struct ib_recv_wr **bad_wr)
 +{
 + struct siw_wqe  *wqe = NULL;
 + struct siw_qp   *qp = siw_qp_ofa2siw(ofa_qp);
 + unsigned long   flags;
 + int rv = 0;
 +
 + dprint(DBG_WR|DBG_TX, (QP%d): state=%d\n, QP_ID(qp),
 + qp-attrs.state);
 +
 + if (qp-srq)
 + return -EOPNOTSUPP; /* what else from errno.h? */

bad_wr is not set.

 + /*
 +  * Try to acquire QP state lock. Must be non-blocking
 +  * to accommodate kernel clients needs.
 +  */
 + if (!down_read_trylock(qp-state_lock)) {
 + *bad_wr = wr;
 + return -ENOTCONN;
 + }
 +
 + if (qp-attrs.state  SIW_QP_STATE_RTS) {
 + up_read(qp-state_lock);
 + dprint(DBG_ON,  (QP%d): state=%d\n, QP_ID(qp),
 + qp-attrs.state);
 + return -EINVAL;

ditto.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

ibv_query_qp returns incorrect sq_sig_all

[frank zago]

Hi,

It seems that sq_sig_all is not properly set when creating the QP or
when querying it. The following programs creates a bunch of QP with
sq_sig_all set to 0 and then queries them. Often sq_sig_all is returned
to be 1.
I tested some other returned values and they seem correct.

I didn't see anything wrong in the libraries nor in the drivers.
Tested with an mthca on OFED 1.5.1 and mlx4 on OFED 1.3.1.

Any idea ?

Frank.



For instance:
[fzago]$ ./testqp
failed at loop 167 (sq_sig_all=1)

Regards,
  Frank.

/*
 * Invalid sq_sig_all test
 *
 * Compile with   gcc -Wall -O2  -libverbs  testqp.c   -o testqp
 */

#include stdio.h
#include assert.h
#include string.h
#include stdlib.h

#include infiniband/verbs.h

#define NUM 1

int main(void)
{
struct ibv_context *context;
struct ibv_pd  *pd;
struct ibv_device **dev_list;
struct ibv_device *ib_dev;
struct ibv_qp_init_attr qp_init_attr;
struct ibv_qp *qp[NUM];
struct ibv_cq *cq;
struct ibv_qp_attr attr;
int rc;
int i;

dev_list = ibv_get_device_list(NULL);

ib_dev = dev_list[0];
assert(ib_dev);

context = ibv_open_device(ib_dev);
assert(context);

pd = ibv_alloc_pd(context);
assert(pd);

cq = ibv_create_cq(context, 10, NULL, NULL, 0);
assert(cq);

for (i=0; iNUM; i++) {
memset(qp_init_attr, 0, sizeof(qp_init_attr));

qp_init_attr.send_cq = cq;
qp_init_attr.recv_cq = cq;
qp_init_attr.cap.max_send_wr  = 10;
qp_init_attr.cap.max_recv_wr  = 10;
qp_init_attr.cap.max_send_sge = 1;
qp_init_attr.cap.max_recv_sge = 1;
qp_init_attr.qp_type = IBV_QPT_RC;

assert(qp_init_attr.sq_sig_all == 0);

qp[i] = ibv_create_qp(pd, qp_init_attr);
assert(qp[i]);

rc = ibv_query_qp(qp[i], attr, 0x, qp_init_attr);
assert(rc == 0);

if (qp_init_attr.sq_sig_all != 0) {
printf(failed at loop %d (sq_sig_all=%d)\n, i,
qp_init_attr.sq_sig_all);
abort();
}
}

return 0;
}
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: where is the perftest tree ?

[frank zago]

On 05/09/2011 10:09 AM, frank zago wrote:
 Hi,
 
 Where is the perftest tree hosted ? The only place I know of 
 (http://git.openfabrics.org/git?p=~shamoya/perftest.git) hasn't been updated 
 in months, and the code differs a lot from what is in OFED 1.5.3.
 Any idea ?
 

No one ?
The maintainer is MIA too (no reply after 3 weeks).

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: where is the perftest tree ?

[frank zago]

On 05/17/2011 12:13 PM, Hefty, Sean wrote:
 Where is the perftest tree hosted ? The only place I know of
 (http://git.openfabrics.org/git?p=~shamoya/perftest.git) hasn't been
 updated in months, and the code differs a lot from what is in OFED 1.5.3.
 Any idea ?


 No one ?
 The maintainer is MIA too (no reply after 3 weeks).
 
 No idea, but the link above has been updated within the past 2 weeks now.  
 Based on the git log dates, there have been several updates this year.  Maybe 
 the updates just hadn't been pushed recently?

OK, so the push is very recent then. I checked last week and the last one was 
still from december.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: where is the perftest tree ?

[frank zago]

On 05/17/2011 02:32 PM, Ido Shamai wrote:
 Hello everyone,
 
 Frank sorry I was gone for a while, I've lost access to this account and 
 didn't saw those mails.
 
 I forget to update to git tree for a a while but now it's up to date,
 with your changes also (Frank thanks for the patches).
 
 The git tree is in -
 
 git://git.openfabrics.org/~shamoya/perftest.git
 

Thanks Ido. 

I've attached a patch for the Makefile so it doesn't rebuild everything every 
time someone types make.
Signed-off-by: Frank Zago fz...@systemfabricworks.com

There's also a typo in README: conatains - contains (which I'll let you fix).

Regards,
  Frank.
diff --git a/Makefile b/Makefile
index b4132c6..134ea26 100644
--- a/Makefile
+++ b/Makefile
@@ -1,6 +1,6 @@
 RDMACM_TESTS = rdma_lat rdma_bw
-MCAST_TESTS = send_bw send_lat
-TESTS = write_bw_postlist write_lat write_bw read_lat read_bw
+MCAST_TESTS = ib_send_bw ib_send_lat
+TESTS = ib_write_bw_postlist ib_write_lat ib_write_bw ib_read_lat ib_read_bw
 UTILS = clock_test
 
 all: ${RDMACM_TESTS} ${MCAST_TESTS} ${TESTS} ${UTILS}
@@ -22,14 +22,19 @@ ${TESTS} ${UTILS}: LOADLIBES += -libverbs
 
 ${RDMACM_TESTS}: %: %.c ${BASIC_FILES} ${BASIC_HEADERS}
 	$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $ ${BASIC_FILES} $(LOADLIBES) $(LDLIBS) -o $@
-${MCAST_TESTS}: %: %.c ${BASIC_FILES} ${EXTRA_FILES} ${MCAST_FILES} ${BASIC_HEADERS} ${EXTRA_HEADERS} ${MCAST_HEADERS}
-	$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $ ${BASIC_FILES} ${EXTRA_FILES} ${MCAST_FILES} $(LOADLIBES) $(LDLIBS) -o ib_$@
-${TESTS} ${UTILS}: %: %.c ${BASIC_FILES} ${EXTRA_FILES} ${BASIC_HEADERS} ${EXTRA_HEADERS}
-	$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $ ${BASIC_FILES} ${EXTRA_FILES} $(LOADLIBES) $(LDLIBS) -o ib_$@
+
+${MCAST_TESTS}: ib_%: %.c ${BASIC_FILES} ${EXTRA_FILES} ${MCAST_FILES} ${BASIC_HEADERS} ${EXTRA_HEADERS} ${MCAST_HEADERS}
+	$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $ ${BASIC_FILES} ${EXTRA_FILES} ${MCAST_FILES} $(LOADLIBES) $(LDLIBS) -o $@
+
+${TESTS}: ib_%: %.c ${BASIC_FILES} ${EXTRA_FILES} ${BASIC_HEADERS} ${EXTRA_HEADERS}
+	$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $ ${BASIC_FILES} ${EXTRA_FILES} $(LOADLIBES) $(LDLIBS) -o $@
+
+${UTILS}: %: %.c ${BASIC_FILES} ${EXTRA_FILES} ${BASIC_HEADERS} ${EXTRA_HEADERS}
+	$(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $ ${BASIC_FILES} ${EXTRA_FILES} $(LOADLIBES) $(LDLIBS) -o $@
 
 clean:
 	$(foreach fname,${RDMACM_TESTS}, rm -f ${fname})
-	$(foreach fname,${MCAST_TESTS}, rm -f ib_${fname})
-	$(foreach fname,${TESTS} ${UTILS}, rm -f ib_${fname})
+	$(foreach fname,${MCAST_TESTS}, rm -f ${fname})
+	$(foreach fname,${TESTS} ${UTILS}, rm -f ${fname})
 .DELETE_ON_ERROR:
 .PHONY: all clean

Re: building RDMA perftests with g++

[frank zago]

On 05/17/2011 02:55 PM, i...@celticblues.com wrote:
 Has this issue been confirmed or debunked?

AFAIK, this style of initialization is not supported by C++.

struct addrinfo hints = {
.ai_family   = AF_UNSPEC,
.ai_socktype = SOCK_STREAM
};

If you want to use a c++ compiler, you'll have to change the code. Something 
like
  struct addrinfo hints = { 0 };
  hints.ai_family   = AF_UNSPEC;
  hints.ai_socktype = SOCK_STREAM;

Frank.


 
 Ed
 
 
 Quoting i...@celticblues.com:
 
 Ido,
 Have you had opportunity to check into this?
 Ed

 Quoting Ido Shamai i...@dev.mellanox.co.il:

 Hey Ed ,

 I'm Ido , perftest maintainer , I will check the issue now , and report
 you back.

 In general , every c program can be compiled with g++ ?

 Ido

 On 11/18/2010 5:20 AM, i...@celticblues.com wrote:
 I have what is probably a silly question

 If I compile the rdma_bw example from perftests with g++, it   doesn't 
 work... granted I have to make a few changes wrt structure   
 initialization, but I would think it should behave as when built   with 
 gcc...  I am getting an error message in   pp_client_init/pp_server_init 
 that ai_family SOCK_STREAM is not   support for the port.  If I use cma I 
 get an unrecognized event on   the client side... Am I missing something?  
 I am trying to  develop  some C++ classes with RDMA/verbs.
 Thanks,
 Ed


 -- 
 To unsubscribe from this list: send the line unsubscribe linux-rdma in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html


 -- 
 To unsubscribe from this list: send the line unsubscribe linux-rdma in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 
 
 
 -- 
 To unsubscribe from this list: send the line unsubscribe linux-rdma in
 the body of a message to majord...@vger.kernel.org
 More majordomo info at  http://vger.kernel.org/majordomo-info.html
 

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC] RDMA CM + XRC, take two

[frank zago]

Hello Sean,

On 05/10/2011 06:17 PM, Hefty, Sean wrote:
 This is an improved version of my previous patch to add XRC support to
 RDMA CM.
 
 I would still like to see support merged upstream and into mainline 
 libibverbs before applying this to my tree.

Agreed. 

  
 This version now support the newer rdma CM interface. It needs some
 polishing but it works.
 
 It's hard to comment on the patch as an attachment.  In several places, 
 if/else statements were added, but indentation was not added to the opposite 
 part of the statement.  There was also a rather large amount of code added to 
 rdma_create_qp() that needs to be split into a separate function.  (I dislike 
 functions longer than a screen.)

Sorry for that attachment. I tried several time to post inline patches but was 
always getting a mangled patch.
I'll address you comments in a new patch.

  
 It modifies the rdma_client/server examples to use xrc instead of rc.

 The big hack is that the SRQ number needs to be transmitted to the remote
 side. This patch hijacks the private data, so it's not acceptable. Ideally 
 the SRQ
 number should be transmitted either in the REQ or REP packet (depending
 on which side the sender or the receiver) alongside the QP number. But that
 would need a change in the specs. Any suggestions ?
 
 What are the QPNs carried in the REQ and REP?

Those for the XRC QPs. Same as a for regular RC connection.

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Converting from RC to UC

[frank zago]

On 03/13/2011 08:08 PM, Alan Cook wrote:
 Or Gerlitz ogerlitz@... writes:
 

 Alan, you haven't specified if you app is user space or kernel space 
 based,  I assume the former, I believe that it either just works or can 
 be just made to work... e.g maybe a minor error on your side, or maybe 
 something small to change/complete on the rdma-cm side...

 I'm copying the maintainer of the kernel rdma-cm and librdmacm, Sean - 
 should there any problem for librdmacm consumers to use UC at their apps?

 Or.

 
 My application is running in user space, but could run in kernel space if need
 be.  I would assume that the error is on my side, but I am unable to find any
 documentation on setting up a UC service as opposed to an RC service.
 

You can look at perftest. A few tests support UC (but none with rdma cm 
support).

Frank.

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB atomic operations

[frank zago]

On 02/25/2011 01:30 AM, Bharath Ramesh wrote:
 Hi,
   I am trying to use IB atomic operations, specifically fetch and add.
 I tried a small example but I am unable to get it to work. The WC status is
 IBV_WC_REM_INV_REQ_ERR. My sample code is as follows:

Did you register your memory with IBV_ACCESS_REMOTE_ATOMIC ?

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB atomic operations

[frank zago]

On 02/25/2011 10:04 AM, Bharath Ramesh wrote:
 -Original Message-
 From: frank zago [mailto:fz...@systemfabricworks.com]

 On 02/25/2011 01:30 AM, Bharath Ramesh wrote:
 Hi,
 I am trying to use IB atomic operations, specifically fetch and
 add.
 I tried a small example but I am unable to get it to work. The WC
 status is
 IBV_WC_REM_INV_REQ_ERR. My sample code is as follows:

 Did you register your memory with IBV_ACCESS_REMOTE_ATOMIC ?
 
 I have registered both the remote and local data with
 IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_ATOMIC | IBV_ACCESS_REMOTE_READ |
 IBV_ACCESS_REMOTE_WRITE
 

Do you create your QPs with RDMA CM or do you transition them by hand ? In the 
later case there is a couple flags to set.

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: IB atomic operations

[frank zago]

Sorry, I don't see what's wrong.
Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH] perftest/write_bw_postlist

[frank zago]

Hello,

in write_bw_postlist.c, my_dest (and possibly rem_dest) must be initialized, 
else the remote node may not be able to parse the connection string and the 
test will fail to start.
The others test are doing that initialization.

Signed-off-by: Frank Zago fz...@systemfabricworks.com
diff --git a/write_bw_postlist.c b/write_bw_postlist.c
index 0349c40..e2e191f 100644
--- a/write_bw_postlist.c
+++ b/write_bw_postlist.c
@@ -738,8 +743,8 @@ int main(int argc, char *argv[])
 	if (!ctx)
 		return 1;
 
-	my_dest  = malloc(sizeof(struct pingpong_dest)*user_param.num_of_qps);
-	rem_dest = malloc(sizeof(struct pingpong_dest)*user_param.num_of_qps);
+	my_dest  = calloc(user_param.num_of_qps, sizeof(struct pingpong_dest));
+	rem_dest = calloc(user_param.num_of_qps, sizeof(struct pingpong_dest));
 if (!my_dest || !rem_dest) {
 		perror(malloc my_dest or rem_dest);
 		return 1;

[PATCH] perftest fixes

[frank zago]

Hello,

This fixes a bug in 3 tests where the result of ibv_open_device is not checked. 
 I added a couple missing endlines too.

Frank.
diff --git a/COPYING b/COPYING
diff --git a/Makefile b/Makefile
diff --git a/README b/README
diff --git a/clock_test.c b/clock_test.c
diff --git a/get_clock.c b/get_clock.c
diff --git a/get_clock.h b/get_clock.h
diff --git a/multicast_resources.c b/multicast_resources.c
diff --git a/multicast_resources.h b/multicast_resources.h
diff --git a/perftest.spec b/perftest.spec
diff --git a/perftest_resources.c b/perftest_resources.c
diff --git a/perftest_resources.h b/perftest_resources.h
diff --git a/rdma_bw.c b/rdma_bw.c
diff --git a/rdma_lat.c b/rdma_lat.c
diff --git a/read_bw.c b/read_bw.c
diff --git a/read_lat.c b/read_lat.c
diff --git a/runme b/runme
diff --git a/send_bw.c b/send_bw.c
index 61ebb52..7d59d8a 100644
--- a/send_bw.c
+++ b/send_bw.c
@@ -1173,9 +1173,13 @@ int main(int argc, char *argv[])
 
 	// Should be a function over here that computes the inline.
 	context = ibv_open_device(ib_dev);
-	
+	if (!context) {
+		fprintf(stderr, Failed to open device %s\n,
+ibv_get_device_name(ib_dev));
+		return 1;
+	}
 	if (ibv_query_device(context, device_attribute)) {
-		fprintf(stderr, Failed to query device props);
+		fprintf(stderr, Failed to query device props\n);
 		return 1;
 	}
 	if ((device_attribute.vendor_part_id == 25408  ||
diff --git a/send_lat.c b/send_lat.c
diff --git a/write_bw.c b/write_bw.c
index 5851835..0f6ca17 100644
--- a/write_bw.c
+++ b/write_bw.c
@@ -773,8 +773,13 @@ int main(int argc, char *argv[])
 		return 7;
 
 	context = ibv_open_device(ib_dev);
+	if (!context) {
+		fprintf(stderr,  Failed to open device %s\n,
+ibv_get_device_name(ib_dev));
+		return 1;
+	}
 	if (ibv_query_device(context, device_attribute)) {
-		fprintf(stderr,  Failed to query device props);
+		fprintf(stderr,  Failed to query device props\n);
 		return 1;
 	}
 	if ((device_attribute.vendor_part_id == 25408 ||
diff --git a/write_bw_postlist.c b/write_bw_postlist.c
index 0349c40..0350082 100644
--- a/write_bw_postlist.c
+++ b/write_bw_postlist.c
@@ -721,8 +721,13 @@ int main(int argc, char *argv[])
 	}
 
 	context = ibv_open_device(ib_dev);
+	if (!context) {
+		fprintf(stderr, Failed to open device %s\n,
+ibv_get_device_name(ib_dev));
+		return 1;
+	}
 	if (ibv_query_device(context, device_attribute)) {
-		fprintf(stderr, Failed to query device props);
+		fprintf(stderr, Failed to query device props\n);
 		return 1;
 	}
 	if ((device_attribute.vendor_part_id == 25408 ||
diff --git a/write_lat.c b/write_lat.c

Re: [PATCH] perftest fixes

[frank zago]

On 02/02/2011 12:19 PM, frank zago wrote:

Hello,

This fixes a bug in 3 tests where the result of ibv_open_device is not checked. 
I added a couple missing endlines too.

Frank.


I forgot:
Signed-off-by: Frank Zago fz...@systemfabricworks.com
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma cm + XRC

[frank zago]

On 08/11/2010 05:22 PM, Hefty, Sean wrote:
 - The server side of the connection also needs an SRQ. It's not 
 obvious whether it's the application or rdma cm to create that SRQ.
 And that SRQ number must be given to the client side, presumably in
 the private data.
 
 The desired mapping of XRC to the librdmacm isn't clear to me.  For 
 example, after 'connecting' is two-way communication possible 
 (setting up INI/TGT pairs on both nodes), or is a connection only 
 one-way (setup local INI to remote TGT)?  Also, as you point out, how
 are SRQ values exchanged?  Does private data carry one SRQ value, all
 SRQ values for remote processes, none?

XRC is one way. There is a send QP (which looks like a regular RC QP),
and a receive QP. Logically, the listen side would create an XRC receive QP,
and for each incoming connection would create/attach an SRQ to it. But the
initiator would have to know the SRQ number to be able to send data, and
the easiest way to pass that informations is in the CM REP private data.

I think one way to see is that you connect a QP on one side to an SRQs 
on the other side. In case of XRC, rdma_get_request() could create that SRQ
instead of a QP. 

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma cm + XRC

[frank zago]

Hello Sean,

On 08/09/2010 03:53 PM, Hefty, Sean wrote:
 This allow rdma ucm to establish an XRC connection between two nodes. Most
 of the changes are related to modify_qp since the API is different
 whether the QP is on the send or receive side.
 To create an XRC receive QP, the cap.max_send_wr must be set to 0.
 Conversely, to create the send XRC QP, that attribute must be non-zero.
 
 I need to give XRC support to the librdmacm more thought, but here are at 
 least the initial concerns:
 
 - XRC support upstream (kernel and user space) is still pending.
   (I can start a librdmacm branch for XRC support.)
 - Changes are needed to the kernel rdma_cm.
   We could start submitting patches against Roland's xrc branch for these.
 - Please update to the latest librdmacm tree.
   More specifically, rdma_getaddrinfo should support XRC as well.

The general parameters would be the same as for RC. Should we create a new 
ai_flag ? or a new port space ?
Is it really necessary to support rdma_getaddrinfo, rdma_create_ep and the new 
APIs ?

 In general, I'd like to find a way to add XRC support to the librdmacm that 
 makes things as simple for the user as possible.

Besides the need to correctly set cap.max_send_wr, the user API is unchanged.

New patch attached.
diff --git a/include/rdma/rdma_cma.h b/include/rdma/rdma_cma.h
index d17ef88..d18685b 100644
--- a/include/rdma/rdma_cma.h
+++ b/include/rdma/rdma_cma.h
@@ -125,6 +125,8 @@ struct rdma_cm_id {
 	struct ibv_cq		*send_cq;
 	struct ibv_comp_channel *recv_cq_channel;
 	struct ibv_cq		*recv_cq;
+ 	struct ibv_xrc_domain *xrc_domain;
+ 	uint32_t xrc_rcv_qpn;
 };
 
 enum {
diff --git a/man/rdma_create_qp.3 b/man/rdma_create_qp.3
index 9d2de76..659e033 100644
--- a/man/rdma_create_qp.3
+++ b/man/rdma_create_qp.3
@@ -39,6 +39,10 @@ a send or receive completion queue is not specified, then a CQ will be
 allocated by the rdma_cm for the QP, along with corresponding completion
 channels.  Completion channels and CQ data created by the rdma_cm are
 exposed to the user through the rdma_cm_id structure.
+.P
+To create an XRC receive QP, and in addition to the XRC QP type,
+ibv_qp_init_attr.cap.max_send_wr must be set to 0. Conversely, to
+create the XRC send QP, that attribute must be non-zero.
 .SH SEE ALSO
 rdma_bind_addr(3), rdma_resolve_addr(3), rdma_destroy_qp(3), ibv_create_qp(3),
 ibv_modify_qp(3)
diff --git a/src/cma.c b/src/cma.c
index a4fd574..b4eec77 100755
--- a/src/cma.c
+++ b/src/cma.c
@@ -948,12 +948,29 @@ static int rdma_init_qp_attr(struct rdma_cm_id *id, struct ibv_qp_attr *qp_attr,
 	return 0;
 }
 
+static int rdma_modify_qp(struct rdma_cm_id *id, 
+		  struct ibv_qp_attr *qp_attr,
+		  int qp_attr_mask)
+{
+	int ret;
+
+	if (id-qp)
+		ret = ibv_modify_qp(id-qp, qp_attr, qp_attr_mask);
+	else if (id-xrc_domain)
+		ret = ibv_modify_xrc_rcv_qp(id-xrc_domain, id-xrc_rcv_qpn,
+	qp_attr, qp_attr_mask);
+	else 
+		ret = EINVAL;
+
+	return ret;
+}
+
 static int ucma_modify_qp_rtr(struct rdma_cm_id *id, uint8_t resp_res)
 {
 	struct ibv_qp_attr qp_attr;
 	int qp_attr_mask, ret;
 
-	if (!id-qp)
+	if (!id-qp  !id-xrc_domain)
 		return ERR(EINVAL);
 
 	/* Need to update QP attributes from default values. */
@@ -962,7 +979,7 @@ static int ucma_modify_qp_rtr(struct rdma_cm_id *id, uint8_t resp_res)
 	if (ret)
 		return ret;
 
-	ret = ibv_modify_qp(id-qp, qp_attr, qp_attr_mask);
+	ret = rdma_modify_qp(id, qp_attr, qp_attr_mask);
 	if (ret)
 		return ERR(ret);
 
@@ -973,7 +990,7 @@ static int ucma_modify_qp_rtr(struct rdma_cm_id *id, uint8_t resp_res)
 
 	if (resp_res != RDMA_MAX_RESP_RES)
 		qp_attr.max_dest_rd_atomic = resp_res;
-	return rdma_seterrno(ibv_modify_qp(id-qp, qp_attr, qp_attr_mask));
+	return rdma_seterrno(rdma_modify_qp(id, qp_attr, qp_attr_mask));
 }
 
 static int ucma_modify_qp_rts(struct rdma_cm_id *id, uint8_t init_depth)
@@ -988,29 +1005,29 @@ static int ucma_modify_qp_rts(struct rdma_cm_id *id, uint8_t init_depth)
 
 	if (init_depth != RDMA_MAX_INIT_DEPTH)
 		qp_attr.max_rd_atomic = init_depth;
-	return rdma_seterrno(ibv_modify_qp(id-qp, qp_attr, qp_attr_mask));
+	return rdma_seterrno(rdma_modify_qp(id, qp_attr, qp_attr_mask));
 }
 
 static int ucma_modify_qp_sqd(struct rdma_cm_id *id)
 {
 	struct ibv_qp_attr qp_attr;
 
-	if (!id-qp)
+	if (!id-qp  !id-xrc_domain)
 		return 0;
 
 	qp_attr.qp_state = IBV_QPS_SQD;
-	return rdma_seterrno(ibv_modify_qp(id-qp, qp_attr, IBV_QP_STATE));
+	return rdma_seterrno(rdma_modify_qp(id, qp_attr, IBV_QP_STATE));
 }
 
 static int ucma_modify_qp_err(struct rdma_cm_id *id)
 {
 	struct ibv_qp_attr qp_attr;
 
-	if (!id-qp)
+	if (!id-qp  !id-xrc_domain)
 		return 0;
 
 	qp_attr.qp_state = IBV_QPS_ERR;
-	return rdma_seterrno(ibv_modify_qp(id-qp, qp_attr, IBV_QP_STATE));
+	return rdma_seterrno(rdma_modify_qp(id, qp_attr, IBV_QP_STATE));
 }
 
 static int ucma_find_pkey(struct cma_device *cma_dev, uint8_t port_num,
@@ -1029,7 +1046,7 @@ static int ucma_find_pkey(struct cma_device *cma_dev, uint8_t 

Re: [PATCH] rdma cm + XRC

[frank zago]

On 08/10/2010 12:14 PM, Jason Gunthorpe wrote:
 On Tue, Aug 10, 2010 at 09:59:50AM -0700, Hefty, Sean wrote:
 
 The general parameters would be the same as for RC. Should we create a new
 ai_flag ? or a new port space ?

 There's a ai_qp_type field available.  I think the RDMA TCP port
 space would work.
 
 Not sure the port space matters at all?
 
 Is there anything additional CM information for XRC other than
 requesting an XRC QP type? (XRCSRQ or something?)

Creating a send or receive XRC QP is using a different API (ibv_create_qp vs
ibv_create_xrc_rcv_qp) so I used the max_send_wr capability attribute to
discriminate between both cases. That's the only visible change to the API.
On 7/30, I posted a patch to perftest rdmb_bw to show how it's used.

 
 Is it really necessary to support rdma_getaddrinfo, rdma_create_ep and the
 new APIs ?

 I think so, yes.  At least XRC needs to be handled, even if some of
 the calls just fail as unsupported.
 
 I'd like to see a strong rational for leaving any of the new API
 unsupported for XRC - IMHO it should all be doable. The new API is
 supposed to be simplifying, we want people to use it..

No rational, besides that someone has to write some code :)
 
Regards,
  Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] rdma cm + XRC

[frank zago]

On 08/10/2010 12:14 PM, Jason Gunthorpe wrote:
 On Tue, Aug 10, 2010 at 09:59:50AM -0700, Hefty, Sean wrote:
 
 The general parameters would be the same as for RC. Should we create a new
 ai_flag ? or a new port space ?

 There's a ai_qp_type field available.  I think the RDMA TCP port
 space would work.
 
 Not sure the port space matters at all?
 
 Is there anything additional CM information for XRC other than
 requesting an XRC QP type? (XRCSRQ or something?)
 
 Is it really necessary to support rdma_getaddrinfo, rdma_create_ep and the
 new APIs ?

 I think so, yes.  At least XRC needs to be handled, even if some of
 the calls just fail as unsupported.
 
 I'd like to see a strong rational for leaving any of the new API
 unsupported for XRC - IMHO it should all be doable. The new API is
 supposed to be simplifying, we want people to use it..

It seems the new API has too many constraints for XRC. There are a couple 
things that don't fit:

- XRC needs a domain, which must be created before creating the QP, but after 
we know 
  the device to use. In addition it also needs a file descriptor. The 
application may 
  want to use a different fd depending on the device. Currently the domain can 
only 
  be created in the middle of rdma_create_ep().

- The server side of the connection also needs an SRQ. It's not obvious whether 
it's
  the application or rdma cm to create that SRQ. And that SRQ number must be
  given to the client side, presumably in the private data.

Frank.

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

RDMA CM and XRC

[frank zago]

Hello,

For a project we consider using RDMA CM to setup XRC connections. 
Currently that combination is not supported. Has anyone done any work on 
adding XRC support to RDMA CM ?


That's something we might also consider doing if not already available 
somewhere. Is there any pitfall to be aware of ?


Regards,
  frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[Announce] rxe dev tree available (soft RDMAoE)

[frank zago]

Hello,

The development tree for a soft RDMA transport over Ethernet driver 
(rxe) is available in the OFA git repository. This is a work in progress 
but has enough functionality for people interested in looking at it to 
be worth sharing. Going forward changes to the driver will be maintained 
here.


To get a copy of the driver type

  git clone git://git.openfabrics.org/~fzago/rxe

This work is being supported by Mellanox and System Fabric Works.

Regards,
  Frank
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 2/6] RDMA/ehca - ib_post_send fixes, v2

[Frank Zago]

Always set ib_post_send()'s bad_wr when an error occurred.
Do not report success if an error occurred.

Signed-off-by: Frank Zago fz...@systemfabricworks.com
---
 drivers/infiniband/hw/ehca/ehca_reqs.c |   26 +++---
 1 file changed, 11 insertions(+), 15 deletions(-)

Index: linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/ehca/ehca_reqs.c2009-12-02
19:20:53.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c 2009-12-02
19:23:19.0 -0600
@@ -400,7 +400,6 @@

 static inline int post_one_send(struct ehca_qp *my_qp,
 struct ib_send_wr *cur_send_wr,
-struct ib_send_wr **bad_send_wr,
 int hidden)
 {
struct ehca_wqe *wqe_p;
@@ -412,8 +411,6 @@
wqe_p = ipz_qeit_get_inc(my_qp-ipz_squeue);
if (unlikely(!wqe_p)) {
/* too many posted work requests: queue overflow */
-   if (bad_send_wr)
-   *bad_send_wr = cur_send_wr;
ehca_err(my_qp-ib_qp.device, Too many posted WQEs 
 qp_num=%x, my_qp-ib_qp.qp_num);
return -ENOMEM;
@@ -433,8 +430,6 @@
 */
if (unlikely(ret)) {
my_qp-ipz_squeue.current_q_offset = start_offset;
-   if (bad_send_wr)
-   *bad_send_wr = cur_send_wr;
ehca_err(my_qp-ib_qp.device, Could not write WQE 
 qp_num=%x, my_qp-ib_qp.qp_num);
return -EINVAL;
@@ -448,7 +443,6 @@
   struct ib_send_wr **bad_send_wr)
 {
struct ehca_qp *my_qp = container_of(qp, struct ehca_qp, ib_qp);
-   struct ib_send_wr *cur_send_wr;
int wqe_cnt = 0;
int ret = 0;
unsigned long flags;
@@ -457,7 +451,8 @@
if (unlikely(my_qp-state  IB_QPS_RTS)) {
ehca_err(qp-device, Invalid QP state  qp_state=%d qpn=%x,
 my_qp-state, qp-qp_num);
-   return -EINVAL;
+   ret = -EINVAL;
+   goto out;
}

/* LOCK the QUEUE */
@@ -476,24 +471,21 @@
struct ib_send_wr circ_wr;
memset(circ_wr, 0, sizeof(circ_wr));
circ_wr.opcode = IB_WR_RDMA_READ;
-   post_one_send(my_qp, circ_wr, NULL, 1); /* ignore retcode */
+   post_one_send(my_qp, circ_wr, 1); /* ignore retcode */
wqe_cnt++;
ehca_dbg(qp-device, posted circ wr  qp_num=%x, qp-qp_num);
my_qp-message_count = my_qp-packet_count = 0;
}

/* loop processes list of send reqs */
-   for (cur_send_wr = send_wr; cur_send_wr != NULL;
-cur_send_wr = cur_send_wr-next) {
-   ret = post_one_send(my_qp, cur_send_wr, bad_send_wr, 0);
+   while(send_wr) {
+   ret = post_one_send(my_qp, send_wr, 0);
if (unlikely(ret)) {
-   /* if one or more WQEs were successful, don't fail */
-   if (wqe_cnt)
-   ret = 0;
goto post_send_exit0;
}
wqe_cnt++;
-   } /* eof for cur_send_wr */
+   send_wr = send_wr-next;
+   }

 post_send_exit0:
iosync(); /* serialize GAL register access */
@@ -503,6 +495,10 @@
 my_qp, qp-qp_num, wqe_cnt, ret);
my_qp-message_count += wqe_cnt;
spin_unlock_irqrestore(my_qp-spinlock_s, flags);
+
+out:
+   if (ret)
+   *bad_send_wr = send_wr;
return ret;
 }


-- 

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 0/6] post_send/recv fixes (v2)

[Frank Zago]

Hello,

This series fixes a few bugs in several IB drivers regarding their
ib_post_send() and ib_post_recv() implementation.

Regards,
  Frank.


--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 2/8] ehca - ib_post_send fixes, Updated

[frank zago]

Always set ib_post_send()'s bad_wr.
Do not report success if an error occurred.

Signed-off-by: Frank Zago fz...@systemfabricworks.com
Index: linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/ehca/ehca_reqs.c	2009-11-24 
18:59:31.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c	2009-11-24 
19:49:19.0 -0600

@@ -400,7 +400,6 @@

 static inline int post_one_send(struct ehca_qp *my_qp,
 struct ib_send_wr *cur_send_wr,
-struct ib_send_wr **bad_send_wr,
 int hidden)
 {
struct ehca_wqe *wqe_p;
@@ -412,8 +411,6 @@
wqe_p = ipz_qeit_get_inc(my_qp-ipz_squeue);
if (unlikely(!wqe_p)) {
/* too many posted work requests: queue overflow */
-   if (bad_send_wr)
-   *bad_send_wr = cur_send_wr;
ehca_err(my_qp-ib_qp.device, Too many posted WQEs 
 qp_num=%x, my_qp-ib_qp.qp_num);
return -ENOMEM;
@@ -433,8 +430,6 @@
 */
if (unlikely(ret)) {
my_qp-ipz_squeue.current_q_offset = start_offset;
-   if (bad_send_wr)
-   *bad_send_wr = cur_send_wr;
ehca_err(my_qp-ib_qp.device, Could not write WQE 
 qp_num=%x, my_qp-ib_qp.qp_num);
return -EINVAL;
@@ -448,7 +443,6 @@
   struct ib_send_wr **bad_send_wr)
 {
struct ehca_qp *my_qp = container_of(qp, struct ehca_qp, ib_qp);
-   struct ib_send_wr *cur_send_wr;
int wqe_cnt = 0;
int ret = 0;
unsigned long flags;
@@ -457,7 +451,8 @@
if (unlikely(my_qp-state  IB_QPS_RTS)) {
ehca_err(qp-device, Invalid QP state  qp_state=%d qpn=%x,
 my_qp-state, qp-qp_num);
-   return -EINVAL;
+   ret = -EINVAL;
+   goto out;
}

/* LOCK the QUEUE */
@@ -476,24 +471,21 @@
struct ib_send_wr circ_wr;
memset(circ_wr, 0, sizeof(circ_wr));
circ_wr.opcode = IB_WR_RDMA_READ;
-   post_one_send(my_qp, circ_wr, NULL, 1); /* ignore retcode */
+   post_one_send(my_qp, circ_wr, 1); /* ignore retcode */
wqe_cnt++;
ehca_dbg(qp-device, posted circ wr  qp_num=%x, qp-qp_num);
my_qp-message_count = my_qp-packet_count = 0;
}

/* loop processes list of send reqs */
-   for (cur_send_wr = send_wr; cur_send_wr != NULL;
-cur_send_wr = cur_send_wr-next) {
-   ret = post_one_send(my_qp, cur_send_wr, bad_send_wr, 0);
+   while(send_wr) {
+   ret = post_one_send(my_qp, send_wr, 0);
if (unlikely(ret)) {
-   /* if one or more WQEs were successful, don't fail */
-   if (wqe_cnt)
-   ret = 0;
goto post_send_exit0;
}
wqe_cnt++;
-   } /* eof for cur_send_wr */
+   send_wr = send_wr-next;
+   }

 post_send_exit0:
iosync(); /* serialize GAL register access */
@@ -503,6 +495,10 @@
 my_qp, qp-qp_num, wqe_cnt, ret);
my_qp-message_count += wqe_cnt;
spin_unlock_irqrestore(my_qp-spinlock_s, flags);
+
+out:
+   if (ret)
+   *bad_send_wr = send_wr;
return ret;
 }


--

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 0/8] post_send/recv fixes

[frank zago]

Hello,

This series fixes a few bugs in several IB drivers regarding their
ib_post_send() and ib_post_recv() implementation.

Regards,
  Frank.

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 1/8] Ammasso - ib_post_send fix

[frank zago]

Always set ib_post_send()'s bad_wr.

Signed-off-by: Frank Zago fz...@systemfabricworks.com
Index: linux-2.6.31/drivers/infiniband/hw/amso1100/c2_qp.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/amso1100/c2_qp.c	2009-11-23 
20:22:19.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/amso1100/c2_qp.c	2009-11-23 
20:25:55.0 -0600

@@ -798,8 +798,10 @@
u8 actual_sge_count;
u32 msg_size;

-   if (qp-state  IB_QPS_RTS)
-   return -EINVAL;
+   if (qp-state  IB_QPS_RTS) {
+   err = -EINVAL;
+   goto out;
+   }

while (ib_wr) {

@@ -930,6 +932,7 @@
ib_wr = ib_wr-next;
}

+out:
if (err)
*bad_wr = ib_wr;
return err;

--

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 5/8] ehca - ib_post_recv fixes Always set ib_post_recv()s bad_wr.

[frank zago]

Signed-off-by: Frank Zago fz...@systemfabricworks.com
Index: linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/ehca/ehca_reqs.c	2009-11-23 
21:23:08.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c	2009-11-23 
21:25:48.0 -0600

@@ -519,7 +519,7 @@
if (unlikely(!HAS_RQ(my_qp))) {
ehca_err(dev, QP has no RQ  ehca_qp=%p qp_num=%x ext_type=%d,
 my_qp, my_qp-real_qp_num, my_qp-ext_type);
-   return -ENODEV;
+   ret = -ENODEV;
}

/* LOCK the QUEUE */
@@ -533,13 +533,9 @@
wqe_p = ipz_qeit_get_inc(my_qp-ipz_rqueue);
if (unlikely(!wqe_p)) {
/* too many posted work requests: queue overflow */
-   if (bad_recv_wr)
-   *bad_recv_wr = cur_recv_wr;
-   if (wqe_cnt == 0) {
-   ret = -ENOMEM;
-   ehca_err(dev, Too many posted WQEs 
-qp_num=%x, my_qp-real_qp_num);
-   }
+   ret = -ENOMEM;
+   ehca_err(dev, Too many posted WQEs 
+   qp_num=%x, my_qp-real_qp_num);
goto post_recv_exit0;
}
/*
@@ -558,11 +554,9 @@
if (unlikely(ret)) {
my_qp-ipz_rqueue.current_q_offset = start_offset;
*bad_recv_wr = cur_recv_wr;
-   if (wqe_cnt == 0) {
-   ret = -EINVAL;
-   ehca_err(dev, Could not write WQE 
-qp_num=%x, my_qp-real_qp_num);
-   }
+   ret = -EINVAL;
+   ehca_err(dev, Could not write WQE 
+   qp_num=%x, my_qp-real_qp_num);
goto post_recv_exit0;
}

@@ -581,6 +575,11 @@
ehca_dbg(dev, ehca_qp=%p qp_num=%x wqe_cnt=%d ret=%i,
 my_qp, my_qp-real_qp_num, wqe_cnt, ret);
spin_unlock_irqrestore(my_qp-spinlock_r, flags);
+
+out:
+   if (ret)
+   *bad_recv_wr = cur_recv_wr;
+
return ret;
 }


--

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 6/8] ammasso - ib_post_recv fix

[frank zago]

Always set ib_post_recv()'s bad_wr.

Signed-off-by: Frank Zago fz...@systemfabricworks.com
Index: linux-2.6.31/drivers/infiniband/hw/amso1100/c2_qp.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/amso1100/c2_qp.c	2009-11-23 
21:26:48.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/amso1100/c2_qp.c	2009-11-23 
21:28:11.0 -0600

@@ -947,8 +947,10 @@
unsigned long lock_flags;
int err = 0;

-   if (qp-state  IB_QPS_RTS)
-   return -EINVAL;
+   if (qp-state  IB_QPS_RTS) {
+   err = -EINVAL;
+   goto out;
+   }

/*
 * Try and post each work request
@@ -1001,6 +1003,7 @@
ib_wr = ib_wr-next;
}

+out:
if (err)
*bad_wr = ib_wr;
return err;

--

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

[patch 8/8] NES - ib_post_recv fixes

[frank zago]

Always set ib_post_recv()'s bad_wr.

Signed-off-by: Frank Zago fz...@systemfabricworks.com
Index: linux-2.6.31/drivers/infiniband/hw/nes/nes_verbs.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/nes/nes_verbs.c	2009-11-23 
21:59:04.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/nes/nes_verbs.c	2009-11-23 
21:59:05.0 -0600

@@ -3554,8 +3554,10 @@
u32 counter;
u32 total_payload_length;

-   if (nesqp-ibqp_state  IB_QPS_RTS)
-   return -EINVAL;
+   if (nesqp-ibqp_state  IB_QPS_RTS) {
+   err = -EINVAL;
+   goto out;
+   }

spin_lock_irqsave(nesqp-lock, flags);

@@ -3618,6 +3620,7 @@

spin_unlock_irqrestore(nesqp-lock, flags);

+out:
if (err)
*bad_wr = ib_wr;
return err;

--

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [patch 5/8] ehca - ib_post_recv fixes Always set ib_post_recv()s bad_wr.

[frank zago]

Alexander Schmidt wrote:

On Tue, 24 Nov 2009 11:36:46 -0600
frank zago fz...@systemfabricworks.com wrote:

Hi Frank,

  

Signed-off-by: Frank Zago fz...@systemfabricworks.com
Index: linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c
===
--- linux-2.6.31.orig/drivers/infiniband/hw/ehca/ehca_reqs.c	2009-11-23 
21:23:08.0 -0600
+++ linux-2.6.31/drivers/infiniband/hw/ehca/ehca_reqs.c	2009-11-23 
21:25:48.0 -0600

@@ -519,7 +519,7 @@
if (unlikely(!HAS_RQ(my_qp))) {
ehca_err(dev, QP has no RQ  ehca_qp=%p qp_num=%x ext_type=%d,
 my_qp, my_qp-real_qp_num, my_qp-ext_type);
-   return -ENODEV;
+   ret = -ENODEV;



aren't you missing a goto statement here?
  


Indeed. I'll resend a patch.

Frank.
--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

ib_post_send in drivers

[frank zago]

Hello,

It seems ib_post_send() is implemented slightly differently in the 
various hardware drivers (as in kernel 2.6.31). Here are the differences 
I've noticed regarding the bad_wr parameter.


amso1100/c2_qp.c : c2_post_send()
* bails out and does not set bad_wr if the 1st check is bad.

cxgb3/iwch_qp.c : post_one_send()
* test for bad_send_wr but it should always be set

cxgb3/iwch_qp.c : iwch_post_send()
* bails out and does not set bad_wr if the 1st 2 checks are bad

ehca/ehca_reqs.c : ehca_post_send()
* bails out and does not set bad_wr if the 1st check is bad.
* test for bad_send_wr but it should always be set
* always return success if at least one post succeeded.

ehca/ehca_reqs.c : post_one_send()
* test for bad_send_wr but it should always be set

nes/nes_verbs.c : nes_post_send()
* bails out and does not set bad_wr if the 1st check is bad.

I think assume most are bugs (especially the ehca driver). I can post a 
patch to fix these if confirmed.


Regards,
Frank

--
To unsubscribe from this list: send the line unsubscribe linux-rdma in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html