[email protected] wrote: > Hi, all > these days I'm puzzled by a weird mistake in atomic operation, when I > use FETCH_AND_ADD, the remote buffer only changed for 9 times at most, > aka if the initial value is 0, when I execute 10 times FETCH_AND_ADD > operations, the value is 9, and no more changes happens to the buffer. > Should I take care of the big-edian from/to litter-edian translation? > No, you should write the value in local host endian. > I tried this, but it seems not the point. > Is there some sample code for atomic operation just like "perftest" in > the libibverbs package? Maybe I missed something? I paste some code > below for reviewing, thanks in advance! > You didn't miss it, not many people use the atomic operations and there isn't any example that uses it.
> struct ibv_send_wr wr; > It will be wise to fill the following delaration here: memset(&wr, 0, sizeof(wr)); > wr.opcode = IBV_WR_ATOMIC_FETCH_AND_ADD; > wr.wr.atomic.remote_addr = rem_dest->vaddr; > wr.wr.atomic.rkey = rem_dest->rkey; > wr.send_flags = IBV_SEND_SIGNALED | > IBV_SEND_FENCE; > wr.wr.atomic.compare_add = 1; > wr.wr.atomic.swap = 0; > since you are using Fetch&Add the swap attribute is useless. > wr.wr_id = id; > wr.sg_list = &list; > wr.num_sge = 1; > wr.next = NULL; > What about the value of the s/g entry? > the qp and memory region is set with atomic flag : > mr = ibv_reg_mr(host->pd, buf, len, > IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE > | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC); > > attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_LOCAL_WRITE > | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_ATOMIC; > At least the QP/MR in the remote side should enable REMOTE_ATOMIC (if you want to enable it locally, it is fine but not really needed). > some attibutes in my ib device about atomic operation are as below: > max_qp_rd_atom is 4, max_qp_init_rd_atom is 128, max_res_rd_atom is 258048 > max_qp_rd_atom: The maximum number of outstanding atomic/RDMA read that a QP can handle as target. max_qp_init_rd_atom: The maximum number of outstanding atomic/RDMA read that a QP can handle as initiator. Obviously, you should use the minimum of the values (use the right value in each side, since different HCAs can have different attributes) I think that max_res_rd_atom is the total outstanding number of atomic/RDMA read that the HCA can handle, but i'm not really sure. If you think that you have a bug, please review the code in remote side (maybe you have problem to read the data...). I hope i helped you a little ... Dotan _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
