Re: [openib-general] [patch] libsdp typo in config_parser
Running an MPI command with LD_PRELOAD=libsdp.so at the beginning won't cause SDP to be used on remote nodes. You have to find a way to load libsdp.so on all nodes, this might work better: LD_PRELOAD=libsdp.so mpirun -np 4 env LD_PRELOAD=libsdp.so /there/vasp/20060503/vasp.4.6/vasp.mpi Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Bernhard Fischer > Sent: Friday, August 18, 2006 12:22 PM > To: Eitan Zahavi > Cc: openib-general@openib.org > Subject: Re: [openib-general] [patch] libsdp typo in config_parser > > On Fri, Aug 18, 2006 at 10:05:35PM +0300, Eitan Zahavi wrote: > >Hi Bernhard > > > >SDP traffic will not show on the IPoIB counters. It does no > go through > >IPoIB. > > That's what i thought, thanks for confirming. > >You can use > >lsmod | grep ib_sdp > >to see how many connections are made over SDP. > > Running lam via 2 nodes, on 2 CPUs each, i see: > # lsmod | grep ib_sdp > ib_sdp 28184 4 > rdma_cm27912 1 ib_sdp > ib_core53632 12 > ib_ucm,ib_uverbs,ib_sdp,rdma_cm,ib_cm,ib_local_sa,ib_umad,ib_i > poib,ib_multicast,ib_sa,ib_mthca,ib_mad > > I did start lamboot with libsdp.so preloaded: > $ LD_PRELOAD=/usr/local/lib64/libsdp.so lamboot l > $ lamnodes C -c -n > node13ib.infiniband > node13ib.infiniband > node15ib.infiniband > node15ib.infiniband > $ LD_PRELOAD=/usr/local/lib64/libsdp.so mpirun -np 4 > /there/vasp/20060503/vasp.4.6/vasp.mpi > > Still, ifconfig ib0 (which hosts node??ib.infiniband on > 10.100.0.0/24) shows that the > communication is being sent over ipoib as ifconfigs counters > constantly > go up when communicating (only one user is active on the system). > $ /sbin/ifconfig ib0 > ib0 Link encap:UNSPEC HWaddr > 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 > inet addr:10.100.0.13 Bcast:10.100.0.255 > Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 > RX packets:182037964 errors:0 dropped:0 overruns:0 frame:0 > TX packets:183607689 errors:0 dropped:0 overruns:0 carrier:0 > collisions:0 txqueuelen:128 > RX bytes:189334244937 (180563.2 Mb) TX > bytes:194777918565 (185754.6 Mb) > > My libsdp.conf looks like this: > $ cat /usr/local/etc/libsdp.conf > #log min-level 1 destination file libsdp.log > use bothconnect * 10.100.0.0/24:* > use bothserver * 10.100.0.0/24:* > > So i fear i'm missing something crucial. > Ideas? > > >Exact number of packets and data can flowing through the IB > port can be > >obtained by : > >/sys/class/infiniband/mthca0/ports/1/counters/port_rcv_packets > >/sys/class/infiniband/mthca0/ports/1/counters/port_xmit_packets > > $ for i in > /sys/class/infiniband/mthca0/ports/1/counters/*packets;do > echo -n $i:' ' ; cat $i;done > /sys/class/infiniband/mthca0/ports/1/counters/port_rcv_packets > : 185010549 > /sys/class/infiniband/mthca0/ports/1/counters/port_xmit_packet > s: 186584856 > > PS: The different pingpong test (which have outdated names in > the openib > wiki, btw) do work just fine if run from the very same user, > so i think > that the basic verbs communication would work proper. > > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] huge pages support
Looks like you don't support regions that have both huge pages and
normal pages either.
And this:
> +while (npages && (ret <= PAGE_SIZE /
> +sizeof (struct page *))) {
> +if (get_user_pages(current, current->mm,
> +cur_base, 1, 1, !write,
> +&page_list[ret], NULL) < 0)
> +goto out;
> +
> +ret++;
> +cur_base += HPAGE_SIZE;
> +npages--;
> +}
seems a little iffy as well.
As I said I think the solution is to add a little more support to the
core stuff in mm/ to make this simpler -- something like a function
like get_user_pages() except returning the page list in a struct
scatterlist so that it can return pages of different sizes would make
sense to me.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH v2][RDMA CM] IB mcast fix
Thanks! committed in 9008. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [patch] libsdp typo in config_parser
On Fri, Aug 18, 2006 at 10:05:35PM +0300, Eitan Zahavi wrote: >Hi Bernhard > >SDP traffic will not show on the IPoIB counters. It does no go through >IPoIB. That's what i thought, thanks for confirming. >You can use >lsmod | grep ib_sdp >to see how many connections are made over SDP. Running lam via 2 nodes, on 2 CPUs each, i see: # lsmod | grep ib_sdp ib_sdp 28184 4 rdma_cm27912 1 ib_sdp ib_core53632 12 ib_ucm,ib_uverbs,ib_sdp,rdma_cm,ib_cm,ib_local_sa,ib_umad,ib_ipoib,ib_multicast,ib_sa,ib_mthca,ib_mad I did start lamboot with libsdp.so preloaded: $ LD_PRELOAD=/usr/local/lib64/libsdp.so lamboot l $ lamnodes C -c -n node13ib.infiniband node13ib.infiniband node15ib.infiniband node15ib.infiniband $ LD_PRELOAD=/usr/local/lib64/libsdp.so mpirun -np 4 /there/vasp/20060503/vasp.4.6/vasp.mpi Still, ifconfig ib0 (which hosts node??ib.infiniband on 10.100.0.0/24) shows that the communication is being sent over ipoib as ifconfigs counters constantly go up when communicating (only one user is active on the system). $ /sbin/ifconfig ib0 ib0 Link encap:UNSPEC HWaddr 00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 inet addr:10.100.0.13 Bcast:10.100.0.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:182037964 errors:0 dropped:0 overruns:0 frame:0 TX packets:183607689 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:128 RX bytes:189334244937 (180563.2 Mb) TX bytes:194777918565 (185754.6 Mb) My libsdp.conf looks like this: $ cat /usr/local/etc/libsdp.conf #log min-level 1 destination file libsdp.log use bothconnect * 10.100.0.0/24:* use bothserver * 10.100.0.0/24:* So i fear i'm missing something crucial. Ideas? >Exact number of packets and data can flowing through the IB port can be >obtained by : >/sys/class/infiniband/mthca0/ports/1/counters/port_rcv_packets >/sys/class/infiniband/mthca0/ports/1/counters/port_xmit_packets $ for i in /sys/class/infiniband/mthca0/ports/1/counters/*packets;do echo -n $i:' ' ; cat $i;done /sys/class/infiniband/mthca0/ports/1/counters/port_rcv_packets: 185010549 /sys/class/infiniband/mthca0/ports/1/counters/port_xmit_packets: 186584856 PS: The different pingpong test (which have outdated names in the openib wiki, btw) do work just fine if run from the very same user, so i think that the basic verbs communication would work proper. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [patch] libsdp typo in config_parser
Hi Bernhard SDP traffic will not show on the IPoIB counters. It does no go through IPoIB. You can use lsmod | grep ib_sdp to see how many connections are made over SDP. Exact number of packets and data can flowing through the IB port can be obtained by : /sys/class/infiniband/mthca0/ports/1/counters/port_rcv_packets /sys/class/infiniband/mthca0/ports/1/counters/port_xmit_packets Eitan Zahavi Senior Engineering Director, Software Architect Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -Original Message- > From: Bernhard Fischer [mailto:[EMAIL PROTECTED] > Sent: Friday, August 18, 2006 6:36 PM > To: Eitan Zahavi > Cc: openib-general@openib.org > Subject: Re: [openib-general] [patch] libsdp typo in config_parser > > On Fri, Aug 18, 2006 at 05:54:04PM +0300, Eitan Zahavi wrote: > >Committed to the trunk (not 1.1 branch) Thanks. > > > Thank you. > PS: I think there is another occurance in srp_daemon that i forgot to include in > the diff, fwiw. > > PPS: IIRC the traffic sent via SDP did not show up in the packet-counters of the > corresponding ipoib device last time i looked. Is this still the case? Asking > because i'm seeing it accounted to my ib0 device now although libsdp's log > indicates that the app is using SDP.. > > TIA for any hint ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH v2][RDMA CM] IB mcast fix
Version 2:
- Updated SIDR CMA code to use global qkey.
- Updated udaddy.c to use global qkey.
- Tested with cmatose, udaddy, mckey over mthca/IB.
-
Set the QKEY to a common global value for all UD QPs and multicast
groups created by the RDMA CM.
Signed-off-by: [EMAIL PROTECTED]
Index: src/userspace/librdmacm/include/rdma/rdma_cma_ib.h
===
--- src/userspace/librdmacm/include/rdma/rdma_cma_ib.h (revision 9004)
+++ src/userspace/librdmacm/include/rdma/rdma_cma_ib.h (working copy)
@@ -59,4 +59,10 @@
struct ibv_ah_attr *ah_attr, uint32_t *remote_qpn,
uint32_t *remote_qkey);
+/*
+ * Global qkey value for all UD QPs and multicast groups created via the
+ * RDMA CM.
+ */
+#define RDMA_UD_QKEY 0x01234567
+
#endif /* RDMA_CMA_IB_H */
Index: src/userspace/librdmacm/src/cma.c
===
--- src/userspace/librdmacm/src/cma.c (revision 9004)
+++ src/userspace/librdmacm/src/cma.c (working copy)
@@ -701,7 +701,7 @@
qp_attr.port_num = id_priv->id.port_num;
qp_attr.qp_state = IBV_QPS_INIT;
- qp_attr.qkey = ntohs(rdma_get_src_port(&id_priv->id));
+ qp_attr.qkey = RDMA_UD_QKEY;
ret = ibv_modify_qp(qp, &qp_attr, IBV_QP_STATE | IBV_QP_PKEY_INDEX |
IBV_QP_PORT | IBV_QP_QKEY);
if (ret)
Index: src/userspace/librdmacm/examples/udaddy.c
===
--- src/userspace/librdmacm/examples/udaddy.c (revision 9004)
+++ src/userspace/librdmacm/examples/udaddy.c (working copy)
@@ -434,7 +434,7 @@
node->ah = ibv_create_ah_from_wc(node->pd, wc, node->mem,
node->cma_id->port_num);
node->remote_qpn = ntohl(wc->imm_data);
- node->remote_qkey = ntohs(rdma_get_dst_port(node->cma_id));
+ node->remote_qkey = RDMA_UD_QKEY;
}
static int poll_cqs(void)
Index: src/linux-kernel/infiniband/include/rdma/rdma_cm_ib.h
===
--- src/linux-kernel/infiniband/include/rdma/rdma_cm_ib.h (revision 9004)
+++ src/linux-kernel/infiniband/include/rdma/rdma_cm_ib.h (working copy)
@@ -82,4 +82,10 @@
*/
int rdma_set_ib_req_info(struct rdma_cm_id *id, struct ib_cm_req_opt *info);
+/*
+ * Global qkey value for all UD QPs and multicast groups created via the
+ * RDMA CM.
+ */
+#define RDMA_UD_QKEY 0x01234567
+
#endif /* RDMA_CM_IB_H */
Index: src/linux-kernel/infiniband/core/cma.c
===
--- src/linux-kernel/infiniband/core/cma.c (revision 9004)
+++ src/linux-kernel/infiniband/core/cma.c (working copy)
@@ -1822,7 +1822,7 @@
break;
}
route = &id_priv->id.route;
- if (rep->qkey != ntohs(cma_port(&route->addr.dst_addr))) {
+ if (rep->qkey != RDMA_UD_QKEY) {
event = RDMA_CM_EVENT_UNREACHABLE;
status = -EINVAL;
break;
@@ -2012,7 +2012,7 @@
rep.status = status;
if (status == IB_SIDR_SUCCESS) {
rep.qp_num = id_priv->qp_num;
- rep.qkey = ntohs(cma_port(&id_priv->id.route.addr.src_addr));
+ rep.qkey = RDMA_UD_QKEY;
}
return ib_send_cm_sidr_rep(id_priv->cm_id.ib, &rep);
@@ -2172,7 +2172,7 @@
ib_addr_get_sgid(dev_addr, &rec.port_gid);
rec.pkey = cpu_to_be16(ib_addr_get_pkey(dev_addr));
rec.join_state = 1;
- rec.qkey = sin->sin_addr.s_addr;
+ rec.qkey = cpu_to_be32(RDMA_UD_QKEY);
comp_mask = IB_SA_MCMEMBER_REC_MGID | IB_SA_MCMEMBER_REC_PORT_GID |
IB_SA_MCMEMBER_REC_PKEY | IB_SA_MCMEMBER_REC_JOIN_STATE |
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] IB/mthca: No userspace SRQs if HCA doesn't have SRQ support
this should fix a crash on HCA's with firwmare too old to support SRQs. commit 5beba53230351b2d77c317c22e66c415f2ebaf02 Author: Roland Dreier <[EMAIL PROTECTED]> Date: Fri Aug 18 10:41:46 2006 -0700 IB/mthca: No userspace SRQs if HCA doesn't have SRQ support Leave all SRQ methods out of the device's uverbs_cmd_mask if the device doesn't have SRQ support (because of ancient firmware) so that we don't allow userspace to call the driver's create_srq method. This fixes a userspace-triggerable oops caused by ib_uverbs_create_srq() following the device's ->create_srq function pointer, which will be NULL if the device doesn't support SRQs. Signed-off-by: Roland Dreier <[EMAIL PROTECTED]> diff --git a/drivers/infiniband/hw/mthca/mthca_provider.c b/drivers/infiniband/hw/mthca/mthca_provider.c index 230ae21..265b1d1 100644 --- a/drivers/infiniband/hw/mthca/mthca_provider.c +++ b/drivers/infiniband/hw/mthca/mthca_provider.c @@ -1287,11 +1287,7 @@ int mthca_register_device(struct mthca_d (1ull << IB_USER_VERBS_CMD_MODIFY_QP) | (1ull << IB_USER_VERBS_CMD_DESTROY_QP) | (1ull << IB_USER_VERBS_CMD_ATTACH_MCAST)| - (1ull << IB_USER_VERBS_CMD_DETACH_MCAST)| - (1ull << IB_USER_VERBS_CMD_CREATE_SRQ) | - (1ull << IB_USER_VERBS_CMD_MODIFY_SRQ) | - (1ull << IB_USER_VERBS_CMD_QUERY_SRQ) | - (1ull << IB_USER_VERBS_CMD_DESTROY_SRQ); + (1ull << IB_USER_VERBS_CMD_DETACH_MCAST); dev->ib_dev.node_type= IB_NODE_CA; dev->ib_dev.phys_port_cnt= dev->limits.num_ports; dev->ib_dev.dma_device = &dev->pdev->dev; @@ -1316,6 +1312,11 @@ int mthca_register_device(struct mthca_d dev->ib_dev.modify_srq = mthca_modify_srq; dev->ib_dev.query_srq= mthca_query_srq; dev->ib_dev.destroy_srq = mthca_destroy_srq; + dev->ib_dev.uverbs_cmd_mask |= + (1ull << IB_USER_VERBS_CMD_CREATE_SRQ) | + (1ull << IB_USER_VERBS_CMD_MODIFY_SRQ) | + (1ull << IB_USER_VERBS_CMD_QUERY_SRQ) | + (1ull << IB_USER_VERBS_CMD_DESTROY_SRQ); if (mthca_is_memfree(dev)) dev->ib_dev.post_srq_recv = mthca_arbel_post_srq_recv; ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] InfiniBand merge plans for 2.6.19
Michael> Cold you oplease consider IB/mthca: recover from device Michael> errors as well? Yes, I will. There's still plenty of time before 2.6.19 opens up. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] InfiniBand merge plans for 2.6.19
Quoting r. Roland Dreier <[EMAIL PROTECTED]>: > o I also have the following minor changes queued in the >for-2.6.19 branch of infiniband.git: Cold you oplease consider IB/mthca: recover from device errors as well? -- MST ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 02/13] IB/ehca: includes
On Friday 18 August 2006 17:35, Christoph Raisch wrote: > we'll change these EDEBs to a wrapper around dev_err, dev_dbg and dev_warn > as it's done in the mthca driver. > > ... > > Hope that's the "official" way how to implement it in ib drivers. I guess it would be even better to just use the dev_* macros directly instead of having your own wrapper. You can do that in both ehca and ehea. Arnd <>< ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM
Bub Thomas wrote: > Can I still use the LID, GUID and SubnetID for connection establishment > then? Then Gen1 counterpart has no IP over IB running. If IPoIB is not running, then you will need to use the IB CM directly. The RDMA CM uses ARP to resolve IP addresses to GIDs. > I'm using OFED-1.0.1. > Do you have a quick link where to find the latest Headers? > (Sorry for the dumb question) https://openfabrics.org/svn/gen2/trunk/src/ https://openfabrics.org/svn/gen2/trunk/src/userspace/libibcm https://openfabrics.org/svn/gen2/trunk/src/userspace/librdmacm https://openfabrics.org/svn/gen2/trunk/src/linux-kernel/infiniband/ - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] libibcm can't open /dev/infiniband/ucm0
Bub Thomas wrote: > It seems as if the problem I had there was not in my code but the > libibcm not being able to open the device /dev/infiniband/ucm0. You will need to load ib_ucm, which exports the IB CM to userspace. - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 13/13] IB/ehca: makefiles/kconfig
[EMAIL PROTECTED] wrote on 18.08.2006 01:34:37: > On Thursday 17 August 2006 22:11, Roland Dreier wrote: > > + > > +CFLAGS += -DEHCA_USE_HCALL -DEHCA_USE_HCALL_KERNEL > > This seems really pointless, since you're always defining these > macros to the same value. > > Just drop the CFLAGS and remove the code that depends on them > being different. Yes, that's true. Those defines are unnecessary. We'll throw them out. Thx Hoang-Nam Nguyen ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [patch] libsdp typo in config_parser
On Fri, Aug 18, 2006 at 05:54:04PM +0300, Eitan Zahavi wrote: >Committed to the trunk (not 1.1 branch) >Thanks. > Thank you. PS: I think there is another occurance in srp_daemon that i forgot to include in the diff, fwiw. PPS: IIRC the traffic sent via SDP did not show up in the packet-counters of the corresponding ipoib device last time i looked. Is this still the case? Asking because i'm seeing it accounted to my ib0 device now although libsdp's log indicates that the app is using SDP.. TIA for any hint ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH 02/13] IB/ehca: includes
abergman > > +#define EDEB_P_GENERIC(level,idstring,format,args...) \ > > These macros are responsible for 61% of the object code size of your module. > ...Please get rid of that crap entirely and replace > it with dev_info/dev_dbg/dev_warn calls where appropriate! > >Arnd <>< we'll change these EDEBs to a wrapper around dev_err, dev_dbg and dev_warn as it's done in the mthca driver. All EDEB_EN and EDEB_EX will be removed, that type of tracing can be done if needed by kprobes. There are a few cases where we won't get to a dev, for these few places we'll use a simple wrapper around printk, as done in ipoib. Hope that's the "official" way how to implement it in ib drivers. Gruss / Regards . . . Christoph R ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [patch] libsdp typo in config_parser
Committed to the trunk (not 1.1 branch) Thanks. > -Original Message- > From: [EMAIL PROTECTED] [mailto:openib-general- > [EMAIL PROTECTED] On Behalf Of Bernhard Fischer > Sent: Thursday, August 17, 2006 9:27 PM > To: openib-general@openib.org > Cc: [EMAIL PROTECTED] > Subject: [openib-general] [patch] libsdp typo in config_parser > > Hi, > > The attached trivial patch fixes a typo in the debugging output of libsdp's > config parser. > > Please apply. > > Signed-off-by: Bernhard Fischer <[EMAIL PROTECTED]> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] libibcm can't open /dev/infiniband/ucm0
Title: libibcm can't open /dev/infiniband/ucm0 I looked into my original libibcm problem today, that made me looking into cmpost, discussed in a sperate thread It seems as if the problem I had there was not in my code but the libibcm not being able to open the device /dev/infiniband/ucm0. All IB modules are installed and the modules loaded the /dev/infiniband devices available are: crw--- 1 root root 231, 64 Aug 17 15:14 issm0 crw-rw-rw- 1 root root 10, 62 Aug 17 15:14 rdma_cm crw--- 1 root root 231, 0 Aug 17 15:14 umad0 crw-rw-rw- 1 root root 231, 192 Aug 17 15:14 uverbs0 The exact error message is: libibcm: error <-1:2> opening device I’m running OFED-1.0.1 on an HP xw9300 running SLES9 SP3 in the x86_64 flawor. Thanks Thomas Bub Thomas Bub Grass Valley Germany GmbH Brunnenweg 9 64331 Weiterstadt, Germany Tel: +49 6150 104 147 Fax: +49 6150 104 656 Email: [EMAIL PROTECTED] www.GrassValley.com ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM
OK Can I still use the LID, GUID and SubnetID for connection establishment then? Then Gen1 counterpart has no IP over IB running. I'm using OFED-1.0.1. Do you have a quick link where to find the latest Headers? (Sorry for the dumb question) Thanks Thomas -Original Message- From: Sean Hefty [mailto:[EMAIL PROTECTED] Sent: Thursday, August 17, 2006 6:50 PM To: Bub Thomas Cc: Sean Hefty; openib-general@openib.org; Erez Cohen Subject: Re: [openib-general] [PATCH] cmpost: allow cmpost to build with latest RDMA CM Bub Thomas wrote: > I'm getting a little puzzled. > For me it seems as if we are moving in the wrong direction. > I don't have a RDMA CM on the Gen1 counterpart that my gen2 application > is talking too. The RDMA CM is only used on the local (active or client) side to obtain a path record, which is needed by the libibcm. Using the librdmacm allows cmpost to get a path record given only the remote IP address or host name. The connection is established using the IB CM through libibcm. > If yes you have to explain me what the two different versions: > rdma_cm.h > and > rdma_cma.h rdma_cm.h defines the kernel interface to the RDMA CM. rdma_cma.h defines the userspace interface. > The cmpost.c was using rdma_cma.h up to now but the missing defines are > located in rdma_cm.h Can you verify that you have the latest version of rdma_cma.h? - Sean ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH] huge pages support
Hello,
I've also worked on the same topic. Here is what I've done so far as I
successfully tested it on mthca and ehca. I'd appreciate for comments and
suggestions.
diff -Nurp a/drivers/infiniband/core/uverbs_mem.c
b/drivers/infiniband/core/uverbs_mem.c
--- old/drivers/infiniband/core/uverbs_mem.c2006-08-15 05:42:06.0
-0700
+++ new/drivers/infiniband/core/uverbs_mem.c2006-08-18 04:22:22.0
-0700
@@ -36,6 +36,7 @@
#include
#include
+#include
#include "uverbs.h"
@@ -73,6 +74,8 @@ int ib_umem_get(struct ib_device *dev, s
unsigned long lock_limit;
unsigned long cur_base;
unsigned long npages;
+ unsigned long region_page_mask, region_page_shift, region_page_size;
+ int use_hugepages;
int ret = 0;
int off;
int i;
@@ -84,19 +87,39 @@ int ib_umem_get(struct ib_device *dev, s
if (!page_list)
return -ENOMEM;
+ down_read(¤t->mm->mmap_sem);
+ if (is_vm_hugetlb_page(find_vma(current->mm, (unsigned long) addr))) {
+ use_hugepages = 1;
+ region_page_mask= HPAGE_MASK;
+ region_page_size= HPAGE_SIZE;
+ } else {
+ use_hugepages = 0;
+ region_page_mask= PAGE_MASK;
+ region_page_size= PAGE_SIZE;
+ }
+ up_read(¤t->mm->mmap_sem);
+
+ region_page_shift = ffs(region_page_size) - 1;
+
mem->user_base = (unsigned long) addr;
mem->length= size;
- mem->offset= (unsigned long) addr & ~PAGE_MASK;
- mem->page_size = PAGE_SIZE;
+ mem->offset= (unsigned long) addr & ~region_page_mask;
+ mem->page_size = region_page_size;
mem->writable = write;
INIT_LIST_HEAD(&mem->chunk_list);
- npages = PAGE_ALIGN(size + mem->offset) >> PAGE_SHIFT;
+ npages = ((size + mem->offset + (region_page_size - 1)) &
+ (~(region_page_size - 1))) >> region_page_shift;
down_write(¤t->mm->mmap_sem);
- locked = npages + current->mm->locked_vm;
+ if (use_hugepages)
+ locked = npages * (HPAGE_SIZE / PAGE_SIZE) +
+ current->mm->locked_vm;
+ else
+ locked = npages + current->mm->locked_vm;
+
lock_limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur >>
PAGE_SHIFT;
if ((locked > lock_limit) && !capable(CAP_IPC_LOCK)) {
@@ -104,19 +127,34 @@ int ib_umem_get(struct ib_device *dev, s
goto out;
}
- cur_base = (unsigned long) addr & PAGE_MASK;
+ cur_base = (unsigned long) addr & region_page_mask;
while (npages) {
- ret = get_user_pages(current, current->mm, cur_base,
-min_t(int, npages,
- PAGE_SIZE / sizeof (struct page *)),
-1, !write, page_list, NULL);
+ if (!use_hugepages) {
+ ret = get_user_pages(current, current->mm, cur_base,
+ min_t(int, npages, PAGE_SIZE
+ / sizeof (struct page *)),
+ 1, !write, page_list, NULL);
- if (ret < 0)
- goto out;
+ if (ret < 0)
+ goto out;
+
+ cur_base += ret * PAGE_SIZE;
+ npages -= ret;
+ } else {
+ while (npages && (ret <= PAGE_SIZE /
+ sizeof (struct page *))) {
+ if (get_user_pages(current, current->mm,
+ cur_base, 1, 1, !write,
+ &page_list[ret], NULL) < 0)
+ goto out;
+
+ ret++;
+ cur_base += HPAGE_SIZE;
+ npages--;
+ }
- cur_base += ret * PAGE_SIZE;
- npages -= ret;
+ }
off = 0;
@@ -133,7 +171,7 @@ int ib_umem_get(struct ib_device *dev, s
for (i = 0; i < chunk->nents; ++i) {
chunk->page_list[i].page = page_list[i + off];
chunk->page_list[i].offset = 0;
- chunk->page_list[i].length = PAGE_SIZE;
+ chunk->page_list[i].length = region_page_size;
}
chunk->nmap = dma_map_sg(dev->dma_device,
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please
[openib-general] How to cross-compile to ppc32 from infiniband driver in IBGD-1.8.2 ?
I am doing cross-compile to ppc440SPe (ppc 32bit) from IBGD-1.8.2. Who ever been cross-compile it? I don’t know exactly how to modify Makefile and source code Help me please. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Ib question
you can find all the required information in the IB spec.sdownload the specification from IBTAjohn t <[EMAIL PROTECTED]> wrote: Hi, In the example code there are things like: attr.max_dest_rd_atomic = 1; attr.min_rnr_timer = 12; attr.ah_attr.is_global = 0; attr.ah_attr.sl = 0; attr.ah_attr.src_path_bits = 0; attr.timeout = 14; attr.retry_cnt = 7; attr.rnr_retry = 7; attr.max_rd_atomic = 1; initattr.cap.max_recv_wr = 1; /* I guess this specifies the length of recv q */ initattr.cap.max_send_wr = tx_depth; /* I guess this specifies the length of send q */ initattr.cap.max_inline_data = MAX_INLINE; /* 400 */ attr.pkey_index = 0; What is the meaning of above fields or where can I find the definition of above fields ? Can I change the value of fields like "timeout" or should it be always set to a fixed value. Like TCP I guess there would be a state transition diagram for IB (QP state machine). Can someone point me to that? In my application, I get an error message "IBV_WC_WR_FLUSH_ERR" and sometimes "IBV_WC_RETRY_EXC_ERR" while polling a CQ after posting some write commands. What could be the reason for that ? Also, can someone point me to a document where I can find the meaning of different error status values (enum ibv_wc_status) returned by "ibv_poll_cq" Regards, John T ___openib-general mailing listopenib-general@openib.orghttp://openib.org/mailman/listinfo/openib-generalTo unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general Here's a new way to find what you're looking for - Yahoo! Answers Send FREE SMS to your friend's mobile from Yahoo! Messenger Version 8. Get it NOW___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] Ib question
Hi, In the example code there are things like: attr.max_dest_rd_atomic = 1; attr.min_rnr_timer = 12; attr.ah_attr.is_global = 0; attr.ah_attr.sl = 0; attr.ah_attr.src_path_bits = 0; attr.timeout = 14; attr.retry_cnt = 7; attr.rnr_retry = 7; attr.max_rd_atomic = 1; initattr.cap.max_recv_wr = 1; /* I guess this specifies the length of recv q */ initattr.cap.max_send_wr = tx_depth; /* I guess this specifies the length of send q */ initattr.cap.max_inline_data = MAX_INLINE; /* 400 */ attr.pkey_index = 0; What is the meaning of above fields or where can I find the definition of above fields ? Can I change the value of fields like "timeout" or should it be always set to a fixed value. Like TCP I guess there would be a state transition diagram for IB (QP state machine). Can someone point me to that? In my application, I get an error message "IBV_WC_WR_FLUSH_ERR" and sometimes "IBV_WC_RETRY_EXC_ERR" while polling a CQ after posting some write commands. What could be the reason for that ? Also, can someone point me to a document where I can find the meaning of different error status values (enum ibv_wc_status) returned by "ibv_poll_cq" Regards, John T ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
