Re: [openib-general] Fork issues with simple MPI program
Arlin Davis wrote: > Any insight would be greatly appreciated. It was our assumption that the > parent process can continue > to use IB resources after the fixes went into 2.6.16 and OFED 1.1. Is this > true? As was discussed over this list in few occasions: in contrast to popular thought the fork support was deployed in libibverbs1.1 where OFED 1.1 contains libibverbs1.0 Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] ofa_1_2_kernel 20070220-0200 daily build status
This email was generated automatically, please do not reply Common build parameters: --with-ipoib-mod --with-sdp-mod --with-srp-mod --with-user_mad-mod --with-user_access-mod --with-mthca-mod --with-core-mod --with-addr_trans-mod --with-cxgb3-mod Passed: Passed on i686 with 2.6.15-23-server Passed on i686 with linux-2.6.17 Passed on i686 with linux-2.6.14 Passed on i686 with linux-2.6.13 Passed on i686 with linux-2.6.16 Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.12 Passed on i686 with linux-2.6.15 Passed on i686 with linux-2.6.19 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.16 Passed on ppc64 with linux-2.6.19 Passed on x86_64 with linux-2.6.14 Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.15 Passed on x86_64 with linux-2.6.19 Passed on powerpc with linux-2.6.19 Passed on powerpc with linux-2.6.18 Passed on x86_64 with linux-2.6.12 Passed on powerpc with linux-2.6.17 Passed on x86_64 with linux-2.6.17 Passed on x86_64 with linux-2.6.13 Passed on ia64 with linux-2.6.12 Passed on powerpc with linux-2.6.14 Passed on powerpc with linux-2.6.15 Passed on ppc64 with linux-2.6.15 Passed on powerpc with linux-2.6.16 Passed on powerpc with linux-2.6.12 Passed on ppc64 with linux-2.6.17 Passed on ppc64 with linux-2.6.13 Passed on ppc64 with linux-2.6.16 Passed on ppc64 with linux-2.6.14 Passed on powerpc with linux-2.6.13 Passed on ppc64 with linux-2.6.12 Passed on ppc64 with linux-2.6.18 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.13 Passed on ia64 with linux-2.6.16 Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.17 Passed on ia64 with linux-2.6.15 Passed on ia64 with linux-2.6.14 Failed: ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Port error rate detection
On Mon, 2007-02-19 at 15:53, Steven Carter wrote: > I have a Nagios module that alerts on connectivity, port errors, > speed/width problems. I would like to give it the ability to change the > severity of the alert depending on whether errors are just present or if > they are increasing faster than a specified rate. The intent is to > equip the module to keep the state of the last query and possibly > history, but I wanted to make sure that I was not re-inventing the wheel > first. Is there an attribute or utility that I am overlooking that will > help me do this? Not currently (to my knowledge). The thresholding of rate aspect is similat to what will be supported in the proposed PerfManager. -- Hal > Thanks, > > Steven. > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED 1.2 dapl and dat.conf
On Fri, 16 Feb 2007, Arlin Davis wrote:
> Doug Ledford wrote:
>
> > On Wed, 2007-02-14 at 13:26 -0800, Arlin Davis wrote:
> >
> > > Steve Wise wrote:
> > >
> > >
> > > > Currently, the dapl rpms don't install dat.conf. I think they probably
> > > > should, eh? Maybe in /etc/dat.conf
> > > >
> > > >
> > > >
> > > my specfile is setup to target sysconfdir which is typically set to
> > > `$(prefix)/etc'
> > >
> > > %{_sysconfdir}/dat.conf
> > >
> > > I am not sure how the 1.2 scripts are building the rpms. Maybe Vladimir
> > > can help explain?
> > >
> >
> > Note that this setup is problematic on multilib arches. Since the
> > dat.conf file hard codes a library path that's different for 32bit/64bit
> > arches, installing both a 32bit and 64bit dapl library is impossible
> > without munging things.
> >
> > For RHEL4U5/RHEL5 I changed the dat library to read dat.conf and
> > have two separate conf files. A probably better approach would be to
> > change the library to use a relative library name that it looks for
> > starting from the libraries own directory. Hence if the dapl library is
> > in /usr/lib, it looks in /usr/lib. Doing that would allow the
> > 32bit/64bit libraries to share the same config file.
> >
> >
> This is a good idea. I will take a look at dladdr options to set
> appropriate starting path for dapl libraries when absolute paths are
> not specified.
>
> James, do you see any issues with this approach?
Nope. The dat registry should be able to handle provider libraries at
any location in the file namespace (provided they are accessible of
course).
> Vladimir, can you tell me how the OFED 1.2 install scripts are
> handling the dat.conf?
>
> -arlin
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Port error rate detection
Hal Rosenstock wrote: > On Mon, 2007-02-19 at 15:53, Steven Carter wrote: > >> I have a Nagios module that alerts on connectivity, port errors, >> speed/width problems. I would like to give it the ability to change the >> severity of the alert depending on whether errors are just present or if >> they are increasing faster than a specified rate. The intent is to >> equip the module to keep the state of the last query and possibly >> history, but I wanted to make sure that I was not re-inventing the wheel >> first. Is there an attribute or utility that I am overlooking that will >> help me do this? >> > > Not currently (to my knowledge). The thresholding of rate aspect is > similat to what will be supported in the proposed PerfManager. > I noticed that in your RFC. How are you planning on presenting the data to other agents (e.g. Nagios, Openview, MRTG, etc.)? One comment that I should have made on your RFC is that I wonder if it is necessary to include the data analysis/reduction part. Just having a central location that collects the values and presents it via SNMP is extremely useful since there are a plethora of monitoring apps (free and commercial) that do what you are proposing. That way, a network manager can leverage existing tools currently used for monitoring Ethernet Nodes, Hosts, etc. You can still include a last change attribute with each counter so that simple utilities (like the one that I am writing) can get an idea of how quickly errors are occurring. Steven. > -- Hal > > >> Thanks, >> >> Steven. >> >> ___ >> openib-general mailing list >> openib-general@openib.org >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> >> > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] osm_vendor_ibumad: termination crash fix
On Mon, 2007-02-19 at 16:46, Sasha Khapyorsky wrote: > When OpenSM is terminated umad_receiver thread still running even after > the structures are destroyed and freed, this causes to random (but easily > reproducible) crashes. The reason is that osm_vendor_delete() does not > care about thread termination. This patch adds the receiver thread > cancellation (by using pthread_cancel() and pthread_join()) and cares to > keep have all mutexes unlocked upon termination. There is also minor > termination code consolidation - osm_vendor_port_close() function. > > Signed-off-by: Sasha Khapyorsky <[EMAIL PROTECTED]> Good find. Thanks! Applied (to both master and ofed_1_2). -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED 1.2 dapl and dat.conf
On Tue, 2007-02-20 at 09:29 -0500, James Lentini wrote:
>
> On Fri, 16 Feb 2007, Arlin Davis wrote:
>
> > Doug Ledford wrote:
> >
> > > On Wed, 2007-02-14 at 13:26 -0800, Arlin Davis wrote:
> > >
> > > > Steve Wise wrote:
> > > >
> > > >
> > > > > Currently, the dapl rpms don't install dat.conf. I think they
> > > > > probably
> > > > > should, eh? Maybe in /etc/dat.conf
> > > > >
> > > > >
> > > > >
> > > > my specfile is setup to target sysconfdir which is typically set to
> > > > `$(prefix)/etc'
> > > >
> > > > %{_sysconfdir}/dat.conf
> > > >
> > > > I am not sure how the 1.2 scripts are building the rpms. Maybe Vladimir
> > > > can help explain?
> > > >
> > >
> > > Note that this setup is problematic on multilib arches. Since the
> > > dat.conf file hard codes a library path that's different for 32bit/64bit
> > > arches, installing both a 32bit and 64bit dapl library is impossible
> > > without munging things.
> > >
> > > For RHEL4U5/RHEL5 I changed the dat library to read dat.conf and
> > > have two separate conf files. A probably better approach would be to
> > > change the library to use a relative library name that it looks for
> > > starting from the libraries own directory. Hence if the dapl library is
> > > in /usr/lib, it looks in /usr/lib. Doing that would allow the
> > > 32bit/64bit libraries to share the same config file.
> > >
> > >
> > This is a good idea. I will take a look at dladdr options to set
> > appropriate starting path for dapl libraries when absolute paths are
> > not specified.
> >
> > James, do you see any issues with this approach?
>
> Nope. The dat registry should be able to handle provider libraries at
> any location in the file namespace (provided they are accessible of
> course).
Yep. Although if you want the 64bit and 32bit dat.conf to be identical,
then the best bet would be something like putting the main library
in /usr/lib or /usr/lib64 and then doing a relative path from there to
the provider libs, such as dapl/provider/libname.so. That way, the same
filespec in the dat.conf file will find either the 64bit or 32bit
provider lib depending on whether the 64bit or 32bit main library is the
one searching for it.
> > Vladimir, can you tell me how the OFED 1.2 install scripts are
> > handling the dat.conf?
> >
> > -arlin
> >
--
Doug Ledford <[EMAIL PROTECTED]>
GPG KeyID: CFBFF194
http://people.redhat.com/dledford
Infiniband specific RPMs available at
http://people.redhat.com/dledford/Infiniband
signature.asc
Description: This is a digitally signed message part
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Port error rate detection
On Tue, 2007-02-20 at 09:44, Steven Carter wrote: > Hal Rosenstock wrote: > > On Mon, 2007-02-19 at 15:53, Steven Carter wrote: > > > >> I have a Nagios module that alerts on connectivity, port errors, > >> speed/width problems. I would like to give it the ability to change the > >> severity of the alert depending on whether errors are just present or if > >> they are increasing faster than a specified rate. The intent is to > >> equip the module to keep the state of the last query and possibly > >> history, but I wanted to make sure that I was not re-inventing the wheel > >> first. Is there an attribute or utility that I am overlooking that will > >> help me do this? > >> > > > > Not currently (to my knowledge). The thresholding of rate aspect is > > similat to what will be supported in the proposed PerfManager. > > > I noticed that in your RFC. How are you planning on presenting the data > to other agents (e.g. Nagios, Openview, MRTG, etc.)? One comment that I > should have made on your RFC is that I wonder if it is necessary to > include the data analysis/reduction part. I think it is because there is too much data to push up the tree to one manager. > Just having a central location that collects the values and presents it via > SNMP is extremely > useful since there are a plethora of monitoring apps (free and > commercial) that do what you are proposing. In general, this information can be exported via SNMP or whatever the management infrastructure is. BTW, are there SNMP MIBs for all of this information ? To my knowledge, some of these were started but never completed. Also, the MIBs were geared at the agents rather than the managers (in the PerfMgt arena). -- Hal > That way, a network manager can leverage existing tools currently used for > monitoring > Ethernet Nodes, Hosts, etc. You can still include a last change > attribute with each counter so that simple utilities (like the one that > I am writing) can get an idea of how quickly errors are occurring. > Steven. > > > -- Hal > > > > > >> Thanks, > >> > >> Steven. > >> > >> ___ > >> openib-general mailing list > >> openib-general@openib.org > >> http://openib.org/mailman/listinfo/openib-general > >> > >> To unsubscribe, please visit > >> http://openib.org/mailman/listinfo/openib-general > >> > >> > > > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED 1.2 dapl and dat.conf
> > Vladimir, can you tell me how the OFED 1.2 install scripts are
> > handling the dat.conf?
> >
> > -arlin
> >
dat.conf updated by rpmbuild process:
/usr/lib is replaced by %{_libdir} (/lib for x86, ppc, ia64 and
/lib64 otherwise).
--
Vladimir Sokolovsky <[EMAIL PROTECTED]>
Mellanox Technologies Ltd.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] librdmacm: fix bug causing failure to work with partial membership pkey
On Mon, 2007-02-19 at 01:40, Or Gerlitz wrote:
> Hi Sean,
>
> this fixes a bug which did not allow to run librdmacm apps over a node
> which is partial member of a partition. The patch takes the approach of the
> kernel ib_find_cached_pkey implementation.
>
> If you approve this, i suggest pushing it also into OFED 1.2 as a bug fix.
>
> Or.
>
> --
> The pkey extracted by the RDMA CM from the IPoIB device hardware address
> always
> has the full membership bit set. However, when looking in the pkey table the
> search must mask out the full membership bit.
>
> Signed-off-by: Or Gerlitz <[EMAIL PROTECTED]>
> Signed-off-by: Olga Shern <[EMAIL PROTECTED]>
>
> diff --git a/src/cma.c b/src/cma.c
> index c5f8cd9..9c24c6a 100644
> --- a/src/cma.c
> +++ b/src/cma.c
> @@ -661,7 +661,7 @@ static int ucma_find_pkey(struct cma_dev
>
> for (i = 0, ret = 0; !ret; i++) {
> ret = ibv_query_pkey(cma_dev->verbs, port_num, i, &chk_pkey);
> - if (!ret && pkey == chk_pkey) {
> + if ((!ret && pkey == chk_pkey) || (!ret && htons(ntohs(pkey) &
> 0x7fff) == chk_pkey)) {
Is this true for both RC and UD QPs ? I thought that at least the UD QPs
were being used for multicast in which case wouldn't full member be
required for this ?
-- Hal
> *pkey_index = (uint16_t) i;
> return 0;
> }
>
> ___
> openib-general mailing list
> openib-general@openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] librdmacm: fix bug causing failure to work with partial membership pkey
Hal Rosenstock wrote: >> The pkey extracted by the RDMA CM from the IPoIB device hardware address >> always >> has the full membership bit set. However, when looking in the pkey table the >> search must mask out the full membership bit. > Is this true for both RC and UD QPs ? I thought that at least the UD QPs > were being used for multicast in which case wouldn't full member be > required for this ? Yes. Its a little bit confusing: partial and full members of an IPoIB IB partition use the same MGID. When an IPoIB MGID is constructed, the pkey placed by the driver is --always-- the full membership one. However, on a node with partial membership, what's plugged into the QP is the pkey index of the partial instance... In the kernel all this is nicely hidden from the IB ULPs in ib_find_cached_pkey(). Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] OFED 1.2 dapl and dat.conf
On Tue, 2007-02-20 at 10:21 -0500, Doug Ledford wrote:
> On Tue, 2007-02-20 at 17:05 +0200, Vladimir Sokolovsky wrote:
> > > > Vladimir, can you tell me how the OFED 1.2 install scripts are
> > > > handling the dat.conf?
> > > >
> > > > -arlin
> > > >
> >
> > dat.conf updated by rpmbuild process:
> > /usr/lib is replaced by %{_libdir} (/lib for x86, ppc, ia64 and
> > /lib64 otherwise).
>
> Which creates a multilib regression, aka when you install both the i386
> and x86_64 versions of the dapl rpm, they both contain a dat.conf file
> at the same location in the filesystem, but with different contents.
> Whether you get the 32bit or 64bit version of the dat.conf file depends
> on which is installed later. Correspondingly, whichever version of the
> library was installed first will be rendered inoperative by this problem
> as it will be either a 32 or 64bit library that is searching for a
> provider library, and the one it finds will be the opposite arch type of
> itself, thereby preventing the dapl library from doing a dlopen on the
> file. Therefore, whatever version of the dapl library is installed
> first will no longer be able to find any valid provider libraries. This
> is considered an error condition by our automated package testing tools
> and we are not allowed to ship a package in this state.
>
I can create /etc/dat32.conf and /etc/dat64.conf.
Currently, in the OFED there is no separation to 32 and 64 bit RPMs.
That is on x86_64, fot example, if 32bit libraries compilation succeeded
then both 32 and 64bit libraries will be a part of the same RPM.
--
Vladimir Sokolovsky <[EMAIL PROTECTED]>
Mellanox Technologies Ltd.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] librdmacm: fix bug causing failure to work with partial membership pkey
On Tue, 2007-02-20 at 10:38, Or Gerlitz wrote: > Hal Rosenstock wrote: > > >> The pkey extracted by the RDMA CM from the IPoIB device hardware address > >> always > >> has the full membership bit set. However, when looking in the pkey table > >> the > >> search must mask out the full membership bit. > > > Is this true for both RC and UD QPs ? I thought that at least the UD QPs > > were being used for multicast in which case wouldn't full member be > > required for this ? > > Yes. Its a little bit confusing: partial and full members of an IPoIB IB > partition use the same MGID. When an IPoIB MGID is constructed, the pkey > placed by the driver is --always-- the full membership one. However, on > a node with partial membership, what's plugged into the QP is the pkey > index of the partial instance... So in this case, do both the full and partial keys need configuring for that port ? -- Hal > In the kernel all this is nicely hidden from the IB ULPs in > ib_find_cached_pkey(). > > Or. > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [2.6 patch] drivers/infiniband/hw/cxgb3/: possible cleanups
On Tue, 2007-02-20 at 01:02 +0100, Adrian Bunk wrote:
> This patch contains the following possible cleanups:
> - don't mark static functions in C files as inline - gcc should know
> best whether inlining makes sense
> - never compile the unused cxio_dbg.c
> - make the following needlessly global functions static:
> - cxio_hal.c: cxio_hal_clear_qp_ctx()
> - iwch_provider.c: iwch_get_qp()
> - #if 0 the following unused global functions:
> - cxio_hal.c: cxio_allocate_stag()
> - cxio_resource.: cxio_hal_get_rhdl()
> - cxio_resource.: cxio_hal_put_rhdl()
>
You could just remove the code instead of #if 0...
> Signed-off-by: Adrian Bunk <[EMAIL PROTECTED]>
>
> ---
>
> drivers/infiniband/hw/cxgb3/Makefile|1
> drivers/infiniband/hw/cxgb3/cxio_hal.c | 22 +++
> drivers/infiniband/hw/cxgb3/cxio_hal.h |5 ---
> drivers/infiniband/hw/cxgb3/cxio_resource.c |8 -
> drivers/infiniband/hw/cxgb3/iwch_cm.c |5 +--
> drivers/infiniband/hw/cxgb3/iwch_provider.c |2 -
> drivers/infiniband/hw/cxgb3/iwch_provider.h |1
> drivers/infiniband/hw/cxgb3/iwch_qp.c | 29
> 8 files changed, 33 insertions(+), 40 deletions(-)
>
> --- linux-2.6.20-mm1/drivers/infiniband/hw/cxgb3/Makefile.old 2007-02-17
> 17:21:03.0 +0100
> +++ linux-2.6.20-mm1/drivers/infiniband/hw/cxgb3/Makefile 2007-02-17
> 17:21:08.0 +0100
> @@ -8,5 +8,4 @@
>
> ifdef CONFIG_INFINIBAND_CXGB3_DEBUG
> EXTRA_CFLAGS += -DDEBUG
> -iw_cxgb3-y += cxio_dbg.o
> endif
> --- linux-2.6.20-mm1/drivers/infiniband/hw/cxgb3/cxio_hal.h.old
> 2007-02-17 17:22:53.0 +0100
> +++ linux-2.6.20-mm1/drivers/infiniband/hw/cxgb3/cxio_hal.h 2007-02-17
> 17:25:08.0 +0100
> @@ -144,7 +144,6 @@
> void cxio_rdev_close(struct cxio_rdev *rdev);
> int cxio_hal_cq_op(struct cxio_rdev *rdev, struct t3_cq *cq,
> enum t3_cq_opcode op, u32 credit);
> -int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev, u32 qpid);
> int cxio_create_cq(struct cxio_rdev *rdev, struct t3_cq *cq);
> int cxio_destroy_cq(struct cxio_rdev *rdev, struct t3_cq *cq);
> int cxio_resize_cq(struct cxio_rdev *rdev, struct t3_cq *cq);
> @@ -155,8 +154,6 @@
> int cxio_destroy_qp(struct cxio_rdev *rdev, struct t3_wq *wq,
> struct cxio_ucontext *uctx);
> int cxio_peek_cq(struct t3_wq *wr, struct t3_cq *cq, int opcode);
> -int cxio_allocate_stag(struct cxio_rdev *rdev, u32 * stag, u32 pdid,
> -enum tpt_mem_perm perm, u32 * pbl_size, u32 * pbl_addr);
> int cxio_register_phys_mem(struct cxio_rdev *rdev, u32 * stag, u32 pdid,
> enum tpt_mem_perm perm, u32 zbva, u64 to, u32 len,
> u8 page_size, __be64 *pbl, u32 *pbl_size,
> @@ -172,8 +169,6 @@
> int cxio_rdma_init(struct cxio_rdev *rdev, struct t3_rdma_init_attr *attr);
> void cxio_register_ev_cb(cxio_hal_ev_callback_func_t ev_cb);
> void cxio_unregister_ev_cb(cxio_hal_ev_callback_func_t ev_cb);
> -u32 cxio_hal_get_rhdl(void);
> -void cxio_hal_put_rhdl(u32 rhdl);
> u32 cxio_hal_get_pdid(struct cxio_hal_resource *rscp);
> void cxio_hal_put_pdid(struct cxio_hal_resource *rscp, u32 pdid);
> int __init cxio_hal_init(void);
> --- linux-2.6.20-mm1/drivers/infiniband/hw/cxgb3/cxio_hal.c.old
> 2007-02-17 17:23:11.0 +0100
> +++ linux-2.6.20-mm1/drivers/infiniband/hw/cxgb3/cxio_hal.c 2007-02-17
> 17:36:40.0 +0100
> @@ -46,7 +46,7 @@
> static LIST_HEAD(rdev_list);
> static cxio_hal_ev_callback_func_t cxio_ev_cb = NULL;
>
> -static inline struct cxio_rdev *cxio_hal_find_rdev_by_name(char *dev_name)
> +static struct cxio_rdev *cxio_hal_find_rdev_by_name(char *dev_name)
> {
> struct cxio_rdev *rdev;
>
> @@ -56,8 +56,7 @@
> return NULL;
> }
>
> -static inline struct cxio_rdev *cxio_hal_find_rdev_by_t3cdev(struct t3cdev
> - *tdev)
> +static struct cxio_rdev *cxio_hal_find_rdev_by_t3cdev(struct t3cdev *tdev)
> {
> struct cxio_rdev *rdev;
>
> @@ -119,7 +118,7 @@
> return 0;
> }
>
> -static inline int cxio_hal_clear_cq_ctx(struct cxio_rdev *rdev_p, u32 cqid)
> +static int cxio_hal_clear_cq_ctx(struct cxio_rdev *rdev_p, u32 cqid)
> {
> struct rdma_cq_setup setup;
> setup.id = cqid;
> @@ -131,7 +130,7 @@
> return (rdev_p->t3cdev_p->ctl(rdev_p->t3cdev_p, RDMA_CQ_SETUP, &setup));
> }
>
> -int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev_p, u32 qpid)
> +static int cxio_hal_clear_qp_ctx(struct cxio_rdev *rdev_p, u32 qpid)
> {
> u64 sge_cmd;
> struct t3_modify_qp_wr *wqe;
> @@ -426,7 +425,7 @@
> }
> }
>
> -static inline int cqe_completes_wr(struct t3_cqe *cqe, struct t3_wq *wq)
> +static int cqe_completes_wr(struct t3_cqe *cqe, struct t3_wq *wq)
> {
> if (CQE_OPCODE(*cqe) == T3_TERMINATE)
> return 0;
> @@ -761,6 +760,7 @@
> return err;
> }
>
> +#if 0
Re: [openib-general] OFED 1.2 dapl and dat.conf
On Tue, 2007-02-20 at 17:44 +0200, Vladimir Sokolovsky wrote:
> On Tue, 2007-02-20 at 10:21 -0500, Doug Ledford wrote:
> > On Tue, 2007-02-20 at 17:05 +0200, Vladimir Sokolovsky wrote:
> > > > > Vladimir, can you tell me how the OFED 1.2 install scripts are
> > > > > handling the dat.conf?
> > > > >
> > > > > -arlin
> > > > >
> > >
> > > dat.conf updated by rpmbuild process:
> > > /usr/lib is replaced by %{_libdir} (/lib for x86, ppc, ia64 and
> > > /lib64 otherwise).
> >
> > Which creates a multilib regression, aka when you install both the i386
> > and x86_64 versions of the dapl rpm, they both contain a dat.conf file
> > at the same location in the filesystem, but with different contents.
> > Whether you get the 32bit or 64bit version of the dat.conf file depends
> > on which is installed later. Correspondingly, whichever version of the
> > library was installed first will be rendered inoperative by this problem
> > as it will be either a 32 or 64bit library that is searching for a
> > provider library, and the one it finds will be the opposite arch type of
> > itself, thereby preventing the dapl library from doing a dlopen on the
> > file. Therefore, whatever version of the dapl library is installed
> > first will no longer be able to find any valid provider libraries. This
> > is considered an error condition by our automated package testing tools
> > and we are not allowed to ship a package in this state.
> >
> I can create /etc/dat32.conf and /etc/dat64.conf.
That's pretty much what I did for our next release, but it's crude. The
other solution we've been discussing would be far preferable.
> Currently, in the OFED there is no separation to 32 and 64 bit RPMs.
> That is on x86_64, fot example, if 32bit libraries compilation succeeded
> then both 32 and 64bit libraries will be a part of the same RPM.
Assuming you actually built both 32 and 64bit dapl libraries, them being
in the same rpm wouldn't solve the problem that the generated dat.conf
would only be correct for one or the other, not for both. And if you
want to create a dat32.conf and dat64.conf, then you need to munge the
dapl source code so that during a 64bit build it looks for dat64.conf
and in a 32bit build it looks for dat32.conf. However, you need to
munge the source code in such a way as to have it be the same on both
32bit and 64bit builds, aka something like:
#ifdef __i386__
default_dapl_file = "/usr/local/ofed/etc/dat32.conf";
#else
default_dapl_file = "/usr/local/ofed/etc/dat64.conf";
#endif
If you make the mistake I made, which was to do one patch on 32bit
arches and a different patch on 64bit arches, then the source code
between the 32 and 64bit arches differs, and guess what, that throws a
multilib regression as well because it breaks debuginfo packages :-/
I'll fix that in our next release.
--
Doug Ledford <[EMAIL PROTECTED]>
GPG KeyID: CFBFF194
http://people.redhat.com/dledford
Infiniband specific RPMs available at
http://people.redhat.com/dledford/Infiniband
signature.asc
Description: This is a digitally signed message part
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [PATCH for-2.6.21] IPoIB/cm: improve small message bandwidth
Avoid overhead of freeing/reallocating and mapping/unmapping for dma
for pages that have not been written to by hardware.
Signed-off-by: Michael S. Tsirkin <[EMAIL PROTECTED]>
---
This gives >10% boost in BW for message sizes up to 32K. Please queue for
2.6.21.
before:
# ./netperf-2.4.2/src/netperf -f M -H 11.4.3.68 -c -C -- -m 32000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68 (11.4.3.68)
port 0 AF_INET : demo
Recv SendSend Utilization Service Demand
Socket Socket Message Elapsed Send Recv SendRecv
Size SizeSize Time Throughput localremote local remote
bytes bytes bytessecs.MBytes /s % S % S us/KB us/KB
87380 16384 3200010.00 716.23 26.2223.941.430 1.306
after:
# ./netperf-2.4.2/src/netperf -f M -H 11.4.3.68 -c -C -- -m 32000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 11.4.3.68 (11.4.3.68)
port 0 AF_INET : demo
Recv SendSend Utilization Service Demand
Socket Socket Message Elapsed Send Recv SendRecv
Size SizeSize Time Throughput localremote local remote
bytes bytes bytessecs.MBytes /s % S % S us/KB us/KB
87380 16384 3200010.00 888.67 24.1325.081.061 1.102
diff --git a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
index 8ee6f06..a23c8e3 100644
--- a/drivers/infiniband/ulp/ipoib/ipoib_cm.c
+++ b/drivers/infiniband/ulp/ipoib/ipoib_cm.c
@@ -68,14 +68,14 @@ struct ipoib_cm_id {
static int ipoib_cm_tx_handler(struct ib_cm_id *cm_id,
struct ib_cm_event *event);
-static void ipoib_cm_dma_unmap_rx(struct ipoib_dev_priv *priv,
+static void ipoib_cm_dma_unmap_rx(struct ipoib_dev_priv *priv, int frags,
u64 mapping[IPOIB_CM_RX_SG])
{
int i;
ib_dma_unmap_single(priv->ca, mapping[0], IPOIB_CM_HEAD_SIZE,
DMA_FROM_DEVICE);
- for (i = 0; i < IPOIB_CM_RX_SG - 1; ++i)
+ for (i = 0; i < frags; ++i)
ib_dma_unmap_single(priv->ca, mapping[i + 1], PAGE_SIZE,
DMA_FROM_DEVICE);
}
@@ -93,7 +93,8 @@ static int ipoib_cm_post_receive(struct net_device *dev, int
id)
ret = ib_post_srq_recv(priv->cm.srq, &priv->cm.rx_wr, &bad_wr);
if (unlikely(ret)) {
ipoib_warn(priv, "post srq failed for buf %d (%d)\n", id, ret);
- ipoib_cm_dma_unmap_rx(priv, priv->cm.srq_ring[id].mapping);
+ ipoib_cm_dma_unmap_rx(priv, IPOIB_CM_RX_SG - 1,
+ priv->cm.srq_ring[id].mapping);
dev_kfree_skb_any(priv->cm.srq_ring[id].skb);
priv->cm.srq_ring[id].skb = NULL;
}
@@ -101,8 +102,8 @@ static int ipoib_cm_post_receive(struct net_device *dev,
int id)
return ret;
}
-static int ipoib_cm_alloc_rx_skb(struct net_device *dev, int id,
-u64 mapping[IPOIB_CM_RX_SG])
+static struct sk_buff *ipoib_cm_alloc_rx_skb(struct net_device *dev, int id,
int frags,
+u64 mapping[IPOIB_CM_RX_SG])
{
struct ipoib_dev_priv *priv = netdev_priv(dev);
struct sk_buff *skb;
@@ -110,7 +111,7 @@ static int ipoib_cm_alloc_rx_skb(struct net_device *dev,
int id,
skb = dev_alloc_skb(IPOIB_CM_HEAD_SIZE + 12);
if (unlikely(!skb))
- return -ENOMEM;
+ return NULL;
/*
* IPoIB adds a 4 byte header. So we need 12 more bytes to align the
@@ -122,10 +123,10 @@ static int ipoib_cm_alloc_rx_skb(struct net_device *dev,
int id,
DMA_FROM_DEVICE);
if (unlikely(ib_dma_mapping_error(priv->ca, mapping[0]))) {
dev_kfree_skb_any(skb);
- return -EIO;
+ return NULL;
}
- for (i = 0; i < IPOIB_CM_RX_SG - 1; i++) {
+ for (i = 0; i < frags; i++) {
struct page *page = alloc_page(GFP_ATOMIC);
if (!page)
@@ -139,7 +140,7 @@ static int ipoib_cm_alloc_rx_skb(struct net_device *dev,
int id,
}
priv->cm.srq_ring[id].skb = skb;
- return 0;
+ return skb;
partial_error:
@@ -148,8 +149,8 @@ partial_error:
for (; i >= 0; --i)
ib_dma_unmap_single(priv->ca, mapping[i + 1], PAGE_SIZE,
DMA_FROM_DEVICE);
- dev_kfree_skb_any(skb);
- return -ENOMEM;
+ dev_kfree_skb_any(skb);
+ return NULL;
}
static struct ib_qp *ipoib_cm_create_rx_qp(struct net_device *dev,
@@ -312,7 +313,7 @@ static int ipoib_cm_rx_handler(struct ib_cm_id *cm_id,
}
/* Adjust length of skb with fragments to match received data */
static void skb_put_frags(struct sk_buff *skb, unsigned int hdr_space,
- unsigned int length)
+
Re: [openib-general] [Fwd: Re: win related [was: Re: [PATCH 1/2] opensm: sigusr1: syslog() fixes]]
Hi Tzachi, On Thu, 2007-02-08 at 16:24, Tzachi Dar wrote: > See bellow. I would like to get back to trying to close on this discussion. > Thanks > Tzachi > > > -Original Message- > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > Sent: Thursday, February 08, 2007 9:47 PM > > To: Tzachi Dar > > Cc: Yossi Leybovich; Gilad Shainer; Yevgeny Kliteynik; > > OPENIB; Michael S. Tsirkin; Hal Rosenstock > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > opensm: sigusr1: syslog() fixes]] > > > > On 20:31 Thu 08 Feb , Tzachi Dar wrote: > > > The windows open IB has decided on using a BSD only license. > > > The common implementation of pthreads as far as I know is > > LGPL, which > > > means that it can not be used in open IB. > > > > Why not? AFAIK it works perfectly (see (5,6 and Preamble)): > > http://www.gnu.org/copyleft/lesser.html > > > > And of course there are tons of examples when BSD software > > links against LGPLed glibc. > > I can of course write you an answer that will be more than 5 pages long > of why *I* don't think that > Using GPL software is bad for everyone, but I guess that my opinion > doesn't really meter, so I > Won't do it. > The page that you have referenced is of the GNU org, and even there it > is hard to say that they > are trying to encourage you to use the LGPL license. In any case, the > main point is that > When open IB windows was formed there was a general decision that it > will use BSD license. If we > Start having components with the LGPL this will break that decision, and > therefore this requires > some voting of the open IB organization. I may be missing your point but is there something in the Windows OpenIB/OpenFabrics license that precludes using Windows OpenIB licensed code (e.g. BSD like license) in concert with non OpenIB code (like LGPL) ? Isn't that essentially what using the Windows pthreads DLL with OpenSM would be like ? As I understand it, I don't think this requires a license change or anything in the OpenIB Windows charter prevents this or needs changing. > > > The only two ways that I see around this are 1) Change the > > license of > > > open IB windows which might be a complicated thing. 2) Find an > > > implementation of pthreads that is BSD. > > > > BTW, just wondering... What is relation between windows open > > IB and OFA (and OFA's "dual-license rule")? > Well, the way I see it one can take code from the Linux part under the > BSD licance and use it in > The windows part. The otherway around seems fine to me but some say that > since the windows BSD liscance > Reqires that some text will always remain there, the other way around is > not possibale. As I'm not an > Expert in that erea I don't know who is right. I don't see how this affects what is being discussed about OpenSM. In all the cases I'm aware of, the portability is from Linux to Windows and not the other way around. -- Hal > > Sasha > > > > > > > > Thanks > > > Tzachi > > > > > > > -Original Message- > > > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > > > Sent: Thursday, February 08, 2007 7:46 PM > > > > To: Tzachi Dar; Yossi Leybovich > > > > Cc: Yevgeny Kliteynik; OPENIB; Michael S. Tsirkin; Hal Rosenstock > > > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > > > opensm: sigusr1: syslog() fixes]] > > > > > > > > On 11:24 Sun 21 Jan , Yevgeny Kliteynik wrote: > > > > > Tzachi, Yossi, please join the thread. > > > > > What do you think about distributing a copy of the pthread DLL > > > > > with opensm? > > > > > > > > Any news here? Thanks. > > > > > > > > Sasha > > > > > > > > > > > > > > -- Yevgeny. > > > > > > > > > > Original Message > > > > > Subject: Re: win related [was: Re: [PATCH 1/2] opensm: sigusr1: > > > > > syslog() fixes] > > > > > Date: Fri, 19 Jan 2007 00:20:32 +0200 > > > > > From: Sasha Khapyorsky <[EMAIL PROTECTED]> > > > > > To: Michael S. Tsirkin <[EMAIL PROTECTED]> > > > > > CC: Yevgeny Kliteynik <[EMAIL PROTECTED]>, > > > > OPENIB > > > > > References: <[EMAIL PROTECTED]> > > > > > <[EMAIL PROTECTED]> > > > > > > > > > > On 23:50 Thu 18 Jan , Michael S. Tsirkin wrote: > > > > > > > Quoting Sasha Khapyorsky <[EMAIL PROTECTED]>: > > > > > > > Subject: Re: win related [was: Re: [PATCH 1/2] > > opensm: sigusr1: > > > > > > > syslog() fixes] > > > > > > > > > > > > > > On 07:00 Thu 18 Jan , Michael S. Tsirkin wrote: > > > > > > > > > What about pure opensource - > > > > > > > > > http://sourceware.org/pthreads-win32/? It is licensed > > > > > > > > > under LGPL, I see on the net many positive reports about > > > > stability and usability. > > > > > > > > > > > > > > > > I used it to do a windows port of linux complib at some > > > > > > > > point and opensm seemed to work fine with it. What it was > > > > lacking at > > > > > > > > that point was support for 64 bit applications, > > and for some > > > > > > > > reason (which i
Re: [openib-general] Port error rate detection
On Tue, 2007-02-20 at 10:25, Steven Carter wrote: > Hal Rosenstock wrote: > > On Tue, 2007-02-20 at 09:44, Steven Carter wrote: > > > >> Hal Rosenstock wrote: > >> > >>> On Mon, 2007-02-19 at 15:53, Steven Carter wrote: > >>> > >>> > I have a Nagios module that alerts on connectivity, port errors, > speed/width problems. I would like to give it the ability to change the > severity of the alert depending on whether errors are just present or if > they are increasing faster than a specified rate. The intent is to > equip the module to keep the state of the last query and possibly > history, but I wanted to make sure that I was not re-inventing the wheel > first. Is there an attribute or utility that I am overlooking that will > help me do this? > > > >>> Not currently (to my knowledge). The thresholding of rate aspect is > >>> similat to what will be supported in the proposed PerfManager. > >>> > >>> > >> I noticed that in your RFC. How are you planning on presenting the data > >> to other agents (e.g. Nagios, Openview, MRTG, etc.)? One comment that I > >> should have made on your RFC is that I wonder if it is necessary to > >> include the data analysis/reduction part. > >> > > > > I think it is because there is too much data to push up the tree to one > > manager. > > > I agree, but does the data need to be pushed to one node? If you go > with a distributed approach where information is aggregated per network > device (switch or group of switches), The proposal includes a distributed approach. > then a third-party monitoring > server can collect and present it in the same way that it does for an > Ethernet network. That way, you do not need to pass information up to a > central node. You can just have a third party monitoring application > collect and present the information. I guess it just depends on how > much you want to leverage existing monitoring solutions and/or how much > capability you want inherent in the OFA software. Third party monitoring agents can hook in at the intermediate nodes in the collection hierarchy if that is what is desired. > >> Just having a central location that collects the values and presents it > >> via SNMP is extremely > >> useful since there are a plethora of monitoring apps (free and > >> commercial) that do what you are proposing. > >> > I should have said 'a location' and not 'a central location'. Since > most monitoring applications support multiple agents, it is not > necessary to aggregate the information into one place. > > > > In general, this information can be exported via SNMP or whatever the > > management infrastructure is. > > > > BTW, are there SNMP MIBs for all of this information ? To my knowledge, > > some of these were started but never completed. Also, the MIBs were > > geared at the agents rather than the managers (in the PerfMgt arena). > > > There are standard MIBS (e.g. mib-2's ifTable) that can present most of > the useful information (in/out octets, errors, etc.) Not most of the useful IB information. > , but I would suspect that you would have to supplement that with a private > MIB as > most other technologies/vendors have. Yes, as this may be data out of a non IBTA specified manager, it is likely a private MIB unless one goes for all the agent (PMA) data. There was a proposed MIB for the PMA at the IETF IPoIB WG. -- Hal > Steven. > > > -- Hal > > > > > >> That way, a network manager can leverage existing tools currently used for > >> monitoring > >> Ethernet Nodes, Hosts, etc. You can still include a last change > >> attribute with each counter so that simple utilities (like the one that > >> I am writing) can get an idea of how quickly errors are occurring. > >> > > > > > >> Steven. > >> > >> > >>> -- Hal > >>> > >>> > >>> > Thanks, > > Steven. > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > > > >>> > >>> > > > > > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IB routing discussion summary
At 02:05 PM 2/15/2007, Sean Hefty wrote: >>Is this first an IBTA problem to solve if you believe there is a problem? > >Based on my interpretation, I do not believe that there's an error in the >architecture. It seems consistent. Additional clarification of what >PathRecord fields mean when the GIDs are on different subnets may be >needed, and a change to the architecture may make things easier to >implement, but that's a separate matter. > >>I contend CM does not require anything that is subnet local other than to >>target a given router port which should be derived from local SM/SA only > >Then please state how the passive side obtains the information (e.g. >SLID/DLID) it needs in order to configure its QP. I claim that >information is carried in the CM REQ. It should not be carried in the CM REQ. The SLID / DLID of the router ports should be derived through local subnet SA / SM query. When a CM REQ traverses one or more subnets there will be potentially many SLID / DLID involved in the communication. Each router should be populating its routing tables in order to build the new LRH attached to the GRH / CM REQ that it is forwarding to the next hop. >The alternatives that I see are: > >1. The passive side extracts the data from the LRH that carries the CM REQ. >2. The passive side issues its own local path record query. > >Will you please clarify where this information comes from? The router protocol determines path to the next hop. As noted in prior e-mails, the router works in conjunction with the SM/SA to populate its database so that any CM or other query for a path record to get to / from the router can be derived and optimized based on local policy, e.g. QoS, within each subnet. >>I will further state that SA-SA communication sans perhaps a >>P_Key / Q_Key service lookup should be avoided wherever possible. > >I agree - which is why my proposal avoided SA-SA communication. I see >nothing in the architecture that prohibits a node from querying an SA that >is not on its local subnet. I'd need to go back but the architecture is predicated that the SM and SA are strictly local and for security purposes their communication should remain local. Higher level management entities built to communicate with SM and SA are responsible for cross subnet communications without exposing the SA or SM to direct interaction. P_Key and Q_Key management across subnets is an example of such communication across subnets that would not be exposed to the SA and SM. Mike ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
>Arlin Davis wrote: >> Any insight would be greatly appreciated. It was our assumption that the >> parent process can >continue >> to use IB resources after the fixes went into 2.6.16 and OFED 1.1. Is this >> true? > >As was discussed over this list in few occasions: in contrast to popular >thought the fork support was deployed in libibverbs1.1 where OFED 1.1 >contains libibverbs1.0 OFED 1.2 alpha (libibverbs 1.1) on 2.6.20 fails the same way. Does the following disclaimer still apply? "Fork support from kernel 2.6.12 and above is available provided that applications do not use threads. The fork() is supported as long as parent process does not run before child exits or calls exec(). The former can be achieved by calling wait(childpid) the later can be achieved by application specific means. Posix system() call is supported." ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
Or Gerlitz wrote: > Arlin Davis wrote: > >> Any insight would be greatly appreciated. It was our assumption that the >> parent process can continue >> to use IB resources after the fixes went into 2.6.16 and OFED 1.1. Is this >> true? >> > > As was discussed over this list in few occasions: in contrast to popular > thought the fork support was deployed in libibverbs1.1 where OFED 1.1 > contains libibverbs1.0 > > Or. > > > The only fork support in OFED 1.1 is system() or fork & exec. Note that the support in OFED 1.2 (actually changes in libibverbs 1.1) needs some change in the application. Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
Arlin Davis wrote: > > OFED 1.2 alpha (libibverbs 1.1) on 2.6.20 fails the same way. Does the > following disclaimer still > apply? > > "Fork support from kernel 2.6.12 and above is available provided > that applications do not use threads. The fork() is supported as long > as parent process does not run before child exits or calls exec(). > The former can be achieved by calling wait(childpid) the later can be > achieved by application specific means. Posix system() call is > supported." > > As replied before - if you want full fork support you need to change the application. Look at the verbs header for details. Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
> As replied before - if you want full fork support you need to change the > application. Look at the verbs header for details. Or you could try setting the IBV_FORK_SAFE environment variable before running your application. I guess for MPI jobs you need to make sure that environment variable is propagated to every process. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
> >Or you could try setting the IBV_FORK_SAFE environment variable before >running your application. I guess for MPI jobs you need to make sure >that environment variable is propagated to every process. Ahh! That's what I was looking for. Thanks! This information is scattered around in various email threads, header files, and code. Can someone please add relevant text to the OFED 1.2 release notes or a Wiki page? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
Arlin Davis wrote: >> Or you could try setting the IBV_FORK_SAFE environment variable before >> running your application. I guess for MPI jobs you need to make sure >> that environment variable is propagated to every process. >> > > Ahh! That's what I was looking for. Thanks! > > This information is scattered around in various email threads, header files, > and code. Can someone > please add relevant text to the OFED 1.2 release notes or a Wiki page? > Roland, If you can send me the details (since you implemented it) I will add it to the Wiki Thanks, Tziporet ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
> If you can send me the details (since you implemented it) I will add > it to the Wiki An application that wants fork() to work with libibverbs should either call ibv_fork_init() before doing anything else with libibverbs, or else a user can set the IBV_FORK_SAFE or RDMAV_FORK_SAFE environment variable to get the same effect. There is some overhead to making fork() work so it is not enabled by default. This is described in the ibv_fork_init manpage in the latest libibverbs git tree. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [Fwd: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] opensm: sigusr1: syslog() fixes]]]
Also, looping in the OpenFabrics Windows email list on this. -- Hal -Forwarded Message- From: Hal Rosenstock <[EMAIL PROTECTED]> To: Tzachi Dar <[EMAIL PROTECTED]> Cc: OPENIB , Gilad Shainer <[EMAIL PROTECTED]> Subject: Re: [openib-general] [Fwd: Re: win related [was: Re: [PATCH 1/2] opensm: sigusr1: syslog() fixes]] Date: 20 Feb 2007 13:21:38 -0500 Hi Tzachi, On Thu, 2007-02-08 at 16:24, Tzachi Dar wrote: > See bellow. I would like to get back to trying to close on this discussion. > Thanks > Tzachi > > > -Original Message- > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > Sent: Thursday, February 08, 2007 9:47 PM > > To: Tzachi Dar > > Cc: Yossi Leybovich; Gilad Shainer; Yevgeny Kliteynik; > > OPENIB; Michael S. Tsirkin; Hal Rosenstock > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > opensm: sigusr1: syslog() fixes]] > > > > On 20:31 Thu 08 Feb , Tzachi Dar wrote: > > > The windows open IB has decided on using a BSD only license. > > > The common implementation of pthreads as far as I know is > > LGPL, which > > > means that it can not be used in open IB. > > > > Why not? AFAIK it works perfectly (see (5,6 and Preamble)): > > http://www.gnu.org/copyleft/lesser.html > > > > And of course there are tons of examples when BSD software > > links against LGPLed glibc. > > I can of course write you an answer that will be more than 5 pages long > of why *I* don't think that > Using GPL software is bad for everyone, but I guess that my opinion > doesn't really meter, so I > Won't do it. > The page that you have referenced is of the GNU org, and even there it > is hard to say that they > are trying to encourage you to use the LGPL license. In any case, the > main point is that > When open IB windows was formed there was a general decision that it > will use BSD license. If we > Start having components with the LGPL this will break that decision, and > therefore this requires > some voting of the open IB organization. I may be missing your point but is there something in the Windows OpenIB/OpenFabrics license that precludes using Windows OpenIB licensed code (e.g. BSD like license) in concert with non OpenIB code (like LGPL) ? Isn't that essentially what using the Windows pthreads DLL with OpenSM would be like ? As I understand it, I don't think this requires a license change or anything in the OpenIB Windows charter prevents this or needs changing. > > > The only two ways that I see around this are 1) Change the > > license of > > > open IB windows which might be a complicated thing. 2) Find an > > > implementation of pthreads that is BSD. > > > > BTW, just wondering... What is relation between windows open > > IB and OFA (and OFA's "dual-license rule")? > Well, the way I see it one can take code from the Linux part under the > BSD licance and use it in > The windows part. The otherway around seems fine to me but some say that > since the windows BSD liscance > Reqires that some text will always remain there, the other way around is > not possibale. As I'm not an > Expert in that erea I don't know who is right. I don't see how this affects what is being discussed about OpenSM. In all the cases I'm aware of, the portability is from Linux to Windows and not the other way around. -- Hal > > Sasha > > > > > > > > Thanks > > > Tzachi > > > > > > > -Original Message- > > > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > > > Sent: Thursday, February 08, 2007 7:46 PM > > > > To: Tzachi Dar; Yossi Leybovich > > > > Cc: Yevgeny Kliteynik; OPENIB; Michael S. Tsirkin; Hal Rosenstock > > > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > > > opensm: sigusr1: syslog() fixes]] > > > > > > > > On 11:24 Sun 21 Jan , Yevgeny Kliteynik wrote: > > > > > Tzachi, Yossi, please join the thread. > > > > > What do you think about distributing a copy of the pthread DLL > > > > > with opensm? > > > > > > > > Any news here? Thanks. > > > > > > > > Sasha > > > > > > > > > > > > > > -- Yevgeny. > > > > > > > > > > Original Message > > > > > Subject: Re: win related [was: Re: [PATCH 1/2] opensm: sigusr1: > > > > > syslog() fixes] > > > > > Date: Fri, 19 Jan 2007 00:20:32 +0200 > > > > > From: Sasha Khapyorsky <[EMAIL PROTECTED]> > > > > > To: Michael S. Tsirkin <[EMAIL PROTECTED]> > > > > > CC: Yevgeny Kliteynik <[EMAIL PROTECTED]>, > > > > OPENIB > > > > > References: <[EMAIL PROTECTED]> > > > > > <[EMAIL PROTECTED]> > > > > > > > > > > On 23:50 Thu 18 Jan , Michael S. Tsirkin wrote: > > > > > > > Quoting Sasha Khapyorsky <[EMAIL PROTECTED]>: > > > > > > > Subject: Re: win related [was: Re: [PATCH 1/2] > > opensm: sigusr1: > > > > > > > syslog() fixes] > > > > > > > > > > > > > > On 07:00 Thu 18 Jan , Michael S. Tsirkin wrote: > > > > > > > > > What about pure opensource - > > > > > > > > > http://sourceware.org/pthreads-win32/? It is licensed > > > > > > > > > under LGPL, I see on t
Re: [openib-general] [ofw] [Fwd: Re: [Fwd: Re: win related [was: Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]]
Submissions to the OFW project are supposed to be bound by the contributor's agreement: http://windows.openib.org/openib/contribute.aspx Contributing code under anything but a BSD license violates condition 1, though there shouldn't be issues with dual licenses as long as one of the available licenses is a BSD license. In any case, we're not talking about putting the pthreads library in source or binary form in the OFW SVN, right? We're just talking about having OpenSM link to the pthreads library that is out-of-tree. So the question is whether there are any licensing issues with having a BSD code include an out-of-tree LGPL file that would affect the ability to retain the BSD license on the OpenSM files. I can see this causing problems for builds, as people would need to find/install the pthreads library before OpenSM would build successfully. -Fab -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hal Rosenstock Sent: Tuesday, February 20, 2007 10:38 AM To: ofw@lists.openfabrics.org Cc: Gilad Shainer; OPENIB Subject: [ofw] [Fwd: Re: [openib-general] [Fwd: Re: win related [was: Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]] Also, looping in the OpenFabrics Windows email list on this. -- Hal -Forwarded Message- From: Hal Rosenstock <[EMAIL PROTECTED]> To: Tzachi Dar <[EMAIL PROTECTED]> Cc: OPENIB , Gilad Shainer <[EMAIL PROTECTED]> Subject: Re: [openib-general] [Fwd: Re: win related [was: Re: [PATCH 1/2] opensm: sigusr1: syslog() fixes]] Date: 20 Feb 2007 13:21:38 -0500 Hi Tzachi, On Thu, 2007-02-08 at 16:24, Tzachi Dar wrote: > See bellow. I would like to get back to trying to close on this discussion. > Thanks > Tzachi > > > -Original Message- > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > Sent: Thursday, February 08, 2007 9:47 PM > > To: Tzachi Dar > > Cc: Yossi Leybovich; Gilad Shainer; Yevgeny Kliteynik; > > OPENIB; Michael S. Tsirkin; Hal Rosenstock > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > opensm: sigusr1: syslog() fixes]] > > > > On 20:31 Thu 08 Feb , Tzachi Dar wrote: > > > The windows open IB has decided on using a BSD only license. > > > The common implementation of pthreads as far as I know is > > LGPL, which > > > means that it can not be used in open IB. > > > > Why not? AFAIK it works perfectly (see (5,6 and Preamble)): > > http://www.gnu.org/copyleft/lesser.html > > > > And of course there are tons of examples when BSD software > > links against LGPLed glibc. > > I can of course write you an answer that will be more than 5 pages long > of why *I* don't think that > Using GPL software is bad for everyone, but I guess that my opinion > doesn't really meter, so I > Won't do it. > The page that you have referenced is of the GNU org, and even there it > is hard to say that they > are trying to encourage you to use the LGPL license. In any case, the > main point is that > When open IB windows was formed there was a general decision that it > will use BSD license. If we > Start having components with the LGPL this will break that decision, and > therefore this requires > some voting of the open IB organization. I may be missing your point but is there something in the Windows OpenIB/OpenFabrics license that precludes using Windows OpenIB licensed code (e.g. BSD like license) in concert with non OpenIB code (like LGPL) ? Isn't that essentially what using the Windows pthreads DLL with OpenSM would be like ? As I understand it, I don't think this requires a license change or anything in the OpenIB Windows charter prevents this or needs changing. > > > The only two ways that I see around this are 1) Change the > > license of > > > open IB windows which might be a complicated thing. 2) Find an > > > implementation of pthreads that is BSD. > > > > BTW, just wondering... What is relation between windows open > > IB and OFA (and OFA's "dual-license rule")? > Well, the way I see it one can take code from the Linux part under the > BSD licance and use it in > The windows part. The otherway around seems fine to me but some say that > since the windows BSD liscance > Reqires that some text will always remain there, the other way around is > not possibale. As I'm not an > Expert in that erea I don't know who is right. I don't see how this affects what is being discussed about OpenSM. In all the cases I'm aware of, the portability is from Linux to Windows and not the other way around. -- Hal > > Sasha > > > > > > > > Thanks > > > Tzachi > > > > > > > -Original Message- > > > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > > > Sent: Thursday, February 08, 2007 7:46 PM > > > > To: Tzachi Dar; Yossi Leybovich > > > > Cc: Yevgeny Kliteynik; OPENIB; Michael S. Tsirkin; Hal Rosenstock > > > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > > > opensm: sigusr1: syslog() fixes]] > > > > > > > > On 11:24 Sun 21 Jan , Yevgeny Kliteynik wrote: > > > >
Re: [openib-general] [ofw] [Fwd: Re: [Fwd: Re: win related [was: Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]]
On Tue, 2007-02-20 at 13:56, Fab Tillier wrote: > Submissions to the OFW project are supposed to be bound by the > contributor's agreement: > > http://windows.openib.org/openib/contribute.aspx > > Contributing code under anything but a BSD license violates condition 1, > though there shouldn't be issues with dual licenses as long as one of > the available licenses is a BSD license. > > In any case, we're not talking about putting the pthreads library in > source or binary form in the OFW SVN, right? Right (we're not). > We're just talking about > having OpenSM link to the pthreads library that is out-of-tree. Yes. > So the > question is whether there are any licensing issues with having a BSD > code include an out-of-tree LGPL file that would affect the ability to > retain the BSD license on the OpenSM files. I don't think this is an issue as there are other instances of this being done (outside of OpenIB). > I can see this causing > problems for builds, as people would need to find/install the pthreads > library before OpenSM would build successfully. Could install documentation for OpenSM on Windows minimize this as an issue ? -- Hal > -Fab > > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of Hal Rosenstock > Sent: Tuesday, February 20, 2007 10:38 AM > To: ofw@lists.openfabrics.org > Cc: Gilad Shainer; OPENIB > Subject: [ofw] [Fwd: Re: [openib-general] [Fwd: Re: win related [was: > Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]] > > Also, looping in the OpenFabrics Windows email list on this. > > -- Hal > > -Forwarded Message- > > From: Hal Rosenstock <[EMAIL PROTECTED]> > To: Tzachi Dar <[EMAIL PROTECTED]> > Cc: OPENIB , Gilad Shainer > <[EMAIL PROTECTED]> > Subject: Re: [openib-general] [Fwd: Re: win related [was: Re: [PATCH > 1/2] opensm: sigusr1: syslog() fixes]] > Date: 20 Feb 2007 13:21:38 -0500 > > Hi Tzachi, > > On Thu, 2007-02-08 at 16:24, Tzachi Dar wrote: > > See bellow. > > I would like to get back to trying to close on this discussion. > > > Thanks > > Tzachi > > > > > -Original Message- > > > From: Sasha Khapyorsky [mailto:[EMAIL PROTECTED] > > > Sent: Thursday, February 08, 2007 9:47 PM > > > To: Tzachi Dar > > > Cc: Yossi Leybovich; Gilad Shainer; Yevgeny Kliteynik; > > > OPENIB; Michael S. Tsirkin; Hal Rosenstock > > > Subject: Re: [Fwd: Re: win related [was: Re: [PATCH 1/2] > > > opensm: sigusr1: syslog() fixes]] > > > > > > On 20:31 Thu 08 Feb , Tzachi Dar wrote: > > > > The windows open IB has decided on using a BSD only license. > > > > The common implementation of pthreads as far as I know is > > > LGPL, which > > > > means that it can not be used in open IB. > > > > > > Why not? AFAIK it works perfectly (see (5,6 and Preamble)): > > > http://www.gnu.org/copyleft/lesser.html > > > > > > And of course there are tons of examples when BSD software > > > links against LGPLed glibc. > > > > I can of course write you an answer that will be more than 5 pages > long > > of why *I* don't think that > > Using GPL software is bad for everyone, but I guess that my opinion > > doesn't really meter, so I > > Won't do it. > > The page that you have referenced is of the GNU org, and even there it > > is hard to say that they > > are trying to encourage you to use the LGPL license. In any case, the > > main point is that > > When open IB windows was formed there was a general decision that it > > will use BSD license. If we > > Start having components with the LGPL this will break that decision, > and > > therefore this requires > > some voting of the open IB organization. > > I may be missing your point but is there something in the Windows > OpenIB/OpenFabrics license that precludes using Windows OpenIB licensed > code (e.g. BSD like license) in concert with non OpenIB code (like LGPL) > ? Isn't that essentially what using the Windows pthreads DLL with OpenSM > would be like ? As I understand it, I don't think this requires a > license change or anything in the OpenIB Windows charter prevents this > or needs changing. > > > > > The only two ways that I see around this are 1) Change the > > > license of > > > > open IB windows which might be a complicated thing. 2) Find an > > > > implementation of pthreads that is BSD. > > > > > > BTW, just wondering... What is relation between windows open > > > IB and OFA (and OFA's "dual-license rule")? > > Well, the way I see it one can take code from the Linux part under the > > BSD licance and use it in > > The windows part. The otherway around seems fine to me but some say > that > > since the windows BSD liscance > > Reqires that some text will always remain there, the other way around > is > > not possibale. As I'm not an > > Expert in that erea I don't know who is right. > > I don't see how this affects what is being discussed about OpenSM. In > all the cases I'm aware of, the portability is from Linux to Windows and > not the other way ar
Re: [openib-general] [ofw] [Fwd: Re: [Fwd: Re: win related [was:Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]]
-Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 20, 2007 10:57 AM On Tue, 2007-02-20 at 13:56, Fab Tillier wrote: > Submissions to the OFW project are supposed to be bound by the > contributor's agreement: > > I can see this causing > problems for builds, as people would need to find/install the pthreads > library before OpenSM would build successfully. Could install documentation for OpenSM on Windows minimize this as an issue ? [ftillier] This isn't just an install issue - it's a build issue. Anyone that wants to build OpenSM will need to find/download/install the pthreads library so that the build will succeed. If linking statically, the resulting executable will not require any special installation. It's only an install issue if you link dynamically to pitheads. -Fab ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [ofw] [Fwd: Re: [Fwd: Re: win related [was:Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]]
On Tue, 2007-02-20 at 16:08, Fab Tillier wrote: > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Tuesday, February 20, 2007 10:57 AM > > On Tue, 2007-02-20 at 13:56, Fab Tillier wrote: > > Submissions to the OFW project are supposed to be bound by the > > contributor's agreement: > > > > I can see this causing > > problems for builds, as people would need to find/install the pthreads > > library before OpenSM would build successfully. > > Could install documentation for OpenSM on Windows minimize this as an > issue ? > > [ftillier] This isn't just an install issue - it's a build issue. > Anyone that wants to build OpenSM will need to find/download/install the > pthreads library so that the build will succeed. If linking statically, > the resulting executable will not require any special installation. > It's only an install issue if you link dynamically to pitheads. OK; then build and install. How big an issue is this ? I thought DLLs were dynamically linked but I'm a Windows plebe. -- Hal > -Fab ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [ofw] [Fwd: Re: [Fwd: Re: win related[was:Re:[PATCH 1/2] opensm: sigusr1: syslog() fixes]]]
-Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Hal Rosenstock Sent: Tuesday, February 20, 2007 1:43 PM On Tue, 2007-02-20 at 16:08, Fab Tillier wrote: > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Tuesday, February 20, 2007 10:57 AM > > On Tue, 2007-02-20 at 13:56, Fab Tillier wrote: > [ftillier] This isn't just an install issue - it's a build issue. > Anyone that wants to build OpenSM will need to find/download/install the > pthreads library so that the build will succeed. If linking statically, > the resulting executable will not require any special installation. > It's only an install issue if you link dynamically to pitheads. OK; then build and install. How big an issue is this ? I thought DLLs were dynamically linked but I'm a Windows plebe. [ftillier] When you build, the linker needs the import library for pthreads so that the functions get resolved as being imported from the pthreads DLL. The dependency on the pthreads DLL is then created and the DLL will be loaded dynamically, assuming it can be found in the path. So for the build process, you need to have the pthreads library available to the build tool (path to the lib). This requires installing the pthreads developer package or however it's done. If you statically link the pthreads lib, rather than dynamically link, then all the pthreads goodies go directly into the executable and you remove the dependency on an external DLL. The build process requirements are no different than for the dynamically linked case. There is also the possibility to remove the link-time dependency by calling GetProcAddress to explicitly resolve the pthreads entrypoints. This method still requires having the DLL loaded on the user's systems. Pesonally, I would rather see static linkage to the pthreads library so that only the builds are affected (something only 'experts' will be doing), while not affecting the common user. -Fab ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] osm/libvendor: compilation fixes
On Mon, 2007-02-19 at 18:04, Sasha Khapyorsky wrote: > This adds needed header files inclusion to prevent compilation failures. > > Signed-off-by: Sasha Khapyorsky <[EMAIL PROTECTED]> > --- Thanks. Applied (to both master and ofed_1_2). -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
>An application that wants fork() to work with libibverbs should >either call ibv_fork_init() before doing anything else with >libibverbs, or else a user can set the IBV_FORK_SAFE or >RDMAV_FORK_SAFE environment variable to get the same effect. There is >some overhead to making fork() work so it is not enabled by default. >This is described in the ibv_fork_init manpage in the latest >libibverbs git tree. Does this require 2.6.16 or better kernel support? ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Fork issues with simple MPI program
> Does this require 2.6.16 or better kernel support? The kernel must support the MADV_DONTFORK flag to madvise(), not sure when exactly that was merged but 2.6.16 or so sounds right. ibv_fork_init() will return an error if the kernel support is missing and fork safety won't actually work. And if you use the environment variable a warning will be printed if ibv_fork_init() fails. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] I created a git tree for the libibverbs man pages
I merged all these manpages into my libibverbs tree and pushed the result out to kernel.org. Please send any future updates as diffs against the libibverbs tree. Thanks, Roland ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
[openib-general] [patch 09/18] IB/mad: Fix race between cancel and receive completion
-stable review patch. If anyone has any objections, please let us know. -- From: Roland Dreier <[EMAIL PROTECTED]> When ib_cancel_mad() is called, it puts the canceled send on a list and schedules a "flushed" callback from process context. However, this leaves a window where a receive completion could be processed before the send is fully flushed. This is fine, except that ib_find_send_mad() will find the MAD and return it to the receive processing, which results in the sender getting both a successful receive and a "flushed" send completion for the same request. Understandably, this confuses the sender, which is expecting only one of these two callbacks, and leads to grief such as a use-after-free in IPoIB. Fix this by changing ib_find_send_mad() to return a send struct only if the status is still successful (and not "flushed"). The search of the send_list already had this check, so this patch just adds the same check to the search of the wait_list. Signed-off-by: Roland Dreier <[EMAIL PROTECTED]> Signed-off-by: Chris Wright <[EMAIL PROTECTED]> Signed-off-by: Greg Kroah-Hartman <[EMAIL PROTECTED]> --- --- drivers/infiniband/core/mad.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) --- linux-2.6.18.7.orig/drivers/infiniband/core/mad.c +++ linux-2.6.18.7/drivers/infiniband/core/mad.c @@ -1750,7 +1750,7 @@ ib_find_send_mad(struct ib_mad_agent_pri */ (is_direct(wc->recv_buf.mad->mad_hdr.mgmt_class) || rcv_has_same_gid(mad_agent_priv, wr, wc))) - return wr; + return (wr->status == IB_WC_SUCCESS) ? wr : NULL; } /* -- ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] Immediate data question
On 2/15/07, Michael Krause <[EMAIL PROTECTED]> wrote: > At 09:37 PM 2/14/2007, Devesh Sharma wrote: > >On 2/14/07, Michael Krause <[EMAIL PROTECTED]> wrote: > >>At 05:37 AM 2/13/2007, Devesh Sharma wrote: > >> >On 2/12/07, Devesh Sharma <[EMAIL PROTECTED]> wrote: > >> >>On 2/10/07, Tang, Changqing <[EMAIL PROTECTED]> wrote: > >> >> > > > > >> >> > > >Not for the receiver, but the sender will be severely slowed down > >> >> > > >by > >> >> > > >having to wait for the RNR timeouts. > >> >> > > > >> >> > > RNR = Receiver Not Ready so by definition, the data flow > >> >> > > isn't going to > >> >> > > progress until the receiver is ready to receive data. If a > >> >> > > receive QP > >> >> > > enters RNR for a RC, then it is likely not progressing as > >> >> > > desired. RNR > >> >> > > was initially put in place to enable a receiver to create > >> >> > > back pressure to the sender without causing a fatal error > >> >> > > condition. It should rarely be entered and therefore should > >> >> > > have negligible impact on overall performance however when a > >> >> > > RNR occurs, no forward progress will occur so performance is > >> >> > > essentially zero. > >> >> > > >> >> > Mike: > >> >> > I still do not quite understand this issue. I have two > >> >> > situations that have RNR triggered. > >> >> > > >> >> > 1. process A and process B is connected with QP. A first post a send > >> >> > to > >> >> > B, B does not post receive. Then A and B are doing a long time > >> >> > RDMA_WRITE each other, A and B just check memory for the RDMA_WRITE > >> >> > message. Finally B will post a receive. Does the first pending send > >> in A > >> >> > block all the later RDMA_WRITE ? > >> >>According to IBTA spec HCA will process WR entries in strict order in > >> >>which they are posted so the send will block all WR posted after this > >> >>send, Until-unless HCA has multiple processing elements, I think even > >> >>then processing order will be maintained by HCA > >> >> If not, since RNR is triggered > >> >> > periodically till B post receive, does it affect the RDMA_WRITE > >> >> > performance between A and B ? > >> >> > > >> >> > 2. extend above to three processes, A connect to B, B connect to C, > >> so B > >> >> > has two QPs, but one CQ.A posts a send to B, B does not post receive, > >> >post ordering accross QP is not guaranteed hence presence of same CQ > >> >or different CQ will not affect any thing. > >> >> > rather B and C are doing a long time RDMA_WRITE,or send/recv. But B > >> >If RDMA WRITE _on_ B, no effect on performance. If RDMA WRITE _on_ C, > >I am sorry I have missed that in both cases same DMA channel is in use. > >> >_may_ affect the performance, since load is on same HCA. In case of > >> >Send/Recv again _may_ affect the performance, with the same reason. > >> > >>Seems orthogonal. Any time h/w is shared, multiple flows will have an > >>impact on one another. That is why we have the different arbitration > >>mechanisms to enable one to control that impact. > >Please, can you explain it more clearly? > > Most I/O devices are shared by multiple applications / kernel > subsystems. Hence, the device acts as a serialization point for what goes > on the wire / link. Sharing = resource contention and in order to add any > structure to that contention, a number of technologies provide arbitration > options. In the case of IB, the arbitration is confined to VL arbitration > where a given data flow is assigned to a VL and that VL is services at some > particular rate. A number of years ago I wrote up how one might also > provide QP arbitration (not part of the IBTA specifications) and I > understand some implementations have incorporated that or a variation of > the mechanisms into their products. Thanks mike for a nice explanation. I am sorry for the late reply, Now I got it, here Chang is trying to find out performance hit due to RNR NAK, performance hit due to device sharing is any how going to be there so "load on same HCA" is not the proper explanation. Am I correct now? > > In addition to IB link contention, there is also PCI link / bus > contention. For PCIe, given most designs did not want to waste resources > on multiple VC, there really isn't any standard arbitration > mechanism. However, many devices, especially a device like a HCA or a > RNIC, already have the concept of separate resource domains, e.g. QP, and > they provide a mechanism to associate how the QP's DMA requests or > interrupts requests are scheduled to the PCIe link. > > > >> >> > must sends RNR periodically to A, right?. So does the pending message > >> >> > from A affects B's overall performance between B and C ? > >> >But RNR NAK is not for very long time.possibly this performance > >> >hit you will not be able to observe even. The moment rnr_counter > >> >expires connection will be broken! > >> > >>Keep in mind the timeout can be infinite. RNR NAK are not expected to be > >>frequent so their perfor
[openib-general] fix SDP bug 108 for OFED 1.2 beta?
Tziporet and Michael, every since the SDP rewrite in OFED 1.0 rc5, SDP throughput drops with message size > 64KB, see attached graph. Can you please fix this for OFED 1.2 beta? Scott Weitzenkamp SQA and Release Manager Server Virtualization Business Unit Cisco Systems sdp2sdp.TCP_STREAM.000.tput_log.pdf Description: sdp2sdp.TCP_STREAM.000.tput_log.pdf ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH] librdmacm: fix bug causing failure to work with partial membership pkey
Hal Rosenstock wrote: > On Tue, 2007-02-20 at 10:38, Or Gerlitz wrote: >> Yes. Its a little bit confusing: partial and full members of an IPoIB IB >> partition use the same MGID. When an IPoIB MGID is constructed, the pkey >> placed by the driver is --always-- the full membership one. However, on >> a node with partial membership, what's plugged into the QP is the pkey >> index of the partial instance... > So in this case, do both the full and partial keys need configuring for > that port ? No. The SM configures --either-- the full or the partial pkey. However, no matter what the SM configures, the core & ipoib code act as the full pkey is there. This is nice simplification and it works well. Or. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [PATCH for-2.6.21] IPoIB/cm: improve small message bandwidth
Thanks, queued for 2.6.21. With this patch I see small-packet latency down almost all the way back to what datagram mode gives -- on a pair of fast woodcrest systems I see latencies for netpipe tcp 1 byte messages like datagram 13.xx original CM 17.xx patched CM 14.xx so there is still a measurable difference but it is much less now. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] [2.6 patch] drivers/infiniband/hw/cxgb3/: possible cleanups
> You could just remove the code instead of #if 0... Steve, can you decide what the right thing to do with these changes is and send me the result (or just tell me to apply Adrian's patch as-is)? Thanks, Roland ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
