[ewg] Stories and myths of pe-enlargers!
http://dujxzqatcjkdt.blogspot.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
> > *OFED 1.4:* > 1. Kernel base: since we target 1.4 release to Sep we target the > kernel base to be 2.6.27 > This is a good target, but we may need to stay with 2.6.26 if the > kernel progress will not be aligned. > > 2. Suggestions for new features: > > * NFS-RDMA > * Verbs: Reliable Multicast (to be presented at Sonoma) > * SDP - Zero copy (There was a question on IPv6 support - seems no > one interested for now) > * IPoIB - continue with performance enhancements > * Xsigo new virtual NIC > * New vendor HW support - non was reported so far (IBM and Chelsio > - do you have something?) > * OpenSM: > o Incremental routing > o Temporary SA DB - to answer queries and a heavy sweep is done > o APM - disjoint paths (?) > o MKey manager (?) > o Sasha to send more management features > * MPI: > o Open MPI 1.3 > o APM support in MPI > o mvapich ??? > * uDAPl > o Extensions for new APIs (like XRC) - ? > o uDAPL provider for interop between Windows & Linux > o 1.2 and 2.0 will stay > As I wrote in an earlier discussion (~2 months ago), we plan to add tgt (SCSI target) with iSCSI over iSER (and TCP of course) support. The git tree for tgt already exists on the ofa server. Erez ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
Can we address multiple-fabrics (physically separated) support ? --CQ Tang > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Erez Zilber > Sent: Thursday, April 03, 2008 8:51 AM > To: Tziporet Koren > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting > summary on OFED 1.4 plans > > > > > *OFED 1.4:* > > 1. Kernel base: since we target 1.4 release to Sep we target the > > kernel base to be 2.6.27 > > This is a good target, but we may need to stay with > 2.6.26 if the > > kernel progress will not be aligned. > > > > 2. Suggestions for new features: > > > > * NFS-RDMA > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > * SDP - Zero copy (There was a question on IPv6 support > - seems no > > one interested for now) > > * IPoIB - continue with performance enhancements > > * Xsigo new virtual NIC > > * New vendor HW support - non was reported so far (IBM > and Chelsio > > - do you have something?) > > * OpenSM: > > o Incremental routing > > o Temporary SA DB - to answer queries and a heavy > sweep is done > > o APM - disjoint paths (?) > > o MKey manager (?) > > o Sasha to send more management features > > * MPI: > > o Open MPI 1.3 > > o APM support in MPI > > o mvapich ??? > > * uDAPl > > o Extensions for new APIs (like XRC) - ? > > o uDAPL provider for interop between Windows & Linux > > o 1.2 and 2.0 will stay > > > > As I wrote in an earlier discussion (~2 months ago), we plan > to add tgt (SCSI target) with iSCSI over iSER (and TCP of > course) support. The git tree for tgt already exists on the > ofa server. > > Erez > > ___ > general mailing list > [EMAIL PROTECTED] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
CQ, On Thu, 2008-04-03 at 14:27 +, Tang, Changqing wrote: > Can we address multiple-fabrics (physically separated) support ? Can you elaborate on what you mean by "physically separated" ? -- Hal > > > --CQ Tang > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of > > Erez Zilber > > Sent: Thursday, April 03, 2008 8:51 AM > > To: Tziporet Koren > > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting > > summary on OFED 1.4 plans > > > > > > > > *OFED 1.4:* > > > 1. Kernel base: since we target 1.4 release to Sep we target the > > > kernel base to be 2.6.27 > > > This is a good target, but we may need to stay with > > 2.6.26 if the > > > kernel progress will not be aligned. > > > > > > 2. Suggestions for new features: > > > > > > * NFS-RDMA > > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > > * SDP - Zero copy (There was a question on IPv6 support > > - seems no > > > one interested for now) > > > * IPoIB - continue with performance enhancements > > > * Xsigo new virtual NIC > > > * New vendor HW support - non was reported so far (IBM > > and Chelsio > > > - do you have something?) > > > * OpenSM: > > > o Incremental routing > > > o Temporary SA DB - to answer queries and a heavy > > sweep is done > > > o APM - disjoint paths (?) > > > o MKey manager (?) > > > o Sasha to send more management features > > > * MPI: > > > o Open MPI 1.3 > > > o APM support in MPI > > > o mvapich ??? > > > * uDAPl > > > o Extensions for new APIs (like XRC) - ? > > > o uDAPL provider for interop between Windows & Linux > > > o 1.2 and 2.0 will stay > > > > > > > As I wrote in an earlier discussion (~2 months ago), we plan > > to add tgt (SCSI target) with iSCSI over iSER (and TCP of > > course) support. The git tree for tgt already exists on the > > ofa server. > > > > Erez > > > > ___ > > general mailing list > > [EMAIL PROTECTED] > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > ___ > general mailing list > [EMAIL PROTECTED] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
You have a system, all HCAs have two ports, all port 1 are connected to the first switch, all port 2 are connected to the second switch, there is NO link between the two switches. We call this system has two physically separated fabrics. If you have a bridge link between the two switches, then it becomes a single fabric. The same thing for multiple HCAs on nodes. The problem is, from MPI side, (and by default), we don't know which port is on which fabric, since the subnet prefix is the same. We rely on system admin to config two different subnet prefixes for HP-MPI to work. No vendor has claimed to support this. --CQ > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Thursday, April 03, 2008 9:32 AM > To: Tang, Changqing > Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; > [EMAIL PROTECTED] > Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting > summary on OFED 1.4 plans > > CQ, > > On Thu, 2008-04-03 at 14:27 +, Tang, Changqing wrote: > > Can we address multiple-fabrics (physically separated) support ? > > Can you elaborate on what you mean by "physically separated" ? > > -- Hal > > > > > > > --CQ Tang > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Erez > > > Zilber > > > Sent: Thursday, April 03, 2008 8:51 AM > > > To: Tziporet Koren > > > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > > > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on > > > OFED 1.4 plans > > > > > > > > > > > *OFED 1.4:* > > > > 1. Kernel base: since we target 1.4 release to Sep we > target the > > > > kernel base to be 2.6.27 > > > > This is a good target, but we may need to stay with > > > 2.6.26 if the > > > > kernel progress will not be aligned. > > > > > > > > 2. Suggestions for new features: > > > > > > > > * NFS-RDMA > > > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > > > * SDP - Zero copy (There was a question on IPv6 support > > > - seems no > > > > one interested for now) > > > > * IPoIB - continue with performance enhancements > > > > * Xsigo new virtual NIC > > > > * New vendor HW support - non was reported so far (IBM > > > and Chelsio > > > > - do you have something?) > > > > * OpenSM: > > > > o Incremental routing > > > > o Temporary SA DB - to answer queries and a heavy > > > sweep is done > > > > o APM - disjoint paths (?) > > > > o MKey manager (?) > > > > o Sasha to send more management features > > > > * MPI: > > > > o Open MPI 1.3 > > > > o APM support in MPI > > > > o mvapich ??? > > > > * uDAPl > > > > o Extensions for new APIs (like XRC) - ? > > > > o uDAPL provider for interop between Windows & Linux > > > > o 1.2 and 2.0 will stay > > > > > > > > > > As I wrote in an earlier discussion (~2 months ago), we > plan to add > > > tgt (SCSI target) with iSCSI over iSER (and TCP of > > > course) support. The git tree for tgt already exists on the ofa > > > server. > > > > > > Erez > > > > > > ___ > > > general mailing list > > > [EMAIL PROTECTED] > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > > ___ > > general mailing list > > [EMAIL PROTECTED] > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
One other thing I hope to talk is some fabric query functionalities for normal user, not only just for root. This is at IB verbs level, not rdma_cm level. for example, in MPI, process A know the HCA guid on another node. After running for some time, the switch is restarted for some reason, and the whole fabric is re-configured. Now process A wants to know if the port lid on another node has changed or not, it knows the HCA guid, is there any function to query this ? I know as root, we can use the mad/umad library to do this kind of query, I want to do such query in MPI, which is a normal user. --CQ Tang, HP-MPI > -Original Message- > From: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] On Behalf Of > Erez Zilber > Sent: Thursday, April 03, 2008 8:51 AM > To: Tziporet Koren > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting > summary on OFED 1.4 plans > > > > > *OFED 1.4:* > > 1. Kernel base: since we target 1.4 release to Sep we target the > > kernel base to be 2.6.27 > > This is a good target, but we may need to stay with > 2.6.26 if the > > kernel progress will not be aligned. > > > > 2. Suggestions for new features: > > > > * NFS-RDMA > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > * SDP - Zero copy (There was a question on IPv6 support > - seems no > > one interested for now) > > * IPoIB - continue with performance enhancements > > * Xsigo new virtual NIC > > * New vendor HW support - non was reported so far (IBM > and Chelsio > > - do you have something?) > > * OpenSM: > > o Incremental routing > > o Temporary SA DB - to answer queries and a heavy > sweep is done > > o APM - disjoint paths (?) > > o MKey manager (?) > > o Sasha to send more management features > > * MPI: > > o Open MPI 1.3 > > o APM support in MPI > > o mvapich ??? > > * uDAPl > > o Extensions for new APIs (like XRC) - ? > > o uDAPL provider for interop between Windows & Linux > > o 1.2 and 2.0 will stay > > > > As I wrote in an earlier discussion (~2 months ago), we plan > to add tgt (SCSI target) with iSCSI over iSER (and TCP of > course) support. The git tree for tgt already exists on the > ofa server. > > Erez > > ___ > general mailing list > [EMAIL PROTECTED] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] physically separate subnets (was: OFED March 24 meeting summary on OFED 1.4 plans)
On Apr 3, 2008, at 10:52 AM, Hal Rosenstock wrote:
On Thu, 2008-04-03 at 10:47 -0400, Jeff Squyres wrote:
In Open MPI, we require physically different ("air gapped") subnets
to
have different subnet ID's so that we can compute reachability
correctly.
Don't understand what the "air gapped" reference means.
There's no physical connection between the two -- there's an "air gap"
between the networks (maybe it's a military term :-) ).
--
Jeff Squyres
Cisco Systems
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, 2008-04-03 at 14:40 +, Tang, Changqing wrote: > You have a system, all HCAs have two ports, all port 1 are connected to the > first switch, > all port 2 are connected to the second switch, there is NO link between the > two switches. > We call this system has two physically separated fabrics. If you have a > bridge link > between the two switches, then it becomes a single fabric. > > The same thing for multiple HCAs on nodes. > > The problem is, from MPI side, (and by default), we don't know which port is > on which > fabric, since the subnet prefix is the same. We rely on system admin to > config two > different subnet prefixes for HP-MPI to work. Yes, these two IB subnets need two different subnet prefixes. (I think it's more than just HP MPI which needs this). -- Hal > No vendor has claimed to support this. > > --CQ > > > -Original Message- > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > > Sent: Thursday, April 03, 2008 9:32 AM > > To: Tang, Changqing > > Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; > > [EMAIL PROTECTED] > > Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting > > summary on OFED 1.4 plans > > > > CQ, > > > > On Thu, 2008-04-03 at 14:27 +, Tang, Changqing wrote: > > > Can we address multiple-fabrics (physically separated) support ? > > > > Can you elaborate on what you mean by "physically separated" ? > > > > -- Hal > > > > > > > > > > > --CQ Tang > > > > > > > -Original Message- > > > > From: [EMAIL PROTECTED] > > > > [mailto:[EMAIL PROTECTED] On Behalf Of Erez > > > > Zilber > > > > Sent: Thursday, April 03, 2008 8:51 AM > > > > To: Tziporet Koren > > > > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > > > > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on > > > > OFED 1.4 plans > > > > > > > > > > > > > > *OFED 1.4:* > > > > > 1. Kernel base: since we target 1.4 release to Sep we > > target the > > > > > kernel base to be 2.6.27 > > > > > This is a good target, but we may need to stay with > > > > 2.6.26 if the > > > > > kernel progress will not be aligned. > > > > > > > > > > 2. Suggestions for new features: > > > > > > > > > > * NFS-RDMA > > > > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > > > > * SDP - Zero copy (There was a question on IPv6 support > > > > - seems no > > > > > one interested for now) > > > > > * IPoIB - continue with performance enhancements > > > > > * Xsigo new virtual NIC > > > > > * New vendor HW support - non was reported so far (IBM > > > > and Chelsio > > > > > - do you have something?) > > > > > * OpenSM: > > > > > o Incremental routing > > > > > o Temporary SA DB - to answer queries and a heavy > > > > sweep is done > > > > > o APM - disjoint paths (?) > > > > > o MKey manager (?) > > > > > o Sasha to send more management features > > > > > * MPI: > > > > > o Open MPI 1.3 > > > > > o APM support in MPI > > > > > o mvapich ??? > > > > > * uDAPl > > > > > o Extensions for new APIs (like XRC) - ? > > > > > o uDAPL provider for interop between Windows & Linux > > > > > o 1.2 and 2.0 will stay > > > > > > > > > > > > > As I wrote in an earlier discussion (~2 months ago), we > > plan to add > > > > tgt (SCSI target) with iSCSI over iSER (and TCP of > > > > course) support. The git tree for tgt already exists on the ofa > > > > server. > > > > > > > > Erez > > > > > > > > ___ > > > > general mailing list > > > > [EMAIL PROTECTED] > > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > > > To unsubscribe, please visit > > > > http://openib.org/mailman/listinfo/openib-general > > > > > > > ___ > > > general mailing list > > > [EMAIL PROTECTED] > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > > ___ > general mailing list > [EMAIL PROTECTED] > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] physically separate subnets (was: OFED March 24 meeting summary on OFED 1.4 plans)
In Open MPI, we require physically different ("air gapped") subnets to
have different subnet ID's so that we can compute reachability
correctly. I don't know how to do it otherwise.
On Apr 3, 2008, at 10:40 AM, Tang, Changqing wrote:
You have a system, all HCAs have two ports, all port 1 are connected
to the first switch,
all port 2 are connected to the second switch, there is NO link
between the two switches.
We call this system has two physically separated fabrics. If you
have a bridge link
between the two switches, then it becomes a single fabric.
The same thing for multiple HCAs on nodes.
The problem is, from MPI side, (and by default), we don't know which
port is on which
fabric, since the subnet prefix is the same. We rely on system admin
to config two
different subnet prefixes for HP-MPI to work.
No vendor has claimed to support this.
--CQ
-Original Message-
From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
Sent: Thursday, April 03, 2008 9:32 AM
To: Tang, Changqing
Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org;
[EMAIL PROTECTED]
Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting
summary on OFED 1.4 plans
CQ,
On Thu, 2008-04-03 at 14:27 +, Tang, Changqing wrote:
Can we address multiple-fabrics (physically separated) support ?
Can you elaborate on what you mean by "physically separated" ?
-- Hal
--CQ Tang
-Original Message-
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of Erez
Zilber
Sent: Thursday, April 03, 2008 8:51 AM
To: Tziporet Koren
Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED]
Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on
OFED 1.4 plans
*OFED 1.4:*
1. Kernel base: since we target 1.4 release to Sep we
target the
kernel base to be 2.6.27
This is a good target, but we may need to stay with
2.6.26 if the
kernel progress will not be aligned.
2. Suggestions for new features:
* NFS-RDMA
* Verbs: Reliable Multicast (to be presented at Sonoma)
* SDP - Zero copy (There was a question on IPv6 support
- seems no
one interested for now)
* IPoIB - continue with performance enhancements
* Xsigo new virtual NIC
* New vendor HW support - non was reported so far (IBM
and Chelsio
- do you have something?)
* OpenSM:
o Incremental routing
o Temporary SA DB - to answer queries and a heavy
sweep is done
o APM - disjoint paths (?)
o MKey manager (?)
o Sasha to send more management features
* MPI:
o Open MPI 1.3
o APM support in MPI
o mvapich ???
* uDAPl
o Extensions for new APIs (like XRC) - ?
o uDAPL provider for interop between Windows & Linux
o 1.2 and 2.0 will stay
As I wrote in an earlier discussion (~2 months ago), we
plan to add
tgt (SCSI target) with iSCSI over iSER (and TCP of
course) support. The git tree for tgt already exists on the ofa
server.
Erez
___
general mailing list
[EMAIL PROTECTED]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
___
general mailing list
[EMAIL PROTECTED]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
--
Jeff Squyres
Cisco Systems
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] physically separate subnets (was: OFED March 24 meeting summary on OFED 1.4 plans)
On Thu, 2008-04-03 at 10:47 -0400, Jeff Squyres wrote:
> In Open MPI, we require physically different ("air gapped") subnets to
> have different subnet ID's so that we can compute reachability
> correctly.
Don't understand what the "air gapped" reference means.
> I don't know how to do it otherwise.
Me neither.
-- Hal
>
>
> On Apr 3, 2008, at 10:40 AM, Tang, Changqing wrote:
> >
> > You have a system, all HCAs have two ports, all port 1 are connected
> > to the first switch,
> > all port 2 are connected to the second switch, there is NO link
> > between the two switches.
> > We call this system has two physically separated fabrics. If you
> > have a bridge link
> > between the two switches, then it becomes a single fabric.
> >
> > The same thing for multiple HCAs on nodes.
> >
> > The problem is, from MPI side, (and by default), we don't know which
> > port is on which
> > fabric, since the subnet prefix is the same. We rely on system admin
> > to config two
> > different subnet prefixes for HP-MPI to work.
> >
> > No vendor has claimed to support this.
> >
> > --CQ
> >
> >> -Original Message-
> >> From: Hal Rosenstock [mailto:[EMAIL PROTECTED]
> >> Sent: Thursday, April 03, 2008 9:32 AM
> >> To: Tang, Changqing
> >> Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org;
> >> [EMAIL PROTECTED]
> >> Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting
> >> summary on OFED 1.4 plans
> >>
> >> CQ,
> >>
> >> On Thu, 2008-04-03 at 14:27 +, Tang, Changqing wrote:
> >>> Can we address multiple-fabrics (physically separated) support ?
> >>
> >> Can you elaborate on what you mean by "physically separated" ?
> >>
> >> -- Hal
> >>
> >>>
> >>>
> >>> --CQ Tang
> >>>
> -Original Message-
> From: [EMAIL PROTECTED]
> [mailto:[EMAIL PROTECTED] On Behalf Of Erez
> Zilber
> Sent: Thursday, April 03, 2008 8:51 AM
> To: Tziporet Koren
> Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED]
> Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on
> OFED 1.4 plans
>
> >
> > *OFED 1.4:*
> > 1. Kernel base: since we target 1.4 release to Sep we
> >> target the
> > kernel base to be 2.6.27
> >This is a good target, but we may need to stay with
> 2.6.26 if the
> > kernel progress will not be aligned.
> >
> > 2. Suggestions for new features:
> >
> >* NFS-RDMA
> >* Verbs: Reliable Multicast (to be presented at Sonoma)
> >* SDP - Zero copy (There was a question on IPv6 support
> - seems no
> > one interested for now)
> >* IPoIB - continue with performance enhancements
> >* Xsigo new virtual NIC
> >* New vendor HW support - non was reported so far (IBM
> and Chelsio
> > - do you have something?)
> >* OpenSM:
> > o Incremental routing
> > o Temporary SA DB - to answer queries and a heavy
> sweep is done
> > o APM - disjoint paths (?)
> > o MKey manager (?)
> > o Sasha to send more management features
> >* MPI:
> > o Open MPI 1.3
> > o APM support in MPI
> > o mvapich ???
> >* uDAPl
> > o Extensions for new APIs (like XRC) - ?
> > o uDAPL provider for interop between Windows & Linux
> > o 1.2 and 2.0 will stay
> >
>
> As I wrote in an earlier discussion (~2 months ago), we
> >> plan to add
> tgt (SCSI target) with iSCSI over iSER (and TCP of
> course) support. The git tree for tgt already exists on the ofa
> server.
>
> Erez
>
> ___
> general mailing list
> [EMAIL PROTECTED]
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
> >>> ___
> >>> general mailing list
> >>> [EMAIL PROTECTED]
> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >>>
> >>> To unsubscribe, please visit
> >>> http://openib.org/mailman/listinfo/openib-general
> >>
> >>
> > ___
> > ewg mailing list
> > ewg@lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>
>
___
ewg mailing list
ewg@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, 2008-04-03 at 14:53 +, Tang, Changqing wrote: > One other thing I hope to talk is some fabric query functionalities for > normal user, > not only just for root. This is at IB verbs level, not rdma_cm level. > > for example, in MPI, process A know the HCA guid on another node. After > running for > some time, the switch is restarted for some reason, and the whole fabric is > re-configured. > > Now process A wants to know if the port lid on another node has changed or > not, it knows > the HCA guid, is there any function to query this ? > I know as root, we can use the mad/umad library to do this kind of query, I > want to do > such query in MPI, which is a normal user. In the IB arch, there are SA registrations and queries for the specific example you used. However, these are not directly exposed to Linux user space directly (for the normal user as opposed to MAD user (note there are some difficulties in making this available to the normal user)) (at least not yet AFAIK). While these are not (direct) fabric query (really SA query), they serve the same function in a different way. -- Hal > --CQ Tang, HP-MPI > > > > > -Original Message- > > From: [EMAIL PROTECTED] > > [mailto:[EMAIL PROTECTED] On Behalf Of > > Erez Zilber > > Sent: Thursday, April 03, 2008 8:51 AM > > To: Tziporet Koren > > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting > > summary on OFED 1.4 plans > > > > > > > > *OFED 1.4:* > > > 1. Kernel base: since we target 1.4 release to Sep we target the > > > kernel base to be 2.6.27 > > > This is a good target, but we may need to stay with > > 2.6.26 if the > > > kernel progress will not be aligned. > > > > > > 2. Suggestions for new features: > > > > > > * NFS-RDMA > > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > > * SDP - Zero copy (There was a question on IPv6 support > > - seems no > > > one interested for now) > > > * IPoIB - continue with performance enhancements > > > * Xsigo new virtual NIC > > > * New vendor HW support - non was reported so far (IBM > > and Chelsio > > > - do you have something?) > > > * OpenSM: > > > o Incremental routing > > > o Temporary SA DB - to answer queries and a heavy > > sweep is done > > > o APM - disjoint paths (?) > > > o MKey manager (?) > > > o Sasha to send more management features > > > * MPI: > > > o Open MPI 1.3 > > > o APM support in MPI > > > o mvapich ??? > > > * uDAPl > > > o Extensions for new APIs (like XRC) - ? > > > o uDAPL provider for interop between Windows & Linux > > > o 1.2 and 2.0 will stay > > > > > > > As I wrote in an earlier discussion (~2 months ago), we plan > > to add tgt (SCSI target) with iSCSI over iSER (and TCP of > > course) support. The git tree for tgt already exists on the > > ofa server. > > > > Erez > > > > ___ > > general mailing list > > [EMAIL PROTECTED] > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > ___ > ewg mailing list > ewg@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
Thanks. When can we have the SA features, very soon, long time, or never ? --CQ > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Thursday, April 03, 2008 10:02 AM > To: Tang, Changqing > Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; > [EMAIL PROTECTED] > Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting > summary on OFED 1.4 plans > > On Thu, 2008-04-03 at 14:53 +, Tang, Changqing wrote: > > One other thing I hope to talk is some fabric query functionalities > > for normal user, not only just for root. This is at IB > verbs level, not rdma_cm level. > > > > for example, in MPI, process A know the HCA guid on another node. > > After running for some time, the switch is restarted for > some reason, and the whole fabric is re-configured. > > > > Now process A wants to know if the port lid on another node has > > changed or not, it knows the HCA guid, is there any > function to query this ? > > > I know as root, we can use the mad/umad library to do this kind of > > query, I want to do such query in MPI, which is a normal user. > > In the IB arch, there are SA registrations and queries for > the specific example you used. However, these are not > directly exposed to Linux user space directly (for the normal > user as opposed to MAD user (note there are some difficulties > in making this available to the normal user)) (at least not > yet AFAIK). While these are not (direct) fabric query (really > SA query), they serve the same function in a different way. > > -- Hal > > > --CQ Tang, HP-MPI > > > > > > > > > -Original Message- > > > From: [EMAIL PROTECTED] > > > [mailto:[EMAIL PROTECTED] On Behalf Of Erez > > > Zilber > > > Sent: Thursday, April 03, 2008 8:51 AM > > > To: Tziporet Koren > > > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > > > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on > > > OFED 1.4 plans > > > > > > > > > > > *OFED 1.4:* > > > > 1. Kernel base: since we target 1.4 release to Sep we > target the > > > > kernel base to be 2.6.27 > > > > This is a good target, but we may need to stay with > > > 2.6.26 if the > > > > kernel progress will not be aligned. > > > > > > > > 2. Suggestions for new features: > > > > > > > > * NFS-RDMA > > > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > > > * SDP - Zero copy (There was a question on IPv6 support > > > - seems no > > > > one interested for now) > > > > * IPoIB - continue with performance enhancements > > > > * Xsigo new virtual NIC > > > > * New vendor HW support - non was reported so far (IBM > > > and Chelsio > > > > - do you have something?) > > > > * OpenSM: > > > > o Incremental routing > > > > o Temporary SA DB - to answer queries and a heavy > > > sweep is done > > > > o APM - disjoint paths (?) > > > > o MKey manager (?) > > > > o Sasha to send more management features > > > > * MPI: > > > > o Open MPI 1.3 > > > > o APM support in MPI > > > > o mvapich ??? > > > > * uDAPl > > > > o Extensions for new APIs (like XRC) - ? > > > > o uDAPL provider for interop between Windows & Linux > > > > o 1.2 and 2.0 will stay > > > > > > > > > > As I wrote in an earlier discussion (~2 months ago), we > plan to add > > > tgt (SCSI target) with iSCSI over iSER (and TCP of > > > course) support. The git tree for tgt already exists on the ofa > > > server. > > > > > > Erez > > > > > > ___ > > > general mailing list > > > [EMAIL PROTECTED] > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > To unsubscribe, please visit > > > http://openib.org/mailman/listinfo/openib-general > > > > > ___ > > ewg mailing list > > ewg@lists.openfabrics.org > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, 2008-04-03 at 15:11 +, Tang, Changqing wrote: > Thanks. When can we have the SA features, very soon, long time, or never ? I'm unaware of any current plans to implement these but my knowledge is far from complete... -- Hal > --CQ > > > -Original Message- > > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > > Sent: Thursday, April 03, 2008 10:02 AM > > To: Tang, Changqing > > Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; > > [EMAIL PROTECTED] > > Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting > > summary on OFED 1.4 plans > > > > On Thu, 2008-04-03 at 14:53 +, Tang, Changqing wrote: > > > One other thing I hope to talk is some fabric query functionalities > > > for normal user, not only just for root. This is at IB > > verbs level, not rdma_cm level. > > > > > > for example, in MPI, process A know the HCA guid on another node. > > > After running for some time, the switch is restarted for > > some reason, and the whole fabric is re-configured. > > > > > > Now process A wants to know if the port lid on another node has > > > changed or not, it knows the HCA guid, is there any > > function to query this ? > > > > > I know as root, we can use the mad/umad library to do this kind of > > > query, I want to do such query in MPI, which is a normal user. > > > > In the IB arch, there are SA registrations and queries for > > the specific example you used. However, these are not > > directly exposed to Linux user space directly (for the normal > > user as opposed to MAD user (note there are some difficulties > > in making this available to the normal user)) (at least not > > yet AFAIK). While these are not (direct) fabric query (really > > SA query), they serve the same function in a different way. > > > > -- Hal > > > > > --CQ Tang, HP-MPI > > > > > > > > > > > > > -Original Message- > > > > From: [EMAIL PROTECTED] > > > > [mailto:[EMAIL PROTECTED] On Behalf Of Erez > > > > Zilber > > > > Sent: Thursday, April 03, 2008 8:51 AM > > > > To: Tziporet Koren > > > > Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] > > > > Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on > > > > OFED 1.4 plans > > > > > > > > > > > > > > *OFED 1.4:* > > > > > 1. Kernel base: since we target 1.4 release to Sep we > > target the > > > > > kernel base to be 2.6.27 > > > > > This is a good target, but we may need to stay with > > > > 2.6.26 if the > > > > > kernel progress will not be aligned. > > > > > > > > > > 2. Suggestions for new features: > > > > > > > > > > * NFS-RDMA > > > > > * Verbs: Reliable Multicast (to be presented at Sonoma) > > > > > * SDP - Zero copy (There was a question on IPv6 support > > > > - seems no > > > > > one interested for now) > > > > > * IPoIB - continue with performance enhancements > > > > > * Xsigo new virtual NIC > > > > > * New vendor HW support - non was reported so far (IBM > > > > and Chelsio > > > > > - do you have something?) > > > > > * OpenSM: > > > > > o Incremental routing > > > > > o Temporary SA DB - to answer queries and a heavy > > > > sweep is done > > > > > o APM - disjoint paths (?) > > > > > o MKey manager (?) > > > > > o Sasha to send more management features > > > > > * MPI: > > > > > o Open MPI 1.3 > > > > > o APM support in MPI > > > > > o mvapich ??? > > > > > * uDAPl > > > > > o Extensions for new APIs (like XRC) - ? > > > > > o uDAPL provider for interop between Windows & Linux > > > > > o 1.2 and 2.0 will stay > > > > > > > > > > > > > As I wrote in an earlier discussion (~2 months ago), we > > plan to add > > > > tgt (SCSI target) with iSCSI over iSER (and TCP of > > > > course) support. The git tree for tgt already exists on the ofa > > > > server. > > > > > > > > Erez > > > > > > > > ___ > > > > general mailing list > > > > [EMAIL PROTECTED] > > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > > > > > > > > To unsubscribe, please visit > > > > http://openib.org/mailman/listinfo/openib-general > > > > > > > ___ > > > ewg mailing list > > > ewg@lists.openfabrics.org > > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > > > > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] how do I use uDAPL with iWARP?
Scott Weitzenkamp (sweitzen) wrote: I have OFED 1.3 and a Chelsio S310E-SR+ iWARP 10GE NIC. I have ib_rdma_lat working, so I know IB verbs are working. How do I use uDAPL, though? All the default /etc/dat.conf entries have IPoIB or bonding interfaces in them. Add an entry like this: cxgb u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "ethx 0" "" Where ethx is the ethernet interface for the chelsio device. Also, last time I ran it you needed this in your env: export DAPL_MAX_INLINE=64 Steve. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?
Scott Weitzenkamp (sweitzen) wrote: I tried that, and it didn't work: [EMAIL PROTECTED] ~]# grep eth /etc/dat.conf OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" [EMAIL PROTECTED] ~]# dtest 10194 Running as server - OpenIB-cma 10194 Error dat_ep_create: DAT_INVALID_HANDLE 10194 Error freeing EP: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_EP try setting DAPL_MAX_INLINE=64 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?
Steve, Thanks, that gets further, but dtest still fails. Client side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest -h 192.168.0.198 13926 Running as client - OpenIB-cma 13926 Server Name: 192.168.0.198 13926 Server Net Address: 192.168.0.198 13926 Waiting for connect response 13926 Error unexpected conn event : DAT_CONNECTION_EVENT_UNREACHABLE 13926 Error connect_ep: DAT_ABORT 13926: DAPL Test Complete. 13926: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: open:36619.19 usec 13926: close: 32500.98 usec 13926: PZ create: 7.87 usec 13926: PZ free: 4.05 usec 13926: LMR create: 58.89 usec 13926: LMR free: 11.92 usec 13926: EVD create: 9.78 usec 13926: EVD free: 14.07 usec 13926: EP create: 78.92 usec 13926: EP free:26.23 usec 13926: TOTAL: 199.79 usec Server side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest 11461 Running as server - OpenIB-cma 11461 Server waiting for connect request.. 11461 Waiting for connect response 11461 CONNECTED! 11461 Send RMR to remote: snd_msg: r_key_ctx=bff,pad=0,va=146db580,len=0x40 11461 Waiting for remote to send RMR data 11461 Error waiting on h_dto_rcv_evd: DAT_TIMEOUT_EXPIRED 11461 Error connect_ep: DAT_TIMEOUT_EXPIRED 11461: DAPL Test Complete. 11461: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: open: 900676.01 usec 11461: close: 31543.97 usec 11461: PZ create: 7.87 usec 11461: PZ free: 5.01 usec 11461: LMR create: 51.98 usec 11461: LMR free: 12.16 usec 11461: EVD create: 10.97 usec 11461: EVD free: 12.87 usec 11461: EP create: 77.01 usec 11461: EP free:30.04 usec 11461: TOTAL: 195.03 usec Scott > -Original Message- > From: Steve Wise [mailto:[EMAIL PROTECTED] > Sent: Thursday, April 03, 2008 9:19 AM > To: Scott Weitzenkamp (sweitzen) > Cc: Joshua Bernstein; OpenFabrics EWG; [ofa_general] > Subject: Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP? > > > > Scott Weitzenkamp (sweitzen) wrote: > > I tried that, and it didn't work: > > > > [EMAIL PROTECTED] ~]# grep eth /etc/dat.conf > > OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 > dapl.1.2 "eth2 0" > > "" > > [EMAIL PROTECTED] ~]# dtest > > 10194 Running as server - OpenIB-cma > > 10194 Error dat_ep_create: DAT_INVALID_HANDLE > > 10194 Error freeing EP: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_EP > > > > try setting DAPL_MAX_INLINE=64 > > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?
What does your network inferface config look like? Does rping work? Scott Weitzenkamp (sweitzen) wrote: Steve, Thanks, that gets further, but dtest still fails. Client side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest -h 192.168.0.198 13926 Running as client - OpenIB-cma 13926 Server Name: 192.168.0.198 13926 Server Net Address: 192.168.0.198 13926 Waiting for connect response 13926 Error unexpected conn event : DAT_CONNECTION_EVENT_UNREACHABLE 13926 Error connect_ep: DAT_ABORT 13926: DAPL Test Complete. 13926: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: open:36619.19 usec 13926: close: 32500.98 usec 13926: PZ create: 7.87 usec 13926: PZ free: 4.05 usec 13926: LMR create: 58.89 usec 13926: LMR free: 11.92 usec 13926: EVD create: 9.78 usec 13926: EVD free: 14.07 usec 13926: EP create: 78.92 usec 13926: EP free:26.23 usec 13926: TOTAL: 199.79 usec Server side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest 11461 Running as server - OpenIB-cma 11461 Server waiting for connect request.. 11461 Waiting for connect response 11461 CONNECTED! 11461 Send RMR to remote: snd_msg: r_key_ctx=bff,pad=0,va=146db580,len=0x40 11461 Waiting for remote to send RMR data 11461 Error waiting on h_dto_rcv_evd: DAT_TIMEOUT_EXPIRED 11461 Error connect_ep: DAT_TIMEOUT_EXPIRED 11461: DAPL Test Complete. 11461: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: open: 900676.01 usec 11461: close: 31543.97 usec 11461: PZ create: 7.87 usec 11461: PZ free: 5.01 usec 11461: LMR create: 51.98 usec 11461: LMR free: 12.16 usec 11461: EVD create: 10.97 usec 11461: EVD free: 12.87 usec 11461: EP create: 77.01 usec 11461: EP free:30.04 usec 11461: TOTAL: 195.03 usec Scott -Original Message- From: Steve Wise [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 9:19 AM To: Scott Weitzenkamp (sweitzen) Cc: Joshua Bernstein; OpenFabrics EWG; [ofa_general] Subject: Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP? Scott Weitzenkamp (sweitzen) wrote: I tried that, and it didn't work: [EMAIL PROTECTED] ~]# grep eth /etc/dat.conf OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" [EMAIL PROTECTED] ~]# dtest 10194 Running as server - OpenIB-cma 10194 Error dat_ep_create: DAT_INVALID_HANDLE 10194 Error freeing EP: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_EP try setting DAPL_MAX_INLINE=64 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?
I can reproduce this. Lemme dig into it... Steve. Scott Weitzenkamp (sweitzen) wrote: Steve, Thanks, that gets further, but dtest still fails. Client side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest -h 192.168.0.198 13926 Running as client - OpenIB-cma 13926 Server Name: 192.168.0.198 13926 Server Net Address: 192.168.0.198 13926 Waiting for connect response 13926 Error unexpected conn event : DAT_CONNECTION_EVENT_UNREACHABLE 13926 Error connect_ep: DAT_ABORT 13926: DAPL Test Complete. 13926: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: open:36619.19 usec 13926: close: 32500.98 usec 13926: PZ create: 7.87 usec 13926: PZ free: 4.05 usec 13926: LMR create: 58.89 usec 13926: LMR free: 11.92 usec 13926: EVD create: 9.78 usec 13926: EVD free: 14.07 usec 13926: EP create: 78.92 usec 13926: EP free:26.23 usec 13926: TOTAL: 199.79 usec Server side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest 11461 Running as server - OpenIB-cma 11461 Server waiting for connect request.. 11461 Waiting for connect response 11461 CONNECTED! 11461 Send RMR to remote: snd_msg: r_key_ctx=bff,pad=0,va=146db580,len=0x40 11461 Waiting for remote to send RMR data 11461 Error waiting on h_dto_rcv_evd: DAT_TIMEOUT_EXPIRED 11461 Error connect_ep: DAT_TIMEOUT_EXPIRED 11461: DAPL Test Complete. 11461: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: open: 900676.01 usec 11461: close: 31543.97 usec 11461: PZ create: 7.87 usec 11461: PZ free: 5.01 usec 11461: LMR create: 51.98 usec 11461: LMR free: 12.16 usec 11461: EVD create: 10.97 usec 11461: EVD free: 12.87 usec 11461: EP create: 77.01 usec 11461: EP free:30.04 usec 11461: TOTAL: 195.03 usec Scott -Original Message- From: Steve Wise [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 9:19 AM To: Scott Weitzenkamp (sweitzen) Cc: Joshua Bernstein; OpenFabrics EWG; [ofa_general] Subject: Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP? Scott Weitzenkamp (sweitzen) wrote: I tried that, and it didn't work: [EMAIL PROTECTED] ~]# grep eth /etc/dat.conf OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" [EMAIL PROTECTED] ~]# dtest 10194 Running as server - OpenIB-cma 10194 Error dat_ep_create: DAT_INVALID_HANDLE 10194 Error freeing EP: DAT_INVALID_HANDLE DAT_INVALID_HANDLE_EP try setting DAPL_MAX_INLINE=64 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP?
Guys, I think this is the same iWARP issue that has been biting me for a while: The client must send the first RDMA message. The dtest app is a peer-2-peer (p2p) application where both sides send immediately after setting up the connection. So dtest doesn't adhere to the iWARP specification (I know: the iWARP spec is broken :). News: I have some prototype FW from chelsio that supports p2p setup and with that FW and my associated iw_cxgb3 driver/library changes, then dtest seems to work fine. These changes will be published upstream soon in order to support Open MPI and other p2p applications for chelsio. For this initial release of p2p support over chelsio, the functionality will be 100% handled in the iw_cxgb3 driver and fw. This is similar to what iw_nes does today with its send_first module option to send a 0B write from the client and defer connection establishment on the server until the 0B write is received. Chelsio will have a similar module option called peer2peer (or I could make it the same option name: send_first) that will use a 0B read to force the client to send first (chelsio cannot use a 0B write for this). The chelsio FW will defer the ESTABLISHED event until the 0B read is received and responded to. The final proper device-independent solution to this will be done in the rdma-cma, the iwarp core and iwarp devices for upstream inclusion as well as for ofed-1.4. Its a much bigger change and will affect the ABI for the rdma_cm probably (app can request p2p behavior). There was a thread a while back driven by Arkady at NetApp with details on how we will implement this (using a small protocol in mpa start req/rep to negotiate this p2p mode). Stay tuned for more on this. Steve. Steve Wise wrote: I can reproduce this. Lemme dig into it... Steve. Scott Weitzenkamp (sweitzen) wrote: Steve, Thanks, that gets further, but dtest still fails. Client side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest -h 192.168.0.198 13926 Running as client - OpenIB-cma 13926 Server Name: 192.168.0.198 13926 Server Net Address: 192.168.0.198 13926 Waiting for connect response 13926 Error unexpected conn event : DAT_CONNECTION_EVENT_UNREACHABLE 13926 Error connect_ep: DAT_ABORT 13926: DAPL Test Complete. 13926: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 13926: open:36619.19 usec 13926: close: 32500.98 usec 13926: PZ create: 7.87 usec 13926: PZ free: 4.05 usec 13926: LMR create: 58.89 usec 13926: LMR free: 11.92 usec 13926: EVD create: 9.78 usec 13926: EVD free: 14.07 usec 13926: EP create: 78.92 usec 13926: EP free:26.23 usec 13926: TOTAL: 199.79 usec Server side: [EMAIL PROTECTED] ~]$ DAPL_MAX_INLINE=64 dtest 11461 Running as server - OpenIB-cma 11461 Server waiting for connect request.. 11461 Waiting for connect response 11461 CONNECTED! 11461 Send RMR to remote: snd_msg: r_key_ctx=bff,pad=0,va=146db580,len=0x40 11461 Waiting for remote to send RMR data 11461 Error waiting on h_dto_rcv_evd: DAT_TIMEOUT_EXPIRED 11461 Error connect_ep: DAT_TIMEOUT_EXPIRED 11461: DAPL Test Complete. 11461: Message RTT: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA write: Total= 0.00 usec, 10 bursts, itime= 0.00 usec, pc= 0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: RDMA read: Total= 0.00 usec, 4 bursts, itime= 0.00 usec, pc =0 11461: open: 900676.01 usec 11461: close: 31543.97 usec 11461: PZ create: 7.87 usec 11461: PZ free: 5.01 usec 11461: LMR create: 51.98 usec 11461: LMR free: 12.16 usec 11461: EVD create: 10.97 usec 11461: EVD free: 12.87 usec 11461: EP create: 77.01 usec 11461: EP free:30.04 usec 11461: TOTAL: 195.03 usec Scott -Original Message- From: Steve Wise [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 9:19 AM To: Scott Weitzenkamp (sweitzen) Cc: Joshua Bernstein; OpenFabrics EWG; [ofa_general] Subject: Re: [ewg] RE: [ofa-general] how do I use uDAPL with iWARP? Scott Weitzenkamp (sweitzen) wrote: I tried that, and it didn't work: [EMAIL PROTECTED] ~]# grep eth /etc/dat.conf OpenIB-cma u1.2 nonthreadsafe default libdaplcma.so.1 dapl.1.2 "eth2 0" "" [EMAIL PROTECTED] ~]# dtest 10194 Runn
[ewg] [ANNOUNCE] management tarballs release
Hi, There is a new release of the management (OpenSM and infiniband diagnostics) tarballs available in: http://www.openfabrics.org/downloads/management/ md5sum: b398ef1246a392338053c8e382b3e6ee libibcommon-1.1.0.tar.gz abce72fbb91530a97493eba7a28a0de6 libibumad-1.2.0.tar.gz fe7a6b80b28e56cf74ffbe09c8819c71 libibmad-1.2.0.tar.gz b0695f75cda10051c8846fd22b77491a opensm-3.2.1.tar.gz 73218ddc536acaaab240a9d51bcd133e infiniband-diags-1.4.0.tar.gz All component versions are from recent master branch. Full change log is below. Sasha Al Chu (6): note cbb means constant bisection bandwidth opensm: multi lid routing balancing for updn/minhop Opensm: minor code cleanup Opensm: switchbalance console option opensm: add lidbalance command to console opens: fix trivial ftree comments Albert Chu (2): check_lft_balance script opensm: enforce routing paths rebalancing on switch reconnection (part 2) Albert L. Chu (2): handle routers in switchbalance console command add router support to check_lft_balance.pl Dotan Barak (1): management: Remove extraneous semicolon from several files Hal Rosenstock (10): OpenSM: Set packet life time to subnet timeout option rather than default infiniband-diags: Fix install of IBswcountlimits.pm script opensm/osm_sw_info_rcv.c: Clarify LinearFDBTop correction log message OpenSM release notes: Clarify QoS firmware support OpenSM/osm_subnet.c: Cosmetic changes to options file OpenSM release notes: Add byacc as alternative to bison for qos parser opensm/doc/partition-config.txt: Update default file name OpenSM release notes: Add in new QLogic HCAs infiniband-diags/ibping.c: Remove extraneous semicolon infiniband-diags/vendstat.c: Fix port xmit wait handling Ira Weiny (17): opensm/libvendor/osm_vendor_ibumad.c: Fix print of Transaction ID Fix 2 potential core dumps now that osm_node_get_physp_ptr can return NULL opensm/libvendor/osm_vendor_ibumad.c: add transaction ID printing to error messages Create script to automate perltidy command opensm/libvendor/osm_vendor_ibumad.c: Add environment variable control for OSM_UMAD_MAX_PENDING infiniband-diags/scripts/ibprintswitch.pl: fix printing of ports Fix bug which prevented some GUIDs from being found due to formating issues. infiniband-diags/scripts/ib[linkinfo][queryerrors].pl: report switch not found Update documentation for guid format Rename ib_gid_t in mad.h to mad_gid_t to prevent name collision with ib_types.h opensm/include/iba/ib_types.h: fix DataDetails definitions based on 1.2 and 1.2.1 specification opensm/include/iba/ib_types.h: update Notice DataDetails for Trap 144 to 1.2.1 Ensure ownership of the /etc/opensm directory infiniband-diags/scripts/set_nodedesc.sh: enhance to be able to set names other than hostname and to provide feedback on the names assigned Add an optional test utility 'ibsendtrap' Add mcm_rereg_test to test-utils option. opensm/opensm/osm_trap_rcv.c: respond to new trap 144 node description update flag Jeremy Brown (1): ibstatus - small script change Sasha Khapyorsky (78): opensm: remove redundant moving_to_master flag opensm: kill drop_mgr, link_mgr and mcast_mgr SM sub-objects opensm: remove unused header files opensm: indentation fixes opensm/osm_sminfo_rcv.c: comments fixing opensm/osm_helper.c: make some static opensm/osm_sm_state_mgr: remove unused function opensm: indentation fixes opensm: label indentation fixes opensm/osm_console.c: indentation fixes opensm/osm_console.c: fix unused func warning opensm: drop unused parameter in OSM_LOG_ENTER macro opensm/osm_log: OSM_LOG() macro opensm: convert to OSM_LOG() macro opensm: Release Notes for 3.1.9 opensm/doc: Remove list of ofed-1.2 bug fixes from OpenSM Release notes. opensm/osm_node: trivial code consolidation opensm/osm_sa_pkey_record: fix typo opensm: fix potential core dumps opensm: check p_physp for null before using opensm/osm_sa_slvl_record.c: fix typo in log print opensm/libvendor: use CL_HTON64() macro for constant conversion opensm/osm_vendor_ibumad: simplify put_madw() prototype opensm/osm_switch.c: comment typo fixing opensm: rename OpenSM startup script to opensmd opensm/scripts: rename all opensm scripts as *.in opensm/scripts: make configurable scripts opensm/doc: rename OpenSM Release notes to 3.1.10 opensm: consolidate osm_sa_vendor_send() status check opensm: move osm_sa_send_error() to osm_sa.c file opensm: cosmetic code clean in SA area opensm/osm_sa_service_record.c: remove unneeded braces libvendor/osm_vendor_ibumad_sa.c: cosmetic opensm: consolidate SA response sending code over SA p
[ewg] 10 ways to control your desire!
http://oactxocmvxwi.blogspot.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] [ANNOUNCE] management tarballs release
On Thu, Apr 03, 2008 at 09:35:11PM +, Sasha Khapyorsky wrote: > Hi, > > There is a new release of the management (OpenSM and infiniband > diagnostics) tarballs available in: I get compile errors for opensm-3.2.1 because osm_console_io.h is missing. Does the make dist target need to be updated to put that file in the tarball? In file included from main.c:61: ../include/opensm/osm_opensm.h:56:35: error: opensm/osm_console_io.h: No such file or directory If you're going to respin the package for that, could you also do a quick test of opensm with no IB cable attached to the HCA? I found that opensm 3.2.0 would spin and hog a CPU when there was no cable attached. It's a pathological case, but sometimes happens in my lab. -- Scott M. Ferris, [EMAIL PROTECTED] ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] [ANNOUNCE] management tarballs release
On Thursday 03 April 2008 21:45:00 Scott M. Ferris wrote: > On Thu, Apr 03, 2008 at 09:35:11PM +, Sasha Khapyorsky wrote: > > Hi, > > > > There is a new release of the management (OpenSM and infiniband > > diagnostics) tarballs available in: > > I get compile errors for opensm-3.2.1 because osm_console_io.h is > missing. Does the make dist target need to be updated to put that > file in the tarball? > > In file included from main.c:61: > ../include/opensm/osm_opensm.h:56:35: error: opensm/osm_console_io.h: No > such file or directory Same here, you can get the file from this link: http://www.openfabrics.org/git/?p=~sashak/management.git;a=tree;f=opensm/include/opensm;h=7dc361f88e573927627c9a394eab4bd95011ee8b;hb=HEAD Cheers, Bernd -- Bernd Schubert Q-Leap Networks GmbH ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] [ANNOUNCE] management tarballs release
On 14:45 Thu 03 Apr , Scott M. Ferris wrote: > > I get compile errors for opensm-3.2.1 because osm_console_io.h is > missing. Does the make dist target need to be updated to put that > file in the tarball? Sure, it should be. I will re upload fixed tarball. > If you're going to respin the package for that, could you also do a > quick test of opensm with no IB cable attached to the HCA? Unfortunately I cannot do it now - don't have any equipment available. > I found > that opensm 3.2.0 would spin and hog a CPU when there was no cable > attached. It's a pathological case, but sometimes happens in my lab. Thanks for reporting (although it would be better to have this report right after 3.2.0). I will look at this after Sonoma. Sasha ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ANNOUNCE] management tarballs release
On 21:35 Thu 03 Apr , Sasha Khapyorsky wrote: > Hi, > > There is a new release of the management (OpenSM and infiniband > diagnostics) tarballs available in: > > http://www.openfabrics.org/downloads/management/ > > md5sum: > > b398ef1246a392338053c8e382b3e6ee libibcommon-1.1.0.tar.gz > abce72fbb91530a97493eba7a28a0de6 libibumad-1.2.0.tar.gz > fe7a6b80b28e56cf74ffbe09c8819c71 libibmad-1.2.0.tar.gz > b0695f75cda10051c8846fd22b77491a opensm-3.2.1.tar.gz OpenSM tarball was replaced by: 997d10f81896a0d70e0f21f0e78eca92 opensm-3.2.1.tar.gz (due to compilation issue). Sorry about inconsistency. Sasha ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Re: [ofa-general] [ANNOUNCE] management tarballs release
On 22:18 Thu 03 Apr , Bernd Schubert wrote: > > > > In file included from main.c:61: > > ../include/opensm/osm_opensm.h:56:35: error: opensm/osm_console_io.h: No > > such file or directory > > Same here, Should be fixed now. Sasha ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] Your mom naked
fukYrBoJSez Watch the video nowCyrfukYrBo___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, Apr 3, 2008 at 5:40 PM, Tang, Changqing <[EMAIL PROTECTED]> wrote: > The problem is, from MPI side, (and by default), we don't know which port is > on which > fabric, since the subnet prefix is the same. We rely on system admin to > config two > different subnet prefixes for HP-MPI to work. > > No vendor has claimed to support this. CQ, not supporting a different subnet prefix per IB subnet is against IB nature, I don't think there should be any problem to configure a different prefix at each open SM instance and the Linux host stack would work perfectly under this config. If you are a ware to any problem in the opensm and/or the host stack please let the community know and the maintainers will fix it. Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, Apr 3, 2008 at 5:53 PM, Tang, Changqing <[EMAIL PROTECTED]> wrote: > for example, in MPI, process A know the HCA guid on another node. After > running for > some time, the switch is restarted for some reason, and the whole fabric is > re-configured. CQ, If by "the whole fabric is re-configured" you refer to a case where a subnet prefix changes while a job runs and a process is detached/reattached to the job so now you want to adopt your design to handle it, is over engineering, why you want to do that? Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
