Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
The SGI Altix ICE cluster system supports 2 InfiniBand fabrics. http://www.sgi.com/products/servers/altix/ice/ Each compute node has 2 HCAs and each is connected to a separate fabric. We recommend that users use one fabric for storage traffic and the other for MPI, but there is no reason why both fabrics could not be used for MPI. OpenMPI requires setting a separate subnet prefix for each fabric to use both fabrics for MPI and OpenSM supports this setting of subnet prefix. Other MPIs do not require this. Edward on 04/04/2008 08:08 AM Tang, Changqing said the following: What I mean claim to support is to have more people to test with this config. --CQ -Original Message- From: Or Gerlitz [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 11:18 PM To: Tang, Changqing Cc: [EMAIL PROTECTED]; ewg@lists.openfabrics.org Subject: Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans On Thu, Apr 3, 2008 at 5:40 PM, Tang, Changqing [EMAIL PROTECTED] wrote: The problem is, from MPI side, (and by default), we don't know which port is on which fabric, since the subnet prefix is the same. We rely on system admin to config two different subnet prefixes for HP-MPI to work. No vendor has claimed to support this. CQ, not supporting a different subnet prefix per IB subnet is against IB nature, I don't think there should be any problem to configure a different prefix at each open SM instance and the Linux host stack would work perfectly under this config. If you are a ware to any problem in the opensm and/or the host stack please let the community know and the maintainers will fix it. Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
What I mean claim to support is to have more people to test with this config. --CQ -Original Message- From: Or Gerlitz [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 11:18 PM To: Tang, Changqing Cc: [EMAIL PROTECTED]; ewg@lists.openfabrics.org Subject: Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans On Thu, Apr 3, 2008 at 5:40 PM, Tang, Changqing [EMAIL PROTECTED] wrote: The problem is, from MPI side, (and by default), we don't know which port is on which fabric, since the subnet prefix is the same. We rely on system admin to config two different subnet prefixes for HP-MPI to work. No vendor has claimed to support this. CQ, not supporting a different subnet prefix per IB subnet is against IB nature, I don't think there should be any problem to configure a different prefix at each open SM instance and the Linux host stack would work perfectly under this config. If you are a ware to any problem in the opensm and/or the host stack please let the community know and the maintainers will fix it. Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
for example, in MPI, process A know the HCA guid on another node. After running for some time, the switch is restarted for some reason, and the whole fabric is re-configured. CQ, If by the whole fabric is re-configured you refer to a case where a subnet prefix changes while a job runs and a process is detached/reattached to the job so now you want to adopt your design to handle it, is over engineering, why you want to do that? I am concerning the port lid change. It is always the best if a process can figure the info it needs by itself, SA query is the right way and is in IB spec. while it is possible to let processes to exchange information(port lid) again, but there are difficulties: during the middle of a long job run, it is hard to let two processes to coordinate such infomation exchange, and it requires a second channel to do so. If the second channel is IPoIB, it is broken as well, and we need to re-establish it again. I just ask for the SA functionalities. If it is not possible, we have to use a very complicated way to let HP-MPI to survive from network failure. --CQ Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, 2008-04-03 at 14:40 +, Tang, Changqing wrote: You have a system, all HCAs have two ports, all port 1 are connected to the first switch, all port 2 are connected to the second switch, there is NO link between the two switches. We call this system has two physically separated fabrics. If you have a bridge link between the two switches, then it becomes a single fabric. The same thing for multiple HCAs on nodes. The problem is, from MPI side, (and by default), we don't know which port is on which fabric, since the subnet prefix is the same. We rely on system admin to config two different subnet prefixes for HP-MPI to work. Yes, these two IB subnets need two different subnet prefixes. (I think it's more than just HP MPI which needs this). -- Hal No vendor has claimed to support this. --CQ -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 9:32 AM To: Tang, Changqing Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans CQ, On Thu, 2008-04-03 at 14:27 +, Tang, Changqing wrote: Can we address multiple-fabrics (physically separated) support ? Can you elaborate on what you mean by physically separated ? -- Hal --CQ Tang -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Erez Zilber Sent: Thursday, April 03, 2008 8:51 AM To: Tziporet Koren Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans *OFED 1.4:* 1. Kernel base: since we target 1.4 release to Sep we target the kernel base to be 2.6.27 This is a good target, but we may need to stay with 2.6.26 if the kernel progress will not be aligned. 2. Suggestions for new features: * NFS-RDMA * Verbs: Reliable Multicast (to be presented at Sonoma) * SDP - Zero copy (There was a question on IPv6 support - seems no one interested for now) * IPoIB - continue with performance enhancements * Xsigo new virtual NIC * New vendor HW support - non was reported so far (IBM and Chelsio - do you have something?) * OpenSM: o Incremental routing o Temporary SA DB - to answer queries and a heavy sweep is done o APM - disjoint paths (?) o MKey manager (?) o Sasha to send more management features * MPI: o Open MPI 1.3 o APM support in MPI o mvapich ??? * uDAPl o Extensions for new APIs (like XRC) - ? o uDAPL provider for interop between Windows Linux o 1.2 and 2.0 will stay As I wrote in an earlier discussion (~2 months ago), we plan to add tgt (SCSI target) with iSCSI over iSER (and TCP of course) support. The git tree for tgt already exists on the ofa server. Erez ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, 2008-04-03 at 14:53 +, Tang, Changqing wrote: One other thing I hope to talk is some fabric query functionalities for normal user, not only just for root. This is at IB verbs level, not rdma_cm level. for example, in MPI, process A know the HCA guid on another node. After running for some time, the switch is restarted for some reason, and the whole fabric is re-configured. Now process A wants to know if the port lid on another node has changed or not, it knows the HCA guid, is there any function to query this ? I know as root, we can use the mad/umad library to do this kind of query, I want to do such query in MPI, which is a normal user. In the IB arch, there are SA registrations and queries for the specific example you used. However, these are not directly exposed to Linux user space directly (for the normal user as opposed to MAD user (note there are some difficulties in making this available to the normal user)) (at least not yet AFAIK). While these are not (direct) fabric query (really SA query), they serve the same function in a different way. -- Hal --CQ Tang, HP-MPI -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Erez Zilber Sent: Thursday, April 03, 2008 8:51 AM To: Tziporet Koren Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans *OFED 1.4:* 1. Kernel base: since we target 1.4 release to Sep we target the kernel base to be 2.6.27 This is a good target, but we may need to stay with 2.6.26 if the kernel progress will not be aligned. 2. Suggestions for new features: * NFS-RDMA * Verbs: Reliable Multicast (to be presented at Sonoma) * SDP - Zero copy (There was a question on IPv6 support - seems no one interested for now) * IPoIB - continue with performance enhancements * Xsigo new virtual NIC * New vendor HW support - non was reported so far (IBM and Chelsio - do you have something?) * OpenSM: o Incremental routing o Temporary SA DB - to answer queries and a heavy sweep is done o APM - disjoint paths (?) o MKey manager (?) o Sasha to send more management features * MPI: o Open MPI 1.3 o APM support in MPI o mvapich ??? * uDAPl o Extensions for new APIs (like XRC) - ? o uDAPL provider for interop between Windows Linux o 1.2 and 2.0 will stay As I wrote in an earlier discussion (~2 months ago), we plan to add tgt (SCSI target) with iSCSI over iSER (and TCP of course) support. The git tree for tgt already exists on the ofa server. Erez ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
Thanks. When can we have the SA features, very soon, long time, or never ? --CQ -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 10:02 AM To: Tang, Changqing Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans On Thu, 2008-04-03 at 14:53 +, Tang, Changqing wrote: One other thing I hope to talk is some fabric query functionalities for normal user, not only just for root. This is at IB verbs level, not rdma_cm level. for example, in MPI, process A know the HCA guid on another node. After running for some time, the switch is restarted for some reason, and the whole fabric is re-configured. Now process A wants to know if the port lid on another node has changed or not, it knows the HCA guid, is there any function to query this ? I know as root, we can use the mad/umad library to do this kind of query, I want to do such query in MPI, which is a normal user. In the IB arch, there are SA registrations and queries for the specific example you used. However, these are not directly exposed to Linux user space directly (for the normal user as opposed to MAD user (note there are some difficulties in making this available to the normal user)) (at least not yet AFAIK). While these are not (direct) fabric query (really SA query), they serve the same function in a different way. -- Hal --CQ Tang, HP-MPI -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Erez Zilber Sent: Thursday, April 03, 2008 8:51 AM To: Tziporet Koren Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans *OFED 1.4:* 1. Kernel base: since we target 1.4 release to Sep we target the kernel base to be 2.6.27 This is a good target, but we may need to stay with 2.6.26 if the kernel progress will not be aligned. 2. Suggestions for new features: * NFS-RDMA * Verbs: Reliable Multicast (to be presented at Sonoma) * SDP - Zero copy (There was a question on IPv6 support - seems no one interested for now) * IPoIB - continue with performance enhancements * Xsigo new virtual NIC * New vendor HW support - non was reported so far (IBM and Chelsio - do you have something?) * OpenSM: o Incremental routing o Temporary SA DB - to answer queries and a heavy sweep is done o APM - disjoint paths (?) o MKey manager (?) o Sasha to send more management features * MPI: o Open MPI 1.3 o APM support in MPI o mvapich ??? * uDAPl o Extensions for new APIs (like XRC) - ? o uDAPL provider for interop between Windows Linux o 1.2 and 2.0 will stay As I wrote in an earlier discussion (~2 months ago), we plan to add tgt (SCSI target) with iSCSI over iSER (and TCP of course) support. The git tree for tgt already exists on the ofa server. Erez ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, 2008-04-03 at 15:11 +, Tang, Changqing wrote: Thanks. When can we have the SA features, very soon, long time, or never ? I'm unaware of any current plans to implement these but my knowledge is far from complete... -- Hal --CQ -Original Message- From: Hal Rosenstock [mailto:[EMAIL PROTECTED] Sent: Thursday, April 03, 2008 10:02 AM To: Tang, Changqing Cc: Erez Zilber; Tziporet Koren; ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: RE: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans On Thu, 2008-04-03 at 14:53 +, Tang, Changqing wrote: One other thing I hope to talk is some fabric query functionalities for normal user, not only just for root. This is at IB verbs level, not rdma_cm level. for example, in MPI, process A know the HCA guid on another node. After running for some time, the switch is restarted for some reason, and the whole fabric is re-configured. Now process A wants to know if the port lid on another node has changed or not, it knows the HCA guid, is there any function to query this ? I know as root, we can use the mad/umad library to do this kind of query, I want to do such query in MPI, which is a normal user. In the IB arch, there are SA registrations and queries for the specific example you used. However, these are not directly exposed to Linux user space directly (for the normal user as opposed to MAD user (note there are some difficulties in making this available to the normal user)) (at least not yet AFAIK). While these are not (direct) fabric query (really SA query), they serve the same function in a different way. -- Hal --CQ Tang, HP-MPI -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Erez Zilber Sent: Thursday, April 03, 2008 8:51 AM To: Tziporet Koren Cc: ewg@lists.openfabrics.org; [EMAIL PROTECTED] Subject: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans *OFED 1.4:* 1. Kernel base: since we target 1.4 release to Sep we target the kernel base to be 2.6.27 This is a good target, but we may need to stay with 2.6.26 if the kernel progress will not be aligned. 2. Suggestions for new features: * NFS-RDMA * Verbs: Reliable Multicast (to be presented at Sonoma) * SDP - Zero copy (There was a question on IPv6 support - seems no one interested for now) * IPoIB - continue with performance enhancements * Xsigo new virtual NIC * New vendor HW support - non was reported so far (IBM and Chelsio - do you have something?) * OpenSM: o Incremental routing o Temporary SA DB - to answer queries and a heavy sweep is done o APM - disjoint paths (?) o MKey manager (?) o Sasha to send more management features * MPI: o Open MPI 1.3 o APM support in MPI o mvapich ??? * uDAPl o Extensions for new APIs (like XRC) - ? o uDAPL provider for interop between Windows Linux o 1.2 and 2.0 will stay As I wrote in an earlier discussion (~2 months ago), we plan to add tgt (SCSI target) with iSCSI over iSER (and TCP of course) support. The git tree for tgt already exists on the ofa server. Erez ___ general mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, Apr 3, 2008 at 5:40 PM, Tang, Changqing [EMAIL PROTECTED] wrote: The problem is, from MPI side, (and by default), we don't know which port is on which fabric, since the subnet prefix is the same. We rely on system admin to config two different subnet prefixes for HP-MPI to work. No vendor has claimed to support this. CQ, not supporting a different subnet prefix per IB subnet is against IB nature, I don't think there should be any problem to configure a different prefix at each open SM instance and the Linux host stack would work perfectly under this config. If you are a ware to any problem in the opensm and/or the host stack please let the community know and the maintainers will fix it. Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ofa-general] Re: [ewg] OFED March 24 meeting summary on OFED 1.4 plans
On Thu, Apr 3, 2008 at 5:53 PM, Tang, Changqing [EMAIL PROTECTED] wrote: for example, in MPI, process A know the HCA guid on another node. After running for some time, the switch is restarted for some reason, and the whole fabric is re-configured. CQ, If by the whole fabric is re-configured you refer to a case where a subnet prefix changes while a job runs and a process is detached/reattached to the job so now you want to adopt your design to handle it, is over engineering, why you want to do that? Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
