Re: [openib-general] IP addressing on InfiniBand networks
On Wed, 29 Jun 2005, Roland Dreier wrote: James> - IB services (both kernel and user space) will be configured using James> IP addresses. By IB services, I'm referring to protocols that are James> layered directly on top of the InfiniBand protocols (e.g. SDP, James> NFS-RDMA, iSER, etc.). James> - IB services will resolve IP addresses to IB addresses using IPoIB James> ARP (all IB nodes would be required to support IPoIB). James> - each IB service would place appropriate IP address information in James> its protocol messages That's mostly right, except that I would leave it up to the individual protocols if and how they want to use IP addressing. For example, the SCSI RDMA Protocol (SRP) published by INCITS T10 does not use IP addresses in any way, and it seems that should be allowed and supported. Thanks for the clarification. I understand your position now. I think it is worth reiterating the two major dependencies imposed by this solution: - IB nodes must run IPoIB - ULPs must add IP addressing information on IB networks Of those two, I see the second as the most difficult. The standardization of additional protocol messages may take a significant amount of time. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
Hello, Comments inline. James Lentini wrote: I'd like to summarize the discussion we've been having around addressing and start a new email thread with a more appropriate title. First off, here is there requirement we are trying to satisfy: kDAPL consumers use an Internet Protocol (IP) addresses to identify remote nodes in an interoperable way. On the active side of a connection, an InfiniBand kDAPL provider must determine the IP address corresponding IB address (see the dat_ep_connect() function). On the passive side of a connection, a InfiniBand kDAPL provider must determine a source IB address for an InfiniBand connection request. This information can be obtain by a kDAPL consumer either in the DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or dat_cr_query()'s dat_cr_param structure. By interoperable, we mean that the solution must not introduce a non-standard protocol or force ULPs using kDAPL to perform special operations when using an InfiniBand network. The first sentence of this requirement is the most important. This is to best support the existing practice of ULPs that use IP addresses: NFS, iSCSI, and and sockets applications. Administrators expect to be able to configure systems using IP addresses. Note that this feature will be used. There has been considerable discussion on how NFS-RDMA software would make use of this feature. Other ULPs may use this feature as well. Of the two mappings, the second (IB address to IP address) has proven the most difficult to implement. Here are the different implementations that have been proposed and a brief critique of each: + CM Private Data The active side of an IB connection could place its source IP address in the CM's private data. The passive side would retrieve the source IP from this location. Analysis: This approach introduces a new protocol that is not hidden from the ULP. The problem becomes where in the software stack this would be implemented. If it was implemented in the DAT provider, kDAPL consumers would not be able to communicate fully with non-kDAPL consumers (the non-kDAPL consumers would not be expecting the IP address and thus interpret the private data incorrectly). If this were implemented in a ULP, the ULP would not be able to use the same code for both an IB interface and an RNIC interface. I think the DAT provider should fill in the CM private-data area with the source IP address (and a bit of additional) information. So, ULPs aren't participating in this aspect of IB communication. Also, it should be possible to have a DAT ULP communicate with a peer that isn't implemented using DAT as long as the location and format of the source IP address (and additional) information in the CM private-data area is specified. For example, there is a draft "iSER on IB" document that specifies the format of additional information sent in the CM private-data area. Similar document could be written for NFS using RDMA and other ULPs. [Perhaps there should be an IBA defined structure for this purpose.] For ULPs written for DAT, this DAT provider does this for the ULP. The security of this is very week. An end node could easily present a false IP address. If the DAT provider is responsible for providing this information where DAT is used, the security is no worse than IP. + Address Translation Service (ATS) The ATS proposal doesn't describe how to select the P_Key that must be specified when installing the SA record. In cases where IPoIB is running on the IB subnet and the IP address is bound to an IPoIB link interface, it's possible to query the IP stack to determine the P_Key associated with the IP address. In cases where IPoIB isn't used on the IB subnet, it's not clear where the IP address and P_Key come from. [I've been told that one solution is to use the IP address bound to an ethernet port and install an SA entry for every IB partition that the node can access based on the contents of its local P_Key table in the HCA.] If an IB fabric chooses not to run IPoIB, perhaps this is a scenario where the IB GID should be used as the IPv6 address as others have suggested. As a result, ATS is not needed. ATS also requires a mechanism on each IB node that monitors the status of IP addresses and updates the SA state. Each IB node places a record containing its IP address and IB address in the SA database. kDAPL consults these records to map between addresses. Analysis: This requires all IB nodes to implement a new protocol/adhere to a new convention. The advantage is that ULPs do not need to be aware of it. Since a GID can have multiple IP address, there may be multiple records with the same GID. The passive side would need a convention for assuming which of these matches to the active side's IP. The security of this solution is also fairly weak. The end nodes must be trusted to place valid
Re: [openib-general] IP addressing on InfiniBand networks
Hello, Caitlin Bestler wrote: On 6/28/05, Roland Dreier <[EMAIL PROTECTED]> wrote: James> First off, here [are the] requirement we are trying to satisfy: James>On the passive side of a connection, a InfiniBand kDAPL James> provider must determine a source IB address for an James> InfiniBand connection request. This information can be James> obtain by a kDAPL consumer either in the James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or James> dat_cr_query()'s dat_cr_param structure. James>By interoperable, we mean that the solution must not James> introduce a non-standard protocol or force ULPs using kDAPL James> to perform special operations when using an InfiniBand James> network. Since these two points are mutually contradictory -- the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address -- we need to figure out which one to drop. I would argue in favor of the solution selected by SDP: when defining the binding of an abstract protocol to the IB transport, put the source and destination IP addresses in the IB-specific connection setup messages. - R. The remote identification is a service being provided *to* the ULP. SDP, or any other application, can provide whatever additional information desired. The "IA Address" is as authenticated as the local network management allows it to be. Private data exchanged by applications is outside the view of the network administrator and therefore can never be authenticated in the same way. For NFSoRDMA, for example, "authenticating" a remote peer based upon an application supplied field in the private data is obviously inappropriate. In proposals that I've seen to use the CM private-data area, the information exchanged for authenticating the remote peer is performed by the provider, not the ULP. The DAPL provider is responsible for figuring out how to handle IA addresses. I think this operation is consistent with your statements above. However, I'm aware of one issue. I'm told that the DAPL implementation already uses the CM private-data area to enable ULPs to exchange information. As a result, there isn't any space left over for a SDP-like Hello messages that could be exchanged by the providers. Reducing the space available to DAPL ULPs to make room for the Hello message may be a change that is visible to ULPs even though this is not an API change. The GID that InfiniBand does supply *does* fully qualify as an IP Address, and can be the address supplied as the "remote address" even *if* there are additional methods of translating IPv4 addresses to GIDs (or even IPv6, but why you would want to *translate* an IPv6 address to a GID rather than just using the GID *as* an IPv6 address is beyond me). I recommend that the mechanism that is chosen be capable of supporting all three cases. Furthermore, I think the impact on IP administrative tools needs to be assessed when using IB GIDs as IPv6 addresses. For example, should it be possible to use ifconfig(1m) to assign an IPv6 address to the GID on an IB port? What is the observed behavior of IP administrative tools when one of the "additional properties / restrictions defined within IBA" related to GID assignment is encountered? What interfaces are required in the SM to enable coordination of IB GIDs with IP layer infrastructure? I think this is all doable, but a bit of work. I think that we should view the ability to use GIDs as IPv6 addresses as an optimization, not a fundamental function of the mechanism. Deployment of the optimization can then occur later. -David To clarify: DAT *does* fully support the use of GIDs as IA Addresses, however it assumes that link local addresses will not be presented to the Consumer (at least when the host is attached to more than one subnet). It assumes that they will be at least upgraded to site-local. The IA Address can be easily sub-divided into IPv4 addresses that need translation, IPv6 addresses that need translation and IPv6 addresses that are also GIDs and therefore do not need translation. The existence of the alternate solutions was largely driven by the need to deploy early solutions that were *not* integrated with the OS naming system, and therefore could not assume that the OS already knew what to do with a GID. That is no longer a problem. Therefore there is no need to change any of the DAT semantics. They are adequate as they are, and in particular there is no need to eliminate reporting of remote peer addresses -- something that is both easy and useful for IP networks. And a feature that is already in use. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
James> I want to make sure I understand your solution. If we choose this James> option: James> - IB services (both kernel and user space) will be configured using James> IP addresses. By IB services, I'm referring to protocols that are James> layered directly on top of the InfiniBand protocols (e.g. SDP, James> NFS-RDMA, iSER, etc.). James> - IB services will resolve IP addresses to IB addresses using IPoIB James> ARP (all IB nodes would be required to support IPoIB). James> - each IB service would place appropriate IP address information in James> its protocol messages That's mostly right, except that I would leave it up to the individual protocols if and how they want to use IP addressing. For example, the SCSI RDMA Protocol (SRP) published by INCITS T10 does not use IP addresses in any way, and it seems that should be allowed and supported. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Tue, 28 Jun 2005, Roland Dreier wrote: James> First off, here [are the] requirement we are trying to satisfy: James>On the passive side of a connection, a InfiniBand kDAPL James> provider must determine a source IB address for an James> InfiniBand connection request. This information can be James> obtain by a kDAPL consumer either in the James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or James> dat_cr_query()'s dat_cr_param structure. James>By interoperable, we mean that the solution must not James> introduce a non-standard protocol or force ULPs using kDAPL James> to perform special operations when using an InfiniBand James> network. Since these two points are mutually contradictory -- the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address -- we need to figure out which one to drop. I would argue in favor of the solution selected by SDP: when defining the binding of an abstract protocol to the IB transport, put the source and destination IP addresses in the IB-specific connection setup messages. I want to make sure I understand your solution. If we choose this option: - IB services (both kernel and user space) will be configured using IP addresses. By IB services, I'm referring to protocols that are layered directly on top of the InfiniBand protocols (e.g. SDP, NFS-RDMA, iSER, etc.). - IB services will resolve IP addresses to IB addresses using IPoIB ARP (all IB nodes would be required to support IPoIB). - each IB service would place appropriate IP address information in its protocol messages Did I get it right? james ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
Roland> Just to be clear, the IBA spec is very clear that a GID Roland> _is_ an IPv6 address. Hal> albeit with additional properties/restrictions on IBA which Hal> do not apply to IPv6 (IBA 1,2 p, 143 lines 11-16). Right: all GIDs are IPv6 addresses. However, as you point out, the converse is not true: all IPv6 addresses are not necessarily GIDs. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Wed, 2005-06-29 at 12:00, Roland Dreier wrote: > Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. albeit with additional properties/restrictions on IBA which do not apply to IPv6 (IBA 1,2 p, 143 lines 11-16). -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
Caitlin> An assigned GID meets all of the requirements for an IA Caitlin> Address. I think taking advantage of that existing Caitlin> capability is just one of many options that can be done Caitlin> by the IB CM rather than forcing IB specific changes up Caitlin> to the application layer. Just to be clear, the IBA spec is very clear that a GID _is_ an IPv6 address. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On 6/29/05, Kanevsky, Arkady <[EMAIL PROTECTED]> wrote: > Most of the existing apps kernel and user space are based on socket > addressing nad naming convention including IP addresses and ports. > All RDMA APIs made a decision to only deal with IP address and hide > IB/MAC > addresses under the covers so Consumer do not have to change their > existing IP addresses based setup. > We will severelly hamper adoption if we request any changes to > application > addressing scheme. > We should strive to the requirement that no ULP changes are needed > in order to use RDMA. I'm in full agreement, but want to add one clarification. The DAT APIs do not require that the user work with an IP Address. They work with an IA Address that has the same format as an IPv6 address and meets all of the semantics of an IPv6 address. But we never *required* it to be an IPv6 address. The actual distinction has more to do with IANA that anything visible to an application. There is one concrete requirement that the IA Address be unique from the perspective on the host. That is, it cannot be a link dependent address. The consensus was that link dependent addresses were something foreign to current network programming and not something that an application should have to deal with. Link-local addresses are intended for bootstrapping, not applications. An assigned GID meets all of the requirements for an IA Address. I think taking advantage of that existing capability is just one of many options that can be done by the IB CM rather than forcing IB specific changes up to the application layer. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
RE: [openib-general] IP addressing on InfiniBand networks
Most of the existing apps kernel and user space are based on socket addressing nad naming convention including IP addresses and ports. All RDMA APIs made a decision to only deal with IP address and hide IB/MAC addresses under the covers so Consumer do not have to change their existing IP addresses based setup. We will severelly hamper adoption if we request any changes to application addressing scheme. We should strive to the requirement that no ULP changes are needed in order to use RDMA. James were nicely summirizes the technical requirements derived from this. Second, we have had 2 DAT plugfests were most of IB and iWARP providers demonstrated interoperability including the IP addressing one and satisfying all requirements that James stated. For IB it was done through ATS. Before we adopt another solution we must ensure that it interoperates. Third, kDAPL and uDAPL are APIs for RDMA transports in general not just IB. Obviously the problem of determining remote IP address and port are IB specific. While currently people contemplating on using DAT Registry as iWARP insertion point sooner or later the insertion point will be pushed down the stack. Since this is IB CM specific issue kDAPL and uDAPL implementation should have the proper design on how to switch the code path based on underlying RDMA network type to support this feature. Arkady Arkady Kanevsky email: [EMAIL PROTECTED] Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 Waltham, MA 02451-2010 central phone: 781-768-5300 > -Original Message- > From: Hal Rosenstock [mailto:[EMAIL PROTECTED] > Sent: Wednesday, June 29, 2005 7:09 AM > To: Lentini, James > Cc: Caitlin Bestler; Christoph Hellwig; openib-general > Subject: Re: [openib-general] IP addressing on InfiniBand networks > > > On Tue, 2005-06-28 at 18:10, James Lentini wrote: > > Are there any tools available for a network administrator > to assign a > > GID? Does OpenSM provider this capability? > > As far as OpenIB OpenSM, there is currrently no capability to > set SM GUIDs. (Note that the SM assigned GID is comprised of > the SM supplied subnet prefix in PortInfo and these additional GUIDs). > > -- Hal > > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Tue, 2005-06-28 at 18:10, James Lentini wrote: > Are there any tools available for a network administrator to assign a > GID? Does OpenSM provider this capability? As far as OpenIB OpenSM, there is currrently no capability to set SM GUIDs. (Note that the SM assigned GID is comprised of the SM supplied subnet prefix in PortInfo and these additional GUIDs). -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
James> I found this limitation in section 3.3 of the "IP over James> InfiniBand(IPoIB) Architecture" draft (April, 2004 version, James> http://www.ietf.org/internet-drafts/draft-ietf-ipoib-architecture-04.txt) James> Until the above conditions are met it is not James> possible to implement IPoIB subnets that span IB James> subnets. The IPoIB standards have however been defined with James> this possibility in mind. James> Am I looking at an old version of the spec? Those conditions are pretty much things that would have to be fixed for it to be possible to communicate between IB subnets at all. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
Are there any tools available for a network administrator to assign a GID? Does OpenSM provider this capability? On Tue, 28 Jun 2005, Caitlin Bestler wrote: Site-local GIDs are just as manageable as site-local IPv6 addresses. In fact they look just like them to /etc/hosts, NIS, DNS, etc. The only real work required of the local network administrator(s) is that they assign *different* site-local network IDs for IPv6 and IB. There are 64K total available, so I doubt anyone is going to run out of available network IDs. On 6/28/05, James Lentini <[EMAIL PROTECTED]> wrote: That would require systems to be configured using IB addresses. Can that be made feasible in non-trivial configurations? For example, how could name resolution be made scalable? If you want to resolve a hostname like foo.bar.com to an IB GID, then information about IB GIDs needs to added to the various name resolution systems (/etc/hosts, NIS, DNS,...). Adding GIDs to these databases doesn't seem like a prudent thing to do. GIDs come in different flavors (adapter specific and subnet manager assigned) none of which appear to be good candidates for large scale management. On Tue, 28 Jun 2005, Christoph Hellwig wrote: hch> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: hch> > hch> > I'd like to summarize the discussion we've been having around hch> > addressing and start a new email thread with a more appropriate title. hch> > hch> > First off, here is there requirement we are trying to satisfy: hch> > hch> > kDAPL consumers use an Internet Protocol (IP) addresses to hch> > identify remote nodes in an interoperable way. hch> hch> I don't think that does belong into the kernel. It's fine to do that in hch> userspace if you want. hch> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
Site-local GIDs are just as manageable as site-local IPv6 addresses. In fact they look just like them to /etc/hosts, NIS, DNS, etc. The only real work required of the local network administrator(s) is that they assign *different* site-local network IDs for IPv6 and IB. There are 64K total available, so I doubt anyone is going to run out of available network IDs. On 6/28/05, James Lentini <[EMAIL PROTECTED]> wrote: > > That would require systems to be configured using IB addresses. Can > that be made feasible in non-trivial configurations? > > For example, how could name resolution be made scalable? If you want > to resolve a hostname like foo.bar.com to an IB GID, then information > about IB GIDs needs to added to the various name resolution systems > (/etc/hosts, NIS, DNS,...). Adding GIDs to these databases doesn't > seem like a prudent thing to do. GIDs come in different flavors > (adapter specific and subnet manager assigned) none of which appear to > be good candidates for large scale management. > > On Tue, 28 Jun 2005, Christoph Hellwig wrote: > > hch> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: > hch> > > hch> > I'd like to summarize the discussion we've been having around > hch> > addressing and start a new email thread with a more appropriate title. > hch> > > hch> > First off, here is there requirement we are trying to satisfy: > hch> > > hch> > kDAPL consumers use an Internet Protocol (IP) addresses to > hch> > identify remote nodes in an interoperable way. > hch> > hch> I don't think that does belong into the kernel. It's fine to do that in > hch> userspace if you want. > hch> > ___ > openib-general mailing list > openib-general@openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Tue, 28 Jun 2005, Hal Rosenstock wrote: On Tue, 2005-06-28 at 15:24, James Lentini wrote: + IPoIB IPoIB encapsulates IP packets in InfiniBand messages. There have been proposals to use the address resolution mechanisms in IPoIB to implement these features. IPv4 subnets use ARP and IPv6 subnets use Neighbor Discovery. Analysis: IPoIB is not free. All nodes would be need to implement it for this to work. The IB address -> IP address mapping on the passive side is problematic. If a reverse lookup were available, IPoIB would require both a GID and QP number as input. The passive side would know the GID but the QP number. Further more, reverse lookup is not well supported. On IPv4 subnets, RARP is quickly becoming (already?) obsolete. The IPoIB HW address includes the QPN (in addition to the GID). This is also problematic. Neighbor Discovery doesn't support reverse lookup at all. [RFC 2461] In addition to all this, IPoIB restricts an IP subnet to the same scope as an IB subnet. IPoIB does not limit an IP subnet to an IB subnet. It can span IB subnets. However, IB routers were not completed in the IB architecture. I found this limitation in section 3.3 of the "IP over InfiniBand(IPoIB) Architecture" draft (April, 2004 version, http://www.ietf.org/internet-drafts/draft-ietf-ipoib-architecture-04.txt) Until the above conditions are met it is not possible to implement IPoIB subnets that span IB subnets. The IPoIB standards have however been defined with this possibility in mind. Am I looking at an old version of the spec? If a kDAPL consumer desired to communicate between IB subnet's, IPoIB may not be sufficient. Are you referring to 2 disjoint IB subnets ? Yes. Your point is valid. If there are no routers, there is no reason to worry about it. What about IB <-> iWARP ? I hadn't been considering that kind of communication. My assumption is that if a translator was created for IB <-> iWARP it would solve this issue...actually, a hypothetical translator is likely to use the solution we choose. + GID as an IPv6 Address See the attachment to Caitlin Bestler's email: http://openib.org/pipermail/openib-general/2005-June/008104.html Analysis: This has been the least discussed option. One issue is that GIDs may not be easy to administer. GIDs can be specific to a particular channel adapter since they can contain EUI-64 identifiers. Administrators avoid using Ethernet MAC addresses in configuration files and they should be able to avoid using adapter specific IB addresses as well. If they don't like ethernet MACs, they really won't like GUIDs/GIDs as they are even longer. Length aside, GUIDs/GIDs are not manageable like IP addresses. Another issue is how dynamically assigned SM GIDs would be managed. Do you mean SM (assigned additional) GUIDs ? No, I was referring to the SM assigned GIDs described in property 3c of section 4.1.1 of the IB spec. -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
James> Which two points? The ability to determine a source IB James> address for an InfiniBand connection request and James> interoperability? James> I don't think those two are contradictory by definition. Yes, those two points I quoted. As I said, "the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address." So if you're not allowed to add anything to the connection establishment protocol, then you don't have enough information to find a source address. Something has to give. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Tue, 28 Jun 2005, Roland Dreier wrote: James> First off, here [are the] requirement we are trying to satisfy: James>On the passive side of a connection, a InfiniBand kDAPL James> provider must determine a source IB address for an James> InfiniBand connection request. This information can be James> obtain by a kDAPL consumer either in the James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or James> dat_cr_query()'s dat_cr_param structure. James>By interoperable, we mean that the solution must not James> introduce a non-standard protocol or force ULPs using kDAPL James> to perform special operations when using an InfiniBand James> network. Since these two points are mutually contradictory Which two points? The ability to determine a source IB address for an InfiniBand connection request and interoperability? I don't think those two are contradictory by definition. -- the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address -- we need to figure out which one to drop. I would argue in favor of the solution selected by SDP: when defining the binding of an abstract protocol to the IB transport, put the source and destination IP addresses in the IB-specific connection setup messages. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
That would require systems to be configured using IB addresses. Can that be made feasible in non-trivial configurations? For example, how could name resolution be made scalable? If you want to resolve a hostname like foo.bar.com to an IB GID, then information about IB GIDs needs to added to the various name resolution systems (/etc/hosts, NIS, DNS,...). Adding GIDs to these databases doesn't seem like a prudent thing to do. GIDs come in different flavors (adapter specific and subnet manager assigned) none of which appear to be good candidates for large scale management. On Tue, 28 Jun 2005, Christoph Hellwig wrote: hch> On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: hch> > hch> > I'd like to summarize the discussion we've been having around hch> > addressing and start a new email thread with a more appropriate title. hch> > hch> > First off, here is there requirement we are trying to satisfy: hch> > hch> > kDAPL consumers use an Internet Protocol (IP) addresses to hch> > identify remote nodes in an interoperable way. hch> hch> I don't think that does belong into the kernel. It's fine to do that in hch> userspace if you want. hch> ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On 6/28/05, Roland Dreier <[EMAIL PROTECTED]> wrote: > James> First off, here [are the] requirement we are trying to satisfy: > > James>On the passive side of a connection, a InfiniBand kDAPL > James> provider must determine a source IB address for an > James> InfiniBand connection request. This information can be > James> obtain by a kDAPL consumer either in the > James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or > James> dat_cr_query()'s dat_cr_param structure. > > James>By interoperable, we mean that the solution must not > James> introduce a non-standard protocol or force ULPs using kDAPL > James> to perform special operations when using an InfiniBand > James> network. > > Since these two points are mutually contradictory -- the IB > communication management protocol does not carry enough information > for a connection request to be mapped uniquely back to a source > address -- we need to figure out which one to drop. > > I would argue in favor of the solution selected by SDP: when defining > the binding of an abstract protocol to the IB transport, put the > source and destination IP addresses in the IB-specific connection > setup messages. > > - R. The remote identification is a service being provided *to* the ULP. SDP, or any other application, can provide whatever additional information desired. The "IA Address" is as authenticated as the local network management allows it to be. Private data exchanged by applications is outside the view of the network administrator and therefore can never be authenticated in the same way. For NFSoRDMA, for example, "authenticating" a remote peer based upon an application supplied field in the private data is obviously inappropriate. The GID that InfiniBand does supply *does* fully qualify as an IP Address, and can be the address supplied as the "remote address" even *if* there are additional methods of translating IPv4 addresses to GIDs (or even IPv6, but why you would want to *translate* an IPv6 address to a GID rather than just using the GID *as* an IPv6 address is beyond me). To clarify: DAT *does* fully support the use of GIDs as IA Addresses, however it assumes that link local addresses will not be presented to the Consumer (at least when the host is attached to more than one subnet). It assumes that they will be at least upgraded to site-local. The IA Address can be easily sub-divided into IPv4 addresses that need translation, IPv6 addresses that need translation and IPv6 addresses that are also GIDs and therefore do not need translation. The existence of the alternate solutions was largely driven by the need to deploy early solutions that were *not* integrated with the OS naming system, and therefore could not assume that the OS already knew what to do with a GID. That is no longer a problem. Therefore there is no need to change any of the DAT semantics. They are adequate as they are, and in particular there is no need to eliminate reporting of remote peer addresses -- something that is both easy and useful for IP networks. And a feature that is already in use. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Tue, 2005-06-28 at 15:24, James Lentini wrote: > + IPoIB > > IPoIB encapsulates IP packets in InfiniBand messages. There have been > proposals to use the address resolution mechanisms in IPoIB to > implement these features. IPv4 subnets use ARP and IPv6 subnets use > Neighbor Discovery. > > Analysis: > > IPoIB is not free. All nodes would be need to implement it for > this to work. > > The IB address -> IP address mapping on the passive side is > problematic. If a reverse lookup were available, IPoIB would require > both a GID and QP number as input. The passive side would know the GID > but the QP number. > > Further more, reverse lookup is not well supported. On IPv4 subnets, > RARP is quickly becoming (already?) obsolete. The IPoIB HW address includes the QPN (in addition to the GID). This is also problematic. > Neighbor Discovery > doesn't support reverse lookup at all. [RFC 2461] > > In addition to all this, IPoIB restricts an IP subnet to the same scope > as an IB subnet. IPoIB does not limit an IP subnet to an IB subnet. It can span IB subnets. However, IB routers were not completed in the IB architecture. > If a kDAPL consumer desired to communicate between > IB subnet's, IPoIB may not be sufficient. Are you referring to 2 disjoint IB subnets ? What about IB <-> iWARP ? > + GID as an IPv6 Address > > See the attachment to Caitlin Bestler's email: > > http://openib.org/pipermail/openib-general/2005-June/008104.html > > Analysis: > > This has been the least discussed option. One issue is > that GIDs may not be easy to administer. GIDs can be specific > to a particular channel adapter since they can contain EUI-64 > identifiers. Administrators avoid using Ethernet MAC addresses > in configuration files and they should be able to avoid using > adapter specific IB addresses as well. If they don't like ethernet MACs, they really won't like GUIDs/GIDs as they are even longer. > Another issue is how > dynamically assigned SM GIDs would be managed. Do you mean SM (assigned additional) GUIDs ? -- Hal ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
James> First off, here [are the] requirement we are trying to satisfy: James>On the passive side of a connection, a InfiniBand kDAPL James> provider must determine a source IB address for an James> InfiniBand connection request. This information can be James> obtain by a kDAPL consumer either in the James> DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or James> dat_cr_query()'s dat_cr_param structure. James>By interoperable, we mean that the solution must not James> introduce a non-standard protocol or force ULPs using kDAPL James> to perform special operations when using an InfiniBand James> network. Since these two points are mutually contradictory -- the IB communication management protocol does not carry enough information for a connection request to be mapped uniquely back to a source address -- we need to figure out which one to drop. I would argue in favor of the solution selected by SDP: when defining the binding of an abstract protocol to the IB transport, put the source and destination IP addresses in the IB-specific connection setup messages. - R. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
Re: [openib-general] IP addressing on InfiniBand networks
On Tue, Jun 28, 2005 at 03:24:35PM -0400, James Lentini wrote: > > I'd like to summarize the discussion we've been having around > addressing and start a new email thread with a more appropriate title. > > First off, here is there requirement we are trying to satisfy: > > kDAPL consumers use an Internet Protocol (IP) addresses to > identify remote nodes in an interoperable way. I don't think that does belong into the kernel. It's fine to do that in userspace if you want. ___ openib-general mailing list openib-general@openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
