Hello,
Comments inline.
James Lentini wrote:
I'd like to summarize the discussion we've been having around
addressing and start a new email thread with a more appropriate title.
First off, here is there requirement we are trying to satisfy:
kDAPL consumers use an Internet Protocol (IP) addresses to
identify remote nodes in an interoperable way.
On the active side of a connection, an InfiniBand kDAPL provider must
determine the IP address corresponding IB address (see the
dat_ep_connect() function).
On the passive side of a connection, a InfiniBand kDAPL provider must
determine a source IB address for an InfiniBand connection request.
This information can be obtain by a kDAPL consumer either in the
DAT_CONNECTION_REQUEST_EVENT's dat_cr_arrival_event_data or
dat_cr_query()'s dat_cr_param structure.
By interoperable, we mean that the solution must not introduce a
non-standard protocol or force ULPs using kDAPL to perform special
operations when using an InfiniBand network.
The first sentence of this requirement is the most important. This is
to best support the existing practice of ULPs that use IP addresses:
NFS, iSCSI, and and sockets applications. Administrators expect to be
able to configure systems using IP addresses.
Note that this feature will be used. There has been considerable
discussion on how NFS-RDMA software would make use of this feature.
Other ULPs may use this feature as well.
Of the two mappings, the second (IB address to IP address) has proven
the most difficult to implement. Here are the different
implementations that have been proposed and a brief critique of each:
+ CM Private Data
The active side of an IB connection could place its source IP
address in the CM's private data. The passive side would retrieve
the source IP from this location.
Analysis: This approach introduces a new protocol that is not
hidden from the ULP. The problem becomes where in the software
stack this would be implemented. If it was implemented in the DAT
provider, kDAPL consumers would not be able to communicate fully
with non-kDAPL consumers (the non-kDAPL consumers would not
be expecting the IP address and thus interpret the private data
incorrectly). If this were implemented in a ULP, the ULP would
not be able to use the same code for both an IB interface and an
RNIC interface.
I think the DAT provider should fill in the CM
private-data area with the source IP address (and a bit of additional)
information. So, ULPs aren't participating in this aspect of IB
communication. Also, it should be possible to have a DAT ULP
communicate with a peer that isn't implemented using DAT as long as the
location and format of the source IP address (and additional)
information in the CM private-data area is specified. For example,
there is a draft "iSER on IB" document that specifies the format of
additional information sent in the CM private-data area. Similar
document could be written for NFS using RDMA and other ULPs. [Perhaps
there should be an IBA defined structure for this purpose.] For ULPs
written for DAT, this DAT provider does this for the ULP.
The security of this is very week. An end node could easily present
a false IP address.
If the DAT provider is responsible for providing this information where
DAT is used, the security is no worse than IP.
+ Address Translation Service (ATS)
The ATS proposal doesn't describe how to select the P_Key that must be
specified when installing the SA record. In cases where IPoIB is
running on the IB subnet and the IP address is bound to an IPoIB link
interface, it's possible to query the IP stack to determine the P_Key
associated with the IP address. In cases where IPoIB isn't used on the
IB subnet, it's not clear where the IP address and P_Key come from.
[I've been told that one solution is to use the IP address bound to an
ethernet port and install an SA entry for every IB partition that the
node can access based on the contents of its local P_Key table in the HCA.]
If an IB fabric chooses not to run IPoIB, perhaps this is a scenario
where the IB GID should be used as the IPv6 address as others have
suggested. As a result, ATS is not needed.
ATS also requires a mechanism on each IB node that monitors the status
of IP addresses and updates the SA state.
Each IB node places a record containing its IP address and IB
address in the SA database. kDAPL consults these records to
map between addresses.
Analysis: This requires all IB nodes to implement a new
protocol/adhere to a new convention. The advantage is that ULPs do
not need to be aware of it.
Since a GID can have multiple IP address, there may be multiple
records with the same GID. The passive side would need a convention
for assuming which of these matches to the active side's IP.
The security of this solution is also fairly weak. The end nodes
must be trusted to place valid records in the SA. Since the
configuration of IP addresses will occur on the end nodes, it would
be difficult for the SA to validate the records. However, the SA is
a central part of the network, so it may be possible to add additional
security features if needed.
It is possible to have the SA authenticate any IB client attempting to
install an entry in the SA. The SA record is protected by a Service
Key. By default, the Service Key value is probably zero - no
protection. However, the IB administrator can assign a value to the
Service Key associated with a Service Name. ATS specifies a well-known
Service Name of "DAPL Address Translation Service", so the ATS
specification could describe steps needed to assign a Service Key value
for it. Also, this Service Key would need to be securely distributed to
all the IB clients that have authority to enter SA records with the
Service Name for ATS.
-David
+ IPoIB
IPoIB encapsulates IP packets in InfiniBand messages. There have been
proposals to use the address resolution mechanisms in IPoIB to
implement these features. IPv4 subnets use ARP and IPv6 subnets use
Neighbor Discovery.
Analysis:
IPoIB is not free. All nodes would be need to implement it for
this to work.
The IB address -> IP address mapping on the passive side is
problematic. If a reverse lookup were available, IPoIB would require
both a GID and QP number as input. The passive side would know the GID
but the QP number.
Further more, reverse lookup is not well supported. On IPv4 subnets,
RARP is quickly becoming (already?) obsolete. Neighbor Discovery
doesn't support reverse lookup at all. [RFC 2461]
In addition to all this, IPoIB restricts an IP subnet to the same
scope
as an IB subnet. If a kDAPL consumer desired to communicate between
IB subnet's, IPoIB may not be sufficient.
+ GID as an IPv6 Address
See the attachment to Caitlin Bestler's email:
http://openib.org/pipermail/openib-general/2005-June/008104.html
Analysis:
This has been the least discussed option. One issue is
that GIDs may not be easy to administer. GIDs can be specific
to a particular channel adapter since they can contain EUI-64
identifiers. Administrators avoid using Ethernet MAC addresses
in configuration files and they should be able to avoid using
adapter specific IB addresses as well. Another issue is how
dynamically assigned SM GIDs would be managed.
Are there other implementations? Is there a way to more tightly
integrate IP addressing with InfiniBand?
_______________________________________________
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
