subject:"\[Rdma\-developers\] Re\: \[openib\-general\] OpenIB and OpenRDMA\: Convergence on common RDMA APIs and ULPs for Linux"

Venkata,
   How will that work? If the RNIC offloads RDMA and
TCP completely from the Operating System and does not
share any state information then the application
running on the host will never be in the position to
utilize the socket interface to use the communication
logic to send and receive data between the remote node
and itself. Some information needs to be shared. How
much of it and what exactly needs to be shared is the
question.

Thanks
SG

--- Venkata Jagana [EMAIL PROTECTED] wrote:
 
 
 
 
 
 
 [EMAIL PROTECTED] wrote on
 05/25/2005 09:47:00
 PM:
 
  Venkata,
  Interesting coincidence: I was talking with
 someone (at HP) today
  who knows substantially more than I do about
 RNICs.
  They indicated RNICs need to manage TCP state on
 the card from userspace.
  I suspect that's only possible through a private
 interface
  (e.g. ioctl() or /proc) or the non-existant (in
 kernel.org)
  TOE implementation. Is this correct?
 
 
 Not correct.
 
 Since RNICs are offloaded adapters with RDMA
 protocols layered on
 top of TCP stack, they do maintain the TCP state
 internally but
 it does not expose to the host. RNIC expose only
 RNIC Verbs interface
 to the host bot not TOE interface.
 
 Thanks
 Venkat
 
 
  hth,
  grant
 
 
 

---
  SF.Net email is sponsored by: GoToMeeting - the
 easiest way to
 collaborate
  online with coworkers and clients while avoiding
 the high cost of travel
 and
  communications. There is no equipment to buy and
 you can meet as often as
  you want. Try it

free.http://ads.osdn.com/?ad_id=7402alloc_id=16135op=click
  ___
  Rdma-developers mailing list
  [EMAIL PROTECTED]
  https://lists.sourceforge.net/lists/listinfo/rdma-developers

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

I completely agree with Michael's viewpoint here.
Doing a clean RDMA focussed implementation would bring
out the exact requirements that need to be changed.
Retrofitting IB/Infiniband based API into the RDMA
based subsystem just to reuse an existing base is not
a good start.

Thanks
SG

--- Michael Krause [EMAIL PROTECTED] wrote:

 At 09:49 AM 5/26/2005, Sean Hefty wrote:
 Roland Dreier wrote:
 I believe the way forward is to evolve the
 existing drivers/infiniband
 code already in Linux into a drivers/rdma that
 supports both IB and
 RNICs.  To be extremely blunt, I believe the
 RNIC-PI is irrelevant to
 the Linux kernel -- no IB vendors will support
 ripping out a working
 midlayer and starting from scratch, and it doesn't
 make sense to have
 two essentially equivalent midlayers in the same
 kernel.
 
 IMO, any APIs need to evolve out of the
 implementation.  Trying to fit an 
 implementation under an existing API tends to lead
 to either a poor 
 implementation, or requires changes to the API
 anyway.
 
 I think a useful path is for someone to implement
 an RNIC driver and 
 provide feedback on what changes would be required
 of the Infiniband/core 
 layer to support it.
 
 There needs to be a balance between establishing a
 good, flexible 
 architecture and how applications or subsystems
 interact with it.  All of 
 this needs to be grounded in implementation
 experience so there is a 
 healthy does of reality.  The problem with the
 iterative-TTM focused API 
 definitions is they rarely produce something that is
 focused on the end 
 game for a solid interface.  They create instant
 legacy baggage that 
 people are unwilling to shed as time goes forward. 
 This legacy inertia is 
 what has stifled quite a bit of innovation when it
 comes to software (some 
 hazard that 90% of the software created today is
 focused primarily on 
 legacy investment protection than really new
 innovation and when you think 
 about it for a bit of time, you can see that much of
 this is re-inventing 
 the wheel or to band-aid over a problem and
 packaging it as something 
 innovative).
 
 So, the problem becomes one of finding balance.  The
 people proposing the 
 various API are people who have implemented various
 types of solutions so 
 they are coming with both real experience as well as
 engineering judgement 
 of what is required to get the right infrastructure
 in place for the end 
 game needs.  From a practical perspective, one
 needs to implement these 
 API and see what is really of value and what should
 be changed.  The key is 
 to avoid creating the legacy inertia that ends up
 reducing fragmenting the 
 interfaces and causes lost productivity, quality
 problems, etc. So before 
 people write off the various API as unnecessary or
 as poor quality, there 
 should be some implementations developed and some
 constructive debate of 
 what features are really of value and what might be
 deprecated or 
 eliminated.  There is no requirement to ever
 implement all of an API as 
 this is where an iterative approach has value. 
 Implement what is needed to 
 demonstrate the desired value and decide then if it
 is worthwhile.  Just 
 please don't discount it or toss it all out because
 it isn't all 
 implemented today or you don't like some aspect.
 
 
 To put a really concrete proposal on the table, I
 would suggest to
 start by extending the current ib_client
 registration structure
 
 Roland's proposal sounds like a reasonable and fair
 way to 
 begin.  Building an abstraction on top of the
 existing layers seems 
 secondary to adding support for other RDMA devices.
 
 Getting the right verbs interface is paramount. 
 Whether one does IT API or 
 DAPL on top of that for a given subsystem is
 secondary.
 
 Ideally, OpenIB and OpenRDMA should be focused on
 developing the RDMA 
 infrastructure - RDMA verbs, connection management,
 IB-specific ULP, 
 etc.  They should not be focused on the upper
 subsystems or ULP.  Those 
 should be done as separate open source projects and
 they can decide where 
 best to intersect with the Open* infrastructure. 
 This one-size-fits-all 
 approach for all subsystems to flow through a given
 interface is simply 
 impractical to execute.  It might happen over time
 but it cannot be 
 force-fit.  Let's focus the two efforts on getting
 as common of an 
 infrastructure as possible.  Some propose RNIC PI;
 other can propose 
 something else if there is desire.  RNIC PI has
 value in that it has 
 focused on the common components between IB and
 iWARP and left the 
 IB-specific and CM outside of its definition to
 allow an OS to decide how 
 best to support or to allow IB to do what is needed
 for its particular usages.
 
 
 Mike 




__ 
Do you Yahoo!? 
Yahoo! Small Business - Try our new Resources site
http://smallbusiness.yahoo.com/resources/
___
openib-general mailing list

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux



At 06:40 AM 5/27/2005, Sukanta ganguly wrote:
Venkata,
 How will that work? If the RNIC offloads RDMA and
TCP completely from the Operating System and does not
share any state information then the application
running on the host will never be in the position to
utilize the socket interface to use the communication
logic to send and receive data between the remote node
and itself. Some information needs to be shared. How
much of it and what exactly needs to be shared is the
question.
Ok. It all depends upon what level of integration / interaction a
TOE and thus a RNIC will have with the host network stack. For
example, if a customer wants to have TCP and IP stats kept for the
off-loaded stack even if it is just being using for RDMA, then there
needs to be a method defined to consolidate these stats back into the
host network stack tool chain. Similarly, if one wants to maintain
a single routing table to manage, etc. on the host, then the RNIC needs
to access / update that information accordingly. One can progress
through other aspects of integration, e.g. connection management,
security interactions (e.g. DOS protection), and so forth. What is
exposed again depends upon the level of integration and how customers
want to manage their services. This problem also exists for IB but
most people have not thought about this from a customer perspective and
how to integrate the IB semantics into the way customers manage their
infrastructures, do billing, etc. For some environments, they
simply do not care but if IB is to be used in the enterprise space, then
some thought will be required here since most IT don't see anything as
being free or self-managed.
Again, Sockets is an application API and not how one communicates to a
TOE or RDMA component. The RNIC PI has been proposed as an
interface to the RDMA functionality. The PI supports all of the
iWARP and IB v 1.2 verbs. 
Mike

Thanks
SG
--- Venkata Jagana [EMAIL PROTECTED] wrote:
 
 
 
 
 
 
 [EMAIL PROTECTED] wrote on
 05/25/2005 09:47:00
 PM:
 
  Venkata,
  Interesting coincidence: I was talking with
 someone (at HP) today
  who knows substantially more than I do about
 RNICs.
  They indicated RNICs need to manage TCP state on
 the card from userspace.
  I suspect that's only possible through a private
 interface
  (e.g. ioctl() or /proc) or the non-existant (in
 kernel.org)
  TOE implementation. Is this correct?
 
 
 Not correct.
 
 Since RNICs are offloaded adapters with RDMA
 protocols layered on
 top of TCP stack, they do maintain the TCP state
 internally but
 it does not expose to the host. RNIC expose only
 RNIC Verbs interface
 to the host bot not TOE interface.
 
 Thanks
 Venkat
 
 
  hth,
  grant
 
 
 

---
  SF.Net email is sponsored by: GoToMeeting - the
 easiest way to
 collaborate
  online with coworkers and clients while avoiding
 the high cost of travel
 and
  communications. There is no equipment to buy and
 you can meet as often as
  you want. Try it

free.
http://ads.osdn.com/?ad_id=7402alloc_id=16135op=click
  ___
  Rdma-developers mailing list
  [EMAIL PROTECTED]
 

https://lists.sourceforge.net/lists/listinfo/rdma-developers
__
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around


http://mail.yahoo.com 

---
This SF.Net email is sponsored by Yahoo.
Introducing Yahoo! Search Developer Network - Create apps using
Yahoo!
Search APIs Find out how you can build Yahoo! directly into your own
Applications - visit

http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
___
Rdma-developers mailing list
[EMAIL PROTECTED]

https://lists.sourceforge.net/lists/listinfo/rdma-developers


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

2005-05-27 Thread Bernard Metzler


Sukanta,

without touching any TOE issues (this
question is about RDMA, right?), after
transforming a TCP connection into RDMA
mode and using an RDMA API,
the socket file descriptor is not longer
to be used for communication.
In fact, on some implementations the
stream socket resources will
even get released. That is, if the socket
was the direct consumer
of the TCP stream, then now it is RDMAP/DDP/MPA.
RDMA APIs such as IT-API defining a
specific call to convert a socket based
connection into RDMA mode (e.g., it_socket_convert()).
Other consumers may directly start via
an RDMA API, never
opening a consumer controlled socket.

So, in RDMA mode, communication will
happen via the RDMA API. At this
stage, the kernel still have to keep
completely in its hands the
synchronisation of state information
related to that offloaded
connection(s) with the host stack (it
would have to protect the
local port used by the offloaded connection
for example, others
are routing, ARP, SNMP...), but it is
not involved at the data path.

With respect to the kernel based TCP
stack, what is not needed is
a hack into the stack and scatter/gather
state information of the live
TCP connection between kernel and RNIC,
but to find one clean interface
to transfer state information out of
that stack and to the RNIC.


With limited benefit, one could of course
also implement native 
sockets over RDMA, where an in-kernel
midlayer on top of kernel
RDMA Verbs is doing the translation
between send(), receive() to 
post_send, post_receive. But usage of
'true RDMA' operations
like RDMA READ or WRITE might be limited,
and I don't see much value 
for the user here. One variety of this
approach with less limited
access to RMDA benefits might be sockets
with extended RDMA 
semantics.

Bernard.

[EMAIL PROTECTED] wrote
on 27.05.2005 15:40:43:

 Venkata,
  How will that work? If the RNIC offloads RDMA and
 TCP completely from the Operating System and does not
 share any state information then the application
 running on the host will never be in the position to
 utilize the socket interface to use the communication
 logic to send and receive data between the remote node
 and itself. Some information needs to be shared. How
 much of it and what exactly needs to be shared is the
 question.
 
 Thanks
 SG
 
 --- Venkata Jagana [EMAIL PROTECTED] wrote:
  
  
  
  
  
  
  [EMAIL PROTECTED] wrote on
  05/25/2005 09:47:00
  PM:
  
   Venkata,
   Interesting coincidence: I was talking with
  someone (at HP) today
   who knows substantially more than I do about
  RNICs.
   They indicated RNICs need to manage TCP state on
  the card from userspace.
   I suspect that's only possible through a private
  interface
   (e.g. ioctl() or /proc) or the non-existant (in
  kernel.org)
   TOE implementation. Is this correct?
  
  
  Not correct.
  
  Since RNICs are offloaded adapters with RDMA
  protocols layered on
  top of TCP stack, they do maintain the TCP state
  internally but
  it does not expose to the host. RNIC expose only
  RNIC Verbs interface
  to the host bot not TOE interface.
  
  Thanks
  Venkat
  
  
   hth,
   grant
  
  
  
 
 ---
   SF.Net email is sponsored by: GoToMeeting - the
  easiest way to
  collaborate
   online with coworkers and clients while avoiding
  the high cost of travel
  and
   communications. There is no equipment to buy and
  you can meet as often as
   you want. Try it
 
 free.http://ads.osdn.com/?ad_id=7402alloc_id=16135op=click
   ___
   Rdma-developers mailing list
   [EMAIL PROTECTED]
   https://lists.sourceforge.net/lists/listinfo/rdma-developers
 
 __
 Do You Yahoo!?
 Tired of spam? Yahoo! Mail has the best spam protection around

 http://mail.yahoo.com 
 
 
 ---
 This SF.Net email is sponsored by Yahoo.
 Introducing Yahoo! Search Developer Network - Create apps using Yahoo!
 Search APIs Find out how you can build Yahoo! directly into your own
 Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
 ___
 Rdma-developers mailing list
 [EMAIL PROTECTED]
 https://lists.sourceforge.net/lists/listinfo/rdma-developers
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

2005-05-27 Thread Grant Grundler

On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote:
...
 Again, Sockets is an application API and not how one communicates to a TOE 
 or RDMA component.

Mike,
What address family is used to open a socket over iWARP? AF_INET?
Or something else?

I understand most of what you wrote but am still missing one bit:
How is the RNIC told what the peer IP is it should communicate with?


 The RNIC PI has been proposed as an interface to the 
 RDMA functionality.  The PI supports all of the iWARP and IB v 1.2 verbs.

That's good. Folks from RDMA consortium will have to look at
openib implementations and see whats missing/wrong.
Then submit proposals to fill in the gaps.
I'm obviously not the first one to say this.

I expect most of the principals involved with openib.org do NOT
have time to browse through RNIC PI at this point. They
are struggling to get openib.org filled in sufficiently
so it can go into a commercial distro (RH/SuSE primarily).

Revenue for them comes from selling IB equipment. Having
openib.org code in kernel.org is a key enabler for getting
into commercial distros.  I expect the same is true for RNIC
vendors as well.

RNIC Vendors (and related switch Vendors) will have to decide which
path is the right one for them to get the support into kernel.org.
Several openib.org people have suggested one (like I have).
RNIC folks need to listen and decide if the advice is good or not.
If RNIC folks think they know better, then please take another look
at where openib.org is today and where rdmaconsortium is.

I'm certain openib.org would be dead now if policies and direction
changes had not made last year as demanded by several key linux
developers and users (Gov Labs).

grant
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux






  
  
  From: 
  [EMAIL PROTECTED] 
  [mailto:[EMAIL PROTECTED] On Behalf Of 
  Michael KrauseSent: Friday, May 27, 2005 7:25 AMTo: 
  Sukanta gangulyCc: openib-general@openib.org; 
  [EMAIL PROTECTED]Subject: Re: [Rdma-developers] 
  Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and 
  ULPs for Linux
  At 06:40 AM 5/27/2005, Sukanta ganguly wrote:
  Venkata, How will 
that work? If the RNIC offloads RDMA andTCP completely from the 
Operating System and does notshare any state information then the 
applicationrunning on the host will never be in the position 
toutilize the socket interface to use the communicationlogic to send 
and receive data between the remote nodeand itself. Some information 
needs to be shared. Howmuch of it and what exactly needs to be shared is 
thequestion.
  Ok. It all depends upon what level of integration / interaction 
  a TOE and thus a RNIC will have with the host network stack. For 
  example, if a customer wants to have TCP and IP stats kept for the off-loaded 
  stack even if it is just being using for RDMA, then there needs to be a method 
  defined to consolidate these stats back into the host network stack tool 
  chain. Similarly, if one wants to maintain a single routing table to 
  manage, etc. on the host, then the RNIC needs to access / update that 
  information accordingly. One can progress through other aspects of 
  integration, e.g. connection management, security interactions (e.g. DOS 
  protection), and so forth. What is exposed again depends upon the level 
  of integration and how customers want to manage their services. This 
  problem also exists for IB but most people have not thought about this from a 
  customer perspective and how to integrate the IB semantics into the way 
  customers manage their infrastructures, do billing, etc. For some 
  environments, they simply do not care but if IB is to be used in the 
  enterprise space, then some thought will be required here since most IT don't 
  see anything as being "free" or self-managed.Again, Sockets is an 
  application API and not how one communicates to a TOE or RDMA component. 
  The RNIC PI has been proposed as an interface to the RDMA functionality. 
  The PI supports all of the iWARP and IB v 1.2 verbs. 
  Mike
  
I'd like to add that RNIC-PI is 
planning onexplicitly defining some of these"obvious" 
dependencies
betweenthe RDMA stack and the 
primary IP stack. For example, the RDMA stack cannot maintain
any connection in a state that 
contradicts current IP stack routing. It has to adapt or break the 
connection.
We can't have an RNIC that has its 
own ARP table that is not in sync with the host's ARP table.

An iWarp RDMA stack gains the benefit of many pre-existing network 
services (such as DNS, ARP
and routing). But that also carries with it the need to not contradict 
those exisiting services. So it is
both a benefit and a restriction -- and a major divergence from an IB 
RDMA stack.


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux





  With respect 
  to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state 
  information of the live TCP connection 
  between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to 
  the RNIC.
Agreed that is an 
OS-specific service that should not be 
implementedin
a device 
dependent way by each hardware vendor. But the 
information
extracted from 
the kernel IP stack cannot be encoded as though it
came from an 
InfiniBand CM.


___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux



At 09:29 AM 5/27/2005, Grant Grundler wrote:
On Fri, May 27, 2005 at
07:24:44AM -0700, Michael Krause wrote:
...
 Again, Sockets is an application API and not how one communicates to
a TOE 
 or RDMA component.
Mike,
What address family is used to open a socket over iWARP? AF_INET?
Or something else?
TCP = AF_INET. Address family != Sockets. Sockets is an API
that can operate over multiple address families. An application can
be coded to Sockets, IT API, DAPL, or a verbs interface like the RNIC
PI. It is a matter of choice as well as what is trying to be
accomplished. The RNIC PI is an acceptable interface for any
RDMA-focused ULP. There are pros / cons to using such a verbs
interface directly but I do not believe any one can deny that a
general-purpose verbs API is a good thing at the end of the day as it
works for the volume verbs definition. Whether one applies further
hardware semantics abstraction such as IT API / DAPL should be a choice
for the individual subsystem as there is no single right answer across
all subsystems. Attempting to force fit isn't practical.
I understand most
of what you wrote but am still missing one bit:
How is the RNIC told what the peer IP is it should communicate
with?
The destination address (IB GID or IP) is derived from the CM
services. This is where the two interconnects differ in what is
required to physical inject a packet on the wire. This is why I
call it out as separate from the verbs interface and something that could
be abstracted to some extent but at the end of the day, really requires
the subsystem to understand the underlying fabric type to make some
intelligent choices. Given this effort is still nascent, most of
the issues beyond basic bootstrap have not really been discussed as
yet.

 The RNIC PI
has been proposed as an interface to the 
 RDMA functionality. The PI supports all of the iWARP and IB v
1.2 verbs.
That's good. Folks from RDMA consortium will have to look at openib
implementations and see whats missing/wrong.
Then submit proposals to fill in the gaps. I'm obviously not the first
one to say this.
There are two open source efforts. The question is whether to move
to a single effort (I tried to get this to occur before OpenIB was
formally launched but it seem to fall on deaf ears for TTM marketing
purposes) or whether to just coordinate on some of the basics. My
preference remains that the efforts remained strictly focused on the RDMA
infrastructure and interconnect-specific components and leave the ULP /
services as separate efforts who will make their own decisions on how
best to interface with the RDMA infrastructure.
I expect most of
the principals involved with openib.org do NOT have time to browse
through RNIC PI at this point. They are struggling to get openib.org
filled in sufficiently so it can go into a commercial distro (RH/SuSE
primarily).
Hence, why OpenRDMA needs to get source being developed to enable the
RNIC community. If people find value in the work, then people can
look at finding the right solution for both IB and iWARP when it makes
sense.

Revenue for them
comes from selling IB equipment. Having openib.org code in kernel.org is
a key enabler for getting
into commercial distros. I expect the same is true for RNIC vendors
as well.
RNIC Vendors (and related switch Vendors) will have to decide which path
is the right one for them to get the support into kernel.org. Several
openib.org people have suggested one (like I have). RNIC folks need to
listen and decide if the advice is good or not. If RNIC folks think they
know better, then please take another look at where openib.org is today
and where rdmaconsortium is.
I'm certain openib.org would be dead now if policies and direction
changes had not made last year as demanded by several key linux
developers and users (Gov Labs).
Understood.
Mike

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux



At 08:05 AM 5/27/2005, Bernard Metzler wrote:

Sukanta, 
without touching any TOE issues (this question is
about RDMA, right?), after 
transforming a TCP connection into RDMA mode and
using an RDMA API, 
the socket file descriptor is not longer to be used
for communication. 
Whether one uses a Socket FD or not is an application issue. A QP
has a handle identifier which may be mapped to a FD or may be used
directly by an application. There is no requirement that anything
flow through the Sockets API - that is a choice for the application to
make. 
In fact, on some
implementations the stream socket resources will
even get released. That is, if the socket was the
direct consumer of the TCP
stream, then now it is RDMAP/DDP/MPA.
RDMA APIs such as IT-API defining a specific call to
convert a socket based connection
into RDMA mode (e.g., it_socket_convert()).
Other consumers may directly start via an RDMA API,
never 
opening a consumer controlled
socket. 
Correct.
So, in RDMA mode,
communication will happen via the RDMA API. At this
stage, the kernel still have to keep completely in
its hands the synchronisation of
state information related to that offloaded
connection(s) with the host stack (it would have to
protect the local port used by
the offloaded connection for example, others
are routing, ARP, SNMP...), but it is not involved at
the data path. 
This is true only when the host and the RNIC share a common IP
address. If they are separate subnets, then there is no need to
coordinate sans some of the reasons I noted in an earlier
e-mail. I think there is great value and customer need to
have an infrastructure that can coordinate information and enable
customers to use the existing tool chains to understand what is going on
in a given device or endnode.
With respect to the
kernel based TCP stack, what is not needed is
a hack into the stack and scatter/gather state
information of the live TCP
connection between kernel and RNIC, but to find one clean
interface to transfer state
information out of that stack and to the RNIC.

At a minimum, one needs:
- A host network stack get state call to extract connection state and
quiesce the port from the host's perspective.
- A host network stack set state call to populate the host structures
with the associated state and enable the port
- A RNIC get state call to extract all transport and RMDA context
- A RNIC set state call to populate all transport and RDMA
context
These calls should be standardized so that both sides of the
infrastructure will be able to utilize without having to hack
anything.
With limited
benefit, one could of course also implement native sockets over RDMA,
where an in-kernel midlayer on top of kernel 
RDMA Verbs is doing the translation between send(),
receive() to post_send, post_receive. But usage of 'true RDMA'
operations 
like RDMA READ or WRITE might be limited, and I don't
see much value for the user here. One variety of this approach with less
limited access to RMDA benefits
might be sockets with extended RDMA semantics.

Sockets with RDMA extensions was proposed a couple of years back by some
of us. There would be quite a bit of benefit as most developers
know how to code to Sockets and the RDMA extensions are relatively simple
when you think about it. It would also eliminate the need to use
SDP by placing the RDMA knowledge requirement on the application.
SDP has benefits for enabling existing Sockets applications to operate
transparently over RDMA but the explicit use of RDMA extensions I think
would provide greater benefit in the end. While it is important to
provide legacy investment protection, real innovation will not occur
until people start developing technology that enables a smarter
customer. Simplification and performance also come with enabling a
smarter customer.  
Mike

Bernard.
 
[EMAIL PROTECTED] wrote on
27.05.2005 15:40:43:
 Venkata,
 How will that work? If the RNIC offloads RDMA
and
 TCP completely from the Operating System and does not
 share any state information then the application
 running on the host will never be in the position to
 utilize the socket interface to use the communication
 logic to send and receive data between the remote node
 and itself. Some information needs to be shared. How
 much of it and what exactly needs to be shared is the
 question.
 
 Thanks
 SG
 
 --- Venkata Jagana [EMAIL PROTECTED] wrote:
  
  
  
  
  
  
  [EMAIL PROTECTED] wrote on
  05/25/2005 09:47:00
  PM:
  
   Venkata,
   Interesting coincidence: I was talking with
  someone (at HP) today
   who knows substantially more than I do about
  RNICs.
   They indicated RNICs need to manage TCP state on
  the card from userspace.
   I suspect that's only possible through a private
  interface
   (e.g. ioctl() or /proc) or the non-existant (in
  kernel.org)
   TOE implementation. Is this correct?
  
  
  Not correct.
  
  Since RNICs are offloaded adapters with RDMA
  protocols layered on
  top of TCP stack, they do maintain the TCP

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

 -Original Message-
 From: [EMAIL PROTECTED] 
 [mailto:[EMAIL PROTECTED] On 
 Behalf Of Grant Grundler
 Sent: Friday, May 27, 2005 9:29 AM
 To: Michael Krause
 Cc: Sukanta ganguly; [EMAIL PROTECTED]; 
 openib-general@openib.org
 Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB 
 and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

 On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote:
 ...
  Again, Sockets is an application API and not how one 
 communicates to a 
  TOE or RDMA component.

 Mike,
 What address family is used to open a socket over iWARP? AF_INET?
 Or something else?

 I understand most of what you wrote but am still missing one bit:
 How is the RNIC told what the peer IP is it should communicate with?

The TCP Connection is already established before the RNIC takes
control of the connection. Therefore the entire TCP state is already
established (source/dest IP Address/Port, negotiated options, EMSS,
local options, etc.).

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

2005-05-27 Thread Grant Grundler

On Fri, May 27, 2005 at 09:45:06AM -0700, Caitlin Bestler wrote:
 I'd like to add that RNIC-PI is planning on explicitly defining some of
 these obvious dependencies 
 between the RDMA stack and the primary IP stack. For example, the RDMA
 stack cannot maintain
 any connection in a state that contradicts current IP stack routing. It
 has to adapt or break the connection.

That's what I was just thinking. Could RNIC driver support both
existing linux NIC interfaces (e.g. use ifconfig/ethtool) *and*
openib RDMA interface?

Essentially that's what openib.org does today but uses ib_ipoib
driver to support TCP/IP communication.

Ie use an AF_INET socket to setup an RDMA connection like
the rdma_lat.c does:
https://openib.org/svn/gen2/trunk/src/userspace/perftest/

Of course, I'm blissfully ignorant of how ugly that might be
in real world implementation of RNIC driver. Seems simple in
concept at least.

 We can't have an RNIC that has its own ARP table that is not in sync
 with the host's ARP table.

Yes, the RDMA and NIC parts of the driver would have to be aware
of each other.

grant

 An iWarp RDMA stack gains the benefit of many pre-existing network
 services (such as DNS, ARP
 and routing). But that also carries with it the need to not contradict
 those exisiting services. So it is
 both a benefit and a restriction -- and a major divergence from an IB
 RDMA stack.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

2005-05-27 Thread Grant Grundler

On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote:
  I understand most of what you wrote but am still missing one bit:
  How is the RNIC told what the peer IP is it should communicate with?
 
 The TCP Connection is already established before the RNIC takes
 control of the connection. Therefore the entire TCP state is already
 established (source/dest IP Address/Port, negotiated options, EMSS,
 local options, etc.).

Would shares the connection be a better description?

I think I get the concepts at least now...this follows
what I was suggesting earlier about having an RNIC
support both NAPI (New API for NICs) (code under drivers/net)
and openib.org Gen2 stack (code in drivers/infiniband).

thanks,
grant
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

 -Original Message-
 From: [EMAIL PROTECTED] 
 [mailto:[EMAIL PROTECTED] On 
 Behalf Of Talpey, Thomas
 Sent: Thursday, May 26, 2005 5:28 AM
 To: Venkata Jagana
 Cc: [EMAIL PROTECTED]; openib-general@openib.org
 Subject: [Rdma-developers] Re: [openib-general] OpenIB and 
 OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

 At 02:20 AM 5/26/2005, Venkata Jagana wrote:
 I would like to really understand the technical reasons why 
 you say RNIC-PI is irrelevant to Linux kernel. 
 RNIC-PI is developed to support not only the RNICs but it is 
 also IB compatible.

 I'm not Roland, but my belief is that until RNIC-PI exists as 
 working code on a reference implementation, it is in fact not 
 yet relevant to Linux. Perhaps you can outline the schedule 
 to get there?

That is a slightly circular argument. The primary reason that
work to implement DAPL and IT-API over RNIC-PI is not already
in progress is the pressure to come up with a one stack solution.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

Mike,
   I am not sure I do understand what your were trying
to communicate. Let me try and decode this. My basic
point was to respond to Venkata's response about
complete offload without any interaction with the host
system. I disagree with that in its totality as I
think there are dependencies and that needs to be
specified in a formal manner, i.e. in the specs, so
that we do not have multiple proprietary interafaces
which change and application users have to change this
consumption based on individual implementations. 
   And you just brought up the reasons why what I was
saying seemed to be justfied. Did I read that
correctly?

Thanks
SG

--- Michael Krause [EMAIL PROTECTED] wrote:

 At 06:40 AM 5/27/2005, Sukanta ganguly wrote:
 Venkata,
 How will that work? If the RNIC offloads RDMA
 and
 TCP completely from the Operating System and does
 not
 share any state information then the application
 running on the host will never be in the position
 to
 utilize the socket interface to use the
 communication
 logic to send and receive data between the remote
 node
 and itself. Some information needs to be shared.
 How
 much of it and what exactly needs to be shared is
 the
 question.
 
 Ok.  It all depends upon what level of integration /
 interaction a TOE and 
 thus a RNIC will have with the host network stack. 
 For example, if a 
 customer wants to have TCP and IP stats kept for the
 off-loaded stack even 
 if it is just being using for RDMA, then there needs
 to be a method defined 
 to consolidate these stats back into the host
 network stack tool 
 chain.  Similarly, if one wants to maintain a single
 routing table to 
 manage, etc. on the host, then the RNIC needs to
 access / update that 
 information accordingly.  One can progress through
 other aspects of 
 integration, e.g. connection management, security
 interactions (e.g. DOS 
 protection), and so forth.  What is exposed again
 depends upon the level of 
 integration and how customers want to manage their
 services.  This problem 
 also exists for IB but most people have not thought
 about this from a 
 customer perspective and how to integrate the IB
 semantics into the way 
 customers manage their infrastructures, do billing,
 etc.  For some 
 environments, they simply do not care but if IB is
 to be used in the 
 enterprise space, then some thought will be required
 here since most IT 
 don't see anything as being free or self-managed.
 
 Again, Sockets is an application API and not how one
 communicates to a TOE 
 or RDMA component.  The RNIC PI has been proposed as
 an interface to the 
 RDMA functionality.  The PI supports all of the
 iWARP and IB v 1.2 verbs.
 
 Mike
 
 
 Thanks
 SG
 
 --- Venkata Jagana [EMAIL PROTECTED] wrote:
  
  
  
  
  
  
   [EMAIL PROTECTED]
 wrote on
   05/25/2005 09:47:00
   PM:
  
Venkata,
Interesting coincidence: I was talking with
   someone (at HP) today
who knows substantially more than I do about
   RNICs.
They indicated RNICs need to manage TCP state
 on
   the card from userspace.
I suspect that's only possible through a
 private
   interface
(e.g. ioctl() or /proc) or the non-existant
 (in
   kernel.org)
TOE implementation. Is this correct?
   
  
   Not correct.
  
   Since RNICs are offloaded adapters with RDMA
   protocols layered on
   top of TCP stack, they do maintain the TCP state
   internally but
   it does not expose to the host. RNIC expose only
   RNIC Verbs interface
   to the host bot not TOE interface.
  
   Thanks
   Venkat
  
   
hth,
grant
   
   
   
  

---
SF.Net email is sponsored by: GoToMeeting -
 the
   easiest way to
   collaborate
online with coworkers and clients while
 avoiding
   the high cost of travel
   and
communications. There is no equipment to buy
 and
   you can meet as often as
you want. Try it
  

free.http://ads.osdn.com/?ad_id=7402alloc_id=16135op=click
   
 ___
Rdma-developers mailing list
[EMAIL PROTECTED]
   

https://lists.sourceforge.net/lists/listinfo/rdma-developers
 
 __
 Do You Yahoo!?
 Tired of spam?  Yahoo! Mail has the best spam
 protection around
 http://mail.yahoo.com
 
 

---
 This SF.Net email is sponsored by Yahoo.
 Introducing Yahoo! Search Developer Network -
 Create apps using Yahoo!
 Search APIs Find out how you can build Yahoo!
 directly into your own
 Applications - visit

http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
 ___
 Rdma-developers mailing list
 [EMAIL PROTECTED]

https://lists.sourceforge.net/lists/listinfo/rdma-developers
 


__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

I agree.

SG
--- Bernard Metzler [EMAIL PROTECTED] wrote:

 Sukanta,
 
 without touching any TOE issues (this question is
 about RDMA, right?), 
 after
 transforming a TCP connection into RDMA mode and
 using an RDMA API,
 the socket file descriptor is not longer to be used
 for communication.
 In fact, on some implementations the stream socket
 resources will
 even get released. That is, if the socket was the
 direct consumer
 of the TCP stream, then now it is RDMAP/DDP/MPA.
 RDMA APIs such as IT-API defining a specific call to
 convert a socket 
 based
 connection into RDMA mode (e.g.,
 it_socket_convert()).
 Other consumers may directly start via an RDMA API,
 never
 opening a consumer controlled socket.
 
 So, in RDMA mode, communication will happen via the
 RDMA API. At this
 stage, the kernel still have to keep completely in
 its hands the
 synchronisation of state information related to that
 offloaded
 connection(s) with the host stack (it would have to
 protect the
 local port used by the offloaded connection for
 example, others
 are routing, ARP, SNMP...), but it is not involved
 at the data path.
 
 With respect to the kernel based TCP stack, what is
 not needed is
 a hack into the stack and scatter/gather state
 information of the live
 TCP connection between kernel and RNIC, but to find
 one clean interface
 to transfer state information out of that stack and
 to the RNIC.
 
 
 With limited benefit, one could of course also
 implement native 
 sockets over RDMA, where an in-kernel midlayer on
 top of kernel
 RDMA Verbs is doing the translation between send(),
 receive() to 
 post_send, post_receive. But usage of 'true RDMA'
 operations
 like RDMA READ or WRITE might be limited, and I
 don't see much value 
 for the user here. One variety of this approach with
 less limited
 access to RMDA benefits might be sockets with
 extended RDMA 
 semantics.
 
 Bernard.
 
 [EMAIL PROTECTED] wrote on
 27.05.2005 15:40:43:
 
  Venkata,
 How will that work? If the RNIC offloads RDMA
 and
  TCP completely from the Operating System and does
 not
  share any state information then the application
  running on the host will never be in the position
 to
  utilize the socket interface to use the
 communication
  logic to send and receive data between the remote
 node
  and itself. Some information needs to be shared.
 How
  much of it and what exactly needs to be shared is
 the
  question.
  
  Thanks
  SG
  
  --- Venkata Jagana [EMAIL PROTECTED] wrote:
   
   
   
   
   
   
   [EMAIL PROTECTED]
 wrote on
   05/25/2005 09:47:00
   PM:
   
Venkata,
Interesting coincidence: I was talking with
   someone (at HP) today
who knows substantially more than I do about
   RNICs.
They indicated RNICs need to manage TCP state
 on
   the card from userspace.
I suspect that's only possible through a
 private
   interface
(e.g. ioctl() or /proc) or the non-existant
 (in
   kernel.org)
TOE implementation. Is this correct?
   
   
   Not correct.
   
   Since RNICs are offloaded adapters with RDMA
   protocols layered on
   top of TCP stack, they do maintain the TCP state
   internally but
   it does not expose to the host. RNIC expose only
   RNIC Verbs interface
   to the host bot not TOE interface.
   
   Thanks
   Venkat
   
   
hth,
grant
   
   
   
  
 

---
SF.Net email is sponsored by: GoToMeeting -
 the
   easiest way to
   collaborate
online with coworkers and clients while
 avoiding
   the high cost of travel
   and
communications. There is no equipment to buy
 and
   you can meet as often as
you want. Try it
  
 

free.http://ads.osdn.com/?ad_id=7402alloc_id=16135op=click
   
 ___
Rdma-developers mailing list
[EMAIL PROTECTED]
   

https://lists.sourceforge.net/lists/listinfo/rdma-developers
  
  __
  Do You Yahoo!?
  Tired of spam?  Yahoo! Mail has the best spam
 protection around 
  http://mail.yahoo.com 
  
  
 

---
  This SF.Net email is sponsored by Yahoo.
  Introducing Yahoo! Search Developer Network -
 Create apps using Yahoo!
  Search APIs Find out how you can build Yahoo!
 directly into your own
  Applications - visit 

http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005
  ___
  Rdma-developers mailing list
  [EMAIL PROTECTED]
 

https://lists.sourceforge.net/lists/listinfo/rdma-developers
 


__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

From:
[EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of
Michael KrauseSent: Friday, May 27, 2005 9:51 AMTo:
Bernard Metzler; Sukanta gangulyCc:
[EMAIL PROTECTED]; openib-general@openib.org;
[EMAIL PROTECTED]Subject: Re:
[Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on
common RDMA APIs and ULPs for Linux
At 08:05 AM 5/27/2005, Bernard Metzler
wrote:
Sukanta, without touching any TOE issues (this
question is about RDMA, right?), after transforming a TCP connection into RDMA mode and using an RDMA
API, the socket file descriptor
is not longer to be used for communication.

Whether one uses a Socket FD or not is an application issue. A
QP has a handle identifier which may be mapped to a FD or may be used directly
by an application. There is no requirement that anything flow through
the Sockets API - that is a choice for the application to
make.

True.

The current RNIC-PI specis deliberately vague
about what an "LLP Handle" is, other than that it must include a Socket
FD.
It may be possible for it to includesome sort of
"connection request handle" for a pending connection on the offload
device,
butyou would have to ensure that this "connection
requestion handle" was reviewed by the Host IP stack so that it
could
forbid connections that contradictthe host's IP
firewall policies.

In fact, on some
implementations the stream socket resources will
even get released. That is, if the socket was the direct
consumer of the TCP stream, then now
it is RDMAP/DDP/MPA. RDMA APIs such
as IT-API defining a specific call to convert a socket based connection into RDMA mode (e.g.,
it_socket_convert()). Other
consumers may directly start via an RDMA API, never
opening a consumer controlled socket.
Correct.

There may never be a*consumer* controlled socket,
but there will be a *host* controlled socket
used
by the Connection Manager and subject to normal host
firewall controls.

It is in fact highly desirable from both a security and
performance basis for the streaming mode TCP
connectionto never be exposed tothe
application. Only specialized applications, such as
iSER,
require this
capability.

So, in RDMA mode,
communication will happen via the RDMA API. At this
stage, the kernel still have to keep completely in its
hands the synchronisation of state
information related to that offloaded connection(s) with the host stack (it would have to protect
the local port used by the offloaded
connection for example, others are
routing, ARP, SNMP...), but it is not involved at the data path.
This is true only when the host and the RNIC share a common IP
address. If they are separate subnets, then there is no need to
coordinate sans some of the reasons I noted in an earlier e-mail.
I think there is great value and customer need to have an infrastructure that
can coordinate information and enable customers to use the existing tool
chains to understand what is going on in a given device or endnode.
Good point. The key is that the RNIC needs to
doco-ordinate all of these with the IP stack that owns the
IPaddress.
If it is that owner, it must co-ordinate with itself.
(Actually that's a clause that needs to be made more explicit in
the
RNIC-PI spec, which was focused on the shared IP
scenario). But when theRNIC is "sub-leasing" the IP it
needs
to co-ordinate its action with the owner of the IP
address.

With respect to the
kernel based TCP stack, what is not needed is
a hack into the stack and scatter/gather state
information of the live TCP
connection between kernel and RNIC, but to find one clean
interface to transfer state
information out of that stack and to the RNIC.

At a minimum, one needs:- A host network stack get state call
to extract connection state and quiesce the port from the host's
perspective.- A host network stack set state call to populate the host
structures with the associated state and enable the port- A RNIC get state
call to extract all transport and RMDA context- A RNIC set state call to
populate all transport and RDMA context
An RNIC"get state" is nice, but not
essential.It enables transferof an RDMA state to another RNIC
for
failover purposes. But in terms of minimal control, the
ability to kill an RDMA controlled connection
is
sufficient.

Additionally, any transfer of RDMA Context will be
limited to transfer between like models for
some
time, andeven then itis not likely to be a
standardfeature.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

 -Original Message-
 From: Grant Grundler [mailto:[EMAIL PROTECTED] 
 Sent: Friday, May 27, 2005 10:06 AM
 To: Caitlin Bestler
 Cc: Michael Krause; Sukanta ganguly; 
 [EMAIL PROTECTED]; openib-general@openib.org
 Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB 
 and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

 On Fri, May 27, 2005 at 09:45:06AM -0700, Caitlin Bestler wrote:
  I'd like to add that RNIC-PI is planning on explicitly 
 defining some 
  of these obvious dependencies between the RDMA stack and 
 the primary 
  IP stack. For example, the RDMA stack cannot maintain any 
 connection 
  in a state that contradicts current IP stack routing. It 
 has to adapt 
  or break the connection.

 That's what I was just thinking. Could RNIC driver support 
 both existing linux NIC interfaces (e.g. use 
 ifconfig/ethtool) *and* openib RDMA interface?

 Essentially that's what openib.org does today but uses 
 ib_ipoib driver to support TCP/IP communication.

 Ie use an AF_INET socket to setup an RDMA connection like the 
 rdma_lat.c does:
   https://openib.org/svn/gen2/trunk/src/userspace/perftest/

 Of course, I'm blissfully ignorant of how ugly that might 
 be in real world implementation of RNIC driver. Seems simple 
 in concept at least.

For time-to-market reasons, and to avoid having to go hacking
deeply into the kernel from a driver, many or most existing
implementations require or at least prefer using TOE sockets.
It is much easier to extract the TCP state from a socket that
you already control than from the host stack.

But that is exactly the sort of limitation that integrating
with the kernel should eliminate. Given a stable interface
to get TCP state from a host controlled connection I can't
imagine why an RNIC vendor would not eagerly embrace it.

___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

RE: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux


--- Caitlin Bestler [EMAIL PROTECTED] wrote:
  
 
 
 
 
   From: [EMAIL PROTECTED]
 [mailto:[EMAIL PROTECTED]
 On Behalf Of
 Michael Krause
   Sent: Friday, May 27, 2005 7:25 AM
   To: Sukanta ganguly
   Cc: openib-general@openib.org;
 [EMAIL PROTECTED]
   Subject: Re: [Rdma-developers] Re: [openib-general]
 OpenIB and
 OpenRDMA: Convergence on common RDMA APIs and ULPs
 for Linux
   
   
   At 06:40 AM 5/27/2005, Sukanta ganguly wrote:
   
 
   Venkata,
  How will that work? If the RNIC offloads RDMA
 and
   TCP completely from the Operating System and does
 not
   share any state information then the application
   running on the host will never be in the position
 to
   utilize the socket interface to use the
 communication
   logic to send and receive data between the remote
 node
   and itself. Some information needs to be shared.
 How
   much of it and what exactly needs to be shared is
 the
   question.
 
 
   Ok.  It all depends upon what level of integration
 / interaction
 a TOE and thus a RNIC will have with the host
 network stack.  For
 example, if a customer wants to have TCP and IP
 stats kept for the
 off-loaded stack even if it is just being using for
 RDMA, then there
 needs to be a method defined to consolidate these
 stats back into the
 host network stack tool chain.  Similarly, if one
 wants to maintain a
 single routing table to manage, etc. on the host,
 then the RNIC needs to
 access / update that information accordingly.  One
 can progress through
 other aspects of integration, e.g. connection
 management, security
 interactions (e.g. DOS protection), and so forth. 
 What is exposed again
 depends upon the level of integration and how
 customers want to manage
 their services.  This problem also exists for IB but
 most people have
 not thought about this from a customer perspective
 and how to integrate
 the IB semantics into the way customers manage their
 infrastructures, do
 billing, etc.  For some environments, they simply do
 not care but if IB
 is to be used in the enterprise space, then some
 thought will be
 required here since most IT don't see anything as
 being free or
 self-managed.
   
   Again, Sockets is an application API and not how
 one
 communicates to a TOE or RDMA component.  The RNIC
 PI has been proposed
 as an interface to the RDMA functionality.  The PI
 supports all of the
 iWARP and IB v 1.2 verbs.  
   
   Mike


 
 I'd like to add that RNIC-PI is planning on
 explicitly defining some of
 these obvious dependencies 
 between the RDMA stack and the primary IP stack. For
 example, the RDMA
 stack cannot maintain
 any connection in a state that contradicts current
 IP stack routing. It
 has to adapt or break the connection.
 We can't have an RNIC that has its own ARP table
 that is not in sync
 with the host's ARP table.
  
 An iWarp RDMA stack gains the benefit of many
 pre-existing network
 services (such as DNS, ARP
 and routing). But that also carries with it the need
 to not contradict
 those exisiting services. So it is
 both a benefit and a restriction -- and a major
 divergence from an IB
 RDMA stack.
  
   
 

This is exactly what I was getting at and seems like
the RNIC-PI is already on its way to do that.


Thanks
SG

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

2005-05-27 Thread Venkata Jagana


[EMAIL PROTECTED] wrote on 05/27/2005 10:25:36 AM:

 Mike,
  I am not sure I do understand what your were trying
 to communicate. Let me try and decode this. My basic
 point was to respond to Venkata's response about
 complete offload without any interaction with the host
 system. I disagree with that in its totality as I
 think there are dependencies and that needs to be
 specified in a formal manner, i.e. in the specs, so
 that we do not have multiple proprietary interafaces
 which change and application users have to change this
 consumption based on individual implementations. 
  And you just brought up the reasons why what I was
 saying seemed to be justfied. Did I read that
 correctly?
 
 Thanks
 SG
 

Hello Sukanta,

Yes, there will be interactions with the host system for an RNIC
but through an interface something similar to RNIC-PI.

The clarification that I was trying to make is that from an RNIC perspective, it 
doesn't have to expose any ToE interface other than the fact that adapters might be providing
multiple services including ToE, RDMA, iSCSI but that's outside the scope of an
RNIC interface. 

Thanks
Venkat___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux

Venkata,
   I understand what you were saying now.

Thanks
SG

--- Venkata Jagana [EMAIL PROTECTED] wrote:
 
 
 
 
 
 [EMAIL PROTECTED] wrote on
 05/27/2005 10:25:36
 AM:
 
  Mike,
 I am not sure I do understand what your were
 trying
  to communicate. Let me try and decode this. My
 basic
  point was to respond to Venkata's response about
  complete offload without any interaction with the
 host
  system. I disagree with that in its totality as I
  think there are dependencies and that needs to be
  specified in a formal manner, i.e. in the specs,
 so
  that we do not have multiple proprietary
 interafaces
  which change and application users have to change
 this
  consumption based on individual implementations.
 And you just brought up the reasons why what I
 was
  saying seemed to be justfied. Did I read that
  correctly?
 
  Thanks
  SG
 
 
 Hello Sukanta,
 
 Yes, there will be interactions with the host system
 for an RNIC
 but through an interface something similar to
 RNIC-PI.
 
 The clarification that I was trying to make is that
 from an RNIC
 perspective, it
 doesn't have to expose any ToE interface other than
 the fact that adapters
 might be providing
 multiple services including ToE, RDMA, iSCSI but
 that's outside the scope
 of an
 RNIC interface.
 
 Thanks
 Venkat

__
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general

To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux