On Sun, 2007-05-13 at 21:26 -0400, Donald Kerr wrote:
>
> Caitlin Bestler wrote:
>
> >Donal Kerr wrote:
> >
> >
> >
> order of business after connection establishment
> (mba_btl_udapl_sendrecv(). The RECV buffer post for this exchange,
> however, should really be done _before_ the
> dat_ep_connect() on the active side, and _before_ the
> dat_cr_accept() on the server side.
> Currently its done after the ESTABLISHED event is dequeued, thus
> allowing the race condition.
>
> I believe the rules are the ULP must ensure that a RECV is posted
> before the client can post a SEND for that buffer.
> And further, the ULP must enforce flow control somehow so that a
> SEND never arrives without a RECV buffer being available.
>
>
>
>
> >>maybe this is a rule iwarp imposes on its ULPs but not uDAPL.
> >>
> >>
> >>
> >
> >It is most assuredly a rule for uDAPL. And it is not a matter
> >of iWARP "imposing" on uDAPL. uDAPL was explicitly designed
> >to support IB, iWARP and VI. To do that DAPL documents its
> >model of what RDMA is.
> >
> >
> (sorry I was off the grid for a couple of days)
> Not to beat a dead horse but you would have to show me where in the Spec
> it says I must post a recv before a send. And thinking about it some I
> don't believe there is a race condition because this is not called out
> as such. Now if posting the handshake recv before the connect call
> speeds things up and helps the iwarp scenario I am all for it.
>
> >This issue is in fact one that is truly fundamental to the
> >efficiency of RDMA -- the transport layer DOES NOT provide
> >buffering. That's the application's job. It is precisely
> >because the application layer does a better job that RDMA
> >can achieve better performance at high bandwidth.
> >
> >For reasons that have been discussed in more depth in the
> >RDMA applicability statement and in RDDP/IPS discussions
> >on iSER, the absence of transport layer buffer throttling
> >places the onus for end-to-end pacing on the application.
> >It is a situation somewhat akin to a car with a broken
> >spedometer that had previously only driven during rush
> >hour bumper-to-bumper traffic. The fact that the spedometer
> >was broken was irrelevant. But if that same car hits the
> >open road the driver will need to come up with some method
> >of regulating their speed.
> >
> >The DAPL semantics are very clear that send/recv operations must
> >be matched one to one, that the receive buffer must be large
> >enough for the received message and that there must be a receive
> >buffer for each incoming send/recv message. That means that
> >the sender needs to have some basis for believing that the
> >RECV has been posted. Usually this is an explicit credit
> >that is decremented per message and incremented per response.
> >
> >
> Matching one to one sure, still does not say a recv must be posted
> before a send. Flow control is handled by the BTL.
>
> >What DAPL does not state is if the transport does explicit flow
> >control so that the sending application's work request is simply
> >not processed (and the sending application continues to provide
> >the buffer, as with InfiniBand) or whether the sender simply
> >transmits and leaves error detection to the receiver (iWARP).
> >There are theoretical advantages to both, but more importantly
> >neither of them is going to change. So the Consumer of RDMA
> >applications needs to use ULP/application layer flow control
> >to pace the transmitter. At the application layer that means
> >that the RECV must be posted *before* the Send/accept that
> >grants ULP credits to the far side.
> >
> >All of that should be clear in the IOV ownership rules and
> >discussion of the semantics of send/recv. If you thought you
> >saw something that implied any guarantees to the contrary
> >then could you point them out in a posting to the DAT reflector?
> >(or just send them to me or Arkady Kanevsky).
> >
> >
> I believe it was either your Steve who claimed a recv must be posted
> before a send thus leading to a race condition. I fail to see this. But
> again, if Steve's patch makes things better I am all for it.
>
For iWARP, the connection may be TERMINATED if a SEND arrives on a QP
and no corresponding RECV buffer is posted.
Steve.
