An open source implementation of the DAT API for the OpenIB stack
(gen2) is available in the OpenIB subversion repository at
https://openib.org/svn/gen2/users/jlentini/
If you are interested in learning more about the DAT API, please see
http://www.datcollaborative.org/
or
http://sourceforge.net/
We are ready to start the cleanup now. I've just checked in some
changes that I think simplify the interface.
james
On Mon, 2 May 2005, Tom Duffy wrote:
James,
When do you plan on cleaning up the dapl code? I am referring to the
following things:
- remove opaque types
- remove use of typedef
- f
On 5/3/05, Tom Duffy <[EMAIL PROTECTED]> wrote:
> On Tue, 2005-05-03 at 09:42 -0700, Caitlin Bestler wrote:
> > Most applications are not written to be able to
> > survice a sudden loss of virtually all of its connections, and then resume
> > all of the suspended sessions with new connections using
On Tue, 2005-05-03 at 09:42 -0700, Caitlin Bestler wrote:
> Most applications are not written to be able to
> survice a sudden loss of virtually all of its connections, and then resume
> all of the suspended sessions with new connections using new memory
> regions.
Most applications *should* be wr
>I'm not sure I like this "usermode" parameter that selects between
>memcpy() and copy_to_user(). For one thing, it means that it is
>impossible to fix the sparse warnings without defeating the sparse
>address space testing: you have to add a casts to and from __user
>pointers all over the place.
On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote:
...
> ionize:~# ibv_pingpong -s 64 -n 10 10.0.0.51
> local address: LID 0x000d, QPN 0x070406, PSN 0x972e9e
> remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543
> 1280 bytes in 1.65 seconds = 62.10 Mbit/sec
> 10 ite
I'm not sure I like this "usermode" parameter that selects between
memcpy() and copy_to_user(). For one thing, it means that it is
impossible to fix the sparse warnings without defeating the sparse
address space testing: you have to add a casts to and from __user
pointers all over the place.
Also
The following patch implements ib_coalesce_recv_mad and fixes an issue
where the mad_len was set incorrectly on the receive side.
Note, you will need to use the attached file for the patch to apply
correctly, but provided inline for review.
Signed-off-by: Sean Hefty <[EMAIL PROTECTED]>
Index: i
On Tue, 2005-05-03 at 18:01, Sean Hefty wrote:
> Hal Rosenstock wrote:
> > Thanks for pointing that out. I am using ib_create_mad_send for this.
> > Not sure why they are not cleared. Maybe it's related to the other issue
> > I am having.
>
> This may or may not be relevant to what you're doing, b
Hal Rosenstock wrote:
Thanks for pointing that out. I am using ib_create_mad_send for this.
Not sure why they are not cleared. Maybe it's related to the other issue
I am having.
This may or may not be relevant to what you're doing, but I should
mention that there's a bug setting the mad_len on the
On Tue, 2005-05-03 at 17:39, Sean Hefty wrote:
> I think what's necessary is for the DATA MAD to be rejected. This check
> is missing, but easily added. (I was deferring some of this until
> STOP/ABORT are implemented.)
OK.
> So... I think that the ACK code may be
> correct, but checks don
Hal Rosenstock wrote:
OK that answers the ACK formatting but not whether an ACK is correct in
this case.
o13-21.1.15 states that if a status code is received with an RMPPType
value other than those it applies to, the Receiver shall discard the
packet, terminate the protocol,and issue an RMPP ABORT
On Tue, 2005-05-03 at 16:55, Sean Hefty wrote:
> Hal Rosenstock wrote:
> > Short RMPP messages (less than 1 MAD's worth) have first bit set but not
> > also last in RMPPFlags.
>
> Can you re-check this? I'm able to successfully receive short segments,
> and the receive code looks for the last bi
Hal Rosenstock wrote:
Short RMPP messages (less than 1 MAD's worth) have first bit set but not
also last in RMPPFlags.
Can you re-check this? I'm able to successfully receive short segments,
and the receive code looks for the last bit to check for completion.
Also, if the initial RMPP send had b
Hi Sean,
Here are some wire based observations so far on RMPP running between a
toy SA client (making some GetTable requests) and OpenSM with RMPP
enabled:
Short RMPP messages (less than 1 MAD's worth) have first bit set but not
also last in RMPPFlags.
Also, if the initial RMPP send had bad stat
On Mon, May 02, 2005 at 05:21:48PM +0300, Michael S. Tsirkin wrote:
> Quoting r. Michael S. Tsirkin <[EMAIL PROTECTED]>:
> > Subject: SDP: unexpected conn state
> >
> > With ttcp, I sometimes see the following messages from sdp_rcvd_disconnect:
> >
> > WARN: : Unexpected conn state. conn <1> stat
On Mon, May 02, 2005 at 03:48:51PM +0100, Steven Wooding wrote:
> Hello Libor,
>
> I've tried your patch, but unfortunately it made no difference to
> the -32/-104 errors I get. I have observed the following features,
> which may help you diagnose my problem:
>
> Platform: 64 bit
> OS: RHEL 4
>
Michael> The application will get a completion with error, and it
Michael> just needs a way to know that its because of the HCA
Michael> going away, so it seems to me you have to signal the
Michael> event before closing the QPs, not after that.
It won't get any completion at all: w
Michael> What about ib_umad? Can you pull the HCA while opensm is
Michael> running?
Don't know... I never thought about it.
- R.
___
openib-general mailing list
openib-general@openib.org
http://openib.org/mailman/listinfo/openib-general
To uns
On Tue, 2005-05-03 at 11:08, Roland Dreier wrote:
> I don't think so, since user_mad's recv_handler() function always
> calls ib_free_recv_mad().
If this only occurs on machines which are running OpenSM, it could be a
problem with the local MAD handling where ib_free_recv_mad is not called
for so
On Tue, 2005-05-03 at 15:07 -0400, James Lentini wrote:
> Tom,
>
> Do you also want to remove the first EXTRA_CLAGS value:
>
>-Idrivers/dat \
>
> or did you leave that there on purpose?
>
> You shouldn't need the
>
>-I$(src)/../../../dat-provider \
>
> li
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: Advice about adapting ibv_pingpong to use UC
>
> Michael> Roland, would you like me to implement this?
>
> It would be great if you wanted to implement UC support. I think the
> main thing that needs to be done is to sit down and g
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: RFD: uverbs and hotplug
> Michael> No, but I know how NIC hotplug works on my laptop: push a
> Michael> button, wait till a little light flashes, pull it out.
>
> Sure, that's easy, because the network layer just throws packets
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: [openib-general] Re: RFD: uverbs and hotplug
>
> Grant> The rx4640 I'm using supports hotplug. What do you want
> Grant> tested? I think you have access to "iowa" via gsyprf10,
> Grant> right?
>
> Yeah, thanks. I'm not sur
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: RFD: uverbs and hotplug
>
> Michael> Hello, Roland, all! How should hotplug work with uverbs?
>
> ...
>
> Another possibility is as you described: free all the resources
> allocated by userspace processes (including remapping any do
Tom,
Do you also want to remove the first EXTRA_CLAGS value:
-Idrivers/dat \
or did you leave that there on purpose?
You shouldn't need the
-I$(src)/../../../dat-provider \
line. DAT consumers only need to be aware of the registry (the code in
the dat directory).
Grant> The rx4640 I'm using supports hotplug. What do you want
Grant> tested? I think you have access to "iowa" via gsyprf10,
Grant> right?
Yeah, thanks. I'm not sure there's anything worth testing: mthca is a
well-behaved PCI driver, so hotplug should work just like removing the
mod
On Tue, May 03, 2005 at 11:43:25AM -0700, Andy Isaacson wrote:
> [1] You might want to allow the child to start a completely new RDMA
> context, but I don't see that as necessary.
Some people use a hybrid OpenMP+MPI model, so some MPI implementations
want to fork and have the child open a new
On 5/3/05, Andy Isaacson <[EMAIL PROTECTED]> wrote:
>
> A consistent statement would be
>
> After fork(2), any regions which were registered are UNDEFINED.
> Region boundaries are byte-accurate; a registration can cover just
> part of a page, in which case the non-registered part of
On Tue, May 03, 2005 at 11:32:18AM -0700, Roland Dreier wrote:
> Sure, that's easy, because the network layer just throws packets away
> when a device disappears.
Mass storage is better example - transactions have "state".
The term "nexus" is used alot in SCSI specs.
> We already have that workin
On Fri, Apr 29, 2005 at 05:31:44PM -0700, Caitlin Bestler wrote:
> Attempting to provide *any* support for applications that fork children
> after doing RDMA registrations is a ratshole best avoided. The general
> rule that application developers should follow is to do RDMA *only*
> in the child pr
Michael> Okay, but you still must blocks the applications from
Michael> accessing the hardware - you cant trust them not to.
Right, that's why I said "including remapping any doorbell pages."
Michael> No, but I know how NIC hotplug works on my laptop: push a
Michael> button, wait
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: RFD: uverbs and hotplug
>
> Michael> Hello, Roland, all! How should hotplug work with uverbs?
>
> It's complicated. Discussion now is useful but I would definitely
> prefer to hold off on any implementation until we have the basic
On 5/3/05, David Addison <[EMAIL PROTECTED]> wrote:
> We believe the IOPROC patch is generic and powerful and would allow other
> RDMA NICs to solve the page registration problems in a different manner.
> For NICs which require page registration, new VM hooks can be used to avoid
> pages being unl
On 5/3/05, Libor Michalek <[EMAIL PROTECTED]> wrote:
> On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote:
> > What advantages is this style of fast memory register supposed to
> > have over the work request style found in IB 1.2 and iWARP?
>
> It's faster since all you're doing is
Michael> Hello, Roland, all! How should hotplug work with uverbs?
It's complicated. Discussion now is useful but I would definitely
prefer to hold off on any implementation until we have the basic
userspace verbs support merged upstream.
One possibility is that it is the administrator's res
Michael> Roland, would you like me to implement this?
It would be great if you wanted to implement UC support. I think the
main thing that needs to be done is to sit down and go over the
description of the modify QP verb in the IB spec and update all the
transitions in state_table[][] in mthc
Hello,
I thought that IB implementations of iSER and NFS over RDMA were using
kDAPL.
-David
Libor Michalek wrote:
On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote:
What advantages is this style of fast memory register supposed to
have over the work request style found in IB 1.2
> On 5/3/05, David Addison <[EMAIL PROTECTED]> wrote:
> > as our recent IOPROC patch on lkml shows, it's not that invasive. There
> > are just 24 hooks added to the Linux VM code paths - which we have been
> > able to
> > maintain outside the mainline tree for many years now.
> > As these hooks
On Mon, May 02, 2005 at 02:59:26PM -0700, Fab Tillier wrote:
> > From: Dror Goldenberg [mailto:[EMAIL PROTECTED]
> >
> > You're welcomed to measure. I think that FMRs will be the fastest way
> > to create mapping. Faster than MWs.
>
> The reason I asked is SDP went down the FMR path, but that can
On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote:
> What advantages is this style of fast memory register supposed to
> have over the work request style found in IB 1.2 and iWARP?
It's faster since all you're doing is rewriting the device's
page table mapping. It would be interest
An ex post facto notification of a PTE change would enable the RDMA
Device driver to know when a Memory Region had been invalidated
so that it could probably declare an access violation and tear all
the connections using it down.
But if the intent is to allow it to migrate the memory region to th
The fundamental question here is what degree of support is desired?
Is it to ensure that the OS can force a memory region to be destroyed
on demand, and it is then up to the application to figure out how to
respond?
Or is it to allow the OS to change the meaning of a Memory Region
so that it can
On Tue, May 03, 2005 at 09:42:12AM +0100, David Addison wrote:
> >This doesn't scale well as more cards are added to the box.
> >I think I understand why it's good for single cards though.
>
> With the IOPROC patch the device driver hooks are registered on a per
> process or perhaps better still,
Michael> Right, but this wont be related to the kmem_cache leak I
Michael> see, will it?
I don't think so, since user_mad's recv_handler() function always
calls ib_free_recv_mad().
- R.
___
openib-general mailing list
openib-general@openib.org
Title: RE: [openib-general] FMR and how they work
> -Original Message-
> From: Fab Tillier [mailto:[EMAIL PROTECTED]]
> Sent: Tuesday, May 03, 2005 12:59 AM
>
> Does the HW2SW_MPT command flush the MTTs referenced by the
> MPT (if any)? That is, if an MPT has mtt_seg_adr_h and
On Tue, 2005-05-03 at 03:27, Michael S. Tsirkin wrote:
> Right, but this wont be related to the kmem_cache leak I see, will it?
Right )it doesn't solve it). This is what the original mail said:
"I don't see a MAD leak but perhaps see a couple of other possible
memory leaks on shutdown." This was o
Grant Grundler wrote:
On Fri, Apr 29, 2005 at 08:22:24PM +0200, Brice Goglin wrote:
For instance, instead of adding PROT_DONT/ALWAYSCOPY, you may use
an ioproc hook in the fork path. This hook (a function in your driver)
would be called for each registered page. It will decide whether
the page shou
Ronald G. Minnich wrote:
On Fri, 29 Apr 2005, Greg Lindahl wrote:
It doesn't imply that there's an MMU, either. I know that Myricom uses a
little lookup routine in software on their nic, which most people
wouldn't call an MMU. I don't know what Mellanox does for this, they
don't talk much about wha
Hello, Roland, all!
How should hotplug work with uverbs?
Currently ib_uverbs_remove_one simply does class_device_unregister.
What will if an application has the device open?
Ideally, once the device is removed, applications
get an error code and so can close the device freeing
up any resources, a
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: Advice about adapting ibv_pingpong to use UC
>
> Steven> Hi, I wonder if someone working on the gen2 uverbs would
> Steven> be so kind as to give me some advice on adapting the
> Steven> ibv_pingpong program to use a UC QP ty
Greg Lindahl wrote:
On Fri, Apr 29, 2005 at 12:33:54PM -0700, Grant Grundler wrote:
Being mostly clueless about Quadrics implementation, I'm probably
missing something that makes Quadrics a MMU but not the IB variants.
Can someone clue me in please?
As far as I can tell it's mostly a marketing dist
Quoting r. Roland Dreier <[EMAIL PROTECTED]>:
> Subject: Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free
> all objects
>
> Hal> Also, does user_mad.c leak memory on the receive side if it
> Hal> is shutdown and the application has not drained the receive
> Hal> queu
53 matches
Mail list logo
