Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Nathan Hjelm
Looks like put and get functions should be added if possible. The MTL layer looks like it is designed for two-sided only with no intention of supporting one-sided. -Nathan On Thu, Nov 06, 2014 at 03:21:32PM -0700, Nathan Hjelm wrote: > > Great! We should probably try to figure out how the mtl l

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Nathan Hjelm
Great! We should probably try to figure out how the mtl layer can be modified to expose those atomics. If possible this should be done before the 1.9 branch to ensure the feature is available in the next release series. -Nathan On Thu, Nov 06, 2014 at 05:15:30PM -0500, Joshua Ladd wrote: >MX

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Joshua Ladd
MXM supports atomics. On Thursday, November 6, 2014, Nathan Hjelm wrote: > I haven't look at that yet. Would be great to get the new osc component > working over both btls and mtls. I know portals supports atomics but I > don't know whether psm does. > > -Nathan > > On Thu, Nov 06, 2014 at 08:

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Nathan Hjelm
Not handling the multi-rail case at this point. Only issue atomics and rdma operations over a single btl module (which should be a single HCA). -Nathan On Thu, Nov 06, 2014 at 12:15:13PM -0700, Howard Pritchard wrote: >HI Nathan, >How would you get things right with atomics and multirail?

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Howard Pritchard
HI Nathan, How would you get things right with atomics and multirail? Getting the memory consistency right would be really difficult. You'd have to keep issuing zero length rdma reads and hoping that that would have the effect of a pci-e flush in the case of multiple updates to a given target add

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Nathan Hjelm
I haven't look at that yet. Would be great to get the new osc component working over both btls and mtls. I know portals supports atomics but I don't know whether psm does. -Nathan On Thu, Nov 06, 2014 at 08:45:15PM +0200, Mike Dubman wrote: >btw, do you plan to add atomics API to MTL layer as

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Mike Dubman
btw, do you plan to add atomics API to MTL layer as well? On Thu, Nov 6, 2014 at 5:23 PM, Nathan Hjelm wrote: > At the moment I select the lowest latency BTL that can reach all of the > ranks in the communicator used to create the window. I can add code to > round-robin windows over the availab

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-06 Thread Nathan Hjelm
At the moment I select the lowest latency BTL that can reach all of the ranks in the communicator used to create the window. I can add code to round-robin windows over the available BTLs on multi-rail systems. -Nathan On Wed, Nov 05, 2014 at 06:38:25PM -0800, Paul Hargrove wrote: >All atomics

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-05 Thread Paul Hargrove
All atomics must be done through not just "the same btl" but the same btl MODULE, since atomics from two IB HCAs, for instance, are not necessarily coherent. So, how is the "best" one to be selected? -Paul [Sent from my phone] On Nov 5, 2014 7:15 AM, "Nathan Hjelm" wrote: > > In the new osc com

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-05 Thread Nathan Hjelm
In the osc component, no. Though it would be straightforward to add that feature. -Nathan On Wed, Nov 05, 2014 at 11:05:48AM -0500, Joshua Ladd wrote: >Does this mean that you maintain a separate channel for 'put' and 'gets' >that can use multiple transports and another for atomics? > >

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-05 Thread Joshua Ladd
Does this mean that you maintain a separate channel for 'put' and 'gets' that can use multiple transports and another for atomics? Josh On Wed, Nov 5, 2014 at 10:15 AM, Nathan Hjelm wrote: > > In the new osc component I don't try to handle that case. All atomics > have to be done through the sa

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-05 Thread Nathan Hjelm
In the new osc component I don't try to handle that case. All atomics have to be done through the same btl (including atomics on self). I did this because with the default setup of Gemini they can not be mixed. If it is possible to mix them with other networks I would be happy to add an atomic fla

Re: [OMPI devel] RFC: revamp btl rdma interface

2014-11-05 Thread Joshua Ladd
Quick question. Out of curiosity, how do you handle the (common) case of mixing network atomics with CPU atomics? Say for a single target with two initiators, one initiator is on host with the target, so goes through the SM BTL, and the other initiator is off host, so goes through the network BTL.

[OMPI devel] RFC: revamp btl rdma interface

2014-11-04 Thread Nathan Hjelm
What: Completely revamp the BTL RDMA interface (btl_put, btl_get) to better match what is needed for MPI one-sided. Why: I am preparing to push an enhanced MPI-3 one-sided component that makes use of network rdma and atomic operations to provide a fast truely one-sided implementation. Before I ca