[OMPI devel] u_int32_t typo in nbc_internal.h?

2012-06-27 Thread Eugene Loh 

ompi/mca/coll/libnbc/nbc_internal.h

   259/* Schedule cache structures/functions */
   260u_int32_t adler32(u_int32_t adler, int8_t *buf, int len);
   261void NBC_SchedCache_args_delete(void *entry);
   262void NBC_SchedCache_args_delete_key_dummy(void *k);

u_int32_t
->
uint32_t

perhaps?

Re: [OMPI devel] RFC: add asynchronous copies for large GPU buffers

2012-06-27 Thread Rolf vandeVaart 
Whoops.  Fixed.

Rolf

>-Original Message-
>From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org]
>On Behalf Of Nathan Hjelm
>Sent: Wednesday, June 27, 2012 6:20 PM
>To: Open MPI Developers
>Subject: Re: [OMPI devel] RFC: add asynchronous copies for large GPU
>buffers
>
>Can you make your repository public or add me to the access list?
>
>-Nathan
>
>On Wed, Jun 27, 2012 at 03:12:34PM -0700, Rolf vandeVaart wrote:
>> WHAT: Add support for doing asynchronous copies of GPU memory with
>larger messages.
>> WHY: Improve performance for sending/receiving of larger GPU messages
>> over IB
>> WHERE: ob1, openib, and convertor code.  All is protected by compiler
>directives
>>so no effect on non-CUDA builds.
>> REFERENCE BRANCH: https://bitbucket.org/rolfv/ompi-trunk-cuda-async
>>
>> DETAILS:
>> When sending/receiving GPU memory through IB, all data first passes into
>host memory.
>> The copy of GPU memory into and out of the host memory can be done
>> asynchronously to improve performance.  This RFC adds that feature for the
>fragments of larger messages.
>>
>> On the sending side, the completion function is essentially broken in
>> two.  The first function is called when the copy completes which then
>> initiates the send.  When the send completes, the second function is called.
>>
>> Likewise, on the receiving side, a callback is called when the
>> fragment arrives which initiates the copy of the data out of the
>> buffer.  When the copy completes, a second function is called which
>> also calls back into the BTL so it can free resources that were being used.
>>
>> M   opal/datatype/opal_datatype_copy.c
>> M   opal/datatype/opal_convertor.c
>> M   opal/datatype/opal_convertor.h
>> M   opal/datatype/opal_datatype_cuda.c
>> M   opal/datatype/opal_datatype_cuda.h
>> M   opal/datatype/opal_datatype_unpack.c
>> M   opal/datatype/opal_datatype_pack.h
>> M   opal/datatype/opal_datatype_unpack.h
>> M   ompi/mca/btl/btl.h
>> M   ompi/mca/btl/openib/btl_openib_component.c
>> M   ompi/mca/btl/openib/btl_openib.c
>> M   ompi/mca/btl/openib/btl_openib.h
>> M   ompi/mca/btl/openib/btl_openib_mca.c
>> M   ompi/mca/pml/ob1/pml_ob1_recvfrag.c
>> M   ompi/mca/pml/ob1/pml_ob1_sendreq.c
>> M   ompi/mca/pml/ob1/pml_ob1_progress.c
>> M   ompi/mca/pml/ob1/pml_ob1_recvreq.c
>> M   ompi/mca/pml/ob1/pml_ob1_cuda.c
>> M   ompi/mca/pml/ob1/pml_ob1_recvreq.h
>> --
>> - This email message is for the sole use of the intended
>> recipient(s) and may contain confidential information.  Any
>> unauthorized review, use, disclosure or distribution is prohibited.
>> If you are not the intended recipient, please contact the sender by
>> reply email and destroy all copies of the original message.
>> --
>> -
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>___
>devel mailing list
>de...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] RFC: add asynchronous copies for large GPU buffers

2012-06-27 Thread Nathan Hjelm 
Can you make your repository public or add me to the access list?

-Nathan

On Wed, Jun 27, 2012 at 03:12:34PM -0700, Rolf vandeVaart wrote:
> WHAT: Add support for doing asynchronous copies of GPU memory with larger 
> messages.
> WHY: Improve performance for sending/receiving of larger GPU messages over IB
> WHERE: ob1, openib, and convertor code.  All is protected by compiler 
> directives
>so no effect on non-CUDA builds.
> REFERENCE BRANCH: https://bitbucket.org/rolfv/ompi-trunk-cuda-async
> 
> DETAILS:
> When sending/receiving GPU memory through IB, all data first passes into host 
> memory.
> The copy of GPU memory into and out of the host memory can be done 
> asynchronously
> to improve performance.  This RFC adds that feature for the fragments of 
> larger messages.
> 
> On the sending side, the completion function is essentially broken in two.  
> The first function
> is called when the copy completes which then initiates the send.  When the 
> send completes,
> the second function is called.
> 
> Likewise, on the receiving side, a callback is called when the fragment 
> arrives which 
> initiates the copy of the data out of the buffer.  When the copy completes, a 
> second
> function is called which also calls back into the BTL so it can free 
> resources that
> were being used.
> 
> M   opal/datatype/opal_datatype_copy.c
> M   opal/datatype/opal_convertor.c
> M   opal/datatype/opal_convertor.h
> M   opal/datatype/opal_datatype_cuda.c
> M   opal/datatype/opal_datatype_cuda.h
> M   opal/datatype/opal_datatype_unpack.c
> M   opal/datatype/opal_datatype_pack.h
> M   opal/datatype/opal_datatype_unpack.h
> M   ompi/mca/btl/btl.h
> M   ompi/mca/btl/openib/btl_openib_component.c
> M   ompi/mca/btl/openib/btl_openib.c
> M   ompi/mca/btl/openib/btl_openib.h
> M   ompi/mca/btl/openib/btl_openib_mca.c
> M   ompi/mca/pml/ob1/pml_ob1_recvfrag.c
> M   ompi/mca/pml/ob1/pml_ob1_sendreq.c
> M   ompi/mca/pml/ob1/pml_ob1_progress.c
> M   ompi/mca/pml/ob1/pml_ob1_recvreq.c
> M   ompi/mca/pml/ob1/pml_ob1_cuda.c
> M   ompi/mca/pml/ob1/pml_ob1_recvreq.h
> ---
> This email message is for the sole use of the intended recipient(s) and may 
> contain
> confidential information.  Any unauthorized review, use, disclosure or 
> distribution
> is prohibited.  If you are not the intended recipient, please contact the 
> sender by
> reply email and destroy all copies of the original message.
> ---
> 
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

[OMPI devel] RFC: add asynchronous copies for large GPU buffers

2012-06-27 Thread Rolf vandeVaart 
WHAT: Add support for doing asynchronous copies of GPU memory with larger 
messages.
WHY: Improve performance for sending/receiving of larger GPU messages over IB
WHERE: ob1, openib, and convertor code.  All is protected by compiler directives
   so no effect on non-CUDA builds.
REFERENCE BRANCH: https://bitbucket.org/rolfv/ompi-trunk-cuda-async

DETAILS:
When sending/receiving GPU memory through IB, all data first passes into host 
memory.
The copy of GPU memory into and out of the host memory can be done 
asynchronously
to improve performance.  This RFC adds that feature for the fragments of larger 
messages.

On the sending side, the completion function is essentially broken in two.  The 
first function
is called when the copy completes which then initiates the send.  When the send 
completes,
the second function is called.

Likewise, on the receiving side, a callback is called when the fragment arrives 
which 
initiates the copy of the data out of the buffer.  When the copy completes, a 
second
function is called which also calls back into the BTL so it can free resources 
that
were being used.

M   opal/datatype/opal_datatype_copy.c
M   opal/datatype/opal_convertor.c
M   opal/datatype/opal_convertor.h
M   opal/datatype/opal_datatype_cuda.c
M   opal/datatype/opal_datatype_cuda.h
M   opal/datatype/opal_datatype_unpack.c
M   opal/datatype/opal_datatype_pack.h
M   opal/datatype/opal_datatype_unpack.h
M   ompi/mca/btl/btl.h
M   ompi/mca/btl/openib/btl_openib_component.c
M   ompi/mca/btl/openib/btl_openib.c
M   ompi/mca/btl/openib/btl_openib.h
M   ompi/mca/btl/openib/btl_openib_mca.c
M   ompi/mca/pml/ob1/pml_ob1_recvfrag.c
M   ompi/mca/pml/ob1/pml_ob1_sendreq.c
M   ompi/mca/pml/ob1/pml_ob1_progress.c
M   ompi/mca/pml/ob1/pml_ob1_recvreq.c
M   ompi/mca/pml/ob1/pml_ob1_cuda.c
M   ompi/mca/pml/ob1/pml_ob1_recvreq.h
---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---

Re: [OMPI devel] Quiet Time on Trunk - ORCA Integration

2012-06-27 Thread Josh Hursey 
This morning Jeff, Brian, Ralph, and I discussed the issues that
emerged with the ORCA branch. We have a clear path forward. It might
take a little while (few weeks) to finish the effort since I am going
to be transiting over the next month, but we will see once we get into
the code.

We will start working on the fixes immediately. There is a new
BitBucket branch for this effort that you can watch at:
  https://bitbucket.org/jjhursey/ompi-orca

(The branch will be filled in later today)

We'll post back when it is ready for further review.

-- Josh

On Tue, Jun 26, 2012 at 9:29 PM, Josh Hursey  wrote:
> ORCA was backed out of the trunk in r26676.
>
> Once we fix the linking issue, we will bring this back.
>
> Sorry for the noise folks. The trunk is open again.
>
> -- Josh
>
> On Tue, Jun 26, 2012 at 9:04 PM, Josh Hursey  wrote:
>> So I'm spinning my wheels on this one. I am going to need someone with
>> more knowledge about linking to help (Jeff or Brian maybe?).
>>
>> I'll see what I can do to back this out of the trunk :(
>>
>> I wish this issue would have come up earlier - I just don't ever build
>> on my mac, so I never saw it.
>>
>> -- Josh
>>
>> On Tue, Jun 26, 2012 at 8:25 PM, Josh Hursey  wrote:
>>> So I can confirm that it is not linking properly on the Mac. It -is-
>>> running correctly on Linux (which is where I have been testing).
>>>
>>> From what I can tell this is a linking issue specific to the Mac. I'm
>>> digging into it a bit at the moment.
>>>
>>> Interesting way to tell is by using ompi_info as a canary. If you
>>> compile with (what is default):
>>>  ../../../ompi/libmpi.la -L/Users/jju/work/open-mpi/ompi-trunk/orte/ 
>>> -lopen-rte
>>> It will display components from OMPI,ORCA,OPAL
>>> If you change that too:
>>>  ../../../ompi/libmpi.la  ../../../orte/libopen-rte.la
>>> you get the same thing, but if you change it to:
>>>  ./../../ompi/libmpi.la  ../../../orte/libopen-rte.la ./../../ompi/libmpi.la
>>> Then it only displays the ORTE,OPAL components.
>>>
>>> So I am thinking that this is an issue with using two .la's in a
>>> single linking - something that is not showing up on Linux.
>>>
>>> Any pointers on what might be going on here would be appreciated as I
>>> dig further.
>>>
>>> -- Josh
>>>
>>>
>>> On Tue, Jun 26, 2012 at 7:40 PM, Josh Hursey  wrote:
 That is odd. I did not see that when testing on Linux. I'll take a look.

 -- josh

 On Tue, Jun 26, 2012 at 7:37 PM, Ralph Castain  wrote:
> FWIW: it built fine on my Mac, but doesn't run. Just hangs when 
> attempting to execute any MPI application. Will execute non-MPI apps 
> though.
>
> Looking deeper, it looks like the app is stuck waiting to receive the 
> sync registration "ack". Mpirun thinks it sent it, but the app is unable 
> to receive it, so I would guess the app is failing to progress the RTE.
>
> If you can't resolve that quickly, I would suggest backing this out and 
> the two of us looking at it in the morning. Somehow, you've lost the 
> progress loop that was driving the RTE thru orte_init.
>
>
> On Jun 26, 2012, at 3:44 PM, Josh Hursey wrote:
>
>> r26670 is the first of the ORCA commits. I am switching machines for
>> testing. Hang on for a couple more hours while the initial testing is
>> underway.
>>
>> -- Josh
>>
>> On Tue, Jun 26, 2012 at 4:34 PM, Josh Hursey  
>> wrote:
>>> I am requesting a quiet time on the trunk for ORCA integration
>>> starting -now- (as previously announced). I will post back when
>>> everything is committed and ready to go.
>>>
>>> Some reading while you are waiting:
>>>  http://www.open-mpi.org/community/lists/devel/2012/06/11109.php
>>>
>>> Thanks,
>>> Josh
>>>
>>> --
>>> Joshua Hursey
>>> Postdoctoral Research Associate
>>> Oak Ridge National Laboratory
>>> http://users.nccs.gov/~jjhursey
>>
>>
>>
>> --
>> Joshua Hursey
>> Postdoctoral Research Associate
>> Oak Ridge National Laboratory
>> http://users.nccs.gov/~jjhursey
>>
>> ___
>> devel mailing list
>> de...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel



 --
 Joshua Hursey
 Postdoctoral Research Associate
 Oak Ridge National Laboratory
 http://users.nccs.gov/~jjhursey
>>>
>>>
>>>
>>> --
>>> Joshua Hursey
>>> Postdoctoral Research Associate
>>> Oak Ridge National Laboratory
>>> http://users.nccs.gov/~jjhursey
>>
>>
>>
>> --
>> Joshua Hursey
>> Postdoctoral Research Associate
>> Oak Ridge National Laboratory