Re: [OMPI users] Concerning the performance of the one-sided communications

[Nathan Hjelm]

How was the latency? That is the best metric to use because osc/pt2pt does put 
aggregation. Makes the result of osu_put_bw relatively garbage. 

> On Feb 16, 2018, at 5:24 PM, Jeff Hammond  wrote:
> 
> 
> 
>> On Fri, Feb 16, 2018 at 8:52 AM, Nathan Hjelm  wrote:
>> It depends on the transport used. If there is a high-performance network 
>> (Cray Aries, Infiniband, etc) then the progress is handled by the hardware. 
>> For other networks (Infinipath, Omnipath, TCP, etc) there are options.
>> 
>> For TCP you can set:
>> 
>>  --mca btl_tcp_progress_thread 1
>> 
>> No such option currently exists for Infinipath and Omnipath. Though with 
>> those networks you can use verbs but Intel does not support that mode of 
>> operation and the performance can be bad for some operations (> 100 us 
>> latency on compare-and-swap for example).
>  
> If Intel Omni Path is the network of interest, I'd recommend using the 
> OFI/libfabric transport layer in Open-MPI and asking the libfabric list 
> (http://lists.openfabrics.org/mailman/listinfo/libfabric-users) for 
> instructions on how to enable asynchronous progress inside of libfabric.
> 
> When I was testing RMA in Open-MPI on Omni Path a while ago, the performance 
> of both PSM2 and OFI was quite good relative to other implementations, as 
> measured by the OSU MPI benchmarks.  I was using Casper 
> (http://www.mcs.anl.gov/project/casper/) for asynchronous progress, although 
> those tests don't necessary reveal a lack of asynchronous progress.
> 
> Best,
> 
> Jeff
> 
>> I don't know how well tested this support is. There haven't been many 
>> requests for this feature so not much work has been done on it.
>> 
>> -Nathan
>> 
>>> On Feb 16, 2018, at 08:46 AM, Ahmed Hamdy Mohamed Eleliemy 
>>>  wrote:
>>> 
>> 
>>> Hi all,
>>> 
>>> I used to work with MPICH. I am trying now to compare and switch to 
>>> OpenMPI. I am using OpenMPI version 2.1.1.
>>> To have progress for one-sided communications (in the passive mode). I used 
>>> to export the following environment variable MPICH_ASYNC_PROGRESS=1
>>> My questions are: 
>>> 1. What would be the equivalent option for openmpi?
>>> 2. How openmpi handle the progress of the one-sided communications?
>>> 
>>> Thanks in advance
>>> 
>>> Best regards
>>> Ahmed
>>> 
>>> ___
>>> users mailing list
>>> users@lists.open-mpi.org
>>> https://lists.open-mpi.org/mailman/listinfo/users
>> 
>> ___
>> users mailing list
>> users@lists.open-mpi.org
>> https://lists.open-mpi.org/mailman/listinfo/users
> 
> 
> 
> -- 
> Jeff Hammond
> jeff.scie...@gmail.com
> http://jeffhammond.github.io/
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] Concerning the performance of the one-sided communications

[Jeff Hammond]

On Fri, Feb 16, 2018 at 8:52 AM, Nathan Hjelm  wrote:

> It depends on the transport used. If there is a high-performance network
> (Cray Aries, Infiniband, etc) then the progress is handled by the hardware.
> For other networks (Infinipath, Omnipath, TCP, etc) there are options.
>
> For TCP you can set:
>
>  --mca btl_tcp_progress_thread 1
>
> No such option currently exists for Infinipath and Omnipath. Though with
> those networks you can use verbs but Intel does not support that mode of
> operation and the performance can be bad for some operations (> 100 us
> latency on compare-and-swap for example).
>

If Intel Omni Path is the network of interest, I'd recommend using the
OFI/libfabric transport layer in Open-MPI and asking the libfabric list (
http://lists.openfabrics.org/mailman/listinfo/libfabric-users) for
instructions on how to enable asynchronous progress inside of libfabric.

When I was testing RMA in Open-MPI on Omni Path a while ago, the
performance of both PSM2 and OFI was quite good relative to other
implementations, as measured by the OSU MPI benchmarks.  I was using Casper
(http://www.mcs.anl.gov/project/casper/) for asynchronous progress,
although those tests don't necessary reveal a lack of asynchronous progress.

Best,

Jeff

I don't know how well tested this support is. There haven't been many
> requests for this feature so not much work has been done on it.
>
> -Nathan
>
> On Feb 16, 2018, at 08:46 AM, Ahmed Hamdy Mohamed Eleliemy <
> ahmed.eleli...@unibas.ch> wrote:
>
> Hi all,
>
> I used to work with MPICH. I am trying now to compare and switch to
> OpenMPI. I am using OpenMPI version 2.1.1.
> To have progress for one-sided communications (in the passive mode). I
> used to export the following environment variable MPICH_ASYNC_PROGRESS=1
> My questions are:
> 1. What would be the equivalent option for openmpi?
> 2. How openmpi handle the progress of the one-sided communications?
>
> Thanks in advance
>
> Best regards
> Ahmed
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
>
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>



-- 
Jeff Hammond
jeff.scie...@gmail.com
http://jeffhammond.github.io/
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] MacOS - Running MPI code with 256 procs - “[warn] select: Invalid argument” message

[Christophe Petit]

Hello Gilles,

I have upgraded to OS X 10.13.3 High Sierra but still have no luck with my
issue.

To increase the limit of open files, from this link

I did :

1) *sudo vim /Library/LaunchDaemons/limit.maxfiles.plist*


2) put into this file :

*
http://www.apple.com/DTDs/PropertyList-1.0.dtd
">

 
 Label
 limit.maxfiles
 ProgramArguments
 
 launchctl
 limit
 maxfiles
 64000
 524288
 
 RunAtLoad
 
 ServiceIPC
 
 
*

3) *sudo launchctl load -w /Library/LaunchDaemons/limit.maxfiles.plist*

4) Checking by  :



*$ sysctl kern.maxfiles kern.maxfiles: 524288*

and



*$ launchctl limit maxfilesmaxfiles64000  524288 *

When I launch ring_c.c executable with :

*mpirun --oversubscribe -np $NB_PROCESS ./a.out *

For a number of processes lower or equal to *142*, it's running fine

For a number of processes greater than *142*, I get the original warning
message :

[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
[warn] select: Invalid argument
...

Why the limit is reached at NB_PROCESS > *142* ? what does it corrrespond
to ?

ps: I have set "*ulimit -n 1*"

Regards, Chris


2018-02-15 1:42 GMT+01:00 Gilles Gouaillardet :

> Christophe,
>
> I can only test this on OS X 10.13.3 High Sierra, and it could differ
> from Maverick.
>
> by default
>
> KA15-002:~ gilles$ ulimit -n
> 256
>
> KA15-002:~ gilles$ ulimit -Hn
> unlimited
>
>
> but surprisingly,
>
> KA15-002:~ gilles$ ulimit -n unlimited
> -bash: ulimit: open files: cannot modify limit: Operation not permitted
>
> KA15-002:~ gilles$ ulimit -n 2
> -bash: ulimit: open files: cannot modify limit: Invalid argument
>
> and finally, a lower value works just fine.
>
> KA15-002:~ gilles$ ulimit -n 1
>
>
> as a consequence, opal_set_max_sys_limits fails in my environment.
> --oversubscribe is mandatory (since there are no 256 cores on my
> laptop), and then
>
> ulimit -n 1; mpirun --oversubscribe -np 256 ./ring_c
>
> works just fine (fwiw, this is an example from Open MPI sources
> examples/ring_c.c)
>
>
>
> So first, I invite you to double check with ulimit -n that your system
> changes are effective.
>
> How did you build/install Open MPI ?
> The message seems to come from libevent, and Open MPI uses an embedded
> version of libevent.
> It is possible to use an external version at configure time.
> If you are using an external libevent, you might want to try
> rebuilding Open MPI with the embedded one.
>
>
> Cheers,
>
> Gilles
>
> On Wed, Feb 14, 2018 at 4:57 PM, Christophe Petit
>  wrote:
> > Hello,
> >
> > Using Open-Mpi 3.0 and following the tutorial on this link, I try to run
> a
> > MPI code under MacOS 10.9.5 (Mavericks) with a number of process equal to
> > 256 : the MPI code allocates for each process a 512x512 2D array, so it
> > requires 256*256kB = 64MB of total used memory.
> >
> > My MacOS has 16GB RAM and 8 cores, so it seems to be weird.
> >
> > For a number of process lower than 256 (I tried : np=2,4,8,16,32,64,128),
> > there is no problem, execution is good and I get expected results.
> >
> > But for np = 256, I get the following message which repeats itself :
> >
> > $ mpirun -np 256 ./explicitPar
> >
> > [warn] select: Invalid argument
> > [warn] select: Invalid argument
> > [warn] select: Invalid argument
> >  ...
> >
> > I tried also to use -mca option by doing :
> >
> > $ mpirun -mca opal_set_max_sys_limits 1 -np 256 ./explicitPar
> >
> > But I get the same warning message.
> >
> > From this link cited above, I did :
> >
> > $ launchctl limit maxfiles
> >
> >   maxfiles65536  20
> >
> > Then, in root user, I created /etc/launchd.conf file and put into :
> >
> > limit maxfiles 65536 20
> >
> > I restarted the system for the new limits to take effect and type as
> normal
> > user :
> >
> > $ launchctl limit maxfiles
> >
> > maxfiles65536  20
> >
> > But unfortunately, these modifications have no effects on the MPI
> function
> > "mpirun" with 256 processes and don't make disappear the warning above.
> >
> > On Linux platform, I can launch my MPI code with np = 256, without
> problem,
> > the issue is only happening on MacOS 10.9.5.
> >
> > I didn't get this issue with previous version of Open-MPI.
> >
> > Any idea ? Thanks
> >
> >
> >
> > ___
> > users mailing list
> > users@lists.open-mpi.org
> > https://lists.open-mpi.org/mailman/listinfo/users
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
>
___
users mailing list

Re: [OMPI users] Concerning the performance of the one-sided communications

[Nathan Hjelm]

It depends on the transport used. If there is a high-performance network (Cray 
Aries, Infiniband, etc) then the progress is handled by the hardware. For other 
networks (Infinipath, Omnipath, TCP, etc) there are options.

For TCP you can set:

 --mca btl_tcp_progress_thread 1

No such option currently exists for Infinipath and Omnipath. Though with those 
networks you can use verbs but Intel does not support that mode of operation and 
the performance can be bad for some operations (> 100 us latency on 
compare-and-swap for example).

I don't know how well tested this support is. There haven't been many requests 
for this feature so not much work has been done on it.

-Nathan

On Feb 16, 2018, at 08:46 AM, Ahmed Hamdy Mohamed Eleliemy 
 wrote:

Hi all,

I used to work with MPICH. I am trying now to compare and switch to OpenMPI. I 
am using OpenMPI version 2.1.1.
To have progress for one-sided communications (in the passive mode). I used to 
export the following environment variable MPICH_ASYNC_PROGRESS=1
My questions are: 
1. What would be the equivalent option for openmpi?

2. How openmpi handle the progress of the one-sided communications?

Thanks in advance

Best regards
Ahmed

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

[OMPI users] Concerning the performance of the one-sided communications

[Ahmed Hamdy Mohamed Eleliemy]

Hi all,

I used to work with MPICH. I am trying now to compare and switch to OpenMPI. I 
am using OpenMPI version 2.1.1.
To have progress for one-sided communications (in the passive mode). I used to 
export the following environment variable MPICH_ASYNC_PROGRESS=1
My questions are: 
1. What would be the equivalent option for openmpi?
2. How openmpi handle the progress of the one-sided communications?

Thanks in advance

Best regards
Ahmed

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users