Hi Gilles,
I've done a big mistake! Compiling the patched version of openMPI and creating a
new module, I've forgotten to add the path to oarshmost command while
OMPI_MCA_plm_rsh_agent=oarshmost was set....
OpenMPI was silently ignoring oarshmost command as it was unable to find it and
so only one node was available!
The good thing is that with your patch, oversuscribing does not occur anymore on
the nodes, it seems to solves efficiently the problem we had. I'll keep this
patched version in prod for the users as the previous one was allowing 2
processes on some cores time to time, and haphazardly bad code performances in
thes cases.
Yes this computer is the biggest one of CIMENT mesocenter, it is called...
froggy and all the nodes are littles frogs :-)
https://ciment.ujf-grenoble.fr/wiki-pub/index.php/Hardware:Froggy
I was using $OAR_NODEFILE and frog.txt to check different syntax, one with a
liste of nodes (on line with a node name for each available core) and the second
with one line per node and the "slots" information for the number of cores. EG:
[begou@frog7 MPI_TESTS]$ cat $OAR_NODEFILE
frog7
frog7
frog7
frog7
frog8
frog8
frog8
frog8
[begou@frog7 MPI_TESTS]$ cat frog.txt
frog7 slots=4
frog8 slots=4
Thanks again for the patch and your help.
Patrick
Gilles Gouaillardet wrote:
Thanks Patrick,
could you please try again with the --hetero-nodes mpirun option ?
(I am afk, and not 100% sure about the syntax)
could you also submit a job with 2 nodes and 4 cores on each node, that does
cat /proc/self/status
oarshmost <remote host> cat /proc/self/status
btw, is there any reason why do you use a machine file (frog.txt) instead of
using $OAR_NODEFILE directly ?
/* not to mention I am surprised a French supercomputer is called "frog" ;-) */
Cheers,
Gilles
On Friday, September 18, 2015, Patrick Begou
<patrick.be...@legi.grenoble-inp.fr
<mailto:patrick.be...@legi.grenoble-inp.fr>> wrote:
Gilles Gouaillardet wrote:
Patrick,
by the way, this will work when running on a single node.
i do not know what will happen when you run on multiple node ...
since there is no OAR integration in openmpi, i guess you are using ssh
to start orted on the remote nodes
(unless you instructed ompi to use an OARified version of ssh)
Yes, OMPI_MCA_plm_rsh_agent=oarshmost
This exports also needed environment instead of multpiple -x options. To
be as similar as possible to the environments on french national
supercomputers.
my concern is the remote orted might not run within the cpuset that was
created by OAR for this job,
so you might end up using all the cores on the remote nodes.
The oar environment does this. With older OpenMPI version all is working
fine.
please let us know how that works for you
Cheers,
Gilles
On 9/18/2015 5:02 PM, Gilles Gouaillardet wrote:
Patrick,
i just filled PR 586 https://github.com/open-mpi/ompi-release/pull/586
for the v1.10 series
this is only a three line patch.
could you please give it a try ?
This patch solve the problem when OpenMPI uses one node but now I'm unable
to use more than one node.
On one node, with 4 cores in the cpuset:
mpirun --bind-to core --hostfile $OAR_NODEFILE ./location.exe |grep
'thread is now running on PU' |sort
(process 0) thread is now running on PU logical index 0 (OS/physical index
12) on system frog26
(process 1) thread is now running on PU logical index 1 (OS/physical index
13) on system frog26
(process 2) thread is now running on PU logical index 2 (OS/physical index
14) on system frog26
(process 3) thread is now running on PU logical index 3 (OS/physical index
15) on system frog26
[begou@frog26 MPI_TESTS]$ mpirun -np 5 --bind-to core --hostfile
$OAR_NODEFILE ./location.exe
--------------------------------------------------------------------------
A request was made to bind to that would result in binding more
processes than cpus on a resource:
Bind to: CORE
Node: frog26
#processes: 2
#cpus: 1
You can override this protection by adding the "overload-allowed"
option to your binding directive.
But if I request two nodes (4 cores one each) only 4 processes can start
on the local cores, none on the second host:
[begou@frog5 MPI_TESTS]$ cat $OAR_NODEFILE
frog5
frog5
frog5
frog5
frog6
frog6
frog6
frog6
[begou@frog5 MPI_TESTS]$ cat ./frog.txt
frog5 slots=4
frog6 slots=4
But only 4 processes are launched:
[begou@frog5 MPI_TESTS]$ mpirun --hostfile frog.txt --bind-to core
./location.exe |grep 'thread is now running on PU'
(process 0) thread is now running on PU logical index 0 (OS/physical index
12) on system frog5
(process 1) thread is now running on PU logical index 1 (OS/physical index
13) on system frog5
(process 2) thread is now running on PU logical index 2 (OS/physical index
14) on system frog5
(process 3) thread is now running on PU logical index 3 (OS/physical index
15) on system frog5
If I ask explicitly 8 processes (one for each 4 cores of the 2 nodes)
[begou@frog5 MPI_TESTS]$ mpirun --hostfile frog.txt -np 8 --bind-to core
./location.exe
--------------------------------------------------------------------------
A request was made to bind to that would result in binding more
processes than cpus on a resource:
Bind to: CORE
Node: frog5
#processes: 2
#cpus: 1
You can override this protection by adding the "overload-allowed"
option to your binding directive.
Cheers,
Gilles
On 9/18/2015 4:54 PM, Patrick Begou wrote:
Ralph Castain wrote:
As I said, if you don't provide an explicit slot count in your
hostfile, we default to allowing oversubscription. We don't have OAR
integration in OMPI, and so mpirun isn't recognizing that you are
running under a resource manager - it thinks this is just being
controlled by a hostfile.
That's look strange for me is that in this case (default)
oversubscription is allowed for the number of core of one cpu (8), not
the number of cores available in the node (16) or unlimited...
If you want us to error out on oversubscription, you can either add
the flag you identified, or simply change your hostfile to:
frog53 slots=4
Either will work.
This syntax in the host file doesn't change anything to the
oversuscribing problem. It is still allowed with the same maximum
amount of processes for this test case:
[begou@frog7 MPI_TESTS]$ mpirun -np 8 *--hostfile frog7.txt* --bind-to
core ./location.exe|grep 'thread is now running on PU' |sort
(process 0) thread is now running on PU logical index 0 (OS/physical
index 0) on system frog7
(process 1) thread is now running on PU logical index 2 (OS/physical
index 6) on system frog7
(process 2) thread is now running on PU logical index 0 (OS/physical
index 0) on system frog7
(process 3) thread is now running on PU logical index 2 (OS/physical
index 6) on system frog7
(process 4) thread is now running on PU logical index 3 (OS/physical
index 7) on system frog7
(process 5) thread is now running on PU logical index 1 (OS/physical
index 5) on system frog7
(process 6) thread is now running on PU logical index 2 (OS/physical
index 6) on system frog7
(process 7) thread is now running on PU logical index 3 (OS/physical
index 7) on system frog7
[begou@frog7 MPI_TESTS]$ *cat frog7.txt*
frog7 slots=4
Patrick
On Sep 16, 2015, at 1:00 AM, Patrick Begou
<patrick.be...@legi.grenoble-inp.fr
<javascript:_e(%7B%7D,'cvml','patrick.be...@legi.grenoble-inp.fr');>>
wrote:
Thanks all for your answers, I've added some details about the tests
I have run. See below.
Ralph Castain wrote:
Not precisely correct. It depends on the environment.
If there is a resource manager allocating nodes, or you provide a
hostfile that specifies the number of slots on the nodes, or you use
-host, then we default to no-oversubscribe.
I'm using a batch scheduler (OAR).
# cat /dev/cpuset/oar/begou_7955553/cpuset.cpus
4-7
So 4 cores allowed. Nodes have two height cores cpus.
Node file contains:
# cat $OAR_NODEFILE
frog53
frog53
frog53
frog53
# mpirun --hostfile $OAR_NODEFILE -bind-to core location.exe
is okay (my test code show one process on each core)
(process 3) thread is now running on PU logical index 1 (OS/physical
index 5) on system frog53
(process 0) thread is now running on PU logical index 3 (OS/physical
index 7) on system frog53
(process 1) thread is now running on PU logical index 0 (OS/physical
index 4) on system frog53
(process 2) thread is now running on PU logical index 2 (OS/physical
index 6) on system frog53
# mpirun -np 5 --hostfile $OAR_NODEFILE -bind-to core location.exe
oversuscribe with:
(process 0) thread is now running on PU logical index 3 (OS/physical
index 7) on system frog53
(process 1) thread is now running on PU logical index 1 (OS/physical
index 5) on system frog53
(*process 3*) thread is now running on PU logical index*2
(OS/physical index 6)*on system frog53
(process 4) thread is now running on PU logical index 0 (OS/physical
index 4) on system frog53
(*process 2*) thread is now running on PU logical index*2
(OS/physical index 6)*on system frog53
This is not allowed with OpenMPI 1.7.3
I can increase until the maximul core number of this first pocessor
(8 cores)
# mpirun -np 8 --hostfile $OAR_NODEFILE -bind-to core location.exe
|grep 'thread is now running on PU'
(process 5) thread is now running on PU logical index 1 (OS/physical
index 5) on system frog53
(process 7) thread is now running on PU logical index 3 (OS/physical
index 7) on system frog53
(process 4) thread is now running on PU logical index 0 (OS/physical
index 4) on system frog53
(process 6) thread is now running on PU logical index 2 (OS/physical
index 6) on system frog53
(process 2) thread is now running on PU logical index 1 (OS/physical
index 5) on system frog53
(process 0) thread is now running on PU logical index 2 (OS/physical
index 6) on system frog53
(process 1) thread is now running on PU logical index 0 (OS/physical
index 4) on system frog53
(process 3) thread is now running on PU logical index 0 (OS/physical
index 4) on system frog53
But I cannot overload more than the 8 cores (max core number of one cpu).
#mpirun -np 9 --hostfile $OAR_NODEFILE -bind-to core location.exe
A request was made to bind to that would result in binding more
processes than cpus on a resource:
Bind to: CORE
Node: frog53
#processes: 2
#cpus: 1
You can override this protection by adding the "overload-allowed"
option to your binding directive.
Now if I add*--nooversubscribe*the problem doesn't exist anymore (no
more than 4 processes, one on each core). So looks like if default
behavior would be a nooversuscribe on cores number of the socket ???
Again, with 1.7.3 this problem doesn't occur at all.
Patrick
If you provide a hostfile that doesn't specify slots, then we use
the number of cores we find on each node, and we allow oversubscription.
What is being described sounds like more of a bug than an intended
feature. I'd need to know more about it, though, to be sure. Can you
tell me how you are specifying this cpuset?
On Sep 15, 2015, at 4:44 PM, Matt Thompson <fort...@gmail.com
<javascript:_e(%7B%7D,'cvml','fort...@gmail.com');>> wrote:
Looking at the Open MPI 1.10.0 man page:
https://www.open-mpi.org/doc/v1.10/man1/mpirun.1.php
it looks like perhaps -oversubscribe (which was an option) is now
the default behavior. Instead we have:
*-nooversubscribe, --nooversubscribe*
Do not oversubscribe any nodes; error (without starting any
processes) if the requested number of processes would cause
oversubscription. This option implicitly sets "max_slots" equal
to the "slots" value for each node.
It also looks like -map-by has a way to implement it as well (see
man page).
Thanks for letting me/us know about this. On a system of mine I
sort of depend on the -nooversubscribe behavior!
Matt
On Tue, Sep 15, 2015 at 11:17 AM, Patrick
Begou<patrick.be...@legi.grenoble-inp.fr
<javascript:_e(%7B%7D,'cvml','patrick.be...@legi.grenoble-inp.fr');>>wrote:
Hi,
I'm runing OpenMPI 1.10.0 built with Intel 2015 compilers on a
Bullx System.
I've some troubles with the bind-to core option when using cpuset.
If the cpuset is less than all the cores of a cpu (ex: 4 cores
allowed on a 8 cores cpus) OpenMPI 1.10.0 allows to overload
these cores until the maximum number of cores of the cpu.
With this config and because the cpuset only allows 4 cores, I
can reach 2 processes/core if I use:
mpirun -np 8 --bind-to core my_application
OpenMPI 1.7.3 doesn't show the problem with the same situation:
mpirun -np 8 --bind-to-core my_application
returns:
/A request was made to bind to that would result in binding more//
//processes than cpus on a resource/
and that's okay of course.
Is there a way to avoid this oveloading with OpenMPI 1.10.0 ?
Thanks
Patrick
--
===================================================================
| Equipe M.O.S.T. | |
| Patrick BEGOU |mailto:patrick.be...@grenoble-inp.fr
<javascript:_e(%7B%7D,'cvml','patrick.be...@grenoble-inp.fr');> |
| LEGI | |
| BP 53 X | Tel 04 76 82 51 35 |
| 38041 GRENOBLE CEDEX | Fax 04 76 82 52 71 |
===================================================================
_______________________________________________
users mailing list
us...@open-mpi.org
<javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27575.php
--
Matt Thompson
Man Among Men
Fulcrum of History
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27579.php
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27580.php
--
===================================================================
| Equipe M.O.S.T. | |
| Patrick BEGOU |mailto:patrick.be...@grenoble-inp.fr
<javascript:_e(%7B%7D,'cvml','patrick.be...@grenoble-inp.fr');> |
| LEGI | |
| BP 53 X | Tel 04 76 82 51 35 |
| 38041 GRENOBLE CEDEX | Fax 04 76 82 52 71 |
===================================================================
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27583.php
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27590.php
--
===================================================================
| Equipe M.O.S.T. | |
| Patrick BEGOU |mailto:patrick.be...@grenoble-inp.fr
<javascript:_e(%7B%7D,'cvml','patrick.be...@grenoble-inp.fr');> |
| LEGI | |
| BP 53 X | Tel 04 76 82 51 35 |
| 38041 GRENOBLE CEDEX | Fax 04 76 82 52 71 |
===================================================================
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27619.php
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27620.php
_______________________________________________
users mailing list
us...@open-mpi.org <javascript:_e(%7B%7D,'cvml','us...@open-mpi.org');>
Subscription:http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this
post:http://www.open-mpi.org/community/lists/users/2015/09/27621.php
--
===================================================================
| Equipe M.O.S.T. | |
| Patrick BEGOU |mailto:patrick.be...@grenoble-inp.fr
<javascript:_e(%7B%7D,'cvml','patrick.be...@grenoble-inp.fr');> |
| LEGI | |
| BP 53 X | Tel 04 76 82 51 35 |
| 38041 GRENOBLE CEDEX | Fax 04 76 82 52 71 |
===================================================================
_______________________________________________
users mailing list
us...@open-mpi.org
Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/users
Link to this post:
http://www.open-mpi.org/community/lists/users/2015/09/27624.php
--
===================================================================
| Equipe M.O.S.T. | |
| Patrick BEGOU | mailto:patrick.be...@grenoble-inp.fr |
| LEGI | |
| BP 53 X | Tel 04 76 82 51 35 |
| 38041 GRENOBLE CEDEX | Fax 04 76 82 52 71 |
===================================================================
