Re: [OMPI users] --map-by

[Noam Bernstein]

> On Nov 21, 2017, at 8:53 AM, r...@open-mpi.org wrote:
> 
>> On Nov 21, 2017, at 5:34 AM, Noam Bernstein > > wrote:
>> 
>>> 
>>> On Nov 20, 2017, at 7:02 PM, r...@open-mpi.org  
>>> wrote:
>>> 
>>> So there are two options here that will work and hopefully provide you with 
>>> the desired pattern:
>>> 
>>> * if you want the procs to go in different NUMA regions:
>>> $ mpirun --map-by numa:PE=2 --report-bindings -n 2 /bin/true
>>> [rhc001:131460] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 
>>> 0[core 1[hwt 0-1]]: 
>>> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
>>> [rhc001:131460] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 
>>> 1[core 13[hwt 0-1]]: 
>>> [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
>>> 
>>> * if you want the procs to go in the same NUMA region:
>>> $ mpirun --map-by ppr:2:numa:PE=2 --report-bindings -n 2 /bin/true
>>> [rhc001:131559] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 
>>> 0[core 1[hwt 0-1]]: 
>>> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
>>> [rhc001:131559] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 
>>> 0[core 3[hwt 0-1]]: 
>>> [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
>>> 
>>> Reason: the level you are mapping by (e.g., NUMA) must have enough cores in 
>>> it to meet your PE=N directive. If you map by core, then there is only one 
>>> core in that object.
>> 
>> Makes sense.  I’ll try that.  However, if I understand your explanation 
>> correctly the docs should probably be changed, because they seem to be 
>> suggesting something that will never work.   In fact, would the ":PE=N" > 1 
>> ever work for "—map-by core”?  I guess maybe if you have hyperthreading on, 
>> but I’d still argue that that’s an unhelpful example, given how rarely 
>> hyperthreading is used in HPC.


I can now confirm that "—map-by numa:PE=2" does indeed work, and seems to give 
good performance.

thanks,
Noam

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[r...@open-mpi.org]

I believe that map-by core with a PE > 1 may have worked at some point in the 
past, but the docs should probably be looked at. I took a (very brief) look at 
the code and re-enabling that particular option would be difficult and not 
really necessary since one can reproduce the desired pattern within the current 
context.

> On Nov 21, 2017, at 5:34 AM, Noam Bernstein  
> wrote:
> 
>> 
>> On Nov 20, 2017, at 7:02 PM, r...@open-mpi.org  
>> wrote:
>> 
>> So there are two options here that will work and hopefully provide you with 
>> the desired pattern:
>> 
>> * if you want the procs to go in different NUMA regions:
>> $ mpirun --map-by numa:PE=2 --report-bindings -n 2 /bin/true
>> [rhc001:131460] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
>> 1[hwt 0-1]]: 
>> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
>> [rhc001:131460] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 
>> 1[core 13[hwt 0-1]]: 
>> [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
>> 
>> * if you want the procs to go in the same NUMA region:
>> $ mpirun --map-by ppr:2:numa:PE=2 --report-bindings -n 2 /bin/true
>> [rhc001:131559] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
>> 1[hwt 0-1]]: 
>> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
>> [rhc001:131559] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 
>> 3[hwt 0-1]]: 
>> [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
>> 
>> Reason: the level you are mapping by (e.g., NUMA) must have enough cores in 
>> it to meet your PE=N directive. If you map by core, then there is only one 
>> core in that object.
> 
> Makes sense.  I’ll try that.  However, if I understand your explanation 
> correctly the docs should probably be changed, because they seem to be 
> suggesting something that will never work.   In fact, would the ":PE=N" > 1 
> ever work for "—map-by core”?  I guess maybe if you have hyperthreading on, 
> but I’d still argue that that’s an unhelpful example, given how rarely 
> hyperthreading is used in HPC.
> 
>   Noam
> 
> 
> 
> 
> 
> ||
> |U.S. NAVAL|
> |_RESEARCH_|
> LABORATORY
> 
> Noam Bernstein, Ph.D.
> Center for Materials Physics and Technology
> U.S. Naval Research Laboratory
> T +1 202 404 8628  F +1 202 404 7546
> https://www.nrl.navy.mil 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[Noam Bernstein]

> On Nov 20, 2017, at 7:02 PM, r...@open-mpi.org wrote:
> 
> So there are two options here that will work and hopefully provide you with 
> the desired pattern:
> 
> * if you want the procs to go in different NUMA regions:
> $ mpirun --map-by numa:PE=2 --report-bindings -n 2 /bin/true
> [rhc001:131460] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
> 1[hwt 0-1]]: 
> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
> [rhc001:131460] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 1[core 
> 13[hwt 0-1]]: 
> [../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]
> 
> * if you want the procs to go in the same NUMA region:
> $ mpirun --map-by ppr:2:numa:PE=2 --report-bindings -n 2 /bin/true
> [rhc001:131559] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
> 1[hwt 0-1]]: 
> [BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
> [rhc001:131559] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 
> 3[hwt 0-1]]: 
> [../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]
> 
> Reason: the level you are mapping by (e.g., NUMA) must have enough cores in 
> it to meet your PE=N directive. If you map by core, then there is only one 
> core in that object.

Makes sense.  I’ll try that.  However, if I understand your explanation 
correctly the docs should probably be changed, because they seem to be 
suggesting something that will never work.   In fact, would the ":PE=N" > 1 
ever work for "—map-by core”?  I guess maybe if you have hyperthreading on, but 
I’d still argue that that’s an unhelpful example, given how rarely 
hyperthreading is used in HPC.

Noam





||
|U.S. NAVAL|
|_RESEARCH_|
LABORATORY
Noam Bernstein, Ph.D.
Center for Materials Physics and Technology
U.S. Naval Research Laboratory
T +1 202 404 8628  F +1 202 404 7546
https://www.nrl.navy.mil 
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[r...@open-mpi.org]

So there are two options here that will work and hopefully provide you with the 
desired pattern:

* if you want the procs to go in different NUMA regions:
$ mpirun --map-by numa:PE=2 --report-bindings -n 2 /bin/true
[rhc001:131460] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]]: 
[BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:131460] MCW rank 1 bound to socket 1[core 12[hwt 0-1]], socket 1[core 
13[hwt 0-1]]: 
[../../../../../../../../../../../..][BB/BB/../../../../../../../../../..]

* if you want the procs to go in the same NUMA region:
$ mpirun --map-by ppr:2:numa:PE=2 --report-bindings -n 2 /bin/true
[rhc001:131559] MCW rank 0 bound to socket 0[core 0[hwt 0-1]], socket 0[core 
1[hwt 0-1]]: 
[BB/BB/../../../../../../../../../..][../../../../../../../../../../../..]
[rhc001:131559] MCW rank 1 bound to socket 0[core 2[hwt 0-1]], socket 0[core 
3[hwt 0-1]]: 
[../../BB/BB/../../../../../../../..][../../../../../../../../../../../..]

Reason: the level you are mapping by (e.g., NUMA) must have enough cores in it 
to meet your PE=N directive. If you map by core, then there is only one core in 
that object.

HTH
Ralph

> On Nov 16, 2017, at 7:08 AM, Noam Bernstein  
> wrote:
> 
> 
>> On Nov 16, 2017, at 9:49 AM, r...@open-mpi.org  
>> wrote:
>> 
>> Do not include the “bind-to core” option.the mapping directive already 
>> forces that 
> 
> Same error message, unfortunately. And no, I’m not setting a global binding 
> policy, as far as I can tell:
> 
> env | grep OMPI_MCA
> OMPI_MCA_hwloc_base_report_bindings=1
> [compute-7-6:15083] SETTING BINDING TO CORE
> --
> A request for multiple cpus-per-proc was given, but a directive
> was also give to map to an object level that cannot support that
> directive.
> 
> Please specify a mapping level that has more than one cpu, or
> else let us define a default mapping that will allow multiple
> cpus-per-proc.
> --
> 
>   Noam
> 
> 
> 
> ||
> |U.S. NAVAL|
> |_RESEARCH_|
> LABORATORY
> 
> Noam Bernstein, Ph.D.
> Center for Materials Physics and Technology
> U.S. Naval Research Laboratory
> T +1 202 404 8628  F +1 202 404 7546
> https://www.nrl.navy.mil 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[Noam Bernstein]

> On Nov 16, 2017, at 9:49 AM, r...@open-mpi.org wrote:
> 
> Do not include the “bind-to core” option.the mapping directive already forces 
> that 

Same error message, unfortunately. And no, I’m not setting a global binding 
policy, as far as I can tell:

env | grep OMPI_MCA
OMPI_MCA_hwloc_base_report_bindings=1
[compute-7-6:15083] SETTING BINDING TO CORE
--
A request for multiple cpus-per-proc was given, but a directive
was also give to map to an object level that cannot support that
directive.

Please specify a mapping level that has more than one cpu, or
else let us define a default mapping that will allow multiple
cpus-per-proc.
--

Noam



||
|U.S. NAVAL|
|_RESEARCH_|
LABORATORY
Noam Bernstein, Ph.D.
Center for Materials Physics and Technology
U.S. Naval Research Laboratory
T +1 202 404 8628  F +1 202 404 7546
https://www.nrl.navy.mil 
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] --map-by

[r...@open-mpi.org]

Do not include the “bind-to core” option.the mapping directive already forces 
that 

Sent from my iPad

> On Nov 16, 2017, at 7:44 AM, Noam Bernstein  
> wrote:
> 
> Hi all - I’m trying to run mixed MPI/OpenMP, so I ideally want binding of 
> each MPI process to a small set of cores (to allow for the OpenMP threads).   
> From the mpirun docs at 
> https://www.open-mpi.org//doc/current/man1/mpirun.1.php
> I got the example that I thought corresponded to what I want,
> % mpirun ... --map-by core:PE=2 --bind-to core
> So I tried
> mpirun -x OMP_NUM_THREADS --map-by core:PE=4 --bind-to core -np 32   python 
> …..
> 
> However, when I run this (with openmpi 3.0.0 or with 1.8.8) I get the 
> following error:
> A request for multiple cpus-per-proc was given, but a directive
> was also give to map to an object level that cannot support that
> directive.
> 
> Please specify a mapping level that has more than one cpu, or
> else let us define a default mapping that will allow multiple
> cpus-per-proc.
> 
> Am I doing something wrong, or is there a mistake in the docs, and it should 
> bind to something other than core?
> 
>   
> thanks,
>   
> Noam
> 
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://lists.open-mpi.org/mailman/listinfo/users
___
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users

Re: [OMPI users] "-map-by socket:PE=1" doesn't do what I expect

[Mark Dixon]

On Fri, 17 Feb 2017, r...@open-mpi.org wrote:

Mark - this is now available in master. Will look at what might be 
required to bring it to 2.0


Thanks Ralph,

To be honest, since you've given me an alternative, there's no rush from 
my point of view.


The logic's embedded in a script and it's been taught "--map-by socket 
--bind-to core" for the special case of 1. It'd be nice to get rid of it 
at some point, but there's no problem waiting for the next stable branch 
:)


Cheers,

Mark
___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] "-map-by socket:PE=1" doesn't do what I expect

[r...@open-mpi.org]

Mark - this is now available in master. Will look at what might be required to 
bring it to 2.0

> On Feb 15, 2017, at 5:49 AM, r...@open-mpi.org wrote:
> 
> 
>> On Feb 15, 2017, at 5:45 AM, Mark Dixon  wrote:
>> 
>> On Wed, 15 Feb 2017, r...@open-mpi.org wrote:
>> 
>>> Ah, yes - I know what the problem is. We weren’t expecting a PE value of 1 
>>> - the logic is looking expressly for values > 1 as we hadn’t anticipated 
>>> this use-case.
>> 
>> Is it a sensible use-case, or am I crazy?
> 
> Not crazy, I’d say. The expected way of doing it would be “--map-by socket 
> --bind-to core”. However, I can see why someone might expect pe=1 to work.
> 
>> 
>>> I can make that change. I’m off to a workshop for the next day or so, but 
>>> can probably do this on the plane.
>> 
>> You're a star - thanks :)
>> 
>> Mark___
>> users mailing list
>> users@lists.open-mpi.org
>> https://rfd.newmexicoconsortium.org/mailman/listinfo/users
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] "-map-by socket:PE=1" doesn't do what I expect

[r...@open-mpi.org]

> On Feb 15, 2017, at 5:45 AM, Mark Dixon  wrote:
> 
> On Wed, 15 Feb 2017, r...@open-mpi.org wrote:
> 
>> Ah, yes - I know what the problem is. We weren’t expecting a PE value of 1 - 
>> the logic is looking expressly for values > 1 as we hadn’t anticipated this 
>> use-case.
> 
> Is it a sensible use-case, or am I crazy?

Not crazy, I’d say. The expected way of doing it would be “--map-by socket 
--bind-to core”. However, I can see why someone might expect pe=1 to work.

> 
>> I can make that change. I’m off to a workshop for the next day or so, but 
>> can probably do this on the plane.
> 
> You're a star - thanks :)
> 
> Mark___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] "-map-by socket:PE=1" doesn't do what I expect

[Mark Dixon]

On Wed, 15 Feb 2017, r...@open-mpi.org wrote:

Ah, yes - I know what the problem is. We weren’t expecting a PE value of 
1 - the logic is looking expressly for values > 1 as we hadn’t 
anticipated this use-case.


Is it a sensible use-case, or am I crazy?

I can make that change. I’m off to a workshop for the next day or so, 
but can probably do this on the plane.


You're a star - thanks :)

Mark___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] "-map-by socket:PE=1" doesn't do what I expect

[r...@open-mpi.org]

Ah, yes - I know what the problem is. We weren’t expecting a PE value of 1 - 
the logic is looking expressly for values > 1 as we hadn’t anticipated this 
use-case.

I can make that change. I’m off to a workshop for the next day or so, but can 
probably do this on the plane.


> On Feb 15, 2017, at 3:17 AM, Mark Dixon  wrote:
> 
> Hi,
> 
> When combining OpenMPI 2.0.2 with OpenMP, I'm interested in launching a 
> number of ranks and allocating a number of cores to each rank. Using "-map-by 
> socket:PE=", switching to "-map-by node:PE=" if I want to allocate 
> more than a single socket to a rank, seems to do what I want.
> 
> Except for "-map-by socket:PE=1". That seems to allocate an entire socket to 
> each rank instead of a single core. Here's the output of a test program on a 
> dual socket non-hyperthreading system that reports rank core bindings (odd 
> cores on one socket, even on the other):
> 
>   $ mpirun -np 2 -map-by socket:PE=1 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2 4 6 8 10 12 14 16 18 20 22
>   Rank 1 bound somehost.somewhere:  1 3 5 7 9 11 13 15 17 19 21 23
> 
>   $ mpirun -np 2 -map-by socket:PE=2 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2
>   Rank 1 bound somehost.somewhere:  1 3
> 
>   $ mpirun -np 2 -map-by socket:PE=3 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2 4
>   Rank 1 bound somehost.somewhere:  1 3 5
> 
>   $ mpirun -np 2 -map-by socket:PE=4 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2 4 6
>   Rank 1 bound somehost.somewhere:  1 3 5 7
> 
> I get the same result if I change "socket" to "numa". Changing "socket" to 
> either "core", "node" or "slot" binds each rank to a single core (good), but 
> doesn't round-robin ranks across sockets like "socket" does (bad).
> 
> Is "-map-by socket:PE=1" doing the right thing, please? I tried reading the 
> man page but I couldn't work out what the expected behaviour is :o
> 
> Cheers,
> 
> Mark
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] "-map-by socket:PE=1" doesn't do what I expect

[r...@open-mpi.org]

Ah, yes - I know what the problem is. We weren’t expecting a PE value of 1 - 
the logic is looking expressly for values > 1 as we hadn’t anticipated this 
use-case.

I can make that change. I’m off to a workshop for the next day or so, but can 
probably do this on the plane.


> On Feb 15, 2017, at 3:17 AM, Mark Dixon  wrote:
> 
> Hi,
> 
> When combining OpenMPI 2.0.2 with OpenMP, I'm interested in launching a 
> number of ranks and allocating a number of cores to each rank. Using "-map-by 
> socket:PE=", switching to "-map-by node:PE=" if I want to allocate 
> more than a single socket to a rank, seems to do what I want.
> 
> Except for "-map-by socket:PE=1". That seems to allocate an entire socket to 
> each rank instead of a single core. Here's the output of a test program on a 
> dual socket non-hyperthreading system that reports rank core bindings (odd 
> cores on one socket, even on the other):
> 
>   $ mpirun -np 2 -map-by socket:PE=1 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2 4 6 8 10 12 14 16 18 20 22
>   Rank 1 bound somehost.somewhere:  1 3 5 7 9 11 13 15 17 19 21 23
> 
>   $ mpirun -np 2 -map-by socket:PE=2 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2
>   Rank 1 bound somehost.somewhere:  1 3
> 
>   $ mpirun -np 2 -map-by socket:PE=3 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2 4
>   Rank 1 bound somehost.somewhere:  1 3 5
> 
>   $ mpirun -np 2 -map-by socket:PE=4 ./report_binding
>   Rank 0 bound somehost.somewhere:  0 2 4 6
>   Rank 1 bound somehost.somewhere:  1 3 5 7
> 
> I get the same result if I change "socket" to "numa". Changing "socket" to 
> either "core", "node" or "slot" binds each rank to a single core (good), but 
> doesn't round-robin ranks across sockets like "socket" does (bad).
> 
> Is "-map-by socket:PE=1" doing the right thing, please? I tried reading the 
> man page but I couldn't work out what the expected behaviour is :o
> 
> Cheers,
> 
> Mark
> 
> ___
> users mailing list
> users@lists.open-mpi.org
> https://rfd.newmexicoconsortium.org/mailman/listinfo/users

___
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users

Re: [OMPI users] map-by node with openmpi-1.7.5a1

[Ralph Castain]

I'm afraid that patch didn't solve the problem when I tested it - it resolved a 
problem of cpus-per-rank > 1, but not the case of descending order of slots. 
Took a little more work, but I believe the patch in r30798 (based on yours) 
completes the job.

FWIW: the "hetero-nodes" flag is a bit of a red herring here. That flag 
indicates a difference in physical topology between the nodes, not a difference 
in number of assigned slots. It would be required if your nodes actually have 
different numbers of cores on them (e.g., have different chips) and you want us 
to bind the processes or map by some object lower than the node level, but only 
in such cases.

Appreciate your help! I scheduled the patch for 1.7.5 and assigned it to you 
for verification - please let me know if you don't haver time to do so.

https://svn.open-mpi.org/trac/ompi/ticket/4296

Ralph

On Feb 19, 2014, at 5:00 AM, tmish...@jcity.maeda.co.jp wrote:

> 
> 
> Hi Ralph, I've found the fix. Please check the attached
> patch file.
> 
> At this moment, nodes in hostfile should be listed in
> ascending order of slot size when we use "map-by node" or
> "map-by obj:span".
> 
> The problem is that the hostfile created by Torque in our
> cluster always lists allocated nodes in descending order...
> 
> Regards,
> Tetsuya Mishima
> 
> (See attached file: patch.rr)
> 
>> Hi Ralph,
>> 
>> I did overall verification of rr_mapper, and I found another problem
>> with "map-by node". As far as I checked, "map-by obj" other than node
>> worked fine. I myself do not use "map-by node", but I'd like to report
>> it to improve reliability of 1.7.5. It seems too difficult for me to
>> resolve it. I hope you could take a look.
>> 
>> The problem occurs when I mixedly use two kinds of node, although I
>> add "-hetero-nodes" to command line:
>> 
>> [mishima@manage work]$ cat pbs_hosts
>> node04 slots=8
>> node05 slots=2
>> node06 slots=2
>> 
>> [mishima@manage work]$ mpirun -np 12 -machinefile pbs_hosts -map-by node
>> -report-bindings -hetero-nodes /home/mishima/mi
>> s/openmpi/demos/myprog
>> [manage.cluster:13113] [[15682,0],0] ORTE_ERROR_LOG: Fatal in file
>> rmaps_rr.c at line 241
>> [manage.cluster:13113] [[15682,0],0] ORTE_ERROR_LOG: Fatal in file
>> base/rmaps_base_map_job.c at line 285
>> 
>> With "-np 11", it works. But rank 10 is bound to the wrong core (which is
>> already used by rank 0). I guess something is wrong with the handling of
>> different topology when "map-by node" is specified. In addition, the
>> calculation of assigning procs to each node has some problems:
>> 
>> [mishima@manage work]$ mpirun -np 11 -machinefile pbs_hosts -map-by node
>> -report-bindings -hetero-nodes /home/mishima/mi
>> s/openmpi/demos/myprog
>> [node04.cluster:13384] MCW rank 3 bound to socket 0[core 1[hwt 0]]:
>> [./B/./././././.][./././././././.][./././././././.][
>> ./././././././.]
>> [node04.cluster:13384] MCW rank 6 bound to socket 0[core 2[hwt 0]]:
>> [././B/././././.][./././././././.][./././././././.][
>> ./././././././.]
>> [node04.cluster:13384] MCW rank 8 bound to socket 0[core 3[hwt 0]]:
>> [./././B/./././.][./././././././.][./././././././.][
>> ./././././././.]
>> [node04.cluster:13384] MCW rank 10 bound to socket 0[core 0[hwt 0]]:
>> [B/././././././.][./././././././.][./././././././.]
>> [./././././././.]
>> [node04.cluster:13384] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
>> [B/././././././.][./././././././.][./././././././.][
>> ./././././././.]
>> [node06.cluster:24192] MCW rank 5 bound to socket 0[core 1[hwt 0]]:
>> [./B/./.][./././.]
>> [node06.cluster:24192] MCW rank 2 bound to socket 0[core 0[hwt 0]]:
>> [B/././.][./././.]
>> [node05.cluster:25655] MCW rank 9 bound to socket 0[core 3[hwt 0]]:
>> [./././B][./././.]
>> [node05.cluster:25655] MCW rank 1 bound to socket 0[core 0[hwt 0]]:
>> [B/././.][./././.]
>> [node05.cluster:25655] MCW rank 4 bound to socket 0[core 1[hwt 0]]:
>> [./B/./.][./././.]
>> [node05.cluster:25655] MCW rank 7 bound to socket 0[core 2[hwt 0]]:
>> [././B/.][./././.]
>> Hello world from process 4 of 11
>> Hello world from process 7 of 11
>> Hello world from process 6 of 11
>> Hello world from process 3 of 11
>> Hello world from process 0 of 11
>> Hello world from process 8 of 11
>> Hello world from process 2 of 11
>> Hello world from process 5 of 11
>> Hello world from process 9 of 11
>> Hello world from process 1 of 11
>> Hello world from process 10 of 11
>> 
>> Regards,
>> Tetsuya Mishima
>> 
>> ___
>> users mailing list
>> us...@open-mpi.org
>> http://www.open-mpi.org/mailman/listinfo.cgi/users
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

Re: [OMPI users] map-by node with openmpi-1.7.5a1

[tmishima]

Hi Ralph, I've found the fix. Please check the attached
patch file.

At this moment, nodes in hostfile should be listed in
ascending order of slot size when we use "map-by node" or
"map-by obj:span".

The problem is that the hostfile created by Torque in our
cluster always lists allocated nodes in descending order...

Regards,
Tetsuya Mishima

(See attached file: patch.rr)

> Hi Ralph,
>
> I did overall verification of rr_mapper, and I found another problem
> with "map-by node". As far as I checked, "map-by obj" other than node
> worked fine. I myself do not use "map-by node", but I'd like to report
> it to improve reliability of 1.7.5. It seems too difficult for me to
> resolve it. I hope you could take a look.
>
> The problem occurs when I mixedly use two kinds of node, although I
> add "-hetero-nodes" to command line:
>
> [mishima@manage work]$ cat pbs_hosts
> node04 slots=8
> node05 slots=2
> node06 slots=2
>
> [mishima@manage work]$ mpirun -np 12 -machinefile pbs_hosts -map-by node
> -report-bindings -hetero-nodes /home/mishima/mi
> s/openmpi/demos/myprog
> [manage.cluster:13113] [[15682,0],0] ORTE_ERROR_LOG: Fatal in file
> rmaps_rr.c at line 241
> [manage.cluster:13113] [[15682,0],0] ORTE_ERROR_LOG: Fatal in file
> base/rmaps_base_map_job.c at line 285
>
> With "-np 11", it works. But rank 10 is bound to the wrong core (which is
> already used by rank 0). I guess something is wrong with the handling of
> different topology when "map-by node" is specified. In addition, the
> calculation of assigning procs to each node has some problems:
>
> [mishima@manage work]$ mpirun -np 11 -machinefile pbs_hosts -map-by node
> -report-bindings -hetero-nodes /home/mishima/mi
> s/openmpi/demos/myprog
> [node04.cluster:13384] MCW rank 3 bound to socket 0[core 1[hwt 0]]:
> [./B/./././././.][./././././././.][./././././././.][
> ./././././././.]
> [node04.cluster:13384] MCW rank 6 bound to socket 0[core 2[hwt 0]]:
> [././B/././././.][./././././././.][./././././././.][
> ./././././././.]
> [node04.cluster:13384] MCW rank 8 bound to socket 0[core 3[hwt 0]]:
> [./././B/./././.][./././././././.][./././././././.][
> ./././././././.]
> [node04.cluster:13384] MCW rank 10 bound to socket 0[core 0[hwt 0]]:
> [B/././././././.][./././././././.][./././././././.]
> [./././././././.]
> [node04.cluster:13384] MCW rank 0 bound to socket 0[core 0[hwt 0]]:
> [B/././././././.][./././././././.][./././././././.][
> ./././././././.]
> [node06.cluster:24192] MCW rank 5 bound to socket 0[core 1[hwt 0]]:
> [./B/./.][./././.]
> [node06.cluster:24192] MCW rank 2 bound to socket 0[core 0[hwt 0]]:
> [B/././.][./././.]
> [node05.cluster:25655] MCW rank 9 bound to socket 0[core 3[hwt 0]]:
> [./././B][./././.]
> [node05.cluster:25655] MCW rank 1 bound to socket 0[core 0[hwt 0]]:
> [B/././.][./././.]
> [node05.cluster:25655] MCW rank 4 bound to socket 0[core 1[hwt 0]]:
> [./B/./.][./././.]
> [node05.cluster:25655] MCW rank 7 bound to socket 0[core 2[hwt 0]]:
> [././B/.][./././.]
> Hello world from process 4 of 11
> Hello world from process 7 of 11
> Hello world from process 6 of 11
> Hello world from process 3 of 11
> Hello world from process 0 of 11
> Hello world from process 8 of 11
> Hello world from process 2 of 11
> Hello world from process 5 of 11
> Hello world from process 9 of 11
> Hello world from process 1 of 11
> Hello world from process 10 of 11
>
> Regards,
> Tetsuya Mishima
>
> ___
> users mailing list
> us...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/users

patch.rr
Description: Binary data