I agree. -----Original Message----- From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Jeff Squyres (jsquyres) Sent: Tuesday, April 21, 2015 7:17 AM To: Open MPI Developers List Subject: Re: [OMPI devel] binding output error
+1 Devendar, you seem to be reporting a different issue than Elena...? FWIW: Open MPI has always used logical CPU numbering. As far as I can tell from your output, it looks like Open MPI did the Right Thing with your examples. Elena's example seemed to show conflicting cpu numbering -- where OMPI said it would bind a process and then where it actually bound it. Ralph mentioned to me that he would look at this as soon as he could; he thinks it might just be an error in the printf output (and that the binding is actually occurring in the right location). > On Apr 20, 2015, at 9:48 PM, tmish...@jcity.maeda.co.jp wrote: > > Hi Devendar, > > As far as I know, the report-bindings option shows the logical cpu > order. On the other hand, you are talking about physical one, I guess. > > Regards, > Tetsuya Mishima > > 2015/04/21 9:04:37、"devel"さんは「Re: [OMPI devel] binding output > error」で書きました >> HT is not enabled. All node are same topo . This is reproducible >> even on > single node. >> >> >> >> I ran osu latency to see if it is really is mapped to other socket or >> not > with –map-by socket. It looks likes mapping is correct as per latency > test. >> >> >> >> $mpirun -np 2 -report-bindings -map-by > socket > /hpc/local/benchmarks/hpc-stack-icc/install/ompi-mellanox-v1.8/tests/o > su-micro-benchmarks-4.4.1/osu_latency > >> >> [clx-orion-001:10084] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > [B/././././././././././././.][./././././././././././././.] >> >> [clx-orion-001:10084] MCW rank 1 bound to socket 1[core 14[hwt 0]]: > [./././././././././././././.][B/././././././././././././.] >> >> # OSU MPI Latency Test v4.4.1 >> >> # Size Latency (us) >> >> 0 0.50 >> >> 1 0.50 >> >> 2 0.50 >> >> 4 0.49 >> >> >> >> >> >> $mpirun -np 2 -report-bindings -cpu-set > 1,7 > /hpc/local/benchmarks/hpc-stack-icc/install/ompi-mellanox-v1.8/tests/o > su-micro-benchmarks-4.4.1/osu_latency > >> >> [clx-orion-001:10155] MCW rank 0 bound to socket 0[core 1[hwt 0]]: > [./B/./././././././././././.][./././././././././././././.] >> >> [clx-orion-001:10155] MCW rank 1 bound to socket 0[core 7[hwt 0]]: > [./././././././B/./././././.][./././././././././././././.] >> >> # OSU MPI Latency Test v4.4.1 >> >> # Size Latency (us) >> >> 0 0.23 >> >> 1 0.24 >> >> 2 0.23 >> >> 4 0.22 >> >> 8 0.23 >> >> >> >> Both hwloc and /proc/cpuinfo indicates following cpu numbering >> >> socket 0 cpus: 0 1 2 3 4 5 6 14 15 16 17 18 19 20 >> >> socket 1 cpus: 7 8 9 10 11 12 13 21 22 23 24 25 26 27 >> >> >> >> $hwloc-info -f >> >> Machine (256GB) >> >> NUMANode L#0 (P#0 128GB) + Socket L#0 + L3 L#0 (35MB) >> >> L2 L#0 (256KB) + L1 L#0 (32KB) + Core L#0 + PU L#0 (P#0) >> >> L2 L#1 (256KB) + L1 L#1 (32KB) + Core L#1 + PU L#1 (P#1) >> >> L2 L#2 (256KB) + L1 L#2 (32KB) + Core L#2 + PU L#2 (P#2) >> >> L2 L#3 (256KB) + L1 L#3 (32KB) + Core L#3 + PU L#3 (P#3) >> >> L2 L#4 (256KB) + L1 L#4 (32KB) + Core L#4 + PU L#4 (P#4) >> >> L2 L#5 (256KB) + L1 L#5 (32KB) + Core L#5 + PU L#5 (P#5) >> >> L2 L#6 (256KB) + L1 L#6 (32KB) + Core L#6 + PU L#6 (P#6) >> >> L2 L#7 (256KB) + L1 L#7 (32KB) + Core L#7 + PU L#7 (P#14) >> >> L2 L#8 (256KB) + L1 L#8 (32KB) + Core L#8 + PU L#8 (P#15) >> >> L2 L#9 (256KB) + L1 L#9 (32KB) + Core L#9 + PU L#9 (P#16) >> >> L2 L#10 (256KB) + L1 L#10 (32KB) + Core L#10 + PU L#10 (P#17) >> >> L2 L#11 (256KB) + L1 L#11 (32KB) + Core L#11 + PU L#11 (P#18) >> >> L2 L#12 (256KB) + L1 L#12 (32KB) + Core L#12 + PU L#12 (P#19) >> >> L2 L#13 (256KB) + L1 L#13 (32KB) + Core L#13 + PU L#13 (P#20) >> >> NUMANode L#1 (P#1 128GB) + Socket L#1 + L3 L#1 (35MB) >> >> L2 L#14 (256KB) + L1 L#14 (32KB) + Core L#14 + PU L#14 (P#7) >> >> L2 L#15 (256KB) + L1 L#15 (32KB) + Core L#15 + PU L#15 (P#8) >> >> L2 L#16 (256KB) + L1 L#16 (32KB) + Core L#16 + PU L#16 (P#9) >> >> L2 L#17 (256KB) + L1 L#17 (32KB) + Core L#17 + PU L#17 (P#10) >> >> L2 L#18 (256KB) + L1 L#18 (32KB) + Core L#18 + PU L#18 (P#11) >> >> L2 L#19 (256KB) + L1 L#19 (32KB) + Core L#19 + PU L#19 (P#12) >> >> L2 L#20 (256KB) + L1 L#20 (32KB) + Core L#20 + PU L#20 (P#13) >> >> L2 L#21 (256KB) + L1 L#21 (32KB) + Core L#21 + PU L#21 (P#21) >> >> L2 L#22 (256KB) + L1 L#22 (32KB) + Core L#22 + PU L#22 (P#22) >> >> L2 L#23 (256KB) + L1 L#23 (32KB) + Core L#23 + PU L#23 (P#23) >> >> L2 L#24 (256KB) + L1 L#24 (32KB) + Core L#24 + PU L#24 (P#24) >> >> L2 L#25 (256KB) + L1 L#25 (32KB) + Core L#25 + PU L#25 (P#25) >> >> L2 L#26 (256KB) + L1 L#26 (32KB) + Core L#26 + PU L#26 (P#26) >> >> L2 L#27 (256KB) + L1 L#27 (32KB) + Core L#27 + PU L#27 (P#27) >> >> >> >> >> >> So, Is --reporting-binding shows one more level of logical CPU numbering? >> >> >> >> >> >> -Devendar >> >> >> >> >> >> From:devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Ralph >> Castain >> Sent: Monday, April 20, 2015 3:52 PM >> To: Open MPI Developers >> Subject: Re: [OMPI devel] binding output error >> >> >> >> Also, was this with HT's enabled? I'm wondering if the print code is > incorrectly computing the core because it isn't correctly accounting > for HT cpus. >> >> >> >> >> >> On Mon, Apr 20, 2015 at 3:49 PM, Jeff Squyres (jsquyres) > <jsquy...@cisco.com> wrote: >> >> Ralph's the authority on this one, but just to be sure: are all nodes >> the > same topology? E.g., does adding "--hetero-nodes" to the mpirun > command line fix the problem? >> >> >> >>> On Apr 20, 2015, at 9:29 AM, Elena Elkina <elena.elk...@itseez.com> > wrote: >>> >>> Hi guys, >>> >>> I faced with an issue on our cluster related to mapping & binding > policies on 1.8.5. >>> >>> The matter is that --report-bindings output doesn't correspond to >>> the > locale. It looks like there is a mistake on the output itself, because > it just puts serial core number while that core can be >> on another socket. For example, >>> >>> mpirun -np 2 --display-devel-map --report-bindings --map-by socket > hostname >>> Data for JOB [43064,1] offset 0 >>> >>> Mapper requested: NULL Last mapper: round_robin Mapping policy: > BYSOCKET Ranking policy: SOCKET >>> Binding policy: CORE Cpu set: NULL PPR: NULL Cpus-per-rank: 1 >>> Num new daemons: 0 New daemon starting vpid INVALID >>> Num nodes: 1 >>> >>> Data for node: clx-orion-001 Launch id: -1 State: 2 >>> Daemon: [[43064,0],0] Daemon launched: True >>> Num slots: 28 Slots in use: 2 Oversubscribed: FALSE >>> Num slots allocated: 28 Max slots: 0 >>> Username on node: NULL >>> Num procs: 2 Next node_rank: 2 >>> Data for proc: [[43064,1],0] >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 0 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 0-6,14-20 Bind location: 0 Binding: 0 >>> Data for proc: [[43064,1],1] >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 1 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 7-13,21-27 Bind location: 7 Binding: 7 >>> [clx-orion-001:26951] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > [B/././././././././././././.][./././././././././././././.] >>> [clx-orion-001:26951] MCW rank 1 bound to socket 1[core 14[hwt 0]]: > [./././././././././././././.][B/././././././././././././.] >>> >>> The second process should be bound at core 7 (not core 14). >>> >>> >>> Another example: >>> mpirun -np 8 --display-devel-map --report-bindings --map-by core > hostname >>> Data for JOB [43202,1] offset 0 >>> >>> Mapper requested: NULL Last mapper: round_robin Mapping policy: > BYCORE Ranking policy: CORE >>> Binding policy: CORE Cpu set: NULL PPR: NULL Cpus-per-rank: 1 >>> Num new daemons: 0 New daemon starting vpid INVALID >>> Num nodes: 1 >>> >>> Data for node: clx-orion-001 Launch id: -1 State: 2 >>> Daemon: [[43202,0],0] Daemon launched: True >>> Num slots: 28 Slots in use: 8 Oversubscribed: FALSE >>> Num slots allocated: 28 Max slots: 0 >>> Username on node: NULL >>> Num procs: 8 Next node_rank: 8 >>> Data for proc: [[43202,1],0] >>> Pid: 0 Local rank: 0 Node rank: 0 App rank: 0 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 0 Bind location: 0 Binding: 0 >>> Data for proc: [[43202,1],1] >>> Pid: 0 Local rank: 1 Node rank: 1 App rank: 1 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 1 Bind location: 1 Binding: 1 >>> Data for proc: [[43202,1],2] >>> Pid: 0 Local rank: 2 Node rank: 2 App rank: 2 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 2 Bind location: 2 Binding: 2 >>> Data for proc: [[43202,1],3] >>> Pid: 0 Local rank: 3 Node rank: 3 App rank: 3 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 3 Bind location: 3 Binding: 3 >>> Data for proc: [[43202,1],4] >>> Pid: 0 Local rank: 4 Node rank: 4 App rank: 4 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 4 Bind location: 4 Binding: 4 >>> Data for proc: [[43202,1],5] >>> Pid: 0 Local rank: 5 Node rank: 5 App rank: 5 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 5 Bind location: 5 Binding: 5 >>> Data for proc: [[43202,1],6] >>> Pid: 0 Local rank: 6 Node rank: 6 App rank: 6 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 6 Bind location: 6 Binding: 6 >>> Data for proc: [[43202,1],7] >>> Pid: 0 Local rank: 7 Node rank: 7 App rank: 7 >>> State: INITIALIZED Restarts: 0 App_context: 0 > Locale: 14 Bind location: 14 Binding: 14 >>> [clx-orion-001:27069] MCW rank 0 bound to socket 0[core 0[hwt 0]]: > [B/././././././././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 1 bound to socket 0[core 1[hwt 0]]: > [./B/./././././././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 2 bound to socket 0[core 2[hwt 0]]: > [././B/././././././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 3 bound to socket 0[core 3[hwt 0]]: > [./././B/./././././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 4 bound to socket 0[core 4[hwt 0]]: > [././././B/././././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 5 bound to socket 0[core 5[hwt 0]]: > [./././././B/./././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 6 bound to socket 0[core 6[hwt 0]]: > [././././././B/././././././.][./././././././././././././.] >>> [clx-orion-001:27069] MCW rank 7 bound to socket 0[core 7[hwt 0]]: > [./././././././B/./././././.][./././././././././././././.] >>> >>> Rank 7 should be bound at core 14 instead of core 7 since core 7 is >>> at > another socket. >>> >>> Best regards, >>> Elena >>> >>> >> >>> _______________________________________________ >>> devel mailing list >>> de...@open-mpi.org >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >>> Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/04/17273.php >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel >> Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/04/17282.php >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/develLink >> to > this post: > http://www.open-mpi.org/community/lists/devel/2015/04/17287.php > _______________________________________________ > devel mailing list > de...@open-mpi.org > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel > Link to this post: > http://www.open-mpi.org/community/lists/devel/2015/04/17291.php -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ _______________________________________________ devel mailing list de...@open-mpi.org Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel Link to this post: http://www.open-mpi.org/community/lists/devel/2015/04/17295.php
