Hi Brice,
Thank you for your comment. I understand what you mean. My opinion was made just considering easy way to adjust the code for inversion of hierarchy in object tree. Tetsuya Mishima > I don't think there's any such difference. > Also, all these NUMA architectures are reported the same by hwloc, and > therefore used the same in Open MPI. > > And yes, L3 and NUMA are topologically-identical on AMD Magny-Cours (and > most recent AMD and Intel platforms). > > Brice > > > > Le 20/12/2013 11:33, tmish...@jcity.maeda.co.jp a écrit : > > > > Hi Ralph, > > > > The numa-node in AMD Mangy-Cours/Interlagos is so called cc(cache > > coherent)NUMA, > > which seems to be a little bit different from the traditional numa defined > > in openmpi. > > > > I notice that ccNUMA object is almost same as L3cache object. > > So "-bind-to l3cache" or "-map-by l3cache" is valid for what I want to do. > > Therefore, "do not touch it" is one of the solution, I think ... > > > > Anyway, mixing up these two types of numa is the problem. > > > > Regards, > > Tetsuya Mishima > > > >> I can wait it'll be fixed in 1.7.5 or later, because putting "-bind-to > >> numa" > >> and "-map-by numa" at the same time works as a workaround. > >> > >> Thanks, > >> Tetsuya Mishima > >> > >>> Yeah, it will impact everything that uses hwloc topology maps, I fear. > >>> > >>> One side note: you'll need to add --hetero-nodes to your cmd line. If > > we > >> don't see that, we assume that all the node topologies are identical - > >> which clearly isn't true here. > >>> I'll try to resolve the hier inversion over the holiday - won't be for > >> 1.7.4, but hopefully for 1.7.5 > >>> Thanks > >>> Ralph > >>> > >>> On Dec 18, 2013, at 9:44 PM, tmish...@jcity.maeda.co.jp wrote: > >>> > >>>> > >>>> I think it's normal for AMD opteron having 8/16 cores such as > >>>> magny cours or interlagos. Because it usually has 2 numa nodes > >>>> in a cpu(socket), numa-node can not include a socket. This type > >>>> of hierarchy would be natural. > >>>> > >>>> (node03 is Dell PowerEdge R815 and maybe quite common, I guess) > >>>> > >>>> By the way, I think this inversion should affect rmaps_lama mapping. > >>>> > >>>> Tetsuya Mishima > >>>> > >>>>> Ick - yeah, that would be a problem. I haven't seen that type of > >>>> hierarchical inversion before - is node03 a different type of chip? > >>>>> Might take awhile for me to adjust the code to handle hier > >>>> inversion... :-( > >>>>> On Dec 18, 2013, at 9:05 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>> > >>>>>> > >>>>>> Hi Ralph, > >>>>>> > >>>>>> I found the reason. I attached the main part of output with 32 > >>>>>> core node(node03) and 8 core node(node05) at the bottom. > >>>>>> > >>>>>> From this information, socket of node03 includes numa-node. > >>>>>> On the other hand, numa-node of node05 includes socket. > >>>>>> The direction of object tree is opposite. > >>>>>> > >>>>>> Since "-map-by socket" may be assumed as default, > >>>>>> for node05, "-bind-to numa and -map-by socket" means > >>>>>> upward search. For node03, this should be downward. > >>>>>> > >>>>>> I guess that openmpi-1.7.4rc1 will always assume numa-node > >>>>>> includes socket. Is it right? Then, upward search is assumed > >>>>>> in orte_rmaps_base_compute_bindings even for node03 when I > >>>>>> put "-bind-to numa and -map-by socket" option. > >>>>>> > >>>>>> [node03.cluster:15508] [[38286,0],0] rmaps:base:compute_usage > >>>>>> [node03.cluster:15508] mca:rmaps: compute bindings for job > > [38286,1] > >>>> with > >>>>>> policy NUMA > >>>>>> [node03.cluster:15508] mca:rmaps: bind upwards for job [38286,1] > > with > >>>>>> bindings NUMA > >>>>>> [node03.cluster:15508] [[38286,0],0] bind:upward target NUMANode > > type > >>>>>> Machine > >>>>>> > >>>>>> That's the reason of this trouble. Therefore, adding "-map-by core" > >>>> works. > >>>>>> (mapping pattern seems to be strange ...) > >>>>>> > >>>>>> [mishima@node03 demos]$ mpirun -np 8 -bind-to numa -map-by core > >>>>>> -report-bindings myprog > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>> Cache > >>>>>> [node03.cluster:15885] [[38679,0],0] bind:upward target NUMANode > > type > >>>>>> NUMANode > >>>>>> [node03.cluster:15885] MCW rank 2 bound to socket 0[core 0[hwt 0]], > >>>> socket > >>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>> cket 0[core 3[hwt 0]]: > >>>>>> > > [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 3 bound to socket 0[core 0[hwt 0]], > >>>> socket > >>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>> cket 0[core 3[hwt 0]]: > >>>>>> > > [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 4 bound to socket 0[core 4[hwt 0]], > >>>> socket > >>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>> cket 0[core 7[hwt 0]]: > >>>>>> > > [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 5 bound to socket 0[core 4[hwt 0]], > >>>> socket > >>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>> cket 0[core 7[hwt 0]]: > >>>>>> > > [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 6 bound to socket 0[core 4[hwt 0]], > >>>> socket > >>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>> cket 0[core 7[hwt 0]]: > >>>>>> > > [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 7 bound to socket 0[core 4[hwt 0]], > >>>> socket > >>>>>> 0[core 5[hwt 0]], socket 0[core 6[hwt 0]], so > >>>>>> cket 0[core 7[hwt 0]]: > >>>>>> > > [././././B/B/B/B][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 0 bound to socket 0[core 0[hwt 0]], > >>>> socket > >>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>> cket 0[core 3[hwt 0]]: > >>>>>> > > [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>> [node03.cluster:15885] MCW rank 1 bound to socket 0[core 0[hwt 0]], > >>>> socket > >>>>>> 0[core 1[hwt 0]], socket 0[core 2[hwt 0]], so > >>>>>> cket 0[core 3[hwt 0]]: > >>>>>> > > [B/B/B/B/./././.][./././././././.][./././././././.][./././././././.] > >>>>>> Hello world from process 6 of 8 > >>>>>> Hello world from process 5 of 8 > >>>>>> Hello world from process 0 of 8 > >>>>>> Hello world from process 7 of 8 > >>>>>> Hello world from process 3 of 8 > >>>>>> Hello world from process 4 of 8 > >>>>>> Hello world from process 2 of 8 > >>>>>> Hello world from process 1 of 8 > >>>>>> > >>>>>> Regards, > >>>>>> Tetsuya Mishima > >>>>>> > >>>>>> [node03.cluster:15508] Type: Machine Number of child objects: 4 > >>>>>> Name=NULL > >>>>>> total=132358820KB > >>>>>> Backend=Linux > >>>>>> OSName=Linux > >>>>>> OSRelease=2.6.18-308.16.1.el5 > >>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >>>>>> Architecture=x86_64 > >>>>>> Cpuset: 0xffffffff > >>>>>> Online: 0xffffffff > >>>>>> Allowed: 0xffffffff > >>>>>> Bind CPU proc: TRUE > >>>>>> Bind CPU thread: TRUE > >>>>>> Bind MEM proc: FALSE > >>>>>> Bind MEM thread: TRUE > >>>>>> Type: Socket Number of child objects: 2 > >>>>>> Name=NULL > >>>>>> total=33071780KB > >>>>>> CPUModel="AMD Opteron(tm) Processor 6136" > >>>>>> Cpuset: 0x000000ff > >>>>>> Online: 0x000000ff > >>>>>> Allowed: 0x000000ff > >>>>>> Type: NUMANode Number of child objects: 1 > >>>>>> > >>>>>> > >>>>>> [node05.cluster:21750] Type: Machine Number of child objects: 2 > >>>>>> Name=NULL > >>>>>> total=33080072KB > >>>>>> Backend=Linux > >>>>>> OSName=Linux > >>>>>> OSRelease=2.6.18-308.16.1.el5 > >>>>>> OSVersion="#1 SMP Tue Oct 2 22:01:43 EDT 2012" > >>>>>> Architecture=x86_64 > >>>>>> Cpuset: 0x000000ff > >>>>>> Online: 0x000000ff > >>>>>> Allowed: 0x000000ff > >>>>>> Bind CPU proc: TRUE > >>>>>> Bind CPU thread: TRUE > >>>>>> Bind MEM proc: FALSE > >>>>>> Bind MEM thread: TRUE > >>>>>> Type: NUMANode Number of child objects: 1 > >>>>>> Name=NULL > >>>>>> local=16532232KB > >>>>>> total=16532232KB > >>>>>> Cpuset: 0x0000000f > >>>>>> Online: 0x0000000f > >>>>>> Allowed: 0x0000000f > >>>>>> Type: Socket Number of child objects: 1 > >>>>>> > >>>>>> > >>>>>>> Hmm...try adding "-mca rmaps_base_verbose 10 -mca ess_base_verbose > >> 5" > >>>> to > >>>>>> your cmd line and let's see what it thinks it found. > >>>>>>> > >>>>>>> On Dec 18, 2013, at 6:55 PM, tmish...@jcity.maeda.co.jp wrote: > >>>>>>> > >>>>>>>> > >>>>>>>> Hi, I report one more problem with openmpi-1.7.4rc1, > >>>>>>>> which is more serious. > >>>>>>>> > >>>>>>>> For our 32 core nodes(AMD magny cours based) which has > >>>>>>>> 8 numa-nodes, "-bind-to numa" does not work. Without > >>>>>>>> this option, it works. For your infomation, at the > >>>>>>>> bottom of this mail, I added the lstopo information > >>>>>>>> of the node. > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Tetsuya Mishima > >>>>>>>> > >>>>>>>> [mishima@manage ~]$ qsub -I -l nodes=1:ppn=32 > >>>>>>>> qsub: waiting for job 8352.manage.cluster to start > >>>>>>>> qsub: job 8352.manage.cluster ready > >>>>>>>> > >>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings -bind-to > > numa > >>>>>> myprog > >>>>>>>> [node03.cluster:15316] [[37582,0],0] bind:upward target NUMANode > >> type > >>>>>>>> Machine > >>>>>>>> > > -------------------------------------------------------------------------- > >>>>>>>> A request was made to bind to NUMA, but an appropriate target > > could > >>>> not > >>>>>>>> be found on node node03. > >>>>>>>> > > -------------------------------------------------------------------------- > >>>>>>>> [mishima@node03 ~]$ cd ~/Desktop/openmpi-1.7/demos/ > >>>>>>>> [mishima@node03 demos]$ mpirun -np 8 -report-bindings myprog > >>>>>>>> [node03.cluster:15282] MCW rank 2 bound to socket 1[core 8[hwt > > 0]]: > >>>>>>>> [./././././././.][B/././././././.][./././././././.][ > >>>>>>>> ./././././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 3 bound to socket 1[core 9[hwt > > 0]]: > >>>>>>>> [./././././././.][./B/./././././.][./././././././.][ > >>>>>>>> ./././././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 4 bound to socket 2[core 16[hwt > >> 0]]: > >>>>>>>> [./././././././.][./././././././.][B/././././././.] > >>>>>>>> [./././././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 5 bound to socket 2[core 17[hwt > >> 0]]: > >>>>>>>> [./././././././.][./././././././.][./B/./././././.] > >>>>>>>> [./././././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 6 bound to socket 3[core 24[hwt > >> 0]]: > >>>>>>>> [./././././././.][./././././././.][./././././././.] > >>>>>>>> [B/././././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 7 bound to socket 3[core 25[hwt > >> 0]]: > >>>>>>>> [./././././././.][./././././././.][./././././././.] > >>>>>>>> [./B/./././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 0 bound to socket 0[core 0[hwt > > 0]]: > >>>>>>>> [B/././././././.][./././././././.][./././././././.][ > >>>>>>>> ./././././././.] > >>>>>>>> [node03.cluster:15282] MCW rank 1 bound to socket 0[core 1[hwt > > 0]]: > >>>>>>>> [./B/./././././.][./././././././.][./././././././.][ > >>>>>>>> ./././././././.] > >>>>>>>> Hello world from process 2 of 8 > >>>>>>>> Hello world from process 5 of 8 > >>>>>>>> Hello world from process 4 of 8 > >>>>>>>> Hello world from process 3 of 8 > >>>>>>>> Hello world from process 1 of 8 > >>>>>>>> Hello world from process 7 of 8 > >>>>>>>> Hello world from process 6 of 8 > >>>>>>>> Hello world from process 0 of 8 > >>>>>>>> [mishima@node03 demos]$ ~/opt/hwloc/bin/lstopo-no-graphics > >>>>>>>> Machine (126GB) > >>>>>>>> Socket L#0 (32GB) > >>>>>>>> NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB) > >>>>>>>> L2 L#0 (512KB) + L1d L#0 (64KB) + L1i L#0 (64KB) + Core L#0 + > > PU > >>>>>> L#0 > >>>>>>>> (P#0) > >>>>>>>> L2 L#1 (512KB) + L1d L#1 (64KB) + L1i L#1 (64KB) + Core L#1 + > > PU > >>>>>> L#1 > >>>>>>>> (P#1) > >>>>>>>> L2 L#2 (512KB) + L1d L#2 (64KB) + L1i L#2 (64KB) + Core L#2 + > > PU > >>>>>> L#2 > >>>>>>>> (P#2) > >>>>>>>> L2 L#3 (512KB) + L1d L#3 (64KB) + L1i L#3 (64KB) + Core L#3 + > > PU > >>>>>> L#3 > >>>>>>>> (P#3) > >>>>>>>> NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB) > >>>>>>>> L2 L#4 (512KB) + L1d L#4 (64KB) + L1i L#4 (64KB) + Core L#4 + > > PU > >>>>>> L#4 > >>>>>>>> (P#4) > >>>>>>>> L2 L#5 (512KB) + L1d L#5 (64KB) + L1i L#5 (64KB) + Core L#5 + > > PU > >>>>>> L#5 > >>>>>>>> (P#5) > >>>>>>>> L2 L#6 (512KB) + L1d L#6 (64KB) + L1i L#6 (64KB) + Core L#6 + > > PU > >>>>>> L#6 > >>>>>>>> (P#6) > >>>>>>>> L2 L#7 (512KB) + L1d L#7 (64KB) + L1i L#7 (64KB) + Core L#7 + > > PU>>>>>> L#7 > >>>>>>>> (P#7) > >>>>>>>> Socket L#1 (32GB) > >>>>>>>> NUMANode L#2 (P#6 16GB) + L3 L#2 (5118KB) > >>>>>>>> L2 L#8 (512KB) + L1d L#8 (64KB) + L1i L#8 (64KB) + Core L#8 + > > PU > >>>>>> L#8 > >>>>>>>> (P#8) > >>>>>>>> L2 L#9 (512KB) + L1d L#9 (64KB) + L1i L#9 (64KB) + Core L#9 + > > PU > >>>>>> L#9 > >>>>>>>> (P#9) > >>>>>>>> L2 L#10 (512KB) + L1d L#10 (64KB) + L1i L#10 (64KB) + Core > > L#10 > >> + > >>>>>> PU > >>>>>>>> L#10 (P#10) > >>>>>>>> L2 L#11 (512KB) + L1d L#11 (64KB) + L1i L#11 (64KB) + Core > > L#11 > >> + > >>>>>> PU > >>>>>>>> L#11 (P#11) > >>>>>>>> NUMANode L#3 (P#7 16GB) + L3 L#3 (5118KB) > >>>>>>>> L2 L#12 (512KB) + L1d L#12 (64KB) + L1i L#12 (64KB) + Core > > L#12 > >> + > >>>>>> PU > >>>>>>>> L#12 (P#12) > >>>>>>>> L2 L#13 (512KB) + L1d L#13 (64KB) + L1i L#13 (64KB) + Core > > L#13 > >> + > >>>>>> PU > >>>>>>>> L#13 (P#13) > >>>>>>>> L2 L#14 (512KB) + L1d L#14 (64KB) + L1i L#14 (64KB) + Core > > L#14 > >> + > >>>>>> PU > >>>>>>>> L#14 (P#14) > >>>>>>>> L2 L#15 (512KB) + L1d L#15 (64KB) + L1i L#15 (64KB) + Core > > L#15 > >> + > >>>>>> PU > >>>>>>>> L#15 (P#15) > >>>>>>>> Socket L#2 (32GB) > >>>>>>>> NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB) > >>>>>>>> L2 L#16 (512KB) + L1d L#16 (64KB) + L1i L#16 (64KB) + Core > > L#16 > >> + > >>>>>> PU > >>>>>>>> L#16 (P#16) > >>>>>>>> L2 L#17 (512KB) + L1d L#17 (64KB) + L1i L#17 (64KB) + Core > > L#17 > >> + > >>>>>> PU > >>>>>>>> L#17 (P#17)> >>>>> L2 L#18 (512KB) + L1d L#18 (64KB) + L1i > > L#18 (64KB) + Core L#18 > >> + > >>>>>> PU > >>>>>>>> L#18 (P#18) > >>>>>>>> L2 L#19 (512KB) + L1d L#19 (64KB) + L1i L#19 (64KB) + Core > > L#19 > >> + > >>>>>> PU > >>>>>>>> L#19 (P#19) > >>>>>>>> NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB) > >>>>>>>> L2 L#20 (512KB) + L1d L#20 (64KB) + L1i L#20 (64KB) + Core > > L#20 > >> + > >>>>>> PU > >>>>>>>> L#20 (P#20) > >>>>>>>> L2 L#21 (512KB) + L1d L#21 (64KB) + L1i L#21 (64KB) + Core > > L#21 > >> + > >>>>>> PU > >>>>>>>> L#21 (P#21) > >>>>>>>> L2 L#22 (512KB) + L1d L#22 (64KB) + L1i L#22 (64KB) + Core > > L#22 > >> + > >>>>>> PU > >>>>>>>> L#22 (P#22) > >>>>>>>> L2 L#23 (512KB) + L1d L#23 (64KB) + L1i L#23 (64KB) + Core > > L#23 > >> + > >>>>>> PU > >>>>>>>> L#23 (P#23) > >>>>>>>> Socket L#3 (32GB) > >>>>>>>> NUMANode L#6 (P#2 16GB) + L3 L#6 (5118KB) > >>>>>>>> L2 L#24 (512KB) + L1d L#24 (64KB) + L1i L#24 (64KB) + Core > > L#24 > >> + > >>>>>> PU > >>>>>>>> L#24 (P#24)>>>>> L2 L#25 (512KB) + L1d L#25 (64KB) + L1i L#25 > >> (64KB) + Core L#25 + > >>>>>> PU > >>>>>>>> L#25 (P#25) > >>>>>>>> L2 L#26 (512KB) + L1d L#26 (64KB) + L1i L#26 (64KB) + Core > > L#26 > >> + > >>>>>> PU > >>>>>>>> L#26 (P#26) > >>>>>>>> L2 L#27 (512KB) + L1d L#27 (64KB) + L1i L#27 (64KB) + Core > > L#27 > >> + > >>>>>> PU > >>>>>>>> L#27 (P#27) > >>>>>>>> NUMANode L#7 (P#3 16GB) + L3 L#7 (5118KB) > >>>>>>>> L2 L#28 (512KB) + L1d L#28 (64KB) + L1i L#28 (64KB) + Core > > L#28 > >> + > >>>>>> PU > >>>>>>>> L#28 (P#28) > >>>>>>>> L2 L#29 (512KB) + L1d L#29 (64KB) + L1i L#29 (64KB) + Core > > L#29 > >> + > >>>>>> PU > >>>>>>>> L#29 (P#29) > >>>>>>>> L2 L#30 (512KB) + L1d L#30 (64KB) + L1i L#30 (64KB) + Core > > L#30 > >> + > >>>>>> PU > >>>>>>>> L#30 (P#30) > >>>>>>>> L2 L#31 (512KB) + L1d L#31 (64KB) + L1i L#31 (64KB) + Core > > L#31 > >> + > >>>>>> PU > >>>>>>>> L#31 (P#31) > >>>>>>>> HostBridge L#0 > >>>>>>>> PCIBridge > >>>>>>>> PCI 14e4:1639 > >>>>>>>> Net L#0 "eth0" > >>>>>>>> PCI 14e4:1639 > >>>>>>>> Net L#1 "eth1" > >>>>>>>> PCIBridge > >>>>>>>> PCI 14e4:1639 > >>>>>>>> Net L#2 "eth2" > >>>>>>>> PCI 14e4:1639 > >>>>>>>> Net L#3 "eth3" > >>>>>>>> PCIBridge > >>>>>>>> PCIBridge > >>>>>>>> PCIBridge > >>>>>>>> PCI 1000:0072 > >>>>>>>> Block L#4 "sdb" > >>>>>>>> Block L#5 "sda" > >>>>>>>> PCI 1002:4390 > >>>>>>>> Block L#6 "sr0" > >>>>>>>> PCIBridge > >>>>>>>> PCI 102b:0532 > >>>>>>>> HostBridge L#7 > >>>>>>>> PCIBridge > >>>>>>>> PCI 15b3:6274 > >>>>>>>> Net L#7 "ib0" > >>>>>>>> OpenFabrics L#8 "mthca0" > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> users mailing list > >>>>>>>> us...@open-mpi.org > >>>>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>>> _______________________________________________ > >>>>>>> users mailing list > >>>>>>> us...@open-mpi.org>> > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>>> _______________________________________________ > >>>>>> users mailing list > >>>>>> us...@open-mpi.org > >>>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>>> _______________________________________________ > >>>>> users mailing list > >>>>> us...@open-mpi.org > >>>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > > users mailing list > > us...@open-mpi.org > > http://www.open-mpi.org/mailman/listinfo.cgi/users > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users
