Hi. I am experincing the following issues when using openmpi on my new AMD machine. **************************************************************************** * hwloc has encountered what looks like an error from the operating system. * * L3 (cpuset 0x000003f0) intersects with NUMANode (P#0 cpuset 0x0000003f) without inclusion! * Error occurred in topology.c line 940 * * Please report this error message to the hwloc user's mailing list, * along with the output from the hwloc-gather-topology script. ****************************************************************************
As suggested by the error message, here is the hwloc-gather-topology attached. If further information is needed please let me know. -- De bedste hilsner / Best regards, Mads Boye HPC Administrator | AAU ITS Phone: (+45) 9940 3453 | Email: m...@its.aau.dk | Web: http://www.its.aau.dk/ Aalborg University | Selma Lagerlöfs Vej 300 – 3.1.14 | Aalborg East
Machine (P#0 total=264135132KB DMIProductName="AS -1042G-TF"
DMIProductVersion=1234567890 DMIBoardVendor=Supermicro DMIBoardName=H8QG6
DMIBoardVersion=1234567890 DMIBoardAssetTag=1234567890
DMIChassisVendor=Supermicro DMIChassisType=1 DMIChassisVersion=1234567890
DMIChassisAssetTag=1234567890 DMIBIOSVendor="American Megatrends Inc."
DMIBIOSVersion="3.5 " DMIBIOSDate=12/16/2013 DMISysVendor=Supermicro
Backend=Linux hwlocVersion=1.10.1)
Socket L#0 (P#0 total=65976888KB CPUModel="AMD Opteron(tm) Processor 6344")
NUMANode L#0 (P#0 local=32947776KB total=32947776KB)
L3Cache L#0 (size=6144KB linesize=64 ways=48)
L2Cache L#0 (size=2048KB linesize=64 ways=16)
L1iCache L#0 (size=64KB linesize=64 ways=2)
L1dCache L#0 (size=16KB linesize=64 ways=4)
Core L#0 (P#0)
PU L#0 (P#0)
L1dCache L#1 (size=16KB linesize=64 ways=4)
Core L#1 (P#1)
PU L#1 (P#1)
L2Cache L#1 (size=2048KB linesize=64 ways=16)
L1iCache L#1 (size=64KB linesize=64 ways=2)
L1dCache L#2 (size=16KB linesize=64 ways=4)
Core L#2 (P#2)
PU L#2 (P#2)
L1dCache L#3 (size=16KB linesize=64 ways=4)
Core L#3 (P#3)
PU L#3 (P#3)
L2Cache L#2 (size=2048KB linesize=64 ways=16)
L1iCache L#2 (size=64KB linesize=64 ways=2)
L1dCache L#4 (size=16KB linesize=64 ways=4)
Core L#4 (P#4)
PU L#4 (P#4)
L1dCache L#5 (size=16KB linesize=64 ways=4)
Core L#5 (P#5)
PU L#5 (P#5)
NUMANode L#1 (P#1 local=33029112KB total=33029112KB)
L2Cache L#3 (size=2048KB linesize=64 ways=16)
L1iCache L#3 (size=64KB linesize=64 ways=2)
L1dCache L#6 (size=16KB linesize=64 ways=4)
Core L#6 (P#0)
PU L#6 (P#6)
L1dCache L#7 (size=16KB linesize=64 ways=4)
Core L#7 (P#1)
PU L#7 (P#7)
L2Cache L#4 (size=2048KB linesize=64 ways=16)
L1iCache L#4 (size=64KB linesize=64 ways=2)
L1dCache L#8 (size=16KB linesize=64 ways=4)
Core L#8 (P#2)
PU L#8 (P#8)
L1dCache L#9 (size=16KB linesize=64 ways=4)
Core L#9 (P#3)
PU L#9 (P#9)
L3Cache L#1 (size=6144KB linesize=64 ways=48)
L2Cache L#5 (size=2048KB linesize=64 ways=16)
L1iCache L#5 (size=64KB linesize=64 ways=2)
L1dCache L#10 (size=16KB linesize=64 ways=4)
Core L#10 (P#4)
PU L#10 (P#10)
L1dCache L#11 (size=16KB linesize=64 ways=4)
Core L#11 (P#5)
PU L#11 (P#11)
Socket L#1 (P#1 total=66058228KB CPUModel="AMD Opteron(tm) Processor 6344
")
NUMANode L#2 (P#2 local=33029116KB total=33029116KB)
L3Cache L#2 (size=6144KB linesize=64 ways=48)
L2Cache L#6 (size=2048KB linesize=64 ways=16)
L1iCache L#6 (size=64KB linesize=64 ways=2)
L1dCache L#12 (size=16KB linesize=64 ways=4)
Core L#12 (P#0)
PU L#12 (P#12)
L1dCache L#13 (size=16KB linesize=64 ways=4)
Core L#13 (P#1)
PU L#13 (P#13)
L2Cache L#7 (size=2048KB linesize=64 ways=16)
L1iCache L#7 (size=64KB linesize=64 ways=2)
L1dCache L#14 (size=16KB linesize=64 ways=4)
Core L#14 (P#2)
PU L#14 (P#14)
L1dCache L#15 (size=16KB linesize=64 ways=4)
Core L#15 (P#3)
PU L#15 (P#15)
L2Cache L#8 (size=2048KB linesize=64 ways=16)
L1iCache L#8 (size=64KB linesize=64 ways=2)
L1dCache L#16 (size=16KB linesize=64 ways=4)
Core L#16 (P#4)
PU L#16 (P#16)
L1dCache L#17 (size=16KB linesize=64 ways=4)
Core L#17 (P#5)
PU L#17 (P#17)
NUMANode L#3 (P#3 local=33029112KB total=33029112KB)
L2Cache L#9 (size=2048KB linesize=64 ways=16)
L1iCache L#9 (size=64KB linesize=64 ways=2)
L1dCache L#18 (size=16KB linesize=64 ways=4)
Core L#18 (P#0)
PU L#18 (P#18)
L1dCache L#19 (size=16KB linesize=64 ways=4)
Core L#19 (P#1)
PU L#19 (P#19)
L3Cache L#3 (size=6144KB linesize=64 ways=48)
L2Cache L#10 (size=2048KB linesize=64 ways=16)
L1iCache L#10 (size=64KB linesize=64 ways=2)
L1dCache L#20 (size=16KB linesize=64 ways=4)
Core L#20 (P#2)
PU L#20 (P#20)
L1dCache L#21 (size=16KB linesize=64 ways=4)
Core L#21 (P#3)
PU L#21 (P#21)
L2Cache L#11 (size=2048KB linesize=64 ways=16)
L1iCache L#11 (size=64KB linesize=64 ways=2)
L1dCache L#22 (size=16KB linesize=64 ways=4)
Core L#22 (P#4)
PU L#22 (P#22)
L1dCache L#23 (size=16KB linesize=64 ways=4)
Core L#23 (P#5)
PU L#23 (P#23)
Socket L#2 (P#2 total=66058228KB CPUModel="AMD Opteron(tm) Processor 6344
")
NUMANode L#4 (P#4 local=33029116KB total=33029116KB)
L3Cache L#4 (size=6144KB linesize=64 ways=48)
L2Cache L#12 (size=2048KB linesize=64 ways=16)
L1iCache L#12 (size=64KB linesize=64 ways=2)
L1dCache L#24 (size=16KB linesize=64 ways=4)
Core L#24 (P#0)
PU L#24 (P#24)
L1dCache L#25 (size=16KB linesize=64 ways=4)
Core L#25 (P#1)
PU L#25 (P#25)
L2Cache L#13 (size=2048KB linesize=64 ways=16)
L1iCache L#13 (size=64KB linesize=64 ways=2)
L1dCache L#26 (size=16KB linesize=64 ways=4)
Core L#26 (P#2)
PU L#26 (P#26)
L1dCache L#27 (size=16KB linesize=64 ways=4)
Core L#27 (P#3)
PU L#27 (P#27)
L2Cache L#14 (size=2048KB linesize=64 ways=16)
L1iCache L#14 (size=64KB linesize=64 ways=2)
L1dCache L#28 (size=16KB linesize=64 ways=4)
Core L#28 (P#4)
PU L#28 (P#28)
L1dCache L#29 (size=16KB linesize=64 ways=4)
Core L#29 (P#5)
PU L#29 (P#29)
NUMANode L#5 (P#5 local=33029112KB total=33029112KB)
L3Cache L#5 (size=6144KB linesize=64 ways=48)
L2Cache L#15 (size=2048KB linesize=64 ways=16)
L1iCache L#15 (size=64KB linesize=64 ways=2)
L1dCache L#30 (size=16KB linesize=64 ways=4)
Core L#30 (P#0)
PU L#30 (P#30)
L1dCache L#31 (size=16KB linesize=64 ways=4)
Core L#31 (P#1)
PU L#31 (P#31)
L2Cache L#16 (size=2048KB linesize=64 ways=16)
L1iCache L#16 (size=64KB linesize=64 ways=2)
L1dCache L#32 (size=16KB linesize=64 ways=4)
Core L#32 (P#2)
PU L#32 (P#32)
L1dCache L#33 (size=16KB linesize=64 ways=4)
Core L#33 (P#3)
PU L#33 (P#33)
L2Cache L#17 (size=2048KB linesize=64 ways=16)
L1iCache L#17 (size=64KB linesize=64 ways=2)
L1dCache L#34 (size=16KB linesize=64 ways=4)
Core L#34 (P#4)
PU L#34 (P#34)
L1dCache L#35 (size=16KB linesize=64 ways=4)
Core L#35 (P#5)
PU L#35 (P#35)
Socket L#3 (P#3 total=66041788KB CPUModel="AMD Opteron(tm) Processor 6344
")
NUMANode L#6 (P#6 local=33029116KB total=33029116KB)
L3Cache L#6 (size=6144KB linesize=64 ways=48)
L2Cache L#18 (size=2048KB linesize=64 ways=16)
L1iCache L#18 (size=64KB linesize=64 ways=2)
L1dCache L#36 (size=16KB linesize=64 ways=4)
Core L#36 (P#0)
PU L#36 (P#36)
L1dCache L#37 (size=16KB linesize=64 ways=4)
Core L#37 (P#1)
PU L#37 (P#37)
L2Cache L#19 (size=2048KB linesize=64 ways=16)
L1iCache L#19 (size=64KB linesize=64 ways=2)
L1dCache L#38 (size=16KB linesize=64 ways=4)
Core L#38 (P#2)
PU L#38 (P#38)
L1dCache L#39 (size=16KB linesize=64 ways=4)
Core L#39 (P#3)
PU L#39 (P#39)
L2Cache L#20 (size=2048KB linesize=64 ways=16)
L1iCache L#20 (size=64KB linesize=64 ways=2)
L1dCache L#40 (size=16KB linesize=64 ways=4)
Core L#40 (P#4)
PU L#40 (P#40)
L1dCache L#41 (size=16KB linesize=64 ways=4)
Core L#41 (P#5)
PU L#41 (P#41)
NUMANode L#7 (P#7 local=33012672KB total=33012672KB)
L2Cache L#21 (size=2048KB linesize=64 ways=16)
L1iCache L#21 (size=64KB linesize=64 ways=2)
L1dCache L#42 (size=16KB linesize=64 ways=4)
Core L#42 (P#0)
PU L#42 (P#42)
L1dCache L#43 (size=16KB linesize=64 ways=4)
Core L#43 (P#1)
PU L#43 (P#43)
L2Cache L#22 (size=2048KB linesize=64 ways=16)
L1iCache L#22 (size=64KB linesize=64 ways=2)
L1dCache L#44 (size=16KB linesize=64 ways=4)
Core L#44 (P#2)
PU L#44 (P#44)
L1dCache L#45 (size=16KB linesize=64 ways=4)
Core L#45 (P#3)
PU L#45 (P#45)
L3Cache L#7 (size=6144KB linesize=64 ways=48)
L2Cache L#23 (size=2048KB linesize=64 ways=16)
L1iCache L#23 (size=64KB linesize=64 ways=2)
L1dCache L#46 (size=16KB linesize=64 ways=4)
Core L#46 (P#4)
PU L#46 (P#46)
L1dCache L#47 (size=16KB linesize=64 ways=4)
Core L#47 (P#5)
PU L#47 (P#47)
Bridge Host->PCI L#0 (P#0 buses=0000:[00-02])
Bridge PCI->PCI (P#208 busid=0000:00:0d.0 id=1002:5a1e class=0604(PCI_B)
buses=0000:[02-02])
PCI 8086:10c9 (P#8192 busid=0000:02:00.0 class=0200(Ether))
Network L#0 (Address=00:25:90:5a:db:b2) "p29p1"
PCI 8086:10c9 (P#8193 busid=0000:02:00.1 class=0200(Ether))
Network L#1 (Address=00:25:90:5a:db:b3) "p29p2"
PCI 1002:4394 (P#272 busid=0000:00:11.0 class=0106(SATA))
Block L#2 "sda"
Block L#3 "sdb"
Block L#4 "sdc"
Bridge PCI->PCI (P#324 busid=0000:00:14.4 id=1002:4384 class=0604(PCI_B)
buses=0000:[01-01])
PCI 102b:0532 (P#4160 busid=0000:01:04.0 class=0300(VGA))
depth 0: 1 Machine (type #1)
depth 1: 4 Socket (type #3)
depth 2: 8 NUMANode (type #2)
depth 3: 8 L3Cache (type #4)
depth 4: 24 L2Cache (type #4)
depth 5: 24 L1iCache (type #4)
depth 6: 48 L1dCache (type #4)
depth 7: 48 Core (type #5)
depth 8: 48 PU (type #6)
Special depth -3: 3 Bridge (type #9)
Special depth -4: 4 PCI Device (type #10)
Special depth -5: 5 OS Device (type #11)
relative latency matrix between NUMANodes (depth 2) by logical indexes:
index 0 1 2 3 4 5 6 7
0 1.000 1.600 1.600 2.200 1.600 2.200 1.600 2.200
1 1.600 1.000 2.200 1.600 2.200 1.600 2.200 1.600
2 1.600 2.200 1.000 1.600 1.600 2.200 1.600 2.200
3 2.200 1.600 1.600 1.000 2.200 1.600 2.200 1.600
4 1.600 2.200 1.600 2.200 1.000 1.600 1.600 2.200
5 2.200 1.600 2.200 1.600 1.600 1.000 2.200 1.600
6 1.600 2.200 1.600 2.200 1.600 2.200 1.000 1.600
7 2.200 1.600 2.200 1.600 2.200 1.600 1.600 1.000
Topology not from this system
thor.tar.bz2
Description: thor.tar.bz2
