Hello: A colleague and I have been running a large F90 application that does an enormous number of mpi_bcast calls during execution. I deny any responsibility for the design of the code and why it needs these calls, but it is what we have inherited and have to work with.
Recently we ported the code to an 8 node, 6 processor/node NUMA system (lstopo output attached) running Debian linux 6.0.3 with Open_MPI 1.5.3, and began having trouble with mysterious 'hangs' in the program inside the mpi_bcast calls. The hangs were always in the same calls, but not necessarily at the same time during integration. We originally didn't have NUMA support, so reinstalled with libnuma support added, but the problem persisted. Finally, just as a wild guess, we inserted 'mpi_barrier' calls just before the 'mpi_bcast' calls, and the program now runs without problems. I believe conventional wisdom is that properly formulated MPI programs should run correctly without barriers, so do you have any thoughts on why we found it necessary to add them? The code has run correctly on other architectures, i.g. Crayxe6, so I don't think there is a bug anywhere. My only explanation is that some internal resource gets exhausted because of the large number of 'mpi_bcast' calls in rapid succession, and the barrier calls force synchronization which allows the resource to be restored. Does this make sense? I'd appreciate any comments and advice you can provide. I have attached compressed copies of config.log and ompi_info for the system. The program is built with ifort 12.0 and typically runs with mpirun -np 36 -bycore -bind-to-core program.exe We have run both interactively and with PBS, but that doesn't seem to make any difference in program behavior. T. Rosmond
Machine (128GB)
Socket L#0 (32GB)
NUMANode L#0 (P#0 16GB) + L3 L#0 (5118KB)
L2 L#0 (512KB) + L1 L#0 (64KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (512KB) + L1 L#1 (64KB) + Core L#1 + PU L#1 (P#1)
L2 L#2 (512KB) + L1 L#2 (64KB) + Core L#2 + PU L#2 (P#2)
L2 L#3 (512KB) + L1 L#3 (64KB) + Core L#3 + PU L#3 (P#3)
L2 L#4 (512KB) + L1 L#4 (64KB) + Core L#4 + PU L#4 (P#4)
L2 L#5 (512KB) + L1 L#5 (64KB) + Core L#5 + PU L#5 (P#5)
NUMANode L#1 (P#1 16GB) + L3 L#1 (5118KB)
L2 L#6 (512KB) + L1 L#6 (64KB) + Core L#6 + PU L#6 (P#6)
L2 L#7 (512KB) + L1 L#7 (64KB) + Core L#7 + PU L#7 (P#7)
L2 L#8 (512KB) + L1 L#8 (64KB) + Core L#8 + PU L#8 (P#8)
L2 L#9 (512KB) + L1 L#9 (64KB) + Core L#9 + PU L#9 (P#9)
L2 L#10 (512KB) + L1 L#10 (64KB) + Core L#10 + PU L#10 (P#10)
L2 L#11 (512KB) + L1 L#11 (64KB) + Core L#11 + PU L#11 (P#11)
Socket L#1 (32GB)
NUMANode L#2 (P#2 16GB) + L3 L#2 (5118KB)
L2 L#12 (512KB) + L1 L#12 (64KB) + Core L#12 + PU L#12 (P#12)
L2 L#13 (512KB) + L1 L#13 (64KB) + Core L#13 + PU L#13 (P#13)
L2 L#14 (512KB) + L1 L#14 (64KB) + Core L#14 + PU L#14 (P#14)
L2 L#15 (512KB) + L1 L#15 (64KB) + Core L#15 + PU L#15 (P#15)
L2 L#16 (512KB) + L1 L#16 (64KB) + Core L#16 + PU L#16 (P#16)
L2 L#17 (512KB) + L1 L#17 (64KB) + Core L#17 + PU L#17 (P#17)
NUMANode L#3 (P#3 16GB) + L3 L#3 (5118KB)
L2 L#18 (512KB) + L1 L#18 (64KB) + Core L#18 + PU L#18 (P#18)
L2 L#19 (512KB) + L1 L#19 (64KB) + Core L#19 + PU L#19 (P#19)
L2 L#20 (512KB) + L1 L#20 (64KB) + Core L#20 + PU L#20 (P#20)
L2 L#21 (512KB) + L1 L#21 (64KB) + Core L#21 + PU L#21 (P#21)
L2 L#22 (512KB) + L1 L#22 (64KB) + Core L#22 + PU L#22 (P#22)
L2 L#23 (512KB) + L1 L#23 (64KB) + Core L#23 + PU L#23 (P#23)
Socket L#2 (32GB)
NUMANode L#4 (P#4 16GB) + L3 L#4 (5118KB)
L2 L#24 (512KB) + L1 L#24 (64KB) + Core L#24 + PU L#24 (P#24)
L2 L#25 (512KB) + L1 L#25 (64KB) + Core L#25 + PU L#25 (P#25)
L2 L#26 (512KB) + L1 L#26 (64KB) + Core L#26 + PU L#26 (P#26)
L2 L#27 (512KB) + L1 L#27 (64KB) + Core L#27 + PU L#27 (P#27)
L2 L#28 (512KB) + L1 L#28 (64KB) + Core L#28 + PU L#28 (P#28)
L2 L#29 (512KB) + L1 L#29 (64KB) + Core L#29 + PU L#29 (P#29)
NUMANode L#5 (P#5 16GB) + L3 L#5 (5118KB)
L2 L#30 (512KB) + L1 L#30 (64KB) + Core L#30 + PU L#30 (P#30)
L2 L#31 (512KB) + L1 L#31 (64KB) + Core L#31 + PU L#31 (P#31)
L2 L#32 (512KB) + L1 L#32 (64KB) + Core L#32 + PU L#32 (P#32)
L2 L#33 (512KB) + L1 L#33 (64KB) + Core L#33 + PU L#33 (P#33)
L2 L#34 (512KB) + L1 L#34 (64KB) + Core L#34 + PU L#34 (P#34)
L2 L#35 (512KB) + L1 L#35 (64KB) + Core L#35 + PU L#35 (P#35)
Socket L#3 (32GB)
NUMANode L#6 (P#6 16GB) + L3 L#6 (5118KB)
L2 L#36 (512KB) + L1 L#36 (64KB) + Core L#36 + PU L#36 (P#36)
L2 L#37 (512KB) + L1 L#37 (64KB) + Core L#37 + PU L#37 (P#37)
L2 L#38 (512KB) + L1 L#38 (64KB) + Core L#38 + PU L#38 (P#38)
L2 L#39 (512KB) + L1 L#39 (64KB) + Core L#39 + PU L#39 (P#39)
L2 L#40 (512KB) + L1 L#40 (64KB) + Core L#40 + PU L#40 (P#40)
L2 L#41 (512KB) + L1 L#41 (64KB) + Core L#41 + PU L#41 (P#41)
NUMANode L#7 (P#7 16GB) + L3 L#7 (5118KB)
L2 L#42 (512KB) + L1 L#42 (64KB) + Core L#42 + PU L#42 (P#42)
L2 L#43 (512KB) + L1 L#43 (64KB) + Core L#43 + PU L#43 (P#43)
L2 L#44 (512KB) + L1 L#44 (64KB) + Core L#44 + PU L#44 (P#44)
L2 L#45 (512KB) + L1 L#45 (64KB) + Core L#45 + PU L#45 (P#45)
L2 L#46 (512KB) + L1 L#46 (64KB) + Core L#46 + PU L#46 (P#46)
L2 L#47 (512KB) + L1 L#47 (64KB) + Core L#47 + PU L#47 (P#47)
config.log.bz2
Description: application/bzip
ompi_info.bz2
Description: application/bzip
