Hello Ben
It will be back, at least for the majority of platforms (those without
heterogeneous memory).
See https://github.com/open-mpi/ompi/issues/8170 and
https://github.com/openpmix/prrte/pull/1141
Brice
Le 11/11/2021 à 05:33, Ben Menadue via devel a écrit :
> Hi,
>
> Quick question: what's the equivalent of "--map-by numa" for the new
> PRRTE-based runtime for v5.0? I can see "package" and "l3cache" in the
> help, which are close, but don't quite match "numa" for our system.
>
> In more detail...
>
> We have dual-socket CLX- and SKL-based nodes with sub-NUMA clustering
> enabled. This shows up in the OS as two packages, each with 1 L3 cache
> domain and 2 NUMA domains. Even worse, each compute node effectively
> has its own unique mapping of the cores of each socket between the NUMA
> domains.
>
> A common way of running for our users is with 1 MPI process per NUMA
> domain and then some form of threading within the cores associated with
> that domain. This effectively gives each MPI process its own memory
> controller and DIMMs.
>
> Using "--map-by numa" worked really well for this, since it took care
> of the unique core numbering of each node. The only way I can think of
> to set up something equivalent without that would be manually
> enumerating the nodes in each job and building a rank file.
>
> I've include an example topology below.
>
> Or do you think this is better as a GitHub issue?
>
> Thanks,
> Ben
>
> [bjm900@gadi-cpu-clx-0143 build]$ lstopo
> Machine (189GB total)
> Package L#0 + L3 L#0 (36MB)
> Group0 L#0
> NUMANode L#0 (P#0 47GB)
> L2 L#0 (1024KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU
> L#0 (P#0)
> L2 L#1 (1024KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU
> L#1 (P#1)
> L2 L#2 (1024KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU
> L#2 (P#2)
> L2 L#3 (1024KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU
> L#3 (P#3)
> L2 L#4 (1024KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU
> L#4 (P#7)
> L2 L#5 (1024KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU
> L#5 (P#8)
> L2 L#6 (1024KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU
> L#6 (P#9)
> L2 L#7 (1024KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU
> L#7 (P#13)
> L2 L#8 (1024KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU
> L#8 (P#14)
> L2 L#9 (1024KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU
> L#9 (P#15)
> L2 L#10 (1024KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10
> + PU L#10 (P#19)
> L2 L#11 (1024KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11
> + PU L#11 (P#20)
> HostBridge
> PCI 00:11.5 (SATA)
> PCI 00:17.0 (SATA)
> Block(Disk) "sda"
> PCIBridge
> PCIBridge
> PCI 02:00.0 (VGA)
> HostBridge
> PCIBridge
> PCIBridge
> PCIBridge
> PCI 08:00.2 (Ethernet)
> Net "eno1"
> Group0 L#1
> NUMANode L#1 (P#1 47GB)
> L2 L#12 (1024KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12
> + PU L#12 (P#4)
> L2 L#13 (1024KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13
> + PU L#13 (P#5)
> L2 L#14 (1024KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14
> + PU L#14 (P#6)
> L2 L#15 (1024KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15
> + PU L#15 (P#10)
> L2 L#16 (1024KB) + L1d L#16 (32KB) + L1i L#16 (32KB) + Core L#16
> + PU L#16 (P#11)
> L2 L#17 (1024KB) + L1d L#17 (32KB) + L1i L#17 (32KB) + Core L#17
> + PU L#17 (P#12)
> L2 L#18 (1024KB) + L1d L#18 (32KB) + L1i L#18 (32KB) + Core L#18
> + PU L#18 (P#16)
> L2 L#19 (1024KB) + L1d L#19 (32KB) + L1i L#19 (32KB) + Core L#19
> + PU L#19 (P#17)
> L2 L#20 (1024KB) + L1d L#20 (32KB) + L1i L#20 (32KB) + Core L#20
> + PU L#20 (P#18)
> L2 L#21 (1024KB) + L1d L#21 (32KB) + L1i L#21 (32KB) + Core L#21
> + PU L#21 (P#21)
> L2 L#22 (1024KB) + L1d L#22 (32KB) + L1i L#22 (32KB) + Core L#22
> + PU L#22 (P#22)
> L2 L#23 (1024KB) + L1d L#23 (32KB) + L1i L#23 (32KB) + Core L#23
> + PU L#23 (P#23)
> HostBridge
> PCIBridge
> PCI 58:00.0 (InfiniBand)
> Net "ib0"
> OpenFabrics "mlx5_0"
> Package L#1 + L3 L#1 (36MB)
> Group0 L#2
> NUMANode L#2 (P#2 47GB)
> L2 L#24 (1024KB) + L1d L#24 (32KB) + L1i L#24 (32KB) + Core L#24
> + PU L#24 (P#24)
> L2 L#25 (1024KB) + L1d L#25 (32KB) + L1i L#25 (32KB) + Core L#25
> + PU L#25 (P#25)
> L2 L#26 (1024KB) + L1d L#26 (32KB) + L1i L#26 (32KB) + Core L#26
> + PU L#26 (P#26)
> L2 L#27 (1024KB) + L1d L#27 (32KB) + L1i L#27 (32KB) + Core L#27
> + PU L#27 (P#27)
> L2 L#28 (1024KB) + L1d L#28 (32KB) + L1i L#28 (32KB) + Core L#28
> + PU L#28 (P#30)
> L2 L#29 (1024KB) + L1d L#29 (32KB) + L1i L#29 (32KB) + Core L#29
> + PU L#29 (P#31)
> L2 L#30 (1024KB) + L1d L#30 (32KB) + L1i L#30 (32KB) + Core L#30
> + PU L#30 (P#35)
> L2 L#31 (1024KB) + L1d L#31 (32KB) + L1i L#31 (32KB) + Core L#31
> + PU L#31 (P#36)
> L2 L#32 (1024KB) + L1d L#32 (32KB) + L1i L#32 (32KB) + Core L#32
> + PU L#32 (P#37)
> L2 L#33 (1024KB) + L1d L#33 (32KB) + L1i L#33 (32KB) + Core L#33
> + PU L#33 (P#42)
> L2 L#34 (1024KB) + L1d L#34 (32KB) + L1i L#34 (32KB) + Core L#34
> + PU L#34 (P#43)
> L2 L#35 (1024KB) + L1d L#35 (32KB) + L1i L#35 (32KB) + Core L#35
> + PU L#35 (P#44)
> Group0 L#3
> NUMANode L#3 (P#3 47GB)
> L2 L#36 (1024KB) + L1d L#36 (32KB) + L1i L#36 (32KB) + Core L#36
> + PU L#36 (P#28)
> L2 L#37 (1024KB) + L1d L#37 (32KB) + L1i L#37 (32KB) + Core L#37
> + PU L#37 (P#29)
> L2 L#38 (1024KB) + L1d L#38 (32KB) + L1i L#38 (32KB) + Core L#38
> + PU L#38 (P#32)
> L2 L#39 (1024KB) + L1d L#39 (32KB) + L1i L#39 (32KB) + Core L#39
> + PU L#39 (P#33)
> L2 L#40 (1024KB) + L1d L#40 (32KB) + L1i L#40 (32KB) + Core L#40
> + PU L#40 (P#34)
> L2 L#41 (1024KB) + L1d L#41 (32KB) + L1i L#41 (32KB) + Core L#41
> + PU L#41 (P#38)
> L2 L#42 (1024KB) + L1d L#42 (32KB) + L1i L#42 (32KB) + Core L#42
> + PU L#42 (P#39)
> L2 L#43 (1024KB) + L1d L#43 (32KB) + L1i L#43 (32KB) + Core L#43
> + PU L#43 (P#40)
> L2 L#44 (1024KB) + L1d L#44 (32KB) + L1i L#44 (32KB) + Core L#44
> + PU L#44 (P#41)
> L2 L#45 (1024KB) + L1d L#45 (32KB) + L1i L#45 (32KB) + Core L#45
> + PU L#45 (P#45)
> L2 L#46 (1024KB) + L1d L#46 (32KB) + L1i L#46 (32KB) + Core L#46
> + PU L#46 (P#46)
> L2 L#47 (1024KB) + L1d L#47 (32KB) + L1i L#47 (32KB) + Core L#47
> + PU L#47 (P#47)
>
>
Hi Brice,
Great, thanks for that!
I think I need to brush up on my searching skills, not sure why I didn't find that issue. Sorry for the noise.
Cheers,
Ben
On 11 Nov. 2021 19:05, Brice Goglin <brice.gog...@inria.fr> wrote:
- [OMPI devel] v5.0 equivalent of --map-by numa Ben Menadue via devel
- Re: [OMPI devel] v5.0 equivalent of --map-by n... Brice Goglin via devel
- Re: [OMPI devel] v5.0 equivalent of --map-... Ben Menadue via devel
