HelloIf I understand correctly, your allocations are going to swap as soon as you use more than half of your available local RAM? I don't see any reason for this unless some additional limit is preventing you from using more. Is there any chance your job was allocated with less memory (for instance SLURM uses Linux cgroups)?
IIRC, cat /proc/self/cgroup will tell you if you are in a "memory" cgroup and its path. Then you can go in something like /sys/fs/cgroup/memory/<path> and dig in files there.
For the record, hwloc takes care of ignoring NUMA nodes that are not available due to cgroups. However cgroup memory limits are more difficult because they don't apply to specific nodes but rather to the entire available memory in the machine, hence it's not easy to expose them in hwloc.
Brice Le 25/09/2022 à 12:40, Mike a écrit :
Dear list,I work on two AMD EPYC 7713 64-Core processors. There are 8 dies per processor containing 8 cores. The machine has 512 GB of main memory available and 112 GB of swap. The operating system is Debian.I use hwloc_set_area_membind() to optimize memory access. Now I came across behavior I cannot explain. When I do not use memory binding this behavior will not occur. I can only use part of the main memory depending on how many dies I use. An example:I use 8 processors for a calculation. I distribute them so that I fill 1 die à 8 cores completely. When I now allocate more than 32 GB (1/16 of main memory) of memory the first 32 GB will be stored in main memory but the remaining memory will be stored in swap.When I use 8 dies à 8 cores the first 256 GB (1/2 of main memory) will be written into the main memory until the rest gets stored into the swap and so on.Is there a way to circumvent this behavior? Thanks in advance. PS: This is the output of lstopo -.synthetic:Package:2 [NUMANode(memory=270369247232)] L3Cache:8(size=33554432) L2Cache:8(size=524288) L1dCache:1(size=32768) L1iCache:1(size=32768) Core:1 PU:2(indexes=2*128:1*2)Mike _______________________________________________ hwloc-users mailing list hwloc-users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/hwloc-users
OpenPGP_signature
Description: OpenPGP digital signature
_______________________________________________ hwloc-users mailing list hwloc-users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/hwloc-users
