Hi Kor,
The nondeterministic nature of your problem is a bit worrying, and I don't have
any insight into that. However, there's an alternative way to set the bindings
as well. Would you mind trying the --hpx:use-process-mask option to see if you
get the expected bindings? By default HPX tries to reconstruct the bindings
based on various environment variables, but if you pass --hpx:use-process-mask
it will use the process mask that srun/mpi/others typically set, and only spawn
worker threads on cores in the process mask. Note that the default, even with
--hpx:use-process-mask, is still to only spawn one worker thread per core (not
per hyperthread), so if you want exactly the binding you ask for with mpirun
you should also add --hpx:threads=all.
Mikael
________________________________
From: hpx-users-boun...@stellar-group.org <hpx-users-boun...@stellar-group.org>
on behalf of Kor de Jong <k.dejo...@uu.nl>
Sent: Friday, May 21, 2021 4:25:29 PM
To: hpx-users@stellar-group.org
Subject: {Spam?} [hpx-users] Assign HPX localities to NUMA nodes, in order
Dear HPX-experts,
I am trying to spawn 8 hpx processes on a cluster node with 8 numa
nodes, containing 6 real cpu cores each. All seems well, but the output
of `--hpx:print-bind` confuses me.
I am using slurm (sbatch command) and openmpi (mpirun command in sbatch
script). The output of mpirun's `--display-map` makes complete sense.
All 8 process ranks get assigned to the 6 cores in the 8 numa nodes, in
order. Process rank 0 is on the first numa node, etc.
The output of `--hpx:print-bind` seems not in sync with this. There is a
correspondence between mpi ranks and hpx locality ids, but the mapping
of hpx localities to cpu cores is different now. For example, it seems
that locality 1 is not on the second numa node (as per mpirun's
`--display-map`), but on the 7-th (as per hpx's `--print-bind`). Also,
the output of `--print-bind` differs per invocation.
It is important for me that hpx localities are assigned to numa nodes in
order. Localities with similar IDs communicate more with each other than
with other localities.
I have attached the slurm script and outputs mentioned above. Does
somebody maybe have an idea what is going on and how to fix things? Does
hpx maybe re-assign the ranks upon initialization? If so, can I
influence this to make this ordering similar to the ordering of the numa
nodes?
BTW, I am pretty sure all this worked fine some time ago, when I was
still using an earlier version of HPX, another version of MPI, and
started HPX processes using srun instead of mpirun.
Thanks for any info!
Kor
_______________________________________________
hpx-users mailing list
hpx-users@stellar-group.org
https://mail.cct.lsu.edu/mailman/listinfo/hpx-users
