Hi, in context of the PRRTE Distributed Virtual Machine, is there a way
to tell the task mapper inside prun to not share a node across separate
prun jobs?
For example, inside a resource allocation from Cobalt/ALPS: 2 nodes with
64 cores each:
prte --daemonize
prun ... &
...
prun ... &
pterm
Scenario A:
$ prun --map-by ppr:64:node -n 64 ./mpitest &
$ prun --map-by ppr:64:node -n 64 ./mpitest &
MPI World size = 64 processes
Hello World from rank 0 running on nid03834 (hostname nid03834)!
...
Hello World from rank 63 running on nid03834 (hostname nid03834)!
MPI World size = 64 processes
Hello World from rank 0 running on nid03835 (hostname nid03835)!
...
Hello World from rank 63 running on nid03835 (hostname nid03835)!
Scenario B:
$ prun --map-by ppr:64:node -n 1 ./mpitest &
$ prun --map-by ppr:64:node -n 1 ./mpitest &
MPI World size = 1 processes
Hello World from rank 0 running on nid03834 (hostname nid03834)!
MPI World size = 1 processes
Hello World from rank 0 running on nid03834 (hostname nid03834)!
The question is: in Scneario B, how to tell prun that node nid03834
should not be used for the second prun job, because this node is already
(partially) occupied by a different prun instance job?
Scenario A implies that the DVM already tracks occupancy, so the
question is just how to tell the mapper to treat a free core on a free
node differently from a free core on a partially occupied node. The
--map-by :NOOVERSUBSCRIBE does not look like the answer since there's
no oversubscription of cores, right? Would need something like --map-by
:exclusive:node? If not supported, how hard would it be for me to patch?
Potential workarounds I can think of is to fill the unoccupied cores on
partially occupied nodes with dummy jobs with --host pointing to the
partially occupied nodes and a -n count matching the number of
unoccupied cores, but is this even doable? also requires dumping the
mapping from each prun which I am unable to achive with --map-by
:DISPLAY (works with mpirun but not with prun).
Or, run a Flux instance [1] instead of the PRRTE DVM on the resource
allocation, which seems similar but features a scheduler with a queue (a
feature proposed for the PRRTE DVM on the list earlier [1]). I am
guessing that Flux has the flexibility to this exclusive node mapping,
but not sure.
The DVM is proving to be very useful to deal with restrictions on
minimum nodecount per job on some HPC clusters, by batching many small
jobs into one job. A queue would be even more useful, but even without a
queue it is still useful for batching sets of jobs which are known to
fit on an allocation in parallel (i.e. without having to wait at all).
[1] https://flux-framework.readthedocs.io/en/latest/quickstart.html
[2] https://www.mail-archive.com/users@lists.open-mpi.org/msg30692.html
OpenMPI: commit 7a922c8774b184ecb3aa1cd06720390bd9200b50
Fri Nov 6 08:48:29 2020 -0800
PRRTE: commit 37dd45c4d9fe973df1000f1a1421c2718fd80050
Fri Nov 6 12:45:38 2020 -0600
Thank you.
