On 2017-04-05 16:00, maviko.wag...@fau.de wrote:
Hello Dani and Thomas,
[ ...]
However i think i did not specify clear enough what my cluster looks
like and what i'm trying to achieve.
Compared to a regular HPC-Cluster my testing cluster consists of as
little as 5 nodes (each having the same "grandscale"-features, so no
IB-nodes etc., and only differ in hardware details like cpu, #RAM
etc., including some MCUs akin to a raspberry pi).
The purpose of this cluster is to investigate how smart distribution
of workloads based on predetermined performance and energy data can
benefit hpc-clusters that consist of heterogenous systems that differ
greatly regarding energy consumption and performance.
Its just a small research project.
IF your only intent is to do this on your 5 node test cluster, you
probably do not need Slurm. If you are looking to have something expand
to real clusters, then you really should be using something like features
and the scheduler. The scheduler is already taking into account what nodes
provide what resources, what resources on the nodes are currently available
for use, and handling the complex (and generally, at least I found it to
be much more complex than I initially and naively thought it should be when
first started thinking about it) task of scheduling the jobs.
My IB example was just an example. You could just as easily assign
Slurm "features" based on the feature set of the CPU, etc. E.g., if only
some nodes have CPUs with support AVX, label those nodes as "avx" and jobs
requiring AVX can restrict themselves to such nodes.
If you start specifying specific nodes in requests to the scheduler, you
are going to find yourself working against the scheduler, and that is not
likely to have a good outcome. You are better off telling Slurm which
nodes have which features (essentially a one-time configuration of the cluster)
and then have your code translate the requirements into a list of "features"
requested for the job under Slurm.
The only part that I see as a potential major problem is that, as I tried to
explain previously, the "features" requested by a job are REQUIREMENTS to
Slurm, not SUGGESTIONS. E.g., if a job can run either with or without AVX
support, but runs better with AVX support, requiring the "avx" feature will
force the job to wait for a node supporting AVX, even if all the AVX nodes
are in use and there are plenty of non-AVX nodes which are idle.
I am not aware of anything in Slurm which handles such as SUGGESTIONS, and
doing such I believe would greatly complicate an already complex algorithm.
I believe anything done would need to modify the actual C code for the
scheduler.
It probably is not _too_ bad to have a situation wherein when a job
"suggesting" avx
starts to run, it picks any avx nodes currently available to it first. But that
is likely to have only limited success. The closest thing currently in the
Slurm code base is the stuff for attempting to keep the nodes for a job on the
same leaf-switch (e.g. https://slurm.schedmd.com/topology.html), but I suspect
that would be quite complicated to handle across a number of "features".
