[slurm-dev] RE: fairshare - memory resource allocation
Bill, As I understand the dilemma you presented, you want to maximize the utilization of node resources when running with Slurm configured for SelectType=select/cons_res. To do this, you would like to nudge users into requesting only the amount of memory they will need for their jobs. The nudge would be in the form of decreased fair-share priority for users' jobs that request only one core but lots of memory. I don't know of a way for Slurm to do this as it exists. I can only offer alternatives that have their pros and cons. One alternative would be to add memory usage support to the multifactor priority plugin. This would be a substantial undertaking as it touches code not just in multifactor/priority_multifactor.c but also in structures that are defined in common/assoc_mgr.h as well as sshare itself. A second less invasive option would be to redefine the multifactor/priority_multifactor.c's raw_usage to make it a configurable blend of cpu and memory usage. These changes could be more localized to the multifactor/priority_multifactor.c module. However you would have a harder time justifying a user's sshare report because the usage numbers would no longer track jobs' historical cpu usage. You response to a user who asked you to justify their sshare usage report would be, "trust me, it's right". A third alternative (as I'm sure you know) is to give up on perfectly packed nodes and make every 4G of memory requested cost 1 cpu of allocation. Perhaps there are other options, but those are the ones that immediately come to mind. Don Lipari > -Original Message- > From: Bill Wichser [mailto:b...@princeton.edu] > Sent: Friday, July 25, 2014 6:14 AM > To: slurm-dev > Subject: [slurm-dev] fairshare - memory resource allocation > > > I'd like to revisit this... > > > After struggling with memory allocations in some flavor of PBS for over > 20 years, it was certainly a wonderful thing to have cgroup support > right out of the box with Slurm. No longer do we have a shared node's > jobs eating all the memory and killing everything running there. But we > have found that there is a cost to this and that is a failure to > adequately feed back this information to the fairshare mechanism. > > In looking at running jobs over the past 4 months, we found a spot where > we could reduce the DefMemPerCPU allocation in slurm.conf to a value > about 1G less than the actual G/core available. This meant that we had > to notify the users close to this max value so that they could adjust > their scripts. We also notified users that if this value was too high > that they'd do best to reduce that limit to exactly what they require. > This has proven much less successful. > > So our default is 3G/core with an actual node having 4G/core available. > This allows some bigger memory jobs and some smaller memory jobs to > make use of the node as there are available cores but not enough memory > for the default case. > > Now that is good. It allows higher utilization of nodes, all the while > protecting the memory of each other's processes. But the problem of > fairshare comes about pretty quickly when there are jobs requiring say > half the node's memory. This is mostly serial jobs requesting a single > core. So this leaves about 11 cores with only about 2G/core left. > Worse, when it comes to fairshare calculations it appears that these > jobs are only using a single core when in fact they are using half a > node. You can see where this is causing issues. > > Fairshare has a number of other issues as well, which I will send under > a different email. > > Now maybe this is just a matter of constant monitoring of user jobs and > proactively going after those users having small memory per core > requirements. We have attempted this in the past and have found that > the first job they run which crashes due to insufficient memory results > in all scripts being increased and so the process is never ending. > > Another solution is to simply trust the users and just keep reminding > them about allocations. They are usually a smart bunch and are quite > creative when it comes to getting jobs to run! So maybe I am concerned > over nothing at all and things will just work out. > > Bill
[slurm-dev] RE: fairshare - memory resource allocation
Bill and Don, We have wondered about this ourselves. I just came up with this idea and haven't thought it through completely, but option two seems like the easiest. For example, you could modify lines like https://github.com/SchedMD/slurm/blob/8a1e1384bacf690aed4c1f384da77a0cd978a63f/src/plugins/priority/multifactor/priority_multifactor.c#L952 to have a MAX() of a few different types. I seem to recall seeing this on the list or in a bug report somewhere already, but you could have different charge rates for memory or GPUs compared to a CPU, maybe on a per partition basis. You could give each of them a charge rate like: PartitionName=p1 ChargePerCPU=1.0 ChargePerGB=0.5 ChargePerGPU=2.0 .. So the line I referenced would be something like the following (except using real code and real struct members, etc): real_decay = run_decay * MAX(CPUs*ChargePerCPU, TotalJobMemory*ChargePerGB, GPUs*ChargePerGPU); In this case, each CPU is 1.0 but each GB of RAM is 0.5. Assuming no GPUs used, if the user requests 1 CPU and 2 GB of RAM the resulting usage is 1.0. But if they use 4 GB of RAM and 1 CPU, it is 2.0 just like they had been using 2 CPUs. Essentially you define every 2 GB of RAM to be equal to 1 CPU, so raw_usage could be redefined to deal with "cpu equivalents". It might be harder to explain to users but I don't think it would be too bad. Ryan On 07/25/2014 10:05 AM, Lipari, Don wrote: Bill, As I understand the dilemma you presented, you want to maximize the utilization of node resources when running with Slurm configured for SelectType=select/cons_res. To do this, you would like to nudge users into requesting only the amount of memory they will need for their jobs. The nudge would be in the form of decreased fair-share priority for users' jobs that request only one core but lots of memory. I don't know of a way for Slurm to do this as it exists. I can only offer alternatives that have their pros and cons. One alternative would be to add memory usage support to the multifactor priority plugin. This would be a substantial undertaking as it touches code not just in multifactor/priority_multifactor.c but also in structures that are defined in common/assoc_mgr.h as well as sshare itself. A second less invasive option would be to redefine the multifactor/priority_multifactor.c's raw_usage to make it a configurable blend of cpu and memory usage. These changes could be more localized to the multifactor/priority_multifactor.c module. However you would have a harder time justifying a user's sshare report because the usage numbers would no longer track jobs' historical cpu usage. You response to a user who asked you to justify their sshare usage report would be, "trust me, it's right". A third alternative (as I'm sure you know) is to give up on perfectly packed nodes and make every 4G of memory requested cost 1 cpu of allocation. Perhaps there are other options, but those are the ones that immediately come to mind. Don Lipari -Original Message- From: Bill Wichser [mailto:b...@princeton.edu] Sent: Friday, July 25, 2014 6:14 AM To: slurm-dev Subject: [slurm-dev] fairshare - memory resource allocation I'd like to revisit this... After struggling with memory allocations in some flavor of PBS for over 20 years, it was certainly a wonderful thing to have cgroup support right out of the box with Slurm. No longer do we have a shared node's jobs eating all the memory and killing everything running there. But we have found that there is a cost to this and that is a failure to adequately feed back this information to the fairshare mechanism. In looking at running jobs over the past 4 months, we found a spot where we could reduce the DefMemPerCPU allocation in slurm.conf to a value about 1G less than the actual G/core available. This meant that we had to notify the users close to this max value so that they could adjust their scripts. We also notified users that if this value was too high that they'd do best to reduce that limit to exactly what they require. This has proven much less successful. So our default is 3G/core with an actual node having 4G/core available. This allows some bigger memory jobs and some smaller memory jobs to make use of the node as there are available cores but not enough memory for the default case. Now that is good. It allows higher utilization of nodes, all the while protecting the memory of each other's processes. But the problem of fairshare comes about pretty quickly when there are jobs requiring say half the node's memory. This is mostly serial jobs requesting a single core. So this leaves about 11 cores with only about 2G/core left. Worse, when it comes to fairshare calculations it appears that these jobs are only using a single core when in fact they are using half a node. You can see where this is causing issues. Fairshare has a number of other issues as well, which I will send under a different email.
[slurm-dev] RE: fairshare - memory resource allocation
Thank you Ryan. Not sure how we will proceed here. Bill On 7/25/2014 12:30 PM, Ryan Cox wrote: Bill and Don, We have wondered about this ourselves. I just came up with this idea and haven't thought it through completely, but option two seems like the easiest. For example, you could modify lines like https://github.com/SchedMD/slurm/blob/8a1e1384bacf690aed4c1f384da77a0cd978a63f/src/plugins/priority/multifactor/priority_multifactor.c#L952 to have a MAX() of a few different types. I seem to recall seeing this on the list or in a bug report somewhere already, but you could have different charge rates for memory or GPUs compared to a CPU, maybe on a per partition basis. You could give each of them a charge rate like: PartitionName=p1 ChargePerCPU=1.0 ChargePerGB=0.5 ChargePerGPU=2.0 .. So the line I referenced would be something like the following (except using real code and real struct members, etc): real_decay = run_decay * MAX(CPUs*ChargePerCPU, TotalJobMemory*ChargePerGB, GPUs*ChargePerGPU); In this case, each CPU is 1.0 but each GB of RAM is 0.5. Assuming no GPUs used, if the user requests 1 CPU and 2 GB of RAM the resulting usage is 1.0. But if they use 4 GB of RAM and 1 CPU, it is 2.0 just like they had been using 2 CPUs. Essentially you define every 2 GB of RAM to be equal to 1 CPU, so raw_usage could be redefined to deal with "cpu equivalents". It might be harder to explain to users but I don't think it would be too bad. Ryan On 07/25/2014 10:05 AM, Lipari, Don wrote: Bill, As I understand the dilemma you presented, you want to maximize the utilization of node resources when running with Slurm configured for SelectType=select/cons_res. To do this, you would like to nudge users into requesting only the amount of memory they will need for their jobs. The nudge would be in the form of decreased fair-share priority for users' jobs that request only one core but lots of memory. I don't know of a way for Slurm to do this as it exists. I can only offer alternatives that have their pros and cons. One alternative would be to add memory usage support to the multifactor priority plugin. This would be a substantial undertaking as it touches code not just in multifactor/priority_multifactor.c but also in structures that are defined in common/assoc_mgr.h as well as sshare itself. A second less invasive option would be to redefine the multifactor/priority_multifactor.c's raw_usage to make it a configurable blend of cpu and memory usage. These changes could be more localized to the multifactor/priority_multifactor.c module. However you would have a harder time justifying a user's sshare report because the usage numbers would no longer track jobs' historical cpu usage. You response to a user who asked you to justify their sshare usage report would be, "trust me, it's right". A third alternative (as I'm sure you know) is to give up on perfectly packed nodes and make every 4G of memory requested cost 1 cpu of allocation. Perhaps there are other options, but those are the ones that immediately come to mind. Don Lipari -Original Message- From: Bill Wichser [mailto:b...@princeton.edu] Sent: Friday, July 25, 2014 6:14 AM To: slurm-dev Subject: [slurm-dev] fairshare - memory resource allocation I'd like to revisit this... After struggling with memory allocations in some flavor of PBS for over 20 years, it was certainly a wonderful thing to have cgroup support right out of the box with Slurm. No longer do we have a shared node's jobs eating all the memory and killing everything running there. But we have found that there is a cost to this and that is a failure to adequately feed back this information to the fairshare mechanism. In looking at running jobs over the past 4 months, we found a spot where we could reduce the DefMemPerCPU allocation in slurm.conf to a value about 1G less than the actual G/core available. This meant that we had to notify the users close to this max value so that they could adjust their scripts. We also notified users that if this value was too high that they'd do best to reduce that limit to exactly what they require. This has proven much less successful. So our default is 3G/core with an actual node having 4G/core available. This allows some bigger memory jobs and some smaller memory jobs to make use of the node as there are available cores but not enough memory for the default case. Now that is good. It allows higher utilization of nodes, all the while protecting the memory of each other's processes. But the problem of fairshare comes about pretty quickly when there are jobs requiring say half the node's memory. This is mostly serial jobs requesting a single core. So this leaves about 11 cores with only about 2G/core left. Worse, when it comes to fairshare calculations it appears that these jobs are only using a single core when in fact they are using half a node. You can see where th
[slurm-dev] RE: fairshare - memory resource allocation
Hi, As a variation on the second option you propose, take a look at the concept of Dominant Resource Fairness [1], which is an algorithm for achieving multi-resource (e.g. cpu's, memory, disk/net BW, ...) fairness. By using "dominant share"-secs instead of cpu-secs in the current accounting code the changes would similarly be limited in scope. [1] http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf https://www.usenix.org/legacy/events/nsdi11/tech/slides/ghodsi.pdf -- Janne Blomqvist From: Lipari, Don [lipa...@llnl.gov] Sent: Friday, July 25, 2014 19:04 To: slurm-dev Subject: [slurm-dev] RE: fairshare - memory resource allocation Bill, As I understand the dilemma you presented, you want to maximize the utilization of node resources when running with Slurm configured for SelectType=select/cons_res. To do this, you would like to nudge users into requesting only the amount of memory they will need for their jobs. The nudge would be in the form of decreased fair-share priority for users' jobs that request only one core but lots of memory. I don't know of a way for Slurm to do this as it exists. I can only offer alternatives that have their pros and cons. One alternative would be to add memory usage support to the multifactor priority plugin. This would be a substantial undertaking as it touches code not just in multifactor/priority_multifactor.c but also in structures that are defined in common/assoc_mgr.h as well as sshare itself. A second less invasive option would be to redefine the multifactor/priority_multifactor.c's raw_usage to make it a configurable blend of cpu and memory usage. These changes could be more localized to the multifactor/priority_multifactor.c module. However you would have a harder time justifying a user's sshare report because the usage numbers would no longer track jobs' historical cpu usage. You response to a user who asked you to justify their sshare usage report would be, "trust me, it's right". A third alternative (as I'm sure you know) is to give up on perfectly packed nodes and make every 4G of memory requested cost 1 cpu of allocation. Perhaps there are other options, but those are the ones that immediately come to mind. Don Lipari > -Original Message- > From: Bill Wichser [mailto:b...@princeton.edu] > Sent: Friday, July 25, 2014 6:14 AM > To: slurm-dev > Subject: [slurm-dev] fairshare - memory resource allocation > > > I'd like to revisit this... > > > After struggling with memory allocations in some flavor of PBS for over > 20 years, it was certainly a wonderful thing to have cgroup support > right out of the box with Slurm. No longer do we have a shared node's > jobs eating all the memory and killing everything running there. But we > have found that there is a cost to this and that is a failure to > adequately feed back this information to the fairshare mechanism. > > In looking at running jobs over the past 4 months, we found a spot where > we could reduce the DefMemPerCPU allocation in slurm.conf to a value > about 1G less than the actual G/core available. This meant that we had > to notify the users close to this max value so that they could adjust > their scripts. We also notified users that if this value was too high > that they'd do best to reduce that limit to exactly what they require. > This has proven much less successful. > > So our default is 3G/core with an actual node having 4G/core available. > This allows some bigger memory jobs and some smaller memory jobs to > make use of the node as there are available cores but not enough memory > for the default case. > > Now that is good. It allows higher utilization of nodes, all the while > protecting the memory of each other's processes. But the problem of > fairshare comes about pretty quickly when there are jobs requiring say > half the node's memory. This is mostly serial jobs requesting a single > core. So this leaves about 11 cores with only about 2G/core left. > Worse, when it comes to fairshare calculations it appears that these > jobs are only using a single core when in fact they are using half a > node. You can see where this is causing issues. > > Fairshare has a number of other issues as well, which I will send under > a different email. > > Now maybe this is just a matter of constant monitoring of user jobs and > proactively going after those users having small memory per core > requirements. We have attempted this in the past and have found that > the first job they run which crashes due to insufficient memory results > in all scripts being increased and so the process is never ending. > > Another solution is to simply trust the users and just keep reminding > them about allocations. They are usually a smart bunch and are quite > creative when it comes to getting jobs to run! So maybe I am concerned > over nothing at all and things will just work out. > > Bill
[slurm-dev] RE: fairshare - memory resource allocation
I'm interested in hearing opinions on this, if any. Basically, I think
there is an easy solution to the problem of a user using few CPUs but a
lot of memory and that not being reflected well in the CPU-centric usage
stats.
Below is my proposal. There are likely some other good approaches out
there too (Don and Janne presented some) so feel free to tell me that
you don't like this idea :)
Short version
I propose that the Raw Usage be modified to *optionally* be ("CPU
equivalents" * time) instead of just (CPUs * time). The "CPU
equivalent" would be a MAX() of CPUs, memory, nodes, GPUs, energy over
that time period, or whatever multiplied by a corresponding charge rate
that an admin can configure on a per partition basis.
I wrote a simple proof of concept patch to demonstrate this (see "Proof
of Concept" below for details).
Longer version
The CPU equivalent would be used in place of total_cpus for calculating
usage_raw. I propose that the default charge rate be 1.0 for each CPU
in a job and 0.0 for everything else. This is the current behavior so
there are no behavior changes if you choose not to define a different
charge rate. The reason I think this should be done on a partition
basis is because different partitions may have nodes with different
memory/core ratios, etc. so one partition may have 2 GB/core and another
may have 8 GB/core nodes and you may want to charge differently on each.
If you define the charge rate for each CPU to be 1.0 and the charge rate
per GB of memory to be 0.5, that is saying that 2 GB of memory will be
equivalent to the charge rate for 1 CPU. 4 GB of memory would be
equivalent to 2 CPUs (4 GB * 0.5/GB). Since it is a MAX() of all the
available (resource * charge_rate) combinations, the largest value is
chosen. If a user uses 1 CPU and 1 TB of RAM out of a 1 TB node, the
user gets charged for using all the RAM. If a user uses 16 CPUs and 1
MB, the user gets charged for 16 CPUs.
Downsides
The problem that is not completely solved is if a user uses 1 CPU but
3/4 of the memory on a node. Then they only get billed for 3/4 of the
node but might make it unusable for others who need a whole or half
node. I'm not sure of a great way to solve that besides modifying the
request in a job submit plugin or requiring exclusive node access.
One other complication is for resources that include a counter rather
than a static allocation value, such as network bandwidth or energy.
This is a problem because the current approach is to immediately begin
decaying the cputime (aka usage) as it accumulates. This means you
would have to keep a delta value for each resource with a counter,
meaning you track that 5 GB have been transmitted since the last decay
thread iteration then only add that 5 GB. This could get messy when
comparing MAX(total_cpus * charge_per_cpu, total_bw * charge_bw_per_gb)
each iteration since the bandwidth may never reach a high enough value
to matter between iterations but might when considered as an entire job.
I don't think this proposal would be too bad for something like energy.
You could define a charge rate per joule (or kilojoule or whatever) that
would equal the node's minimum power divided by core count. Then you
look at the delta of that time period. If they were allocated all cores
and used minimum power, they get charged 1.0 * core count. If they were
allocated all cores and used maximum power, they effectively get charged
for the difference in the node's max energy and min energy times the
energy charge rate. This calculation, as with others, would occur once
per decay thread iteration.
User Education
The reason I like this approach is that it is incredibly simple to
implement and I don't think it takes much effort to explain to users.
It would be easy to add other resources you want to charge for (it would
require a code addition, though it would be pretty simple if the data is
available in the right structs). It doesn't require any RPC changes.
sshare, etc only need manpage clarifications to say that the usage data
is "CPU equivalents". No new fields are required.
As for user education, you just need to explain the concept of "CPU
equivalents", something that can be easily done in the documentation.
The slurm.conf partition lines would be relatively easy to read too. If
you don't need to change the behavior, no slurm.conf changes or
explanations to users are required.
Proof of Concept
I did a really quick proof of concept (attached) based on the master
branch. It is very simple to charge for most things as long as the data
is there in the existing structs. One caveat for the test patch is that
I didn't see a float handler in the config parser so I skipped over that
for the test. Instead, each config parameter in slurm.conf should be
set to (desired_value * 1000) for now. Proper float handling can be
added if this is the route people want to take.
[slurm-dev] RE: fairshare - memory resource allocation
Hi,
if I understand it correctly, this is actually very close to Dominant Resource
Fairness (DRF) which I mentioned previously, with the difference that in DRF
the charge rates are determined automatically from the available resources (in
a partition) rather than being specified explicitly by the administrator. So
for an example, say we have a partition with 100 cores and 400 GB memory. Now
for a job requesting (10CPU's, 20 GB) the domination calculation proceeds as
follows:
1) Calculate the "domination vector" by dividing each element in the request
vector (here, CPU & MEM) with the available resources. That is (10/100, 20/400)
= (0.1, 0.05).
2) The MAX element in the domination vector is chosen (it "dominates" the
others, hence the name of the algorithm) as the one to use in fairshare
calculations, accounting etc. In this case, the CPU element (0.1).
Now for another job request, (1CPU, 20 GB) the domination vector is (0.01,
0.05) and the MAX element is the memory element (0.05), so in this case the
memory part of the request dominates.
In your patch you have used "cpu-sec equivalents" rather than "dominant share
secs", but that's just a difference of a scaling factor. From a backwards
compatibility and user education point of view cpu-sec equivalents seem like a
better choice to me, actually.
So while you patch is more flexible than DRF in that it allows arbitrary charge
rates to be specified, I'm not sure it makes sense to specify rates different
from the DRF ones? Or if one does specify different rates, it might end up
breaking some of the fairness properties that are described in the DRF paper
and opens up the algorithm for gaming?
--
Janne Blomqvist
From: Ryan Cox [ryan_...@byu.edu]
Sent: Tuesday, July 29, 2014 18:47
To: slurm-dev
Subject: [slurm-dev] RE: fairshare - memory resource allocation
I'm interested in hearing opinions on this, if any. Basically, I think
there is an easy solution to the problem of a user using few CPUs but a
lot of memory and that not being reflected well in the CPU-centric usage
stats.
Below is my proposal. There are likely some other good approaches out
there too (Don and Janne presented some) so feel free to tell me that
you don't like this idea :)
Short version
I propose that the Raw Usage be modified to *optionally* be ("CPU
equivalents" * time) instead of just (CPUs * time). The "CPU
equivalent" would be a MAX() of CPUs, memory, nodes, GPUs, energy over
that time period, or whatever multiplied by a corresponding charge rate
that an admin can configure on a per partition basis.
I wrote a simple proof of concept patch to demonstrate this (see "Proof
of Concept" below for details).
Longer version
The CPU equivalent would be used in place of total_cpus for calculating
usage_raw. I propose that the default charge rate be 1.0 for each CPU
in a job and 0.0 for everything else. This is the current behavior so
there are no behavior changes if you choose not to define a different
charge rate. The reason I think this should be done on a partition
basis is because different partitions may have nodes with different
memory/core ratios, etc. so one partition may have 2 GB/core and another
may have 8 GB/core nodes and you may want to charge differently on each.
If you define the charge rate for each CPU to be 1.0 and the charge rate
per GB of memory to be 0.5, that is saying that 2 GB of memory will be
equivalent to the charge rate for 1 CPU. 4 GB of memory would be
equivalent to 2 CPUs (4 GB * 0.5/GB). Since it is a MAX() of all the
available (resource * charge_rate) combinations, the largest value is
chosen. If a user uses 1 CPU and 1 TB of RAM out of a 1 TB node, the
user gets charged for using all the RAM. If a user uses 16 CPUs and 1
MB, the user gets charged for 16 CPUs.
Downsides
The problem that is not completely solved is if a user uses 1 CPU but
3/4 of the memory on a node. Then they only get billed for 3/4 of the
node but might make it unusable for others who need a whole or half
node. I'm not sure of a great way to solve that besides modifying the
request in a job submit plugin or requiring exclusive node access.
One other complication is for resources that include a counter rather
than a static allocation value, such as network bandwidth or energy.
This is a problem because the current approach is to immediately begin
decaying the cputime (aka usage) as it accumulates. This means you
would have to keep a delta value for each resource with a counter,
meaning you track that 5 GB have been transmitted since the last decay
thread iteration then only add that 5 GB. This could get messy when
comparing MAX(total_cpus * charge_per_cpu, total_bw * charge_bw_per_gb)
each iteration since the bandwidth may never reach a high enough value
to matter between iterations but might when c
[slurm-dev] RE: fairshare - memory resource allocation
Just a short note about terminology. I believe "processor equivalents" (PE) is a much used term for this. It is at least what Maui and Moab uses, if I recall correctly. The "resource*time" would then be PE seconds (or hours, or whatever). -- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo
[slurm-dev] RE: fairshare - memory resource allocation
Janne,
I appreciate the feedback. I agree that it makes the most sense to
specify rates like DRF most of the time. However, there are some use
cases that I'm aware of and others that are probably out there that
would make a DRF imitation difficult or less desirable if it's the only
option.
We happen to have one partition that has mixed memory amounts per node,
32 GB and 64 GB. Besides the memory differences (long story), the nodes
are homogeneous and each have 16 cores. I'm not sure I would like the
DRF approach for this particular scenario. In this case we would like
to set the charge rate to be .5/GB, or 1 core == 2 GB RAM. If someone
needs 64 GB per node, they are contending for a more limited resource
and we would be happy to double the charge rate for the 64 GB nodes. If
they need all 64 GB, they would end up being charged for 32
CPU/processor equivalents instead of 16. With DRF that wouldn't be
possible if I understand correctly.
One other feature that could be interesting is to have a "baseline"
standard for a CPU charge on a per-partition basis. Let's say that you
have three partitions: old_hardware, new_hardware, and
super_cooled_overclocked_awesomeness. You could set the per CPU charges
to be 0.8, 1.0, and 20.0. That would reflect that a cpu-hour on one
partition doesn't result in the same amount of computation as in another
partition. You could accomplish the same thing automatically by using a
QOS (and maybe some other parameter I'm not aware of) and maybe a job
submit plugin but this would make it easier. I don't know that we would
do this in our setup but it would be possible.
It would be possible to add a config parameter that is something like
Mem=DRF that would auto-configure it to match. The one question I have
about that approach is what to do about partitions with non-homogeneous
nodes. Does it make sense to sum the total cores and memory, etc or
should it default to a charge rate that is the min() of the node
configurations? Of course, partitions with mixed node types could be
difficult to support no matter what method is used for picking charge rates.
So yes, having a DRF-like auto-configuration could be nice and we might
even use it for most of our partitions. I don't think I'll attempt it
for the initial implementation but we'll see.
Thanks,
Ryan
On 07/30/2014 03:31 PM, Blomqvist Janne wrote:
Hi,
if I understand it correctly, this is actually very close to Dominant Resource
Fairness (DRF) which I mentioned previously, with the difference that in DRF
the charge rates are determined automatically from the available resources (in
a partition) rather than being specified explicitly by the administrator. So
for an example, say we have a partition with 100 cores and 400 GB memory. Now
for a job requesting (10CPU's, 20 GB) the domination calculation proceeds as
follows:
1) Calculate the "domination vector" by dividing each element in the request vector
(here, CPU & MEM) with the available resources. That is (10/100, 20/400) = (0.1, 0.05).
2) The MAX element in the domination vector is chosen (it "dominates" the
others, hence the name of the algorithm) as the one to use in fairshare calculations,
accounting etc. In this case, the CPU element (0.1).
Now for another job request, (1CPU, 20 GB) the domination vector is (0.01,
0.05) and the MAX element is the memory element (0.05), so in this case the
memory part of the request dominates.
In your patch you have used "cpu-sec equivalents" rather than "dominant share
secs", but that's just a difference of a scaling factor. From a backwards compatibility and
user education point of view cpu-sec equivalents seem like a better choice to me, actually.
So while you patch is more flexible than DRF in that it allows arbitrary charge
rates to be specified, I'm not sure it makes sense to specify rates different
from the DRF ones? Or if one does specify different rates, it might end up
breaking some of the fairness properties that are described in the DRF paper
and opens up the algorithm for gaming?
--
Janne Blomqvist
____________________
From: Ryan Cox [ryan_...@byu.edu]
Sent: Tuesday, July 29, 2014 18:47
To: slurm-dev
Subject: [slurm-dev] RE: fairshare - memory resource allocation
I'm interested in hearing opinions on this, if any. Basically, I think
there is an easy solution to the problem of a user using few CPUs but a
lot of memory and that not being reflected well in the CPU-centric usage
stats.
Below is my proposal. There are likely some other good approaches out
there too (Don and Janne presented some) so feel free to tell me that
you don't like this idea :)
Short version
I propose that the Raw Usage be modified to *optionally* be ("CPU
equivalents" * time) instead of just (CPUs * time). Th
[slurm-dev] RE: fairshare - memory resource allocation
Thanks. I can certainly call it that. My understanding is that this
would be a slightly different implementation from Moab/Maui, but I don't
know those as well so I could be wrong. Either way, the concept is
similar enough that a more recognizable term might be good.
Does anyone else have thoughts on this? I called it "CPU equivalents"
because the calculation in the code is currently ("total_cpus" * time)
so I stuck with CPUs. Slurm seems to use lots of terms somewhat
interchangeably so I couldn't really decide. I don't really have an
opinion on the name so I'll just accept what others decide.
Ryan
On 07/31/2014 02:28 AM, Bjørn-Helge Mevik wrote:
Just a short note about terminology. I believe "processor equivalents"
(PE) is a much used term for this. It is at least what Maui and Moab
uses, if I recall correctly. The "resource*time" would then be PE seconds
(or hours, or whatever).
[slurm-dev] RE: fairshare - memory resource allocation
All, There has been more conversation on http://bugs.schedmd.com/show_bug.cgi?id=858. It might be good to post future comments there so we have just one central location for everything. No worries if you'd rather reply on the list. Once a solution is ready I'll post something to the list so everyone is aware. Ryan
[slurm-dev] RE: fairshare - memory resource allocation
Hi all, this is a very interesting approach. I hope we find a chance to discuss it in Lugano. Ulf -- ___ Dr. Ulf Markwardt Technische Universität Dresden Center for Information Services and High Performance Computing (ZIH) 01062 Dresden, Germany Phone: (+49) 351/463-33640 WWW: http://www.tu-dresden.de/zih smime.p7s Description: S/MIME Cryptographic Signature
