Re: [slurm-users] Terminating Jobs based on GrpTRESMins
On 28-04-2023 18:28, Hoot Thompson wrote: I’m somewhat confused by your statement "This would only occur if you lower the GrpTRESMins limit after a job has started.”. My test case had the limits established before job submittal and the job was terminated when the threshold was crossed. According to the sacctmgr manual page describing the GrpTRESMins limit: When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. Therefore a job requesting more minutes than the GrpTRESMins limit should not be permitted to start. /Ole On Apr 28, 2023, at 6:43 AM, Ole Holm Nielsen wrote: Hi Hoot, I'm glad that you have figured out that GrpTRESMins is working as documented and kills running jobs when the limit is exceeded. This would only occur if you lower the GrpTRESMins limit after a job has started. /Ole On 4/27/23 22:39, Hoot Thompson wrote: I have GrpTRESMins working and terminating jobs as expected. I was working under the belief that the limit “current value” was only updated upon job completion. That is not the case, it’s actually updated every 5 minutes it appears. If and when the limit/threshold is crossed, jobs are in fact canceled. Thanks for your help. On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen wrote: On 24-04-2023 18:33, Hoot Thompson wrote: In my reading of the Slurm documentation, it seems that exceeding the limits set in GrpTRESMins should result in terminating a running job. However, in testing this, The ‘current value’ of the GrpTRESMins only updates upon job completion and is not updated as the job progresses. Therefore jobs aren’t being stopped. On the positive side, no new jobs are started if the limit is exceeded. Here’s the documentation that is confusing me….. I think the jobs resource usage will only be added to the Slurm database upon job completion. I believe that Slurm doesn't update the resource usage continually as you seem to expect. If any limit is reached, all running jobs with that TRES in this group will be killed, and no new jobs will be allowed to run. Perhaps there is a setting or misconfiguration on my part. The sacctmgr manual page states: GrpTRESMins=TRES=[,TRES=,...] The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children. To clear a previously set value use the modify command with a new value of -1 for each TRES id. NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output, it will not be enforced. ALSO NOTE: This limit only applies when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. Can you please confirm that you have configured the "Priority Multifactor" plugin? Your jobs should not be able to start if the user's GrpTRESMins has been exceeded. Hence they won't be killed! Can you explain step by step what you observe? It may be that the above documentation of killing jobs is in error, in which case we should make a bug report to SchedMD.
Re: [slurm-users] Terminating Jobs based on GrpTRESMins
I’m somewhat confused by your statement "This would only occur if you lower the GrpTRESMins limit after a job has started.”. My test case had the limits established before job submittal and the job was terminated when the threshold was crossed. Hoot > On Apr 28, 2023, at 6:43 AM, Ole Holm Nielsen > wrote: > > Hi Hoot, > > I'm glad that you have figured out that GrpTRESMins is working as documented > and kills running jobs when the limit is exceeded. This would only occur if > you lower the GrpTRESMins limit after a job has started. > > /Ole > > On 4/27/23 22:39, Hoot Thompson wrote: >> I have GrpTRESMins working and terminating jobs as expected. I was working >> under the belief that the limit “current value” was only updated upon job >> completion. That is not the case, it’s actually updated every 5 minutes it >> appears. If and when the limit/threshold is crossed, jobs are in fact >> canceled. >> Thanks for your help. >>> On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen >>> wrote: >>> >>> On 24-04-2023 18:33, Hoot Thompson wrote: In my reading of the Slurm documentation, it seems that exceeding the limits set in GrpTRESMins should result in terminating a running job. However, in testing this, The ‘current value’ of the GrpTRESMins only updates upon job completion and is not updated as the job progresses. Therefore jobs aren’t being stopped. On the positive side, no new jobs are started if the limit is exceeded. Here’s the documentation that is confusing me….. >>> >>> I think the jobs resource usage will only be added to the Slurm database >>> upon job completion. I believe that Slurm doesn't update the resource >>> usage continually as you seem to expect. >>> If any limit is reached, all running jobs with that TRES in this group will be killed, and no new jobs will be allowed to run. Perhaps there is a setting or misconfiguration on my part. >>> >>> The sacctmgr manual page states: >>> GrpTRESMins=TRES=[,TRES=,...] The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children. To clear a previously set value use the modify command with a new value of -1 for each TRES id. NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output, it will not be enforced. ALSO NOTE: This limit only applies when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. >>> >>> Can you please confirm that you have configured the "Priority Multifactor" >>> plugin? >>> >>> Your jobs should not be able to start if the user's GrpTRESMins has been >>> exceeded. Hence they won't be killed! >>> >>> Can you explain step by step what you observe? It may be that the above >>> documentation of killing jobs is in error, in which case we should make a >>> bug report to SchedMD. >
Re: [slurm-users] Terminating Jobs based on GrpTRESMins
Hi Hoot, I'm glad that you have figured out that GrpTRESMins is working as documented and kills running jobs when the limit is exceeded. This would only occur if you lower the GrpTRESMins limit after a job has started. /Ole On 4/27/23 22:39, Hoot Thompson wrote: I have GrpTRESMins working and terminating jobs as expected. I was working under the belief that the limit “current value” was only updated upon job completion. That is not the case, it’s actually updated every 5 minutes it appears. If and when the limit/threshold is crossed, jobs are in fact canceled. Thanks for your help. On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen wrote: On 24-04-2023 18:33, Hoot Thompson wrote: In my reading of the Slurm documentation, it seems that exceeding the limits set in GrpTRESMins should result in terminating a running job. However, in testing this, The ‘current value’ of the GrpTRESMins only updates upon job completion and is not updated as the job progresses. Therefore jobs aren’t being stopped. On the positive side, no new jobs are started if the limit is exceeded. Here’s the documentation that is confusing me….. I think the jobs resource usage will only be added to the Slurm database upon job completion. I believe that Slurm doesn't update the resource usage continually as you seem to expect. If any limit is reached, all running jobs with that TRES in this group will be killed, and no new jobs will be allowed to run. Perhaps there is a setting or misconfiguration on my part. The sacctmgr manual page states: GrpTRESMins=TRES=[,TRES=,...] The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children. To clear a previously set value use the modify command with a new value of -1 for each TRES id. NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output, it will not be enforced. ALSO NOTE: This limit only applies when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. Can you please confirm that you have configured the "Priority Multifactor" plugin? Your jobs should not be able to start if the user's GrpTRESMins has been exceeded. Hence they won't be killed! Can you explain step by step what you observe? It may be that the above documentation of killing jobs is in error, in which case we should make a bug report to SchedMD.
Re: [slurm-users] Terminating Jobs based on GrpTRESMins
I have GrpTRESMins working and terminating jobs as expected. I was working under the belief that the limit “current value” was only updated upon job completion. That is not the case, it’s actually updated every 5 minutes it appears. If and when the limit/threshold is crossed, jobs are in fact canceled. Thanks for your help. > On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen > wrote: > > On 24-04-2023 18:33, Hoot Thompson wrote: >> In my reading of the Slurm documentation, it seems that exceeding the limits >> set in GrpTRESMins should result in terminating a running job. However, in >> testing this, The ‘current value’ of the GrpTRESMins only updates upon job >> completion and is not updated as the job progresses. Therefore jobs aren’t >> being stopped. On the positive side, no new jobs are started if the limit is >> exceeded. Here’s the documentation that is confusing me….. > > I think the jobs resource usage will only be added to the Slurm database upon > job completion. I believe that Slurm doesn't update the resource usage > continually as you seem to expect. > >> If any limit is reached, all running jobs with that TRES in this group will >> be killed, and no new jobs will be allowed to run. >> Perhaps there is a setting or misconfiguration on my part. > > The sacctmgr manual page states: > >> GrpTRESMins=TRES=[,TRES=,...] >> The total number of TRES minutes that can possibly be used by past, present >> and future jobs running from this association and its children. To clear a >> previously set value use the modify command with a new value of -1 for each >> TRES id. >> NOTE: This limit is not enforced if set on the root association of a >> cluster. So even though it may appear in sacctmgr output, it will not be >> enforced. >> ALSO NOTE: This limit only applies when using the Priority Multifactor >> plugin. The time is decayed using the value of PriorityDecayHalfLife or >> PriorityUsageResetPeriod as set in the slurm.conf. When this limit is >> reached all associated jobs running will be killed and all future jobs >> submitted with associations in the group will be delayed until they are able >> to run inside the limit. > > Can you please confirm that you have configured the "Priority Multifactor" > plugin? > > Your jobs should not be able to start if the user's GrpTRESMins has been > exceeded. Hence they won't be killed! > > Can you explain step by step what you observe? It may be that the above > documentation of killing jobs is in error, in which case we should make a bug > report to SchedMD. > > /Ole > > > >
Re: [slurm-users] Terminating Jobs based on GrpTRESMins
So Ole, any thoughts on the config info I sent? I’m still not certain if terminating a running job based on GrpTRESMins is even possible or supposed to work. Hoot > On Apr 24, 2023, at 3:21 PM, Hoot Thompson wrote: > > See below…... > >> On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen >> wrote: >> >> On 24-04-2023 18:33, Hoot Thompson wrote: >>> In my reading of the Slurm documentation, it seems that exceeding the >>> limits set in GrpTRESMins should result in terminating a running job. >>> However, in testing this, The ‘current value’ of the GrpTRESMins only >>> updates upon job completion and is not updated as the job progresses. >>> Therefore jobs aren’t being stopped. On the positive side, no new jobs are >>> started if the limit is exceeded. Here’s the documentation that is >>> confusing me….. >> >> I think the jobs resource usage will only be added to the Slurm database >> upon job completion. I believe that Slurm doesn't update the resource usage >> continually as you seem to expect. >> >>> If any limit is reached, all running jobs with that TRES in this group will >>> be killed, and no new jobs will be allowed to run. >>> Perhaps there is a setting or misconfiguration on my part. >> >> The sacctmgr manual page states: >> >>> GrpTRESMins=TRES=[,TRES=,...] >>> The total number of TRES minutes that can possibly be used by past, present >>> and future jobs running from this association and its children. To clear a >>> previously set value use the modify command with a new value of -1 for each >>> TRES id. >>> NOTE: This limit is not enforced if set on the root association of a >>> cluster. So even though it may appear in sacctmgr output, it will not be >>> enforced. >>> ALSO NOTE: This limit only applies when using the Priority Multifactor >>> plugin. The time is decayed using the value of PriorityDecayHalfLife or >>> PriorityUsageResetPeriod as set in the slurm.conf. When this limit is >>> reached all associated jobs running will be killed and all future jobs >>> submitted with associations in the group will be delayed until they are >>> able to run inside the limit. >> >> Can you please confirm that you have configured the "Priority Multifactor" >> plugin? > Here’s relevant items from slurm.conf > > > # Activate the Multifactor Job Priority Plugin with decay > PriorityType=priority/multifactor > > # apply no decay > PriorityDecayHalfLife=0 > > # reset usage after 1 month > PriorityUsageResetPeriod=MONTHLY > > # The larger the job, the greater its job size priority. > PriorityFavorSmall=NO > > # The job's age factor reaches 1.0 after waiting in the > # queue for 2 weeks. > PriorityMaxAge=14-0 > > # This next group determines the weighting of each of the > # components of the Multifactor Job Priority Plugin. > # The default value for each of the following is 1. > PriorityWeightAge=1000 > PriorityWeightFairshare=1 > PriorityWeightJobSize=1000 > PriorityWeightPartition=1000 > PriorityWeightQOS=0 # don't use the qos factor > > >> >> Your jobs should not be able to start if the user's GrpTRESMins has been >> exceeded. Hence they won't be killed! > > Yes, this works fine >> >> Can you explain step by step what you observe? It may be that the above >> documentation of killing jobs is in error, in which case we should make a >> bug report to SchedMD. > > I set the GrpTRESMins limit to a very small number and then ran a sleep job > that exceeded the limit. The job continued to run past the limits until I > killed it. It was the only job in the queue. And if it makes any difference, > this testing is being done in AWS on a parallel cluster. >> >> /Ole
Re: [slurm-users] Terminating Jobs based on GrpTRESMins
See below…... > On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen > wrote: > > On 24-04-2023 18:33, Hoot Thompson wrote: >> In my reading of the Slurm documentation, it seems that exceeding the limits >> set in GrpTRESMins should result in terminating a running job. However, in >> testing this, The ‘current value’ of the GrpTRESMins only updates upon job >> completion and is not updated as the job progresses. Therefore jobs aren’t >> being stopped. On the positive side, no new jobs are started if the limit is >> exceeded. Here’s the documentation that is confusing me….. > > I think the jobs resource usage will only be added to the Slurm database upon > job completion. I believe that Slurm doesn't update the resource usage > continually as you seem to expect. > >> If any limit is reached, all running jobs with that TRES in this group will >> be killed, and no new jobs will be allowed to run. >> Perhaps there is a setting or misconfiguration on my part. > > The sacctmgr manual page states: > >> GrpTRESMins=TRES=[,TRES=,...] >> The total number of TRES minutes that can possibly be used by past, present >> and future jobs running from this association and its children. To clear a >> previously set value use the modify command with a new value of -1 for each >> TRES id. >> NOTE: This limit is not enforced if set on the root association of a >> cluster. So even though it may appear in sacctmgr output, it will not be >> enforced. >> ALSO NOTE: This limit only applies when using the Priority Multifactor >> plugin. The time is decayed using the value of PriorityDecayHalfLife or >> PriorityUsageResetPeriod as set in the slurm.conf. When this limit is >> reached all associated jobs running will be killed and all future jobs >> submitted with associations in the group will be delayed until they are able >> to run inside the limit. > > Can you please confirm that you have configured the "Priority Multifactor" > plugin? Here’s relevant items from slurm.conf # Activate the Multifactor Job Priority Plugin with decay PriorityType=priority/multifactor # apply no decay PriorityDecayHalfLife=0 # reset usage after 1 month PriorityUsageResetPeriod=MONTHLY # The larger the job, the greater its job size priority. PriorityFavorSmall=NO # The job's age factor reaches 1.0 after waiting in the # queue for 2 weeks. PriorityMaxAge=14-0 # This next group determines the weighting of each of the # components of the Multifactor Job Priority Plugin. # The default value for each of the following is 1. PriorityWeightAge=1000 PriorityWeightFairshare=1 PriorityWeightJobSize=1000 PriorityWeightPartition=1000 PriorityWeightQOS=0 # don't use the qos factor > > Your jobs should not be able to start if the user's GrpTRESMins has been > exceeded. Hence they won't be killed! Yes, this works fine > > Can you explain step by step what you observe? It may be that the above > documentation of killing jobs is in error, in which case we should make a bug > report to SchedMD. I set the GrpTRESMins limit to a very small number and then ran a sleep job that exceeded the limit. The job continued to run past the limits until I killed it. It was the only job in the queue. And if it makes any difference, this testing is being done in AWS on a parallel cluster. > > /Ole > > > >
Re: [slurm-users] Terminating Jobs based on GrpTRESMins
On 24-04-2023 18:33, Hoot Thompson wrote: In my reading of the Slurm documentation, it seems that exceeding the limits set in GrpTRESMins should result in terminating a running job. However, in testing this, The ‘current value’ of the GrpTRESMins only updates upon job completion and is not updated as the job progresses. Therefore jobs aren’t being stopped. On the positive side, no new jobs are started if the limit is exceeded. Here’s the documentation that is confusing me….. I think the jobs resource usage will only be added to the Slurm database upon job completion. I believe that Slurm doesn't update the resource usage continually as you seem to expect. If any limit is reached, all running jobs with that TRES in this group will be killed, and no new jobs will be allowed to run. Perhaps there is a setting or misconfiguration on my part. The sacctmgr manual page states: GrpTRESMins=TRES=[,TRES=,...] The total number of TRES minutes that can possibly be used by past, present and future jobs running from this association and its children. To clear a previously set value use the modify command with a new value of -1 for each TRES id. NOTE: This limit is not enforced if set on the root association of a cluster. So even though it may appear in sacctmgr output, it will not be enforced. ALSO NOTE: This limit only applies when using the Priority Multifactor plugin. The time is decayed using the value of PriorityDecayHalfLife or PriorityUsageResetPeriod as set in the slurm.conf. When this limit is reached all associated jobs running will be killed and all future jobs submitted with associations in the group will be delayed until they are able to run inside the limit. Can you please confirm that you have configured the "Priority Multifactor" plugin? Your jobs should not be able to start if the user's GrpTRESMins has been exceeded. Hence they won't be killed! Can you explain step by step what you observe? It may be that the above documentation of killing jobs is in error, in which case we should make a bug report to SchedMD. /Ole
[slurm-users] Terminating Jobs based on GrpTRESMins
In my reading of the Slurm documentation, it seems that exceeding the limits set in GrpTRESMins should result in terminating a running job. However, in testing this, The ‘current value’ of the GrpTRESMins only updates upon job completion and is not updated as the job progresses. Therefore jobs aren’t being stopped. On the positive side, no new jobs are started if the limit is exceeded. Here’s the documentation that is confusing me….. If any limit is reached, all running jobs with that TRES in this group will be killed, and no new jobs will be allowed to run. Perhaps there is a setting or misconfiguration on my part. Thanks in advance!
