Re: [slurm-users] Terminating Jobs based on GrpTRESMins

2023-04-28 Thread Hoot Thompson
Ole Holm Nielsen > wrote: > > Hi Hoot, > > I'm glad that you have figured out that GrpTRESMins is working as documented > and kills running jobs when the limit is exceeded. This would only occur if > you lower the GrpTRESMins limit after a job has started. > > /

Re: [slurm-users] Terminating Jobs based on GrpTRESMins

2023-04-27 Thread Hoot Thompson
canceled. Thanks for your help. > On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen > wrote: > > On 24-04-2023 18:33, Hoot Thompson wrote: >> In my reading of the Slurm documentation, it seems that exceeding the limits >> set in GrpTRESMins should result in terminating a

[slurm-users] Terminate active job

2023-04-27 Thread Hoot Thompson
Is there a mechanism for terminating an active job based on a predetermined threshold - CPU usage, budget, etc? GrpTRESMins reads like it should work but in practice it doesn’t seem to. Thanks in advance!

Re: [slurm-users] Terminating Jobs based on GrpTRESMins

2023-04-25 Thread Hoot Thompson
So Ole, any thoughts on the config info I sent? I’m still not certain if terminating a running job based on GrpTRESMins is even possible or supposed to work. Hoot > On Apr 24, 2023, at 3:21 PM, Hoot Thompson wrote: > > See below…... > >> On Apr 24, 2023, at 1:55 PM

Re: [slurm-users] Terminating Jobs based on GrpTRESMins

2023-04-24 Thread Hoot Thompson
See below…... > On Apr 24, 2023, at 1:55 PM, Ole Holm Nielsen > wrote: > > On 24-04-2023 18:33, Hoot Thompson wrote: >> In my reading of the Slurm documentation, it seems that exceeding the limits >> set in GrpTRESMins should result in terminating a running job. Howe

[slurm-users] Terminating Jobs based on GrpTRESMins

2023-04-24 Thread Hoot Thompson
In my reading of the Slurm documentation, it seems that exceeding the limits set in GrpTRESMins should result in terminating a running job. However, in testing this, The ‘current value’ of the GrpTRESMins only updates upon job completion and is not updated as the job progresses. Therefore jobs a

Re: [slurm-users] Resource LImits

2023-04-21 Thread Hoot Thompson
installation solution. > > Since requirements can be so different, and because Slurm is a fantastic > software that can be configured for many different scenarios, IMHO a support > contract with SchedMD is the best way to get consulting services, get general > help, and report bugs.

Re: [slurm-users] Resource LImits

2023-04-20 Thread Hoot Thompson
So an update, GrpTRES registers a value while a job is running but GRpTRESMins does not. So I still have something wrong. GrpTRESMins reads in the docs like it is in fact an aggregate number. > On Apr 20, 2023, at 1:01 PM, Ole Holm Nielsen > wrote: > > On 20-04-2023 18:23, H

Re: [slurm-users] Resource LImits

2023-04-20 Thread Hoot Thompson
And it indeed does show current value for a running job!! Do I feel stupid :-) > On Apr 20, 2023, at 1:01 PM, Ole Holm Nielsen > wrote: > > On 20-04-2023 18:23, Hoot Thompson wrote: >> Ole, >> Earlier I found your Slurm_tools posting and found it very useful. Thi

Re: [slurm-users] Resource LImits

2023-04-20 Thread Hoot Thompson
Ah, I thought that was the aggregate of past and current jobs. > On Apr 20, 2023, at 1:01 PM, Ole Holm Nielsen > wrote: > > On 20-04-2023 18:23, Hoot Thompson wrote: >> Ole, >> Earlier I found your Slurm_tools posting and found it very useful. This >> rema

Re: [slurm-users] Resource LImits

2023-04-20 Thread Hoot Thompson
ubuntu 1 0 0.00 0.00 Clearly I’m still missing something or I don’t understand how it’s supposed to work. Hoot > On Apr 20, 2023, at 2:10 AM, Ole Holm Nielsen > wrote: > > Hi Hoot, > > On 4/20/23 00:15, Hoot Thompson wrote:

Re: [slurm-users] Resource LImits

2023-04-20 Thread Hoot Thompson
Thank you for this. I’ll give it a read but no promises that I won’t be back with more questions! Hoot > On Apr 20, 2023, at 2:10 AM, Ole Holm Nielsen > wrote: > > Hi Hoot, > > On 4/20/23 00:15, Hoot Thompson wrote: >> Is there a ‘how to’ or recipe document for

[slurm-users] Resource LImits

2023-04-19 Thread Hoot Thompson
Is there a ‘how to’ or recipe document for setting up and enforcing resource limits? I can establish accounts, users, and set limits but 'current value' is not incrementing after running jobs. Thanks in advance

[slurm-users] PrologSlurmctld race condition

2022-09-22 Thread Hoot Thompson
I'm running slurm on an AWS cluster and there seems to be a race condition whereby the PrologSlurmctld script runs on occasion when compute nodes try to transition to CG from P but fall back to P when nodes are not available. Have you seen this behavior and is there a way to prevent it? I'm run

[slurm-users] slurm prolog script

2022-09-15 Thread Hoot Thompson
Can the prolog script be configured to only run on a cluster head node as opposed to compute nodes?

Re: [slurm-users] memory per node default

2022-01-21 Thread Hoot Thompson
Thank you for the support. I will be back with any additional questions. BTW, if it changes or adds to your thoughts, I'm working in AWS on a parallelcluster. Hoot On 1/21/22 4:12 AM, Ole Holm Nielsen wrote: On 1/21/22 10:05, Diego Zuccato wrote: Il 21/01/2022 07:51, Ole Holm Nielsen ha scr

[slurm-users] memory per node default

2022-01-20 Thread Hoot Thompson
How do you change the default memory per node from the current 1MB to something much higher? Thanks in advance. *ubuntu@node*:*/shared*$ sinfo -o "%20N%10c%10m%25f%10G " NODELISTCPUSMEMORYAVAIL_FEATURES GRES hpc-demand-dy-c5n18x361 dynamic,c5n.18xlarge,c5n1(null)

Re: [slurm-users] Error when upgrading to 21.08.1

2021-09-23 Thread Hoot Thompson
Ok, a fresh start after installing the two recommended packages and things appear to be working. Thank for the help! On 9/23/21, 3:04 PM, "slurm-users on behalf of Hoot Thompson" wrote: Do I need to specify the json path in the configure process? On 9/23/21, 2:45 PM, &q

Re: [slurm-users] Error when upgrading to 21.08.1

2021-09-23 Thread Hoot Thompson
Do I need to specify the json path in the configure process? On 9/23/21, 2:45 PM, "slurm-users on behalf of Hoot Thompson" wrote: If this useful, note that there's no attempt to build anything in the serializer/json directory. Making all in serializer ma

Re: [slurm-users] Error when upgrading to 21.08.1

2021-09-23 Thread Hoot Thompson
7; make[5]: Nothing to be done for 'all-am'. make[5]: Leaving directory '/home/ubuntu/slurm-21.08.1/src/plugins/serializer' make[4]: Leaving directory '/home/ubuntu/slurm-21.08.1/src/plugins/serializer' Making all in site_factor make[4]: Entering directory '/home/u

Re: [slurm-users] Error when upgrading to 21.08.1

2021-09-23 Thread Hoot Thompson
What's getting built is serializer_url_encoded.a serializer_url_encoded.la serializer_url_encoded.so if this helps. On 9/23/21, 2:10 PM, "slurm-users on behalf of Hoot Thompson" wrote: On Ubuntu 20.04 I installed ... libjson-c-dev Libhttp-parser-dev Th

Re: [slurm-users] Error when upgrading to 21.08.1

2021-09-23 Thread Hoot Thompson
On Ubuntu 20.04 I installed ... libjson-c-dev Libhttp-parser-dev That work? No joy if so. On 9/23/21, 1:30 PM, "slurm-users on behalf of Ole Holm Nielsen" wrote: On 23-09-2021 16:01, Hoot Thompson wrote: > In upgrading to 21.08.1, slurmctld status reports: >

[slurm-users] Error when upgrading to 21.08.1

2021-09-23 Thread Hoot Thompson
In upgrading to 21.08.1, slurmctld status reports: Sep 23 13:49:52 ip-10-10-7-17 systemd[1]: Started Slurm controller daemon. Sep 23 13:49:52 ip-10-10-7-17 slurmctld[1323]: fatal: Unable to find plugin: serializer/json Sep 23 13:49:52 ip-10-10-7-17 systemd[1]: slurmctld.service: Main process

[slurm-users] REST API

2021-05-28 Thread Hoot Thompson
I have the REST API basically working but I am having a problem with job submission syntax. The error I receive is ‘Unable to parse query”. I have followed the guides found on-line to no avail. Is there somewhere to look for what the issue may be?