Re: [slurm-users] slurm and singularity

[Markus Kötter]

Hi,


On 08.02.23 05:00, Carl Ponder wrote:

Take a look at this extension to SLURM:

https://github.com/NVIDIA/pyxis


https://slurm.schedmd.com/SLUG19/NVIDIA_Containers.pdf

enroot & pyxis - great recommendation for rootless containerized runtime 
environments in HPC.


Free software, no license or DGX required.


Some things to consider

Cache in /tmp so it's free'd upon reboot:
# /etc/enroot/enroot.conf
ENROOT_RUNTIME_PATH /tmp/enroot/user-$(id -u)
ENROOT_CACHE_PATH /tmp/enroot-cache/user-$(id -u)
ENROOT_DATA_PATH /tmp/enroot-data/user-$(id -u)


When using a local container repo, the image urls port is seperated 
using # :



srun … --container-image mygitlab:5005#path/pytorch:22.12-py3 …



MfG
--
Markus Kötter, +49 681 870832434
30159 Hannover, Lange Laube 6
Helmholtz Center for Information Security


smime.p7s
Description: S/MIME Cryptographic Signature

Re: [slurm-users] slurm and singularity

[Carl Ponder]

Take a look at this extension to SLURM:

   https://github.com/NVIDIA/pyxis

You put the container path on the srun command-line and each rank runs 
inside it's own copy of the image.



Subject:[slurm-users] slurm and singularity
Date:   Tue, 7 Feb 2023 17:31:45 +
From:   Groner, Rob 
Reply-To:   Slurm User Community List 
To: slurm-users@lists.schedmd.com 



*External email: Use caution opening links or attachments*


I'm trying to setup the capability where a user can execute:

$: sbatch  script_to_run.sh

and the end result is that a job is created on a node, and that job will 
execute "singularity exec  script_to_run.sh"


Also, that they could execute:

$: salloc 

and would end up on a node per their parameters, and instead of a bash 
prompt, they have the singularity prompt because they're inside a 
running container.


Oddly, I ran:  salloc  /usr/bin/singularity shell sif> and that allocated and said the node was ready and gave me an 
apptainer prompt...cool!  But when I asked it what hostname I was on, I 
was NOT on the node that it had said was ready, I was still on the 
submit node.  When I exit out of the apptainer shell, it ends my 
allocation. Sooo...it gave me the allocation and started the apptainer 
shell, but somehow I was still on the submit node.


As far as the job, I've done some experiments with using job_submit.lua 
to replace the script with one that has a singularity call in it 
instead, and that might hold some promise.  But I'd have to write the 
passed-in script to a temp file or something, and then have singularity 
exec that. That MIGHT work.


The results for "slurm and singularity" do not describe what I'm trying 
to do.  The closest thing I can find is what slurm touts on their 
website, a leftover from Slurm 2017 talking about a spank plugin that, 
as near as I can figure, doesn't exist.  I read through the OCI docs on 
the slurm website, but it shows that using singularity with that 
requires all commands to have sudo. That's not going to work.


I'm running out of ideas here.

Thanks,

Rob

Re: [slurm-users] slurm and singularity

[Jeffrey T Frey]

> The remaining issue then is how to put them into an allocation that is 
> actually running a singularity container.  I don't get how what I'm doing now 
> is resulting in an allocation where I'm in a container on the submit node 
> still!

Try prefixing the singularity command with "srun" e.g.


salloc  srun  /usr/bin/singularity shell 

Re: [slurm-users] slurm and singularity

[Groner, Rob]

Looks like we can go the route of a wrapper script, since our users don't 
specifically need to know they're running an sbatch.  Thanks for the suggestion.

The remaining issue then is how to put them into an allocation that is actually 
running a singularity container.  I don't get how what I'm doing now is 
resulting in an allocation where I'm in a container on the submit node still!


From: slurm-users  on behalf of Brian 
Andrus 
Sent: Tuesday, February 7, 2023 12:52 PM
To: slurm-users@lists.schedmd.com 
Subject: Re: [slurm-users] slurm and singularity


You should have the job script itself have the singularity/apptainer command.


I am guessing you don't want your users to have to deal with that part for 
their scripts, so I would suggest using a wrapper script.


You could just have them run something like: cluster_run.sh 

Then cluster_run.sh would call sbatch along with the appropriate commands.


Brian Andrus



On 2/7/2023 9:31 AM, Groner, Rob wrote:
I'm trying to setup the capability where a user can execute:

$: sbatch  script_to_run.sh

and the end result is that a job is created on a node, and that job will 
execute "singularity exec  script_to_run.sh"

Also, that they could execute:

$: salloc 

and would end up on a node per their parameters, and instead of a bash prompt, 
they have the singularity prompt because they're inside a running container.

Oddly, I ran:  salloc  /usr/bin/singularity shell  and 
that allocated and said the node was ready and gave me an apptainer 
prompt...cool!  But when I asked it what hostname I was on, I was NOT on the 
node that it had said was ready, I was still on the submit node.  When I exit 
out of the apptainer shell, it ends my allocation.  Sooo...it gave me the 
allocation and started the apptainer shell, but somehow I was still on the 
submit node.

As far as the job, I've done some experiments with using job_submit.lua to 
replace the script with one that has a singularity call in it instead, and that 
might hold some promise.  But I'd have to write the passed-in script to a temp 
file or something, and then have singularity exec that. That MIGHT work.

The results for "slurm and singularity" do not describe what I'm trying to do.  
The closest thing I can find is what slurm touts on their website, a leftover 
from Slurm 2017 talking about a spank plugin that, as near as I can figure, 
doesn't exist.  I read through the OCI docs on the slurm website, but it shows 
that using singularity with that requires all commands to have sudo. That's not 
going to work.

I'm running out of ideas here.

Thanks,

Rob

Re: [slurm-users] [External] Hibernating a whole cluster

[Analabha Roy]

Howdy,

On Tue, 7 Feb 2023 at 20:18, Sean Mc Grath  wrote:

> Hi Analabha,
>
> Yes, unfortunately for your needs, I expect a time limited reservation
> along my suggestion would not accept jobs that would be scheduled to end
> outside of the reservations availability times. I'd suggest looking at
> check-pointing in this case, e.g. with DMTCP: Distributed MultiThreaded
> Checkpointing, http://dmtcp.sourceforge.net/. That could allow jobs to
> have their state saved and then re-loaded when they are started again.
>
>
Checkpointing sounds intriguing. Many thanks for the suggestion.

A bit of googling turned up this cluster page
where they've
set it up to work with slurm. However, I also noticed this presentation
hosted on the slurm website
that indicates that DMTCP doesn't work with containers, and the other
checkpointing tools that do support containers don't support MPI.
I also took a gander at CRIU , but this paper

indicates that it too, has similar limitations, and BLCR seems to have died
.


Unless some or all of this information is dated or obsolete, these
drawbacks would be deal-breakers, since most of us have been spoiled by
containerization, and MPI is, of course, bread and butter for all.

I'd be mighty grateful for any other insights regarding my predicament. In
the meantime, I'm going to give the ugly hack of launching scontrol
suspend-resume scripts a whirl.


AR




> Best
>
> Sean
>
> ---
> Sean McGrath
> Senior Systems Administrator, IT Services
>
> --
> *From:* slurm-users  on behalf of
> Analabha Roy 
> *Sent:* Tuesday 7 February 2023 12:14
> *To:* Slurm User Community List 
> *Subject:* Re: [slurm-users] [External] Hibernating a whole cluster
>
> Hi Sean,
>
> Thanks for your awesome suggestion! I'm going through the reservation docs
> now. At first glance, it seems like a daily reservation would turn down
> jobs that are too big for the reservation. It'd be nice if
> slurm could suspend (in the manner of 'scontrol suspend') jobs during
> reserved downtime and resume them after. That way, folks can submit large
> jobs without having to worry about the downtimes. Perhaps the FLEX option
> in reservations can accomplish this somehow?
>
>
> I suppose that I can do it using a shell script iterator and a cron job,
> but that seems like an ugly hack. I was hoping if there is a way to config
> this in slurm itself?
>
> AR
>
> On Tue, 7 Feb 2023 at 16:06, Sean Mc Grath  wrote:
>
> Hi Analabha,
>
> Could you do something like create a daily reservation for 8 hours that
> starts at 9am, or whatever times work for you like the following untested
> command:
>
> scontrol create reservation starttime=09:00:00 duration=8:00:00 nodecnt=1
> flags=daily ReservationName=daily
>
> Daily option at https://slurm.schedmd.com/scontrol.html#OPT_DAILY
>
> Some more possible helpful documentation at
> https://slurm.schedmd.com/reservations.html, search for "daily".
>
> My idea being that jobs can only run in that reservation, (that would have
> to be configured separately, not sure how from the top of my head), which
> is only active during the times you want the node to be working. So the
> cronjob that hibernates/shuts it down will do so when there are no jobs
> running. At least in theory.
>
> Hope that helps.
>
> Sean
>
> ---
> Sean McGrath
> Senior Systems Administrator, IT Services
>
> --
> *From:* slurm-users  on behalf of
> Analabha Roy 
> *Sent:* Tuesday 7 February 2023 10:05
> *To:* Slurm User Community List 
> *Subject:* Re: [slurm-users] [External] Hibernating a whole cluster
>
> Hi,
>
> Thanks. I had read the Slurm Power Saving Guide before. I believe the
> configs enable slurmctld to check other nodes for idleness and
> suspend/resume them. Slurmctld must run on a separate, always-on server for
> this to work, right?
>
> My issue might be a little different. I literally have only one node that
> runs everything: slurmctld, slurmd, slurmdbd, everything.
>
> This node must be set to "sudo systemctl hibernate"after business hours,
> regardless of whether jobs are queued or running. The next business day, it
> can be switched on manually.
>
> systemctl hibernate is supposed to save the entire run state of the sole
> node to swap and poweroff. When powered on again, it should restore
> everything to its previous running state.
>
> When the job queue is empty, this works well. I'm not sure how well this
> hibernate/resume will work with running jobs and would appreciate any
> suggestions or insights.
>
> AR
>
>
> On Tue, 7 Feb 2023 at 01:39, Florian Zillner  wrote:
>
> Hi,
>
> follow this guide: https://slurm.schedmd.com/power_save.html
>
> Create poweroff / poweron scripts and configure slu

Re: [slurm-users] slurm and singularity

[Brian Andrus]

You should have the job script itself have the singularity/apptainer 
command.



I am guessing you don't want your users to have to deal with that part 
for their scripts, so I would suggest using a wrapper script.



You could just have them run something like: cluster_run.sh 

Then cluster_run.sh would call sbatch along with the appropriate commands.


Brian Andrus



On 2/7/2023 9:31 AM, Groner, Rob wrote:

I'm trying to setup the capability where a user can execute:

$: sbatch  script_to_run.sh

and the end result is that a job is created on a node, and that job 
will execute "singularity exec  script_to_run.sh"


Also, that they could execute:

$: salloc 

and would end up on a node per their parameters, and instead of a bash 
prompt, they have the singularity prompt because they're inside a 
running container.


Oddly, I ran:  salloc  /usr/bin/singularity shell sif> and that allocated and said the node was ready and gave me an 
apptainer prompt...cool!  But when I asked it what hostname I was on, 
I was NOT on the node that it had said was ready, I was still on the 
submit node.  When I exit out of the apptainer shell, it ends my 
allocation.  Sooo...it gave me the allocation and started the 
apptainer shell, but somehow I was still on the submit node.


As far as the job, I've done some experiments with using 
job_submit.lua to replace the script with one that has a singularity 
call in it instead, and that might hold some promise.  But I'd have to 
write the passed-in script to a temp file or something, and then have 
singularity exec that. That MIGHT work.


The results for "slurm and singularity" do not describe what I'm 
trying to do.  The closest thing I can find is what slurm touts on 
their website, a leftover from Slurm 2017 talking about a spank plugin 
that, as near as I can figure, doesn't exist.  I read through the OCI 
docs on the slurm website, but it shows that using singularity with 
that requires all commands to have sudo. That's not going to work.


I'm running out of ideas here.

Thanks,

Rob

[slurm-users] slurm and singularity

[Groner, Rob]

I'm trying to setup the capability where a user can execute:

$: sbatch  script_to_run.sh

and the end result is that a job is created on a node, and that job will 
execute "singularity exec  script_to_run.sh"

Also, that they could execute:

$: salloc 

and would end up on a node per their parameters, and instead of a bash prompt, 
they have the singularity prompt because they're inside a running container.

Oddly, I ran:  salloc  /usr/bin/singularity shell  and 
that allocated and said the node was ready and gave me an apptainer 
prompt...cool!  But when I asked it what hostname I was on, I was NOT on the 
node that it had said was ready, I was still on the submit node.  When I exit 
out of the apptainer shell, it ends my allocation.  Sooo...it gave me the 
allocation and started the apptainer shell, but somehow I was still on the 
submit node.

As far as the job, I've done some experiments with using job_submit.lua to 
replace the script with one that has a singularity call in it instead, and that 
might hold some promise.  But I'd have to write the passed-in script to a temp 
file or something, and then have singularity exec that. That MIGHT work.

The results for "slurm and singularity" do not describe what I'm trying to do.  
The closest thing I can find is what slurm touts on their website, a leftover 
from Slurm 2017 talking about a spank plugin that, as near as I can figure, 
doesn't exist.  I read through the OCI docs on the slurm website, but it shows 
that using singularity with that requires all commands to have sudo. That's not 
going to work.

I'm running out of ideas here.

Thanks,

Rob

Re: [slurm-users] [External] Hibernating a whole cluster

[Sean Mc Grath]

Hi Analabha,

Yes, unfortunately for your needs, I expect a time limited reservation along my 
suggestion would not accept jobs that would be scheduled to end outside of the 
reservations availability times. I'd suggest looking at check-pointing in this 
case, e.g. with DMTCP: Distributed MultiThreaded Checkpointing, 
http://dmtcp.sourceforge.net/. That could allow jobs to have their state saved 
and then re-loaded when they are started again.

Best

Sean

---
Sean McGrath
Senior Systems Administrator, IT Services


From: slurm-users  on behalf of Analabha 
Roy 
Sent: Tuesday 7 February 2023 12:14
To: Slurm User Community List 
Subject: Re: [slurm-users] [External] Hibernating a whole cluster

Hi Sean,

Thanks for your awesome suggestion! I'm going through the reservation docs now. 
At first glance, it seems like a daily reservation would turn down jobs that 
are too big for the reservation. It'd be nice if
slurm could suspend (in the manner of 'scontrol suspend') jobs during reserved 
downtime and resume them after. That way, folks can submit large jobs without 
having to worry about the downtimes. Perhaps the FLEX option in reservations 
can accomplish this somehow?


I suppose that I can do it using a shell script iterator and a cron job, but 
that seems like an ugly hack. I was hoping if there is a way to config this in 
slurm itself?

AR

On Tue, 7 Feb 2023 at 16:06, Sean Mc Grath 
mailto:smcg...@tcd.ie>> wrote:
Hi Analabha,

Could you do something like create a daily reservation for 8 hours that starts 
at 9am, or whatever times work for you like the following untested command:

scontrol create reservation starttime=09:00:00 duration=8:00:00 nodecnt=1 
flags=daily ReservationName=daily

Daily option at https://slurm.schedmd.com/scontrol.html#OPT_DAILY

Some more possible helpful documentation at 
https://slurm.schedmd.com/reservations.html, search for "daily".

My idea being that jobs can only run in that reservation, (that would have to 
be configured separately, not sure how from the top of my head), which is only 
active during the times you want the node to be working. So the cronjob that 
hibernates/shuts it down will do so when there are no jobs running. At least in 
theory.

Hope that helps.

Sean

---
Sean McGrath
Senior Systems Administrator, IT Services


From: slurm-users 
mailto:slurm-users-boun...@lists.schedmd.com>>
 on behalf of Analabha Roy 
mailto:hariseldo...@gmail.com>>
Sent: Tuesday 7 February 2023 10:05
To: Slurm User Community List 
mailto:slurm-users@lists.schedmd.com>>
Subject: Re: [slurm-users] [External] Hibernating a whole cluster

Hi,

Thanks. I had read the Slurm Power Saving Guide before. I believe the configs 
enable slurmctld to check other nodes for idleness and suspend/resume them. 
Slurmctld must run on a separate, always-on server for this to work, right?

My issue might be a little different. I literally have only one node that runs 
everything: slurmctld, slurmd, slurmdbd, everything.

This node must be set to "sudo systemctl hibernate"after business hours, 
regardless of whether jobs are queued or running. The next business day, it can 
be switched on manually.

systemctl hibernate is supposed to save the entire run state of the sole node 
to swap and poweroff. When powered on again, it should restore everything to 
its previous running state.

When the job queue is empty, this works well. I'm not sure how well this 
hibernate/resume will work with running jobs and would appreciate any 
suggestions or insights.

AR


On Tue, 7 Feb 2023 at 01:39, Florian Zillner 
mailto:fzill...@lenovo.com>> wrote:
Hi,

follow this guide: https://slurm.schedmd.com/power_save.html

Create poweroff / poweron scripts and configure slurm to do the poweroff after 
X minutes. Works well for us. Make sure to set an appropriate time 
(ResumeTimeout) to allow the node to come back to service.
Note that we did not achieve good power saving with suspending the nodes, 
powering them off and on saves way more power. The downside is it takes ~ 5 
mins to resume (= power on) the nodes when needed.

Cheers,
Florian

From: slurm-users 
mailto:slurm-users-boun...@lists.schedmd.com>>
 on behalf of Analabha Roy 
mailto:hariseldo...@gmail.com>>
Sent: Monday, 6 February 2023 18:21
To: slurm-users@lists.schedmd.com 
mailto:slurm-users@lists.schedmd.com>>
Subject: [External] [slurm-users] Hibernating a whole cluster

Hi,

I've just finished  setup of a single node "cluster" with slurm on ubuntu 
20.04. Infrastructural limitations  prevent me from running it 24/7, and it's 
only powered on during business hours.


Currently, I have a cron job running that hibernates that sole node before 
closing time.

The hibernation is done with standard systemd, and hibernates to the swap 
partition.

 I have not run any lengthy slurm jobs on it yet. Before I do, can I get some 
though

[slurm-users] job_res_rm_job: plugin still initializing

[Loris Bennett]

Hi,

The other day we updated to 22.05.8.  We are interested in using
sharding with our GPUs, so after the update had finished, we
changed

  SelectType=select/cons_res

to

  SelectType=select/cons_tres

This seemed to cause the slurmctld to loose contact with the
slurmstepds, so that a large number of jobs were requeued, although they
were in fact still running.

The slurmstepds reported

  slurmd: error: Malformed RPC of type REQUEST_TERMINATE_JOB(6011) received
  slurmd: error: select_g_select_jobinfo_unpack: select plugin cons_tres not 
found
  slurmd: error: select_g_select_jobinfo_unpack: unpack error

In the slurmctld log multiple lines of 

  select/cons_res: job_res_rm_job: plugin still initializing
  
occurred.  This line also occurs in the following bug report

  https://bugs.schedmd.com/show_bug.cgi?id=10980

which is however related to something else, but of the line the SchedMD
employee writes

  I don't think this should ever happen.

Has anyone else seen this issue?

Cheers,

Loris

-- 
Dr. Loris Bennett (Herr/Mr)
ZEDAT, Freie Universität Berlin

Re: [slurm-users] [External] Hibernating a whole cluster

[Diego Zuccato]

That's probably not optimal, but could work. I'd go with brutal 
preemption: swapping 90+G can be quite time-consuming.


Diego

Il 07/02/2023 14:18, Analabha Roy ha scritto:



On Tue, 7 Feb 2023, 18:12 Diego Zuccato, > wrote:


RAM used by a suspended job is not released. At most it can be swapped
out (if enough swap is available).



There should be enough swap available. I have 93 gigs of Ram and as big 
a swap partition. I can top it off with swap files if needed.






Il 07/02/2023 13:14, Analabha Roy ha scritto:
 > Hi Sean,
 >
 > Thanks for your awesome suggestion! I'm going through the
reservation
 > docs now. At first glance, it seems like a daily reservation
would turn
 > down jobs that are too big for the reservation. It'd be nice if
 > slurm could suspend (in the manner of 'scontrol suspend') jobs
during
 > reserved downtime and resume them after. That way, folks can submit
 > large jobs without having to worry about the downtimes. Perhaps
the FLEX
 > option in reservations can accomplish this somehow?
 >
 >
 > I suppose that I can do it using a shell script iterator and a
cron job,
 > but that seems like an ugly hack. I was hoping if there is a way to
 > config this in slurm itself?
 >
 > AR
 >
 > On Tue, 7 Feb 2023 at 16:06, Sean Mc Grath mailto:smcg...@tcd.ie>
 > >> wrote:
 >
 >     Hi Analabha,
 >
 >     Could you do something like create a daily reservation for 8
hours
 >     that starts at 9am, or whatever times work for you like the
 >     following untested command:
 >
 >     scontrol create reservation starttime=09:00:00 duration=8:00:00
 >     nodecnt=1 flags=daily ReservationName=daily
 >
 >     Daily option at
https://slurm.schedmd.com/scontrol.html#OPT_DAILY

 >     >
 >
 >     Some more possible helpful documentation at
 > https://slurm.schedmd.com/reservations.html

 >     >, search for "daily".
 >
 >     My idea being that jobs can only run in that reservation, (that
 >     would have to be configured separately, not sure how from the
top of
 >     my head), which is only active during the times you want the
node to
 >     be working. So the cronjob that hibernates/shuts it down will
do so
 >     when there are no jobs running. At least in theory.
 >
 >     Hope that helps.
 >
 >     Sean
 >
 >     ---
 >     Sean McGrath
 >     Senior Systems Administrator, IT Services
 >
 >   
  

 >     *From:* slurm-users mailto:slurm-users-boun...@lists.schedmd.com>
 >     >> on behalf of
 >     Analabha Roy mailto:hariseldo...@gmail.com> >>
 >     *Sent:* Tuesday 7 February 2023 10:05
 >     *To:* Slurm User Community List
mailto:slurm-users@lists.schedmd.com>
 >     >>
 >     *Subject:* Re: [slurm-users] [External] Hibernating a whole
cluster
 >     Hi,
 >
 >     Thanks. I had read the Slurm Power Saving Guide before. I believe
 >     the configs enable slurmctld to check other nodes for
idleness and
 >     suspend/resume them. Slurmctld must run on a separate, always-on
 >     server for this to work, right?
 >
 >     My issue might be a little different. I literally have only
one node
 >     that runs everything: slurmctld, slurmd, slurmdbd, everything.
 >
 >     This node must be set to "sudo systemctl hibernate"after business
 >     hours, regardless of whether jobs are queued or running. The next
 >     business day, it can be switched on manually.
 >
 >     systemctl hibernate is supposed to save the entire run state
of the
 >     sole node to swap and poweroff. When powered on again, it should
 >     restore everything to its previous running state.
 >
 >     When the job queue is empty, this works well. I'm not sure
how well
 >     this hibernate/resume will work with running jobs and would
 >     appreciate any suggestions or insights.
 >
 >     AR
 >
 >
 >     On Tue, 7 Feb 2023 at 01:39, Florian Zillner
mailto:fzill...@lenovo.com>
 >     

Re: [slurm-users] Debian dist-upgrade?

[Steffen Grunewald]

Hi Loris,

On Tue, 2023-01-24 at 16:48:26 +0100, Loris Bennett wrote:
> Hi Steffen,
> 
> Could you create/find a deb-package for a Slurm 19.x version to use as
> an intermediate?  Never having built such a package, I don't now how
> much trouble that would be.

Actually, I found a 20.02.6 version that spent quite some time in the
Debian repositories (that would be two major releases after 18.08.5
and therefore should be OK).
I was able to build Debian packages for Buster from that source package,
and *might* be able to use the dbd package to obtain a database dump
that could be further processed by Bullseye's proper slurmdbd (20.11.4).
There would be no need to use a VM, or a RH system.

> That would still be a bit of a faff and so still may not be worth it.

It's perhaps worth the attempt - but if that fails, I'd just go on with
a fresh setup (which could easily be scripted for most of the gory details).

Thanks,
 Steffen

-- 
Steffen Grunewald, Cluster Administrator
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Am Mühlenberg 1 * D-14476 Potsdam-Golm * Germany
~~~
Fon: +49-331-567 7274
Mail: steffen.grunewald(at)aei.mpg.de
~~~

Re: [slurm-users] [External] Hibernating a whole cluster

[Analabha Roy]

On Tue, 7 Feb 2023, 18:12 Diego Zuccato,  wrote:

> RAM used by a suspended job is not released. At most it can be swapped
> out (if enough swap is available).
>


There should be enough swap available. I have 93 gigs of Ram and as big a
swap partition. I can top it off with swap files if needed.




>
> Il 07/02/2023 13:14, Analabha Roy ha scritto:
> > Hi Sean,
> >
> > Thanks for your awesome suggestion! I'm going through the reservation
> > docs now. At first glance, it seems like a daily reservation would turn
> > down jobs that are too big for the reservation. It'd be nice if
> > slurm could suspend (in the manner of 'scontrol suspend') jobs during
> > reserved downtime and resume them after. That way, folks can submit
> > large jobs without having to worry about the downtimes. Perhaps the FLEX
> > option in reservations can accomplish this somehow?
> >
> >
> > I suppose that I can do it using a shell script iterator and a cron job,
> > but that seems like an ugly hack. I was hoping if there is a way to
> > config this in slurm itself?
> >
> > AR
> >
> > On Tue, 7 Feb 2023 at 16:06, Sean Mc Grath  > > wrote:
> >
> > Hi Analabha,
> >
> > Could you do something like create a daily reservation for 8 hours
> > that starts at 9am, or whatever times work for you like the
> > following untested command:
> >
> > scontrol create reservation starttime=09:00:00 duration=8:00:00
> > nodecnt=1 flags=daily ReservationName=daily
> >
> > Daily option at https://slurm.schedmd.com/scontrol.html#OPT_DAILY
> > 
> >
> > Some more possible helpful documentation at
> > https://slurm.schedmd.com/reservations.html
> > , search for "daily".
> >
> > My idea being that jobs can only run in that reservation, (that
> > would have to be configured separately, not sure how from the top of
> > my head), which is only active during the times you want the node to
> > be working. So the cronjob that hibernates/shuts it down will do so
> > when there are no jobs running. At least in theory.
> >
> > Hope that helps.
> >
> > Sean
> >
> > ---
> > Sean McGrath
> > Senior Systems Administrator, IT Services
> >
> >
>  
> > *From:* slurm-users  > > on behalf of
> > Analabha Roy mailto:hariseldo...@gmail.com
> >>
> > *Sent:* Tuesday 7 February 2023 10:05
> > *To:* Slurm User Community List  > >
> > *Subject:* Re: [slurm-users] [External] Hibernating a whole cluster
> > Hi,
> >
> > Thanks. I had read the Slurm Power Saving Guide before. I believe
> > the configs enable slurmctld to check other nodes for idleness and
> > suspend/resume them. Slurmctld must run on a separate, always-on
> > server for this to work, right?
> >
> > My issue might be a little different. I literally have only one node
> > that runs everything: slurmctld, slurmd, slurmdbd, everything.
> >
> > This node must be set to "sudo systemctl hibernate"after business
> > hours, regardless of whether jobs are queued or running. The next
> > business day, it can be switched on manually.
> >
> > systemctl hibernate is supposed to save the entire run state of the
> > sole node to swap and poweroff. When powered on again, it should
> > restore everything to its previous running state.
> >
> > When the job queue is empty, this works well. I'm not sure how well
> > this hibernate/resume will work with running jobs and would
> > appreciate any suggestions or insights.
> >
> > AR
> >
> >
> > On Tue, 7 Feb 2023 at 01:39, Florian Zillner  > > wrote:
> >
> > Hi,
> >
> > follow this guide: https://slurm.schedmd.com/power_save.html
> > 
> >
> > Create poweroff / poweron scripts and configure slurm to do the
> > poweroff after X minutes. Works well for us. Make sure to set an
> > appropriate time (ResumeTimeout) to allow the node to come back
> > to service.
> > Note that we did not achieve good power saving with suspending
> > the nodes, powering them off and on saves way more power. The
> > downside is it takes ~ 5 mins to resume (= power on) the nodes
> > when needed.
> >
> > Cheers,
> > Florian
> >
>  
> > *From:* slurm-users  > > on behalf of
> > Analabha Roy  > >
> > *Sent:* Monday, 6 February 2023 18:21
> > *To:* slurm-users@lists.schedmd.com
> > 

Re: [slurm-users] [External] Hibernating a whole cluster

[Diego Zuccato]

RAM used by a suspended job is not released. At most it can be swapped 
out (if enough swap is available).



Il 07/02/2023 13:14, Analabha Roy ha scritto:

Hi Sean,

Thanks for your awesome suggestion! I'm going through the reservation 
docs now. At first glance, it seems like a daily reservation would turn 
down jobs that are too big for the reservation. It'd be nice if
slurm could suspend (in the manner of 'scontrol suspend') jobs during 
reserved downtime and resume them after. That way, folks can submit 
large jobs without having to worry about the downtimes. Perhaps the FLEX 
option in reservations can accomplish this somehow?



I suppose that I can do it using a shell script iterator and a cron job, 
but that seems like an ugly hack. I was hoping if there is a way to 
config this in slurm itself?


AR

On Tue, 7 Feb 2023 at 16:06, Sean Mc Grath > wrote:


Hi Analabha,

Could you do something like create a daily reservation for 8 hours
that starts at 9am, or whatever times work for you like the
following untested command:

scontrol create reservation starttime=09:00:00 duration=8:00:00
nodecnt=1 flags=daily ReservationName=daily

Daily option at https://slurm.schedmd.com/scontrol.html#OPT_DAILY


Some more possible helpful documentation at
https://slurm.schedmd.com/reservations.html
, search for "daily".

My idea being that jobs can only run in that reservation, (that
would have to be configured separately, not sure how from the top of
my head), which is only active during the times you want the node to
be working. So the cronjob that hibernates/shuts it down will do so
when there are no jobs running. At least in theory.

Hope that helps.

Sean

---
Sean McGrath
Senior Systems Administrator, IT Services


*From:* slurm-users mailto:slurm-users-boun...@lists.schedmd.com>> on behalf of
Analabha Roy mailto:hariseldo...@gmail.com>>
*Sent:* Tuesday 7 February 2023 10:05
*To:* Slurm User Community List mailto:slurm-users@lists.schedmd.com>>
*Subject:* Re: [slurm-users] [External] Hibernating a whole cluster
Hi,

Thanks. I had read the Slurm Power Saving Guide before. I believe
the configs enable slurmctld to check other nodes for idleness and
suspend/resume them. Slurmctld must run on a separate, always-on
server for this to work, right?

My issue might be a little different. I literally have only one node
that runs everything: slurmctld, slurmd, slurmdbd, everything.

This node must be set to "sudo systemctl hibernate"after business
hours, regardless of whether jobs are queued or running. The next
business day, it can be switched on manually.

systemctl hibernate is supposed to save the entire run state of the
sole node to swap and poweroff. When powered on again, it should
restore everything to its previous running state.

When the job queue is empty, this works well. I'm not sure how well
this hibernate/resume will work with running jobs and would
appreciate any suggestions or insights.

AR


On Tue, 7 Feb 2023 at 01:39, Florian Zillner mailto:fzill...@lenovo.com>> wrote:

Hi,

follow this guide: https://slurm.schedmd.com/power_save.html


Create poweroff / poweron scripts and configure slurm to do the
poweroff after X minutes. Works well for us. Make sure to set an
appropriate time (ResumeTimeout) to allow the node to come back
to service.
Note that we did not achieve good power saving with suspending
the nodes, powering them off and on saves way more power. The
downside is it takes ~ 5 mins to resume (= power on) the nodes
when needed.

Cheers,
Florian

*From:* slurm-users mailto:slurm-users-boun...@lists.schedmd.com>> on behalf of
Analabha Roy mailto:hariseldo...@gmail.com>>
*Sent:* Monday, 6 February 2023 18:21
*To:* slurm-users@lists.schedmd.com

mailto:slurm-users@lists.schedmd.com>>
*Subject:* [External] [slurm-users] Hibernating a whole cluster
Hi,

I've just finished  setup of a single node "cluster" with slurm
on ubuntu 20.04. Infrastructural limitations  prevent me from
running it 24/7, and it's only powered on during business hours.


Currently, I have a cron job running that hibernates that sole
node before closing time.

The hibernation is done with standard systemd, and hibernates to
the swap partition.

  I have not run any lengthy slurm

Re: [slurm-users] [External] Hibernating a whole cluster

[Analabha Roy]

Hi Sean,

Thanks for your awesome suggestion! I'm going through the reservation docs
now. At first glance, it seems like a daily reservation would turn down
jobs that are too big for the reservation. It'd be nice if
slurm could suspend (in the manner of 'scontrol suspend') jobs during
reserved downtime and resume them after. That way, folks can submit large
jobs without having to worry about the downtimes. Perhaps the FLEX option
in reservations can accomplish this somehow?


I suppose that I can do it using a shell script iterator and a cron job,
but that seems like an ugly hack. I was hoping if there is a way to config
this in slurm itself?

AR

On Tue, 7 Feb 2023 at 16:06, Sean Mc Grath  wrote:

> Hi Analabha,
>
> Could you do something like create a daily reservation for 8 hours that
> starts at 9am, or whatever times work for you like the following untested
> command:
>
> scontrol create reservation starttime=09:00:00 duration=8:00:00 nodecnt=1
> flags=daily ReservationName=daily
>
> Daily option at https://slurm.schedmd.com/scontrol.html#OPT_DAILY
>
> Some more possible helpful documentation at
> https://slurm.schedmd.com/reservations.html, search for "daily".
>
> My idea being that jobs can only run in that reservation, (that would have
> to be configured separately, not sure how from the top of my head), which
> is only active during the times you want the node to be working. So the
> cronjob that hibernates/shuts it down will do so when there are no jobs
> running. At least in theory.
>
> Hope that helps.
>
> Sean
>
> ---
> Sean McGrath
> Senior Systems Administrator, IT Services
>
> --
> *From:* slurm-users  on behalf of
> Analabha Roy 
> *Sent:* Tuesday 7 February 2023 10:05
> *To:* Slurm User Community List 
> *Subject:* Re: [slurm-users] [External] Hibernating a whole cluster
>
> Hi,
>
> Thanks. I had read the Slurm Power Saving Guide before. I believe the
> configs enable slurmctld to check other nodes for idleness and
> suspend/resume them. Slurmctld must run on a separate, always-on server for
> this to work, right?
>
> My issue might be a little different. I literally have only one node that
> runs everything: slurmctld, slurmd, slurmdbd, everything.
>
> This node must be set to "sudo systemctl hibernate"after business hours,
> regardless of whether jobs are queued or running. The next business day, it
> can be switched on manually.
>
> systemctl hibernate is supposed to save the entire run state of the sole
> node to swap and poweroff. When powered on again, it should restore
> everything to its previous running state.
>
> When the job queue is empty, this works well. I'm not sure how well this
> hibernate/resume will work with running jobs and would appreciate any
> suggestions or insights.
>
> AR
>
>
> On Tue, 7 Feb 2023 at 01:39, Florian Zillner  wrote:
>
> Hi,
>
> follow this guide: https://slurm.schedmd.com/power_save.html
>
> Create poweroff / poweron scripts and configure slurm to do the poweroff
> after X minutes. Works well for us. Make sure to set an appropriate time
> (ResumeTimeout) to allow the node to come back to service.
> Note that we did not achieve good power saving with suspending the nodes,
> powering them off and on saves way more power. The downside is it takes ~ 5
> mins to resume (= power on) the nodes when needed.
>
> Cheers,
> Florian
> --
> *From:* slurm-users  on behalf of
> Analabha Roy 
> *Sent:* Monday, 6 February 2023 18:21
> *To:* slurm-users@lists.schedmd.com 
> *Subject:* [External] [slurm-users] Hibernating a whole cluster
>
> Hi,
>
> I've just finished  setup of a single node "cluster" with slurm on ubuntu
> 20.04. Infrastructural limitations  prevent me from running it 24/7, and
> it's only powered on during business hours.
>
>
> Currently, I have a cron job running that hibernates that sole node before
> closing time.
>
> The hibernation is done with standard systemd, and hibernates to the swap
> partition.
>
>  I have not run any lengthy slurm jobs on it yet. Before I do, can I get
> some thoughts on a couple of things?
>
> If it hibernated when slurm still had jobs running/queued, would they
> resume properly when the machine powers back on?
>
> Note that my swap space is bigger than my  RAM.
>
> Is it necessary to perhaps setup a pre-hibernate script for systemd to
> iterate scontrol to suspend all the jobs before hibernating and resume them
> post-resume?
>
> What about the wall times? I'm uessing that slurm will count the downtime
> as elapsed for each job. Is there a way to config this, or is the only
> alternative a post-hibernate script that iteratively updates the wall times
> of the running jobs using scontrol again?
>
> Thanks for your attention.
> Regards
> AR
>
>
>
> --
> Analabha Roy
> Assistant Professor
> Department of Physics
> 
> The University of Burdwan 
> Golapbag Campus, Barddhaman 713104
> W

Re: [slurm-users] [External] Hibernating a whole cluster

[Sean Mc Grath]

Hi Analabha,

Could you do something like create a daily reservation for 8 hours that starts 
at 9am, or whatever times work for you like the following untested command:

scontrol create reservation starttime=09:00:00 duration=8:00:00 nodecnt=1 
flags=daily ReservationName=daily

Daily option at https://slurm.schedmd.com/scontrol.html#OPT_DAILY

Some more possible helpful documentation at 
https://slurm.schedmd.com/reservations.html, search for "daily".

My idea being that jobs can only run in that reservation, (that would have to 
be configured separately, not sure how from the top of my head), which is only 
active during the times you want the node to be working. So the cronjob that 
hibernates/shuts it down will do so when there are no jobs running. At least in 
theory.

Hope that helps.

Sean

---
Sean McGrath
Senior Systems Administrator, IT Services


From: slurm-users  on behalf of Analabha 
Roy 
Sent: Tuesday 7 February 2023 10:05
To: Slurm User Community List 
Subject: Re: [slurm-users] [External] Hibernating a whole cluster

Hi,

Thanks. I had read the Slurm Power Saving Guide before. I believe the configs 
enable slurmctld to check other nodes for idleness and suspend/resume them. 
Slurmctld must run on a separate, always-on server for this to work, right?

My issue might be a little different. I literally have only one node that runs 
everything: slurmctld, slurmd, slurmdbd, everything.

This node must be set to "sudo systemctl hibernate"after business hours, 
regardless of whether jobs are queued or running. The next business day, it can 
be switched on manually.

systemctl hibernate is supposed to save the entire run state of the sole node 
to swap and poweroff. When powered on again, it should restore everything to 
its previous running state.

When the job queue is empty, this works well. I'm not sure how well this 
hibernate/resume will work with running jobs and would appreciate any 
suggestions or insights.

AR


On Tue, 7 Feb 2023 at 01:39, Florian Zillner 
mailto:fzill...@lenovo.com>> wrote:
Hi,

follow this guide: https://slurm.schedmd.com/power_save.html

Create poweroff / poweron scripts and configure slurm to do the poweroff after 
X minutes. Works well for us. Make sure to set an appropriate time 
(ResumeTimeout) to allow the node to come back to service.
Note that we did not achieve good power saving with suspending the nodes, 
powering them off and on saves way more power. The downside is it takes ~ 5 
mins to resume (= power on) the nodes when needed.

Cheers,
Florian

From: slurm-users 
mailto:slurm-users-boun...@lists.schedmd.com>>
 on behalf of Analabha Roy 
mailto:hariseldo...@gmail.com>>
Sent: Monday, 6 February 2023 18:21
To: slurm-users@lists.schedmd.com 
mailto:slurm-users@lists.schedmd.com>>
Subject: [External] [slurm-users] Hibernating a whole cluster

Hi,

I've just finished  setup of a single node "cluster" with slurm on ubuntu 
20.04. Infrastructural limitations  prevent me from running it 24/7, and it's 
only powered on during business hours.


Currently, I have a cron job running that hibernates that sole node before 
closing time.

The hibernation is done with standard systemd, and hibernates to the swap 
partition.

 I have not run any lengthy slurm jobs on it yet. Before I do, can I get some 
thoughts on a couple of things?

If it hibernated when slurm still had jobs running/queued, would they resume 
properly when the machine powers back on?

Note that my swap space is bigger than my  RAM.

Is it necessary to perhaps setup a pre-hibernate script for systemd to  iterate 
scontrol to suspend all the jobs before hibernating and resume them post-resume?

What about the wall times? I'm uessing that slurm will count the downtime as 
elapsed for each job. Is there a way to config this, or is the only alternative 
a post-hibernate script that iteratively updates the wall times of the running 
jobs using scontrol again?

Thanks for your attention.
Regards
AR


--
Analabha Roy
Assistant Professor
Department of Physics
The University of Burdwan
Golapbag Campus, Barddhaman 713104
West Bengal, India
Emails: dan...@utexas.edu, 
a...@phys.buruniv.ac.in, 
hariseldo...@gmail.com
Webpage: http://www.ph.utexas.edu/~daneel/

Re: [slurm-users] [External] Hibernating a whole cluster

[Analabha Roy]

Hi,

Thanks. I had read the Slurm Power Saving Guide before. I believe the
configs enable slurmctld to check other nodes for idleness and
suspend/resume them. Slurmctld must run on a separate, always-on server for
this to work, right?

My issue might be a little different. I literally have only one node that
runs everything: slurmctld, slurmd, slurmdbd, everything.

This node must be set to "sudo systemctl hibernate"after business hours,
regardless of whether jobs are queued or running. The next business day, it
can be switched on manually.

systemctl hibernate is supposed to save the entire run state of the sole
node to swap and poweroff. When powered on again, it should restore
everything to its previous running state.

When the job queue is empty, this works well. I'm not sure how well this
hibernate/resume will work with running jobs and would appreciate any
suggestions or insights.

AR


On Tue, 7 Feb 2023 at 01:39, Florian Zillner  wrote:

> Hi,
>
> follow this guide: https://slurm.schedmd.com/power_save.html
>
> Create poweroff / poweron scripts and configure slurm to do the poweroff
> after X minutes. Works well for us. Make sure to set an appropriate time
> (ResumeTimeout) to allow the node to come back to service.
> Note that we did not achieve good power saving with suspending the nodes,
> powering them off and on saves way more power. The downside is it takes ~ 5
> mins to resume (= power on) the nodes when needed.
>
> Cheers,
> Florian
> --
> *From:* slurm-users  on behalf of
> Analabha Roy 
> *Sent:* Monday, 6 February 2023 18:21
> *To:* slurm-users@lists.schedmd.com 
> *Subject:* [External] [slurm-users] Hibernating a whole cluster
>
> Hi,
>
> I've just finished  setup of a single node "cluster" with slurm on ubuntu
> 20.04. Infrastructural limitations  prevent me from running it 24/7, and
> it's only powered on during business hours.
>
>
> Currently, I have a cron job running that hibernates that sole node before
> closing time.
>
> The hibernation is done with standard systemd, and hibernates to the swap
> partition.
>
>  I have not run any lengthy slurm jobs on it yet. Before I do, can I get
> some thoughts on a couple of things?
>
> If it hibernated when slurm still had jobs running/queued, would they
> resume properly when the machine powers back on?
>
> Note that my swap space is bigger than my  RAM.
>
> Is it necessary to perhaps setup a pre-hibernate script for systemd to
> iterate scontrol to suspend all the jobs before hibernating and resume them
> post-resume?
>
> What about the wall times? I'm uessing that slurm will count the downtime
> as elapsed for each job. Is there a way to config this, or is the only
> alternative a post-hibernate script that iteratively updates the wall times
> of the running jobs using scontrol again?
>
> Thanks for your attention.
> Regards
> AR
>


-- 
Analabha Roy
Assistant Professor
Department of Physics

The University of Burdwan 
Golapbag Campus, Barddhaman 713104
West Bengal, India
Emails: dan...@utexas.edu, a...@phys.buruniv.ac.in, hariseldo...@gmail.com
Webpage: http://www.ph.utexas.edu/~daneel/