In addition to the other suggestions, there's this:
https://slurm.schedmd.com/faq.html#tmpfs_jobcontainer
https://slurm.schedmd.com/job_container.conf.html
I would be interested in hearing how well it works - it's so buried in the
documentation that unfortunately I didn't see it until after I rolled a
solution similar to Diego's (which can be extended such that TaskProlog
sets the TMPDIR environment variable appropriately, and limit the disk
space used by the job).
All the best,
Mark
On Mon, 23 May 2022, Diego Zuccato wrote:
[EXTERNAL EMAIL]
Hi Arsene.
I did something like that some weeks ago.
I used the lines
Prolog=/home/conf/Prolog.sh
TaskProlog=/home/conf/TaskProlog.sh
Epilog=/home/conf/Epilog.sh
The scripts for prolog and epilog manage the creation (and permissions
assignment) of a directory in local storage (including the job ID, so
that different jobs don't get messed up).
TaskProlog script should export an environment variable but I couldn't
make it work :(
In your case, TaskProlog should copy the dataset to the local storage
and then you should add a TaskEpilog script to copy back the result. I
don't know if the TaskEpilog gets run for aborted jobs.
Moreover, IIRC you shouldn't do slow operations in task prolog or
epilog, so in your case a state machine implemented as a job array could
probably be better suited than TaskProlog/TaskEpilog (you'd need
Prolog/Epilog anyway): the first "job" copies to scratch, the second
does the number crunching and the third copies back the results.
HIH,
Diego
Il 23/05/2022 11:30, Arsene Marian Alain ha scritto:
Dear SLURM users,
I am IT Administrator of a small scientific computing center. We
recently installed SLURM as a job scheduler on our Cluster and
everything seems to be working fine. I just have a question about how to
create temporary directories with SLURM.
We use some programs for scientific calculation (such as Gromacs,
Gaussian, NAMD, etc.). So, the process is the following:
When we need to launch a calculation the first step is to copy all the
necessary files from the local "$SLURM_SUBMIT_DIR" directory to the
"/scratch" of the remote node, second step is to access the "/scratch"
of the remote node and then run the program. Finally, when the program
finishes we copy all the output files from the remote node's "/scratch"
back to the local "$SLURM_SUBMIT_DIR" directory.
So, is there any way to automatically generate a temporary directory
inside the "/scratch" of the remote node?
At the moment I am creating that directory manually as follows:
"export HOMEDIR=$SLURM_SUBMIT_DIR
export SCRATCHDIR=/scratch/job.$SLURM_JOB_ID.$USER
export WORKDIR=$SCRATCHDIR
mkdir -p $WORKDIR
cp $HOMEDIR/* $WORKDIR
cd $WORKDIR
$NAMD/namd2 +idlepoll +p11 run_eq.namd > run_eq.log
wait
cp $WORKDIR/* $HOMEDIR"
The main problem when you create the "/scratch" manually is that in case
the calculation ends (successfully or unsuccessfully), users have to
check the "/scratch" and remove the directory manually. I know I could
include a line at the end of my script to delete that directory when the
calculation is done, but I'm sure there must be a better way to do this.
Thanks in advance for the help.
best regards,
Alain
--
Diego Zuccato
DIFA - Dip. di Fisica e Astronomia
Servizi Informatici
Alma Mater Studiorum - Università di Bologna
V.le Berti-Pichat 6/2 - 40127 Bologna - Italy
tel.: +39 051 20 95786
