Re: [slurm-users] propose environment variables SLURM_STDOUT, SLURM_STDERR, SLURM_STDIN
I think it would be useful, yes, and mostly for the epilog script. In the job script itself, you are creating such files, so some of the proposed use cases are a bit tricky to get right in the way you described them. For example, if you scp these files, you are scp'ing them to their status before scp is run. Something else might happen (e.g. scp warnings) which will be added to the files after the command is run, and those would not be included. Also, the buffers might not have flushed, so the scp'ed version can be incomplete. Even worse for post-processing, which can be covered better with something like the following in the slurm script program_that_creates_lots_of_output | tee full_output_just_in_case_is_needed.txt | post_processing_script So that the original slurm file will automatically contain the post-processing version, and the "just in case" file will contain the full log. Of course the name of the latter does not need to be hardcoded and can use things like $SLURM_JOB_ID to make it unique for each job. On Mon, Jan 22, 2024 at 12:11 AM Bjørn-Helge Mevik wrote: > I would find that useful, yes. Especially if the variables were made > available for the Prolog and Epilog scripts. > > -- > Regards, > Bjørn-Helge Mevik, dr. scient, > Department for Research Computing, University of Oslo > >
Re: [slurm-users] propose environment variables SLURM_STDOUT, SLURM_STDERR, SLURM_STDIN
I would find that useful, yes. Especially if the variables were made available for the Prolog and Epilog scripts. -- Regards, Bjørn-Helge Mevik, dr. scient, Department for Research Computing, University of Oslo signature.asc Description: PGP signature
[slurm-users] propose environment variables SLURM_STDOUT, SLURM_STDERR, SLURM_STDIN
RE: Placing the full pathname of the job stdout in an environment variable Would others find it useful if new variables were added that contained the full pathnames of the standard input, error and input files of batch jobs? ## SYNOPSIS Proposed new environment variables SLURM_STDOUT,SLURM_STDERR, and SLURM_STDIN pointing to the full pathnames of the output, error output, and input files of a task would be a useful feature for batch jobs in particular. ## PROBLEM There are cases where it is desirable to have a job process the standard files (stdin, stdout, stderr) from a batch job from within the job instead of later via an epilog or via post-processing. Just a few examples where the job might want to reference the full pathname of stdout: * copy the file to a remote machine (via scp(1), for example) * move the file from local or scratch space to permanent globally accessible storage * remove the file depending on completion status * mail the output * archive it * post-process it Slurm commands like scontrol(1) and squeue(1) can help you locate the file names. But the names returned do not always expand all the macros allowed when specifying the names, and require calling a Slurm command (which might overload Slurm if millions of jobs are submitted or cause problems if Slurm is not responding for any reason(?)). For example: #!/bin/bash # Minimalist submit script for sbatch(1) #SBATCH --nodes 1-1 --ntasks=1 --time 0-0:1:00 --chdir=/tmp #SBATCH --output=stdout.%j #SBATCH --error=stderr.%A:%a:%J:%j:%N:%n:%s:%t:%u.out # query filenames via squeue(1) export SLURM_STDOUT=$(squeue --noheader --Format=STDOUT: --job=$SLURM_JOBID) export SLURM_STDERR=$(squeue --noheader --Format=STDERR: --job=$SLURM_JOBID) # PS: Current documentation for squeue(1) does not explicitly tell the # user a null size works. # query filenames via scontrol(1) declare -x $(scontrol show job=$SLURM_JOBID|grep StdOut) declare -x $(scontrol show job=$SLURM_JOBID|grep StdErr) cat
