Re: [slurm-users] Notify users about job submit plugin actions
Hi Lorenzo, On 7/20/23 12:16, Lorenzo Bosio wrote: > One more thing I'd like to point out, is that I need to monitor jobs going > from pending to running state (after waiting in the jobs queue). I > currently have a separate pthread to achieve this, but I think at this > point the job_submit()/job_modify() function has already exited. > I do get the output of the slurm_kill_job() function when called, but > that's not useful for the user and I couldn't find a way to append custom > messages. Maybe it's useful to have E-mail notifications sent to your users when the job transitions its state? According to the sbatch man-page the user can specify himself what mail alerts he would like: --mail-type= Notify user by email when certain event types occur. Valid type values are NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, INVALID_DEPEND, REQUEUE, and STAGE_OUT), INVALID_DEPEND (dependency never satisfied), STAGE_OUT (burst buffer stage out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), TIME_LIMIT_50 (reached 50 percent of time limit) and ARRAY_TASKS (send emails for each array task). /Ole
Re: [slurm-users] Notify users about job submit plugin actions
Hello everyone,
thanks for all the answers.
To elaborate further: I'm developing in C, but that's not a problem
since I can find an equivalent to LUA as Jeffrey T Frey said.
One more thing I'd like to point out, is that I need to monitor jobs
going from pending to running state (after waiting in the jobs queue). I
currently have a separate pthread to achieve this, but I think at this
point the job_submit()/job_modify() function has already exited.
I do get the output of the slurm_kill_job() function when called, but
that's not useful for the user and I couldn't find a way to append
custom messages.
Again, thanks everyone who helped.
Regards,
Lorenzo
On 19/07/23 16:00, Jeffrey T Frey wrote:
In case you're developing the plugin in C and not LUA, behind the scenes the
LUA mechanism is concatenating all log_user() strings into a single variable
(user_msg). When the LUA code completes, the C code sets the *err_msg argument
to the job_submit()/job_modify() function to that string, then NULLs-out
user-msg. (There's a mutex around all of that code so slurmctld never executes
LUA job submit/modify scripts concurrently.) The slurmctld then communicates
that returned string back to sbatch/salloc/srun for display to the user.
Your C plugin would do likewise — set *err_msg before returning from
job_submit()/job_modify() — and needn't be mutex'ed if the code is reentrant.
On Jul 19, 2023, at 08:37, Angel de Vicente wrote:
Hello Lorenzo,
Lorenzo Bosio writes:
I'm developing a job submit plugin to check if some conditions are met before a
job runs.
I'd need a way to notify the user about the plugin actions (i.e. why its jobs
was killed and what to do), but after a lot of research I could only write to
logs and not the user shell.
The user gets the output of slurm_kill_job but I can't find a way to add a
custom note.
Can anyone point me to the right api/function in the code?
In our "job_submit.lua" script we have the following for that purpose:
,
| slurm.log_user("%s: WARNING: [...]", log_prefix)
`
--
Ángel de Vicente
Research Software Engineer (Supercomputing and BigData)
Tel.: +34 922-605-747
Web.:http://research.iac.es/proyecto/polmag/
GPG: 0x8BDC390B69033F52
--
*/Dott. Mag. Lorenzo Bosio/*
Tecnico di Ricerca
Dipartimento di Informatica
Università degli Studi di Torino
Corso Svizzera, 185 - 10149 Torino
tel. +39 011 670 6836
Re: [slurm-users] Notify users about job submit plugin actions
In case you're developing the plugin in C and not LUA, behind the scenes the
LUA mechanism is concatenating all log_user() strings into a single variable
(user_msg). When the LUA code completes, the C code sets the *err_msg argument
to the job_submit()/job_modify() function to that string, then NULLs-out
user-msg. (There's a mutex around all of that code so slurmctld never executes
LUA job submit/modify scripts concurrently.) The slurmctld then communicates
that returned string back to sbatch/salloc/srun for display to the user.
Your C plugin would do likewise — set *err_msg before returning from
job_submit()/job_modify() — and needn't be mutex'ed if the code is reentrant.
> On Jul 19, 2023, at 08:37, Angel de Vicente wrote:
>
> Hello Lorenzo,
>
> Lorenzo Bosio writes:
>
>> I'm developing a job submit plugin to check if some conditions are met
>> before a job runs.
>> I'd need a way to notify the user about the plugin actions (i.e. why its
>> jobs was killed and what to do), but after a lot of research I could only
>> write to logs and not the user shell.
>> The user gets the output of slurm_kill_job but I can't find a way to add a
>> custom note.
>>
>> Can anyone point me to the right api/function in the code?
>
> In our "job_submit.lua" script we have the following for that purpose:
>
> ,
> | slurm.log_user("%s: WARNING: [...]", log_prefix)
> `
>
> --
> Ángel de Vicente
> Research Software Engineer (Supercomputing and BigData)
> Tel.: +34 922-605-747
> Web.: http://research.iac.es/proyecto/polmag/
>
> GPG: 0x8BDC390B69033F52
Re: [slurm-users] Notify users about job submit plugin actions
Hello Lorenzo,
Lorenzo Bosio writes:
> I'm developing a job submit plugin to check if some conditions are met before
> a job runs.
> I'd need a way to notify the user about the plugin actions (i.e. why its jobs
> was killed and what to do), but after a lot of research I could only write to
> logs and not the user shell.
> The user gets the output of slurm_kill_job but I can't find a way to add a
> custom note.
>
> Can anyone point me to the right api/function in the code?
In our "job_submit.lua" script we have the following for that purpose:
,
| slurm.log_user("%s: WARNING: [...]", log_prefix)
`
--
Ángel de Vicente
Research Software Engineer (Supercomputing and BigData)
Tel.: +34 922-605-747
Web.: http://research.iac.es/proyecto/polmag/
GPG: 0x8BDC390B69033F52
smime.p7s
Description: S/MIME cryptographic signature
Re: [slurm-users] Notify users about job submit plugin actions
Hi Lorenzo, On 7/19/23 14:22, Lorenzo Bosio wrote: > I'm developing a job submit plugin to check if some conditions are met > before a job runs. > I'd need a way to notify the user about the plugin actions (i.e. why its > jobs was killed and what to do), but after a lot of research I could only > write to logs and not the user shell. > The user gets the output of slurm_kill_job but I can't find a way to add a > custom note. > > Can anyone point me to the right api/function in the code? I've written a fairly general job submit plugin which you can copy and customize for your needs: https://github.com/OleHolmNielsen/Slurm_tools/tree/master/plugins The slurm.log_user() function prints an error message to the user's terminal. I hope this helps. /Ole
[slurm-users] Notify users about job submit plugin actions
Hello everyone, I'm developing a job submit plugin to check if some conditions are met before a job runs. I'd need a way to notify the user about the plugin actions (i.e. why its jobs was killed and what to do), but after a lot of research I could only write to logs and not the user shell. The user gets the output of slurm_kill_job but I can't find a way to add a custom note. Can anyone point me to the right api/function in the code? Thanks in advance, Lorenzo -- */Dott. Mag. Lorenzo Bosio/* Tecnico di Ricerca Dipartimento di Informatica Università degli Studi di Torino Corso Svizzera, 185 - 10149 Torino tel. +39 011 670 6836
