[slurm-dev] Re: Job ends successfully but spawned processes still run?

2017-05-23 Thread Christopher Samuel 

On 24/05/17 13:45, Lachlan Musicman wrote:

> Not yet - that's part of the next update cycle :/

Ah well that might help, along with pam_slurm_adopt so that users
SSH'ing into nodes they have jobs on are put into a cgroup of theirs on
that node.

Helps catch any legacy SSH based MPI launchers (and other naughtiness).

Good luck!
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 Melbourne Bioinformatics - The University of Melbourne
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545

[slurm-dev] Re: Job ends successfully but spawned processes still run?

2017-05-23 Thread Lachlan Musicman 
On 24 May 2017 at 13:18, Christopher Samuel  wrote:

>
> Hiya,
>
> On 24/05/17 13:10, Lachlan Musicman wrote:
>
> > Occasionally I'll see a bunch of processes "running" (sleeping) on a
> > node well after the job they are associated with has finished.
> >
> > How does this happen - does slurm not make sure all processes spawned by
> > a job have finished at completion?
>
> Are you not using cgroups for enforcement?
>
> Usually that picks everything up.
>
>

Not yet - that's part of the next update cycle :/

L.

--
"Mission Statement: To provide hope and inspiration for collective action,
to build collective power, to achieve collective transformation, rooted in
grief and rage but pointed towards vision and dreams."

 - Patrice Cullors, *Black Lives Matter founder*

[slurm-dev] Re: Job ends successfully but spawned processes still run?

2017-05-23 Thread Christopher Samuel 

Hiya,

On 24/05/17 13:10, Lachlan Musicman wrote:

> Occasionally I'll see a bunch of processes "running" (sleeping) on a
> node well after the job they are associated with has finished.
> 
> How does this happen - does slurm not make sure all processes spawned by
> a job have finished at completion?

Are you not using cgroups for enforcement?

Usually that picks everything up.

cheers,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 Melbourne Bioinformatics - The University of Melbourne
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545