[slurm-dev] Re: SLRUM as a load balancer for interactive use
This isn't exactly what you're looking for but I'll chime in anyway with how we do things. We decided to buy a few slightly beefier interactive nodes and set up cgroups, /tmp and /dev/shm namespaces (/tmp and /dev/shm are per-user), cputime limits, /tmp quotas, etc to sanely oversubscribe resources. This ended up being cheaper than other options and it has worked really well. We currently use LVS to load balance between interactive nodes but may switch to something else at some point. We allow users to edit files, compile code, transfer files around, etc. and also test their code for a little while. Anything beyond that requires submitting a job. We limit users to 1/4 of the RAM on the node and only 60 CPU-minutes per process via ulimit. The cpu cgroup (not cpuset) is used to set a soft limit of just a few cores for each user but allows them to burst to 100% of the cores when there is no contention on the node. It would take at least four users all using the maximum amount of RAM plus some extra use before the node crashes. The memory per node ratio and other settings could easily be changed if necessary. In practice, these settings have made it so that no user has crashed an interactive node since everything was deployed. Obviously I didn't answer the original question about SLURM but this is an alternative approach that has worked well for us. If you're interested in the code we used to set everything up, it is available at: https://github.com/BYUHPC/uft Ryan On 03/24/2014 01:32 PM, Olli-Pekka Lehto wrote: I can foresee the screen issue as well. One could fairly simply add a check when the user logs in to see if the the user has a node assigned to them already and force the session to use that node. It could perhaps even prompt if they want to do access this session or get a new one. The immediate issue we encountered when testing with screen, however, is the fact that when you detach and exit the interactive session SLURM faithfully cleans all the processes. In most cases this would be preferred but in this case I want the screen session (and the associated interactive job) to persist. Any ideas how to do this? In our case there is no real time limit on the current interactive use nodes so setting runtime as unlimited is probably the way to go, at least initially. Of course one needs to have a sufficiently large oversubscription factor of slots in this case. Olli-Pekka On Mar 24, 2014, at 3:48 PM, Schmidtmann, Carl carl.schmidtm...@rochester.edu wrote: We considered this option as well but the problem we saw with it is what happens when a user tries to use screen? Many of our users login, start screen, do some work and then disconnect. Whenever they reconnect they can pick up from where they left off. If you are allocated to a compute node based on loads, you likely won't be on the same node where your last session was. This is inconvenient for the users but then also leaves screen sessions open, at least until the time limit expires, on compute nodes. The other issue is the time limit. Do you make it 1 hour, 4 hours, 8 hours? How long does a user get to be logged in? If the time limit expires, what happens to the open editor session? Can this be recovered on a different compute node? We are still looking for a good way to balance users on login nodes. Right now we are working on a method of redirecting ssh logins based on user IDs which feels extremely hacky as well. Carl -- Carl Schmidtmann Center for Integrated Research Computing University of Rochester On Mar 24, 2014, at 5:44 AM, Olli-Pekka Lehto wrote: Dear devs, We are testing a concept where we are dynamically allocating a portion of our compute nodes with oversubscribed interactive nodes for low-intensity use. To make the use as simple as possible, we are testing redirecting user login sessions directly to these nodes via SLURM. Basically the shell initialization on the actual login node contains a SLURM srun command to spawn an interactive session and the user gets transparently dropped into a shell session on a compute node. This would offer more flexibility than physically setting up a set of login nodes. Furthermore, SLURM should be able make better decisions on where to assign each incoming session based on resource usage than a more naive round-robin load balancer. This way also all interactive use can be tracked with SLURM's accounting. Based on simple initial testing this seems to work but it's still a bit hacky. My question is has anyone been doing similar things and what are your experiences? Are there some caveats that we should be aware of? Best regards, Olli-Pekka -- Olli-Pekka Lehto Development Manager Computing Platforms CSC - IT Center for Science Ltd. E-Mail: olli-pekka.le...@csc.fi Tel: +358 50 381 8604 skype: oplehto // twitter: ople -- Ryan Cox Operations Director Fulton Supercomputing Lab Brigham Young University
[slurm-dev] Re: SLRUM as a load balancer for interactive use
We considered this option as well but the problem we saw with it is what happens when a user tries to use screen? Many of our users login, start screen, do some work and then disconnect. Whenever they reconnect they can pick up from where they left off. If you are allocated to a compute node based on loads, you likely won't be on the same node where your last session was. This is inconvenient for the users but then also leaves screen sessions open, at least until the time limit expires, on compute nodes. The other issue is the time limit. Do you make it 1 hour, 4 hours, 8 hours? How long does a user get to be logged in? If the time limit expires, what happens to the open editor session? Can this be recovered on a different compute node? We are still looking for a good way to balance users on login nodes. Right now we are working on a method of redirecting ssh logins based on user IDs which feels extremely hacky as well. Carl -- Carl Schmidtmann Center for Integrated Research Computing University of Rochester On Mar 24, 2014, at 5:44 AM, Olli-Pekka Lehto wrote: Dear devs, We are testing a concept where we are dynamically allocating a portion of our compute nodes with oversubscribed interactive nodes for low-intensity use. To make the use as simple as possible, we are testing redirecting user login sessions directly to these nodes via SLURM. Basically the shell initialization on the actual login node contains a SLURM srun command to spawn an interactive session and the user gets transparently dropped into a shell session on a compute node. This would offer more flexibility than physically setting up a set of login nodes. Furthermore, SLURM should be able make better decisions on where to assign each incoming session based on resource usage than a more naive round-robin load balancer. This way also all interactive use can be tracked with SLURM's accounting. Based on simple initial testing this seems to work but it's still a bit hacky. My question is has anyone been doing similar things and what are your experiences? Are there some caveats that we should be aware of? Best regards, Olli-Pekka -- Olli-Pekka Lehto Development Manager Computing Platforms CSC - IT Center for Science Ltd. E-Mail: olli-pekka.le...@csc.fi Tel: +358 50 381 8604 skype: oplehto // twitter: ople
[slurm-dev] Re: SLRUM as a load balancer for interactive use
I can foresee the screen issue as well. One could fairly simply add a check when the user logs in to see if the the user has a node assigned to them already and force the session to use that node. It could perhaps even prompt if they want to do access this session or get a new one. The immediate issue we encountered when testing with screen, however, is the fact that when you detach and exit the interactive session SLURM faithfully cleans all the processes. In most cases this would be preferred but in this case I want the screen session (and the associated interactive job) to persist. Any ideas how to do this? In our case there is no real time limit on the current interactive use nodes so setting runtime as unlimited is probably the way to go, at least initially. Of course one needs to have a sufficiently large oversubscription factor of slots in this case. Olli-Pekka On Mar 24, 2014, at 3:48 PM, Schmidtmann, Carl carl.schmidtm...@rochester.edu wrote: We considered this option as well but the problem we saw with it is what happens when a user tries to use screen? Many of our users login, start screen, do some work and then disconnect. Whenever they reconnect they can pick up from where they left off. If you are allocated to a compute node based on loads, you likely won't be on the same node where your last session was. This is inconvenient for the users but then also leaves screen sessions open, at least until the time limit expires, on compute nodes. The other issue is the time limit. Do you make it 1 hour, 4 hours, 8 hours? How long does a user get to be logged in? If the time limit expires, what happens to the open editor session? Can this be recovered on a different compute node? We are still looking for a good way to balance users on login nodes. Right now we are working on a method of redirecting ssh logins based on user IDs which feels extremely hacky as well. Carl -- Carl Schmidtmann Center for Integrated Research Computing University of Rochester On Mar 24, 2014, at 5:44 AM, Olli-Pekka Lehto wrote: Dear devs, We are testing a concept where we are dynamically allocating a portion of our compute nodes with oversubscribed interactive nodes for low-intensity use. To make the use as simple as possible, we are testing redirecting user login sessions directly to these nodes via SLURM. Basically the shell initialization on the actual login node contains a SLURM srun command to spawn an interactive session and the user gets transparently dropped into a shell session on a compute node. This would offer more flexibility than physically setting up a set of login nodes. Furthermore, SLURM should be able make better decisions on where to assign each incoming session based on resource usage than a more naive round-robin load balancer. This way also all interactive use can be tracked with SLURM's accounting. Based on simple initial testing this seems to work but it's still a bit hacky. My question is has anyone been doing similar things and what are your experiences? Are there some caveats that we should be aware of? Best regards, Olli-Pekka -- Olli-Pekka Lehto Development Manager Computing Platforms CSC - IT Center for Science Ltd. E-Mail: olli-pekka.le...@csc.fi Tel: +358 50 381 8604 skype: oplehto // twitter: ople
