[slurm-dev] Segfault when using accounting_storage/filetxt in 15.08.0-0pre4

[Eric Lund]

Folks,

I have a reasonably up to date version of the master branch (builds as 
15.08.0-0pre4) with some of my own changes in it, and I am getting a 
segfault with the following backtrace when I try to start a job:


#0  0x7fffbf9c53a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x004e8221 in list_find_first (l=0x0, f=0x56f5d7 
, key=0x7fffb70e86bc) at list.c:466
#2  0x0056f0e4 in slurmdb_make_tres_string_from_simple 
(tres_in=0x7fffa40054b0 "1=24,2=0", full_tres_list=0x0) at 
slurmdb_defs.c:2938
#3  0x00457b2c in job_set_tres (job_ptr=0x7fffa4002a40) at 
job_mgr.c:7123
#4  0x004888ac in select_nodes (job_ptr=0x7fffa4002a40, 
test_only=false, select_node_bitmap=0x0, err_msg=0x7fffb70e8c90) at 
node_scheduler.c:1988
#5  0x0044f0ae in _select_nodes_parts (job_ptr=0x7fffa4002a40, 
test_only=false, select_node_bitmap=0x0, err_msg=0x7fffb70e8c90) at 
job_mgr.c:3855
#6  0x0044f698 in job_allocate (job_specs=0x7fffa40017a0, 
immediate=0, will_run=0, resp=0x0, allocate=1, submit_uid=10931, 
job_pptr=0x7fffb70e8c98, err_msg=0x7fffb70e8c90, protocol_version=7424)

at job_mgr.c:4017
#7  0x00497fd2 in _slurm_rpc_allocate_resources 
(msg=0x7fffa40008d0) at proc_req.c:1047
#8  0x004958ba in slurmctld_req (msg=0x7fffa40008d0, 
arg=0x7fffb8f0) at proc_req.c:262
#9  0x0043a4db in _service_connection (arg=0x7fffb8f0) at 
controller.c:1086

#10 0x7fffbf9c39d1 in start_thread () from /lib64/libpthread.so.0
#11 0x7fffbf710b7d in clone () from /lib64/libc.so.6

The immediate cause of the failure is that list_find_first() is trying 
to lock the list 'l' which is a NULL pointer (not sure why the asserts() 
in list_find_first() don't fire, maybe they are compiled out?) using a 
field in the list structure.


The reason that 'l' is a NULL pointer appears to be that it comes from 
'assoc_mgr_tres_list' which is set by calling acct_storage_p_get_tres() 
when accounting_storage/filetxt is in use.


The thing is, acct_storage_p_get_tres() returns NULL, which seems to be 
perfectly acceptable up until the code segfaults, and 
accounting_storage/filetxt seemed to work fine when I was running 
15.08.0-0pre3 with my same changes in place...


None of this has anything to do with any of the changes I made.  Is this 
a known problem with some kind of workaround, or should I file a bug on it?


Eric

[slurm-dev] using linux username as default account

[Jordan Willis]

Hello, 

Since we only have a few users on our cluster, I’m only using 
accounting_storage/filetxt. When I submit jobs and look at sacct, the account 
says (null). If I provide an account using sbatch/srun. It will list this in 
sacct. Is there a way to default the linux username as the account in filetxt?

Also, since this string is arbitrary and you can list anything you want, is 
there no notion of fair share using the filetxt format? It is highly 
recommended that we start using the mysql or slurmdb plugins?


Jordan

[slurm-dev] Re: old version of scontrol, sacct keeps resurrecting

[Andrew Petersen]

I was able to narrow down my problem further.  I managed to stop all slurm
daemons on head and slave nodes.  On head node, if I start the new slurm:
/.../new-slurm/etc/init.d/slurm start
the slurmctld daemon runs ok, but with error messages in the
/var/log/slurmctld file about Incompatible versions.  If I do "ps -u root
-F |grep slurm", I can see why.  Periodically, the /../old-slurm/scontrol
and squeue run.  It seems like somehow, the "new-slurm/etc/init.d/slurm
start" is triggering these old daemons to run.  I check the path in my
$PATH, slurm.conf, and /../new-slurm/etc/init.d/slurm file, and there is
nothing like that.  I would appreciate any guidance on resolving this.

Regards
Andrew

On Wed, Jun 3, 2015 at 4:52 PM, Andrew Petersen  wrote:

> Hello
>
> I installed a new version of slurm, 14.11.3.  It works fine.  However I
> noticed that my log file /var/log/slurmctld shows
> error: slurm_receive_msg: Incompatible versions of client and server code
> This led me to discover that old slurm scontrol, squeue and sacct are
> still running on the head node, using
> ps -u root -F |grep slurm
>
> I have tried to kill this every which way, but they wont die, they keep
> resurrecting with different pid's.  I tried
>  /old-slurm-version/bin/scontrol shutdown
> but this gives me
> slurm_shutdown error: Zero Bytes were transmitted or received
>
> It seems like something is automatically restarting the old slurm.  I am
> using Bright Cluster Manager, and I set it so that it does NOT auto-start
> or run the slurm daemon, but that did not help.
>
> Can someone help me kill this thing?  It is causing the creation of big
> log zip files, and using up cpu capacity on the head node.
>
> Regards
> Andrew Petersen
>
>

[slurm-dev] old version of scontrol, sacct keeps resurrecting

[Andrew Petersen]

Hello

I installed a new version of slurm, 14.11.3.  It works fine.  However I
noticed that my log file /var/log/slurmctld shows
error: slurm_receive_msg: Incompatible versions of client and server code
This led me to discover that old slurm scontrol, squeue and sacct are still
running on the head node, using
ps -u root -F |grep slurm

I have tried to kill this every which way, but they wont die, they keep
resurrecting with different pid's.  I tried
 /old-slurm-version/bin/scontrol shutdown
but this gives me
slurm_shutdown error: Zero Bytes were transmitted or received

It seems like something is automatically restarting the old slurm.  I am
using Bright Cluster Manager, and I set it so that it does NOT auto-start
or run the slurm daemon, but that did not help.

Can someone help me kill this thing?  It is causing the creation of big log
zip files, and using up cpu capacity on the head node.

Regards
Andrew Petersen

[slurm-dev] Re: FAIR_TREE in SLURM 14.11

[Ryan Cox]

Trey,

In http://slurm.schedmd.com/fair_tree.html#fairshare, take a look at the 
definition for "S".  Basically, the normalized shares only matters 
between sibling associations and will equal 1.0 when summed.  If an 
association has no siblings, the value is 1.0.  If each of the four 
siblings in an account has the same Raw Shares (as defined in sacctmgr) 
value, the normalized shares value for each is 0.25.  The reason why is 
because the Level Fairshare calculations are only done within in 
account, comparing siblings to each other. Note that Norm Usage is still 
presented in sshare but not used in the calculations.


The sshare manpage has a section about the Fair Tree modifications to 
existing columns: 
http://slurm.schedmd.com/sshare.html#SECTION_FAIR_TREE%20MODIFICATIONS


Ryan

On 06/03/2015 02:47 PM, Trey Dockendorf wrote:

FAIR_TREE in SLURM 14.11
My site is currently on 14.03.10 and we are evaluating and testing 
14.11.7 as well as moving from 
PriorityFlags=DEPTH_OBLIVIOUS,SMALL_RELATIVE_TO_TIME to using 
PriorityFlags=FAIR_TREE,SMALL_RELATIVE_TO_TIME.


Our account hierarchy is very deep and is intended to represent the 
org structure of departments and research organizations that are using 
our cluster [1].  We were able to make the normalized share ratio 
match up so all non-stakeholders were equal (0.000323) and all 
stakeholders had the correct ratio based on their contributions to the 
cluster.  The Shares value assigned represents CPUs funded. All the 
CPUs no longer belonging to stakeholders were given to the "mgmt" 
group so that the Shares given to the top level (tamu) had a 
meaningful value when divided up amongst all the accounts.


While testing FAIR_TREE I noticed the normalized shares were 
drastically different [2].  In particular the current stakeholders 
(idhcm and hepx) both ended up with 1.0.  I'm guessing this is due to 
having no sibling accounts.


The docs for FAIR_TREE only describe the formula used to calculate the 
Level FairShare.  Does the method for calculating normalized shares 
change for FAIR_TREE? Is the hierarchy we are using not a good fit for 
FAIR_TREE?  The description and benefits of FAIR_TREE appeal to our 
use case, so modifying our hierarchy is within the realm of things I'm 
willing to change.


Any advice on migrating into FAIR_TREE is more than welcome.  Right 
now I've been running "sleep" jobs under different UIDs to simulate 
usage to try and work out how we may need to adjust things for a 
migration to FAIR_TREE.


I used the attached spreadsheet to work out the share values we are 
using with 14.03.10.


Thanks,
- Trey

[1]:
 Account User Raw Shares Norm Shares   Raw Usage 
Effectv Usage  FairShare
 -- -- --- --- 
- --

root1.00   114089982  1.00   0.870551
 rootroot  10.000323   0  0.00 1.00
 grid  10.0003233688  0.32 0.986174
  cms 100.0002693688  0.27 0.986155
  suragrid   10.27   0  0.00   
1.00

 tamu   30960.999354   114086294  0.68 0.870477
  agriculture 200.0066712697  0.24 
0.999507
   aglife  10.0033362697  0.12 
0.999507
   genomics  10.003336   0  0.00 
1.00
  engineering 100.003336   0  0.00 
1.00

   pete  10.003336   0  0.00 1.00
  general 100.0033365542  0.49 
0.997977

  geo 100.003336   2  0.00 0.99
   atmo  10.003336   2  0.00 0.99
  liberalarts1280.042696   0  0.00 
1.00
   idhmc   10.042696   0  0.00   
1.00
  mgmt20580.686472  16  0.00   
1.00
  science7600.253508   114078034  0.999895 
0.578806

   acad 100.003336   0  0.00 1.00
   chem 100.003336   0  0.00 1.00
   iamcs  100.003336 3506549  0.030649   
0.279777
   math-dept  200.00667111735411  0.102422 
  0.119035
math  100.00333611735411  0.102422   
0.014169
secant  100.003336   0  0.00   
1.00
   physics 7000.23349498836073  0.919795   
0.579205
hepx 7000.23349498836073  0.919795   
0.579205

   stat 100.003336   0  0.00 1.00
carroll 100.003336   0  0.00 
1.00


[2]:
  

[slurm-dev] Re: Slurm and docker/containers

[Christopher Samuel]

On 20/05/15 03:15, Kilian Cavalotti wrote:

> One major downside to running Docker containers in a shared HPC
> cluster (to me at least), is that the default user in a container is
> root.

One thing that has occurred to me is that the whole point of containers
is that they are using the kernel namespace features and so whilst the
user inside the container is root that is only inside their own user
namespace, that does not (should not!) correspond to root on the host
itself (there's a mapping file to determine who they are mapped to).

There is an excellent LWN article on user namespaces here:

https://lwn.net/Articles/532593/

Your filesystem has to support it though, otherwise it looks like you'll
get EINVAL back - as this comment from a user who was trying it on
filesystems not yet ported to  it reports:

https://lwn.net/Articles/541787/

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: Segfault when using accounting_storage/filetxt in 15.08.0-0pre4

[Eric Lund]

I should note, acct_storage_p_get_tres() _always_ returns NULL for 
accounting_storage/filetxt, so it is not just doing that in this case.


Eric

On 6/4/15 11:43 AM, Eric Lund wrote:


Folks,

I have a reasonably up to date version of the master branch (builds as
15.08.0-0pre4) with some of my own changes in it, and I am getting a
segfault with the following backtrace when I try to start a job:

#0  0x7fffbf9c53a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x004e8221 in list_find_first (l=0x0, f=0x56f5d7
, key=0x7fffb70e86bc) at list.c:466
#2  0x0056f0e4 in slurmdb_make_tres_string_from_simple
(tres_in=0x7fffa40054b0 "1=24,2=0", full_tres_list=0x0) at
slurmdb_defs.c:2938
#3  0x00457b2c in job_set_tres (job_ptr=0x7fffa4002a40) at
job_mgr.c:7123
#4  0x004888ac in select_nodes (job_ptr=0x7fffa4002a40,
test_only=false, select_node_bitmap=0x0, err_msg=0x7fffb70e8c90) at
node_scheduler.c:1988
#5  0x0044f0ae in _select_nodes_parts (job_ptr=0x7fffa4002a40,
test_only=false, select_node_bitmap=0x0, err_msg=0x7fffb70e8c90) at
job_mgr.c:3855
#6  0x0044f698 in job_allocate (job_specs=0x7fffa40017a0,
immediate=0, will_run=0, resp=0x0, allocate=1, submit_uid=10931,
job_pptr=0x7fffb70e8c98, err_msg=0x7fffb70e8c90, protocol_version=7424)
 at job_mgr.c:4017
#7  0x00497fd2 in _slurm_rpc_allocate_resources
(msg=0x7fffa40008d0) at proc_req.c:1047
#8  0x004958ba in slurmctld_req (msg=0x7fffa40008d0,
arg=0x7fffb8f0) at proc_req.c:262
#9  0x0043a4db in _service_connection (arg=0x7fffb8f0) at
controller.c:1086
#10 0x7fffbf9c39d1 in start_thread () from /lib64/libpthread.so.0
#11 0x7fffbf710b7d in clone () from /lib64/libc.so.6

The immediate cause of the failure is that list_find_first() is trying
to lock the list 'l' which is a NULL pointer (not sure why the asserts()
in list_find_first() don't fire, maybe they are compiled out?) using a
field in the list structure.

The reason that 'l' is a NULL pointer appears to be that it comes from
'assoc_mgr_tres_list' which is set by calling acct_storage_p_get_tres()
when accounting_storage/filetxt is in use.

The thing is, acct_storage_p_get_tres() returns NULL, which seems to be
perfectly acceptable up until the code segfaults, and
accounting_storage/filetxt seemed to work fine when I was running
15.08.0-0pre3 with my same changes in place...

None of this has anything to do with any of the changes I made.  Is this
a known problem with some kind of workaround, or should I file a bug on it?

Eric

[slurm-dev] Re: Segfault when using accounting_storage/filetxt in 15.08.0-0pre4

[David Bigagli]

Hi,
   please file a bug and append the diffs of your changes.

On 06/04/2015 09:43 AM, Eric Lund wrote:


Folks,

I have a reasonably up to date version of the master branch (builds as
15.08.0-0pre4) with some of my own changes in it, and I am getting a
segfault with the following backtrace when I try to start a job:

#0  0x7fffbf9c53a0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1  0x004e8221 in list_find_first (l=0x0, f=0x56f5d7
, key=0x7fffb70e86bc) at list.c:466
#2  0x0056f0e4 in slurmdb_make_tres_string_from_simple
(tres_in=0x7fffa40054b0 "1=24,2=0", full_tres_list=0x0) at
slurmdb_defs.c:2938
#3  0x00457b2c in job_set_tres (job_ptr=0x7fffa4002a40) at
job_mgr.c:7123
#4  0x004888ac in select_nodes (job_ptr=0x7fffa4002a40,
test_only=false, select_node_bitmap=0x0, err_msg=0x7fffb70e8c90) at
node_scheduler.c:1988
#5  0x0044f0ae in _select_nodes_parts (job_ptr=0x7fffa4002a40,
test_only=false, select_node_bitmap=0x0, err_msg=0x7fffb70e8c90) at
job_mgr.c:3855
#6  0x0044f698 in job_allocate (job_specs=0x7fffa40017a0,
immediate=0, will_run=0, resp=0x0, allocate=1, submit_uid=10931,
job_pptr=0x7fffb70e8c98, err_msg=0x7fffb70e8c90, protocol_version=7424)
 at job_mgr.c:4017
#7  0x00497fd2 in _slurm_rpc_allocate_resources
(msg=0x7fffa40008d0) at proc_req.c:1047
#8  0x004958ba in slurmctld_req (msg=0x7fffa40008d0,
arg=0x7fffb8f0) at proc_req.c:262
#9  0x0043a4db in _service_connection (arg=0x7fffb8f0) at
controller.c:1086
#10 0x7fffbf9c39d1 in start_thread () from /lib64/libpthread.so.0
#11 0x7fffbf710b7d in clone () from /lib64/libc.so.6

The immediate cause of the failure is that list_find_first() is trying
to lock the list 'l' which is a NULL pointer (not sure why the asserts()
in list_find_first() don't fire, maybe they are compiled out?) using a
field in the list structure.

The reason that 'l' is a NULL pointer appears to be that it comes from
'assoc_mgr_tres_list' which is set by calling acct_storage_p_get_tres()
when accounting_storage/filetxt is in use.

The thing is, acct_storage_p_get_tres() returns NULL, which seems to be
perfectly acceptable up until the code segfaults, and
accounting_storage/filetxt seemed to work fine when I was running
15.08.0-0pre3 with my same changes in place...

None of this has anything to do with any of the changes I made.  Is this
a known problem with some kind of workaround, or should I file a bug on it?

Eric


--

Thanks,
  /David/Bigagli

www.schedmd.com

[slurm-dev] using linux username as default account

[Jordan Willis]

Hello, 

Since we only have a few users on our cluster, I’m only using 
accounting_storage/filetxt. When I submit jobs and look at sacct, the account 
says (null). If I provide an account using sbatch/srun. It will list this in 
sacct. Is there a way to default the linux username as the account in filetxt?

Also, since this string is arbitrary and you can list anything you want, is 
there no notion of fair share using the filetxt format? It is highly 
recommended that we start using the mysql or slurmdb plugins?


Jordan

[slurm-dev] Re: Slurm 14.11.7 no more "exceeded memory" error?

[Will French]

The “slurmstepd: error: Exceeded step memory limit at some point. Step may have 
been partially swapped out to disk.” message was removed in 14.11.7 because 
users found this message to be misleading when jobs were not subsequently 
killed. I personally prefer to see this message because (1) it makes 
troubleshooting much easier if the job is eventually killed (MaxRSS is not 
always super accurate) and (2) even if the job is not killed you’re made aware 
of swap usage and the potential for performance degradation in your job.

http://bugs.schedmd.com/show_bug.cgi?id=1682

Will
 

On Jun 3, 2015, at 3:48 PM, Michael Kit Gilbert  wrote:

> Hello everyone,
> 
> We just upgraded to Slurm 14.11.7 recently and have noticed an issue with 
> jobs that exceed their specified memory limit. Instead of the familiar 
> "Exceeded step memory limit..." error that we normally see, we are now seeing 
> that the job simply was killed.
> 
> Here is a job script I ran to reproduce the issue, which previously was found 
> to need around 500MB of memory:
> 
> #!/bin/bash
> #SBATCH -o /home/mkg52/mpi_test.%J.out
> #SBATCH --workdir=/home/mkg52/namd_testing
> #SBATCH --ntasks=8
> #SBATCH --mem=200
> 
> module load namd/2.10
> 
> srun namd2 apoa1/apoa1.namd
> -
> 
> Here is the end of the output file produced by the job:
> -
> srun: error: cn4: task 6: Killed
> srun: error: cn4: task 7: Killed
> srun: error: cn4: task 1: Killed
> srun: Force Terminated job step 130.0
> -
> 
> Does anyone have any insight into why we aren't seeing memory-related errors 
> in the output like we used to?
> 
> Thanks a bunch for any help,
> 
> Mike

[slurm-dev] Re: using linux username as default account

[Bruce Roberts]

Highly recommended using a SlurmDBD with mysql as the backend.  The 
setup is fairly easy and will give you what you are looking for. filetxt 
is very simple and only does simple accounting for jobs/steps.  It 
doesn't do any access control or such.


On 06/04/15 09:55, Jordan Willis wrote:

Hello,

Since we only have a few users on our cluster, I’m only using 
accounting_storage/filetxt. When I submit jobs and look at sacct, the account 
says (null). If I provide an account using sbatch/srun. It will list this in 
sacct. Is there a way to default the linux username as the account in filetxt?

Also, since this string is arbitrary and you can list anything you want, is 
there no notion of fair share using the filetxt format? It is highly 
recommended that we start using the mysql or slurmdb plugins?


Jordan

[slurm-dev] Re: Slurm 14.11.7 no more "exceeded memory" error?

[Michael Kit Gilbert]

Will,

Thank you for the explanation and that link. I'll be sure to check for bug
reports before posting questions like this in the future. I do agree with
your thoughts about the logging of the message. We've gotten several
instances of users asking what the "killed" errors meant, when in the past,
they were able to solve the issue themselves due to Slurm's "memory
exceeded" message.


On Thu, Jun 4, 2015 at 9:56 AM, Will French 
wrote:

>
> The “slurmstepd: error: Exceeded step memory limit at some point. Step may
> have been partially swapped out to disk.” message was removed in 14.11.7
> because users found this message to be misleading when jobs were not
> subsequently killed. I personally prefer to see this message because (1) it
> makes troubleshooting much easier if the job is eventually killed (MaxRSS
> is not always super accurate) and (2) even if the job is not killed you’re
> made aware of swap usage and the potential for performance degradation in
> your job.
>
> http://bugs.schedmd.com/show_bug.cgi?id=1682
>
> Will
>
>
> On Jun 3, 2015, at 3:48 PM, Michael Kit Gilbert  wrote:
>
> > Hello everyone,
> >
> > We just upgraded to Slurm 14.11.7 recently and have noticed an issue
> with jobs that exceed their specified memory limit. Instead of the familiar
> "Exceeded step memory limit..." error that we normally see, we are now
> seeing that the job simply was killed.
> >
> > Here is a job script I ran to reproduce the issue, which previously was
> found to need around 500MB of memory:
> >
> 
> > #!/bin/bash
> > #SBATCH -o /home/mkg52/mpi_test.%J.out
> > #SBATCH --workdir=/home/mkg52/namd_testing
> > #SBATCH --ntasks=8
> > #SBATCH --mem=200
> >
> > module load namd/2.10
> >
> > srun namd2 apoa1/apoa1.namd
> >
> -
> >
> > Here is the end of the output file produced by the job:
> >
> -
> > srun: error: cn4: task 6: Killed
> > srun: error: cn4: task 7: Killed
> > srun: error: cn4: task 1: Killed
> > srun: Force Terminated job step 130.0
> >
> -
> >
> > Does anyone have any insight into why we aren't seeing memory-related
> errors in the output like we used to?
> >
> > Thanks a bunch for any help,
> >
> > Mike
>

[slurm-dev] Re: Slurm and docker/containers

[Michael Jennings]

On Thu, Jun 4, 2015 at 9:52 AM, Christopher Samuel
 wrote:

> One thing that has occurred to me is that the whole point of containers
> is that they are using the kernel namespace features and so whilst the
> user inside the container is root that is only inside their own user
> namespace, that does not (should not!) correspond to root on the host
> itself (there's a mapping file to determine who they are mapped to).

While strictly true, depending on what elements are provided/allowed
within the container, host-level root can be obtained even within the
protected namespace.  Device access, for example, or write access to
/proc or /sys.  There are messages fairly regularly on the Docker
mailing list from the Docker Inc. security team which advise folks not
to rely on container security to protect the host from container-root.

root access also gets interesting when you factor in networking and NFS.  :-)

The real solution is for Docker to facilitate execution of containers
as non-root, but it's not clear how far down on their roadmap it
is

Michael

-- 
Michael Jennings 
Senior HPC Systems Engineer
High-Performance Computing Services
Lawrence Berkeley National Laboratory
Bldg 50B-3209EW: 510-495-2687
MS 050B-3209  F: 510-486-8615

[slurm-dev] Re: old version of scontrol, sacct keeps resurrecting

[Andy Riebs]

   Having never seen your system, here are a couple of shots in the
 dark:
 
 1. Do you have a cron job that might be starting Slurm, perhaps for
 a simple failure-restart solution?
 2. If "ps --forest" works on your system, it may suggest that some
 other process is responsible for running the old Slurm tools and
 daemon.
 
 Andy
 On 06/04/2015 12:49 PM, Andrew Petersen
   wrote:
   Re: old version of scontrol, sacct keeps resurrecting
 I was able to narrow down my problem further.  I
   managed to stop all slurm daemons on head and slave
   nodes.  On head node, if I start the new slurm:
 
 /.../new-slurm/etc/init.d/slurm start
   
   the slurmctld daemon runs ok, but with error messages in
   the /var/log/slurmctld file about Incompatible versions. 
   If I do "ps -u root -F |grep slurm", I can see why. 
   Periodically, the /../old-slurm/scontrol and squeue run. 
   It seems like somehow, the "new-slurm/etc/init.d/slurm
   start" is triggering these old daemons to run.  I check
   the path in my $PATH, slurm.conf, and
   /../new-slurm/etc/init.d/slurm file, and there is nothing
   like that.  I would appreciate any guidance on resolving
   this.
   Regards 
 
 Andrew
 On Wed, Jun 3, 2015 at 4:52 PM, Andrew
   Petersen 
   wrote:
 Hello
 I installed a new version of slurm,
 14.11.3.  It works fine.  However I noticed
 that my log file /var/log/slurmctld shows
 error: slurm_receive_msg: Incompatible
 versions of client and server code
   
   This led me to discover that old slurm
   scontrol, squeue and sacct are still running
   on the head node, using
   ps -u root -F |grep slurm
 I have tried to kill this every which way, but
 they wont die, they keep resurrecting with
 different pid's.  I tried
  /old-slurm-version/bin/scontrol shutdown
   
   but this gives me
   slurm_shutdown error: Zero Bytes were transmitted
   or received
 It seems like something is automatically restarting
 the old slurm.  I am using Bright Cluster Manager,
 and I set it so that it does NOT auto-start or run
 the slurm daemon, but that did not help.
   Can someone help me kill this thing?  It is causing
   the creation of big log zip files, and using up cpu
   capacity on the head node.
 Regards 
   
   Andrew Petersen

[slurm-dev] Re: FAIR_TREE in SLURM 14.11

[Trey Dockendorf]

Ryan,

Thanks, I think that clarification plus the slide I found here,
http://slurm.schedmd.com/SUG14/fair_tree.pdf, offered the insight I
needed.  With the deep hierarchy the goal is that a deeply nested account
be considered a sibling with an account elsewhere in the tree.  The key was
setting FairShare=parent for the accounts that contain no users and that
are only used for organizational purposes.  Once I made that change the
normalized shares of accounts that contain users looked correct.  The basic
idea is such that all accounts containing users end up being considered
siblings under the root account.  Using FairShare=parent achieves this
perfectly in 14.11.

The only issue I'm seeing may be a non-issue but could use some
clarification for better understanding.  I ran 100 5-minute jobs as
"test-idhmc" (stakeholder) and "test-hepx" (stakeholder) and 50 5-minute
jobs as "test-general" (non-stakeholder).  I looked at the sshare -l output
for these 3 users and noticed while the FairShare values are correctly
ordered such that test-hepx > test-idhmc > test-general, the ratio is not
proportional for test-hepx vs test-idhmc.  The hepx account has 700 shares
while the idhmc account has 128 shares.  The ratio for the account's
normalized shares is correct, and so is the Level FS of the account.  In
this case the calculated FairShare for test-hepx is not much higher than
test-idhmc but it's still greater so in the end I guess that's what matters.

Thanks,
- Trey

# sshare -u test-hepx,test-idhmc,test-general -l
 Account   User Raw Shares Norm Shares   Raw Usage  Norm
Usage Effectv Usage  FairShare   Level FS  GrpCPUMins  CPURunMins
 -- -- --- ---
--- - -- -- --- ---
root  0.00   11217
 1.00 0
 grid   parent0.00   0
 0.00  0.00 0
  cms   100.003315   0
 0.00  0.00   inf   0
  suragrid  100.003315   0
 0.00  0.00   inf   0
 tamu   parent0.00   11217
 1.00  1.00 0
  agriculture   parent0.00   0
 0.00  0.00 0
   aglife   100.003315   0
 0.00  0.00   inf   0
   genomics 100.003315   0
 0.00  0.00   inf   0
  engineering   parent0.00   0
 0.00  0.00 0
   pete 100.003315   0
 0.00  0.00   inf   0
  general   100.0033152262
 0.201728  0.201728  0.016431   0
   general test-general  10.0078742262
 0.201728  1.00   0.001572   0.007874   0
  geo   parent0.00   0
 0.00  0.00 0
   atmo 100.003315   0
 0.00  0.00   inf   0
  liberalarts   parent0.004507
 0.401794  0.401794 0
   idhmc   1280.0424264507
 0.401794  0.401794  0.105592   0
idhmc  test-idhmc10.0625004507
 0.401794  1.00   0.201258   0.062500   0
  mgmt20580.682135   0
 0.00  0.00   inf   0
  science   parent0.004447
 0.396478  0.396478 0
   acad 100.003315   0
 0.00  0.00   inf   0
   chem 100.003315   0
 0.00  0.00   inf   0
   iamcs100.003315   0
 0.00  0.00   inf   0
   math-deptparent0.00   0
 0.00  0.

[slurm-dev] How to unsubscribe?

[Troy Baer]

How does one unsubscribe from this mailing list?  I have changed jobs 
and need to remove my old email address from the list.


http://slurm.schedmd.com/mail.html asserts the following:

   The slurm-dev@schedmd.com  mailing
   list is a good mechanism for the user community to share
   information. Visit the link below, sign up and get involved!

   http://lists.schedmd.com/cgi-bin/dada/mail.cgi/list

   You can likewise unsubscribe from either list at the same link.


However, http://lists.schedmd.com/cgi-bin/dada/mail.cgi/list has no 
unsubscribe function that I can find, and I can't find anything in my 
back email describing how to get off the list either.  What am I missing?


Thanks,
--Troy

[slurm-dev] Re: Slurm and docker/containers

[Michael Jennings]

On Sunday, 31 May 2015, at 17:27:44 (-0700),
Christopher Samuel wrote:

> Sorry for being absent for a while after starting this thread, pressures
> of work.
> 
> Michael hit the nail on the head for me there.
> 
> The security side of things is an issue, though I'm not sure how much
> the fact that the program is running in a separate UID namespace helps,
> presumably if you've got to give it HPC filesystem access then the
> answer is probably "not at all".
> 
> One of my concerns has always been that as these images age without
> updates then their exposure to known security bugs increases.
> 
> That seems to be born out by this recent survey:
> 
> http://www.banyanops.com/blog/analyzing-docker-hub/
> 
> # Over 30% of Official Images in Docker Hub Contain High Priority
> # Security Vulnerabilities
> #
> # [...] Surprisingly, we found that more than 30% of images in
> # official repositories are highly susceptible to a variety of
> # security attacks (e.g., Shellshock, Heartbleed, Poodle, etc.).
> # For general images ??? images pushed by docker users, but not
> # explicitly verified by any authority ??? this number jumps up
> # to ~40% with a sampling error bound of 3%. [...]
> 
> If anything that puts me off liking them even more. :-(

One option we've considered is insisting that users provide us with
their Dockerfiles which we would then audit, build, and manage
ourselves.  Then users would be allowed to run only in containers
based on curated images.  Could potentially make for a management
nightmare, but solves a lot of security considerations by sheer
avoidance.

My team had a very (!!) productive and interesting discussion yesterday
with some folks who have succeeded in integrating Docker and SLURM to
the point that users can specify Docker repositories in which they
want SLURM to run their jobs, and SLURM will do so.  I'm not sure how
much I'm at liberty to say at this point, but my understanding is that
they have submitted an abstract for SUG15.  So stay tuned!  :-)

Michael

-- 
Michael Jennings 
Linux Systems and Cluster Engineer
High-Performance Computing Services
Lawrence Berkeley National Laboratory

[slurm-dev] Re: FAIR_TREE in SLURM 14.11

[Ryan Cox]

Trey,

The reason why the Fairshare values are not proportionally different is 
because they are essentially that user's ranking compared to other 
users, divided by the total number of user associations.  The fairshare 
value for a user is just (rank-- / user_assoc_count) where rank is 
initialized to user_assoc_count.  That's why it's not proportional for 
the fairshare value.  There may be some other ways to do this but the 
ranking method guarantees proper ordering and eliminates any worries 
over floating point issues but, as you observed, it is not proportional.


http://slurm.schedmd.com/fair_tree.html#algorithm and page 36 onward in 
the presentation you linked to describe it a little more, especially the 
"animation" that starts on page 38.  The numbers that pop up in the 
bottom left when visiting a user show the final fairshare calculation.


Ryan

On 06/04/2015 11:38 AM, Trey Dockendorf wrote:

Re: [slurm-dev] Re: FAIR_TREE in SLURM 14.11
Ryan,

Thanks, I think that clarification plus the slide I found here, 
http://slurm.schedmd.com/SUG14/fair_tree.pdf, offered the insight I 
needed.  With the deep hierarchy the goal is that a deeply nested 
account be considered a sibling with an account elsewhere in the 
tree.  The key was setting FairShare=parent for the accounts that 
contain no users and that are only used for organizational purposes.  
Once I made that change the normalized shares of accounts that contain 
users looked correct.  The basic idea is such that all accounts 
containing users end up being considered siblings under the root 
account.  Using FairShare=parent achieves this perfectly in 14.11.


The only issue I'm seeing may be a non-issue but could use some 
clarification for better understanding.  I ran 100 5-minute jobs as 
"test-idhmc" (stakeholder) and "test-hepx" (stakeholder) and 50 
5-minute jobs as "test-general" (non-stakeholder).  I looked at the 
sshare -l output for these 3 users and noticed while the FairShare 
values are correctly ordered such that test-hepx > test-idhmc > 
test-general, the ratio is not proportional for test-hepx vs 
test-idhmc. The hepx account has 700 shares while the idhmc account 
has 128 shares.  The ratio for the account's normalized shares is 
correct, and so is the Level FS of the account.  In this case the 
calculated FairShare for test-hepx is not much higher than test-idhmc 
but it's still greater so in the end I guess that's what matters.


Thanks,
- Trey

# sshare -u test-hepx,test-idhmc,test-general -l
 Account User Raw Shares Norm Shares   Raw Usage  Norm 
Usage Effectv Usage  FairShare   Level FS  GrpCPUMins  CPURunMins
 -- -- --- --- 
--- - -- -- --- 
---

root0.00   112171.00   0
 grid parent0.00   00.00  0.00 0
  cms 100.003315   00.00  0.00 
  inf 0
  suragrid  100.003315   00.00   
 0.00   inf   0

 tamu parent0.00   112171.00  1.00 0
  agriculture parent0.00   00.00 
 0.00 0
   aglife 100.003315   00.00 
 0.00   inf 0
   genomics 100.003315   00.00 
 0.00   inf 0
  engineering parent0.00   00.00 
 0.00 0
   pete 100.003315   00.00 
 0.00   inf 0
  general 100.00331522620.201728 
 0.201728  0.016431 0
   general test-general  10.00787422620.201728   
 1.00   0.001572   0.007874   0

  geo parent0.00   00.00  0.00 0
   atmo 100.003315   00.00 
 0.00   inf 0
  liberalarts parent0.0045070.401794 
 0.401794 0
   idhmc 1280.04242645070.401794   
 0.401794  0.105592   0
idhmc  test-idhmc10.06250045070.401794   
 1.00   0.201258   0.062500   0
  mgmt20580.682135   00.00   
 0.00   inf   0
  science parent0.0044470.396478 
 0.396478 0
   acad 100.003315   00.00 
 0.00   inf 0
   chem 100.003315   00.00 
 0.00   inf 0
   iamcs  100.003315   00.00   
 0.00   inf   0
   math-dept  parent0.00   00.00   
 0.00   0
math  100.003315   00.00   
 0.00

[slurm-dev] Re: How to unsubscribe?

[Christopher Samuel]

Hi Troy,

On 05/06/15 04:37, Troy Baer wrote:
> 
> How does one unsubscribe from this mailing list?  I have changed jobs
> and need to remove my old email address from the list.

You need to go to:

http://lists.schedmd.com/cgi-bin/dada/mail.cgi/profile_login/

reset your password (from memory) and then you can login to unsubscribe.
 No direct link from what I can see sorry!

I think Danny, Moe, David et. al monitor the list and unsubscribe people
who ask here, so it might have already happened for you (hence the CC).

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: Slurm and docker/containers

[Christopher Samuel]

On 01/06/15 13:24, Ralph Castain wrote:

> I sympathize with the problem. In addition, although I am not a lawyer,
> it is my understanding that Docker’s license is incompatible with
> Slurm’s GPL, and thus you cannot distribute such an integration.

Looks like you're right, both Docker and CoreOS's rkt are Apache2. :-(

https://www.apache.org/licenses/GPL-compatibility.html

Oh well, makes the whole thing academic really for us.

Thanks for pointing that out Ralph!

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: Slurm and docker/containers

[Christopher Samuel]

On 05/06/15 06:59, Michael Jennings wrote:

> My team had a very (!!) productive and interesting discussion yesterday
> with some folks who have succeeded in integrating Docker and SLURM to
> the point that users can specify Docker repositories in which they
> want SLURM to run their jobs, and SLURM will do so.

How did they deal with the license incompatibility that Ralph mentioned?

All the best
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: Slurm and docker/containers

[Danny Auble]

Were they linking to slurm or just calling commands? Unless they link there 
shouldn't be any license issues. 

On June 4, 2015 4:40:14 PM PDT, Christopher Samuel  
wrote:
>
>On 05/06/15 06:59, Michael Jennings wrote:
>
>> My team had a very (!!) productive and interesting discussion
>yesterday
>> with some folks who have succeeded in integrating Docker and SLURM to
>> the point that users can specify Docker repositories in which they
>> want SLURM to run their jobs, and SLURM will do so.
>
>How did they deal with the license incompatibility that Ralph
>mentioned?
>
>All the best
>Chris
>-- 
> Christopher SamuelSenior Systems Administrator
> VLSCI - Victorian Life Sciences Computation Initiative
> Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
> http://www.vlsci.org.au/  http://twitter.com/vlsci

[slurm-dev] Re: old version of scontrol, sacct keeps resurrecting

[Loris Bennett]

Hi Andrew,

Andrew Petersen  writes:

> Re: old version of scontrol, sacct keeps resurrecting 
>
> I was able to narrow down my problem further. I managed to stop all slurm
> daemons on head and slave nodes. On head node, if I start the new slurm:
> /.../new-slurm/etc/init.d/slurm start
> the slurmctld daemon runs ok, but with error messages in the 
> /var/log/slurmctld
> file about Incompatible versions. If I do "ps -u root -F |grep slurm", I can 
> see
> why. Periodically, the /../old-slurm/scontrol and squeue run. It seems like
> somehow, the "new-slurm/etc/init.d/slurm start" is triggering these old 
> daemons
> to run.

'scontrol' and 'squeue' are not daemons, so they must be being run
directly.

> I check the path in my $PATH, slurm.conf, and
> /../new-slurm/etc/init.d/slurm file, and there is nothing like that. I would
> appreciate any guidance on resolving this.

Have you updated your module files?  Some users could even be explicity
loading the old version in their .bashrc.

Cheers,

Loris

> Regards 
> Andrew
>
> On Wed, Jun 3, 2015 at 4:52 PM, Andrew Petersen  wrote:
>
> 
> 
> 
> 
> 
> 
> 
> 
> Hello
> 
> 
> I installed a new version of slurm, 14.11.3. It works fine. However I
> noticed that my log file /var/log/slurmctld shows
> error: slurm_receive_msg: Incompatible versions of client and server code
> 
> This led me to discover that old slurm scontrol, squeue and sacct are 
> still
> running on the head node, using
> ps -u root -F |grep slurm
> 
> 
> I have tried to kill this every which way, but they wont die, they keep
> resurrecting with different pid's. I tried
> /old-slurm-version/bin/scontrol shutdown
> 
> but this gives me
> slurm_shutdown error: Zero Bytes were transmitted or received
> 
> 
> It seems like something is automatically restarting the old slurm. I am
> using Bright Cluster Manager, and I set it so that it does NOT auto-start 
> or
> run the slurm daemon, but that did not help.
> 
> 
> Can someone help me kill this thing? It is causing the creation of big log
> zip files, and using up cpu capacity on the head node.
> 
> 
> Regards 
> 
> Andrew Petersen
> 
> 
> 
> 
>
>

-- 
This signature is currently under construction.

[slurm-dev] Re: old version of scontrol, sacct keeps resurrecting

[Christopher Samuel]

On 05/06/15 16:22, Loris Bennett wrote:

> 'scontrol' and 'squeue' are not daemons, so they must be being run
> directly.

It really sounds like they're being run from a cron job somewhere, do
you have some sort of health check or accounting checks that are getting
done regularly with the old paths hardcoded in them?

What users are they running as?

All the best,
Chris
-- 
 Christopher SamuelSenior Systems Administrator
 VLSCI - Victorian Life Sciences Computation Initiative
 Email: sam...@unimelb.edu.au Phone: +61 (0)3 903 55545
 http://www.vlsci.org.au/  http://twitter.com/vlsci