I have included my login node in the list of nodes. Not all cores are included though. Please see the output of "scontrol" below
[mahmood@rocks7 ~]$ scontrol show nodes NodeName=compute-0-0 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUTot=32 CPULoad=31.96 AvailableFeatures=rack-0,32CPUs ActiveFeatures=rack-0,32CPUs Gres=(null) NodeAddr=10.1.1.254 NodeHostName=compute-0-0 Version=18.08 OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 RealMemory=64261 AllocMem=0 FreeMem=5187 Sockets=32 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=444124 Weight=20511900 Owner=N/A MCS_label=N/A Partitions=CLUSTER,WHEEL BootTime=2018-12-24T18:16:49 SlurmdStartTime=2019-01-02T23:53:20 CfgTRES=cpu=32,mem=64261M,billing=47 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-1 Arch=x86_64 CoresPerSocket=1 CPUAlloc=6 CPUTot=32 CPULoad=25.90 AvailableFeatures=rack-0,32CPUs ActiveFeatures=rack-0,32CPUs Gres=(null) NodeAddr=10.1.1.253 NodeHostName=compute-0-1 Version=18.08 OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 RealMemory=64261 AllocMem=4096 FreeMem=509 Sockets=32 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=444124 Weight=20511899 Owner=N/A MCS_label=N/A Partitions=CLUSTER,WHEEL,RUBY,EMERALD,QEMU BootTime=2018-12-24T18:07:22 SlurmdStartTime=2019-01-02T23:53:20 CfgTRES=cpu=32,mem=64261M,billing=47 AllocTRES=cpu=6,mem=4G CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-2 Arch=x86_64 CoresPerSocket=1 CPUAlloc=6 CPUTot=32 CPULoad=5.95 AvailableFeatures=rack-0,32CPUs ActiveFeatures=rack-0,32CPUs Gres=(null) NodeAddr=10.1.1.252 NodeHostName=compute-0-2 Version=18.08 OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 RealMemory=64261 AllocMem=4096 FreeMem=7285 Sockets=32 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=444124 Weight=20511898 Owner=N/A MCS_label=N/A Partitions=CLUSTER,WHEEL,RUBY,EMERALD,QEMU BootTime=2018-12-24T18:10:56 SlurmdStartTime=2019-01-02T23:53:20 CfgTRES=cpu=32,mem=64261M,billing=47 AllocTRES=cpu=6,mem=4G CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-3 Arch=x86_64 CoresPerSocket=1 CPUAlloc=6 CPUTot=56 CPULoad=36.87 AvailableFeatures=rack-0,56CPUs ActiveFeatures=rack-0,56CPUs Gres=(null) NodeAddr=10.1.1.251 NodeHostName=compute-0-3 Version=18.08 OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 RealMemory=64147 AllocMem=4096 FreeMem=15274 Sockets=56 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=913567 Weight=20535897 Owner=N/A MCS_label=N/A Partitions=CLUSTER,WHEEL,RUBY,EMERALD,QEMU BootTime=2018-12-24T18:02:51 SlurmdStartTime=2019-01-02T23:53:20 CfgTRES=cpu=56,mem=64147M,billing=71 AllocTRES=cpu=6,mem=4G CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=compute-0-4 Arch=x86_64 CoresPerSocket=1 CPUAlloc=6 CPUTot=56 CPULoad=37.47 AvailableFeatures=rack-0,56CPUs ActiveFeatures=rack-0,56CPUs Gres=(null) NodeAddr=10.1.1.250 NodeHostName=compute-0-4 Version=18.08 OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 RealMemory=64147 AllocMem=4096 FreeMem=15233 Sockets=56 Boards=1 State=MIXED ThreadsPerCore=1 TmpDisk=50268 Weight=20535896 Owner=N/A MCS_label=N/A Partitions=CLUSTER,WHEEL,RUBY,EMERALD,QEMU BootTime=2018-12-24T18:05:38 SlurmdStartTime=2019-01-02T23:53:20 CfgTRES=cpu=56,mem=64147M,billing=71 AllocTRES=cpu=6,mem=4G CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s NodeName=rocks7 Arch=x86_64 CoresPerSocket=1 CPUAlloc=0 CPUTot=8 CPULoad=23.40 AvailableFeatures=(null) ActiveFeatures=(null) Gres=(null) NodeAddr=10.1.1.1 NodeHostName=rocks7 Version=18.08 OS=Linux 3.10.0-693.5.2.el7.x86_64 #1 SMP Fri Oct 20 20:32:50 UTC 2017 RealMemory=64261 AllocMem=0 FreeMem=322 Sockets=8 Boards=1 State=IDLE ThreadsPerCore=1 TmpDisk=272013 Weight=1 Owner=N/A MCS_label=N/A Partitions=WHEEL,QEMU BootTime=2018-12-24T17:47:14 SlurmdStartTime=2019-01-02T23:53:20 CfgTRES=cpu=8,mem=64261M,billing=8 AllocTRES= CapWatts=n/a CurrentWatts=0 LowestJoules=0 ConsumedJoules=0 ExtSensorsJoules=n/s ExtSensorsWatts=0 ExtSensorsTemp=n/s And here are some salloc examples: [mahmood@rocks7 ~]$ salloc salloc: Granted job allocation 275 [mahmood@rocks7 ~]$ exit exit salloc: Relinquishing job allocation 275 [mahmood@rocks7 ~]$ salloc -n1 salloc: Granted job allocation 276 [mahmood@rocks7 ~]$ exit exit salloc: Relinquishing job allocation 276 [mahmood@rocks7 ~]$ salloc --nodelist=compute-0-2 salloc: Granted job allocation 277 [mahmood@rocks7 ~]$ exit exit salloc: Relinquishing job allocation 277 [mahmood@rocks7 ~]$ salloc -n1 hostname salloc: Granted job allocation 278 rocks7.jupiterclusterscu.com salloc: Relinquishing job allocation 278 salloc: Job allocation 278 has been revoked. [mahmood@rocks7 ~]$ As you can see whenever I run salloc, I see the rocks7 prompt which is the login node. Regards, Mahmood On Wed, Jan 2, 2019 at 10:13 PM Brian Johanson <bjoha...@psc.edu> wrote: > sallocdefaultcommand specified in slurm.conf will change the default > behavior when salloc is executed without appending a command and also > explain conflicting behavior between installations. > > > SallocDefaultCommand > Normally, salloc(1) will run the user's default shell > when a command to execute is not specified on the salloc command line. > If SallocDefaultCommand is specified, salloc will instead > run the configured command. The command is passed to > '/bin/sh -c', so shell metacharacters are allowed, and commands with > multiple arguments should be quoted. For instance: > > SallocDefaultCommand = "$SHELL" > > would run the shell in the user's $SHELL environment > variable. and > > SallocDefaultCommand = "srun -n1 -N1 --mem-per-cpu=0 > --pty --preserve-env --mpi=none $SHELL" > > would run spawn the user's default shell on the > allocated resources, but not consume any of the CPU or memory resources, > configure it as a pseudo-terminal, and preserve all of the job's > environment variables (i.e. and not over-write them with > the job step's allocation information). > > For systems with generic resources (GRES) defined, the > SallocDefaultCommand value should explicitly specify a zero count for > the configured GRES. Failure to do so will result in the > launched shell consuming those GRES and preventing > subsequent srun commands from using them. For example, on Cray systems > add "--gres=craynetwork:0" as shown below: > SallocDefaultCommand = "srun -n1 -N1 --mem-per-cpu=0 > --gres=craynetwork:0 --pty --preserve-env --mpi=none $SHELL" > > For systems with TaskPlugin set, adding an option of > "--cpu-bind=no" is recommended if the default shell should have access > to all of the CPUs allocated to the job on that node, other‐ > wise the shell may be limited to a single cpu or core. > > On 1/2/2019 12:38 PM, Ryan Novosielski wrote: > > I don’t think that’s true (and others have shared documentation > regarding interactive jobs and the S commands). There was documentation > shared for how this works, and it seems as if it has been ignored. > > > > [novosirj@amarel2 ~]$ salloc -n1 > > salloc: Pending job allocation 83053985 > > salloc: job 83053985 queued and waiting for resources > > salloc: job 83053985 has been allocated resources > > salloc: Granted job allocation 83053985 > > salloc: Waiting for resource configuration > > salloc: Nodes slepner012 are ready for job > > > > This is the behavior I’ve always seen. If I include a command at the end > of the line, it appears to simply run it in the “new” shell that is created > by salloc (which you’ll notice you can exit via CTRL-D or exit). > > > > [novosirj@amarel2 ~]$ salloc -n1 hostname > > salloc: Pending job allocation 83054458 > > salloc: job 83054458 queued and waiting for resources > > salloc: job 83054458 has been allocated resources > > salloc: Granted job allocation 83054458 > > salloc: Waiting for resource configuration > > salloc: Nodes slepner012 are ready for job > > amarel2.amarel.rutgers.edu > > salloc: Relinquishing job allocation 83054458 > > > > You can, however, tell it to srun something in that shell instead: > > > > [novosirj@amarel2 ~]$ salloc -n1 srun hostname > > salloc: Pending job allocation 83054462 > > salloc: job 83054462 queued and waiting for resources > > salloc: job 83054462 has been allocated resources > > salloc: Granted job allocation 83054462 > > salloc: Waiting for resource configuration > > salloc: Nodes node073 are ready for job > > node073.perceval.rutgers.edu > > salloc: Relinquishing job allocation 83054462 > > > > When you use salloc, it starts an allocation and sets up the environment: > > > > [novosirj@amarel2 ~]$ env | grep SLURM > > SLURM_NODELIST=slepner012 > > SLURM_JOB_NAME=bash > > SLURM_NODE_ALIASES=(null) > > SLURM_MEM_PER_CPU=4096 > > SLURM_NNODES=1 > > SLURM_JOBID=83053985 > > SLURM_NTASKS=1 > > SLURM_TASKS_PER_NODE=1 > > SLURM_JOB_ID=83053985 > > SLURM_SUBMIT_DIR=/cache/home/novosirj > > SLURM_NPROCS=1 > > SLURM_JOB_NODELIST=slepner012 > > SLURM_CLUSTER_NAME=amarel > > SLURM_JOB_CPUS_PER_NODE=1 > > SLURM_SUBMIT_HOST=amarel2.amarel.rutgers.edu > > SLURM_JOB_PARTITION=main > > SLURM_JOB_NUM_NODES=1 > > > > If you run “srun” subsequently, it will run on the compute node, but a > regular command will run right where you are: > > > > [novosirj@amarel2 ~]$ srun hostname > > slepner012.amarel.rutgers.edu > > > > [novosirj@amarel2 ~]$ hostname > > amarel2.amarel.rutgers.edu > > > > Again, I’d advise Mahmood to read the documentation that was already > provided. It really doesn’t matter what behavior is requested — that’s not > what this command does. If one wants to run a script on a compute node, the > correct command is sbatch. I’m not sure what advantage salloc with srun > has. I assume it’s so you can open an allocation and then occasionally send > srun commands over to it. > > > > -- > > ____ > > || \\UTGERS, > |---------------------------*O*--------------------------- > > ||_// the State | Ryan Novosielski - novos...@rutgers.edu > > || \\ University | Sr. Technologist - 973/972.0922 (2x0922) ~*~ RBHS > Campus > > || \\ of NJ | Office of Advanced Research Computing - MSB > C630, Newark > > `' > > > >