slurm_spank_local_user_init() is being invoked by srun, so that's the
only place it's output gets logged. Spank functions invoked by the
daemons would be logged by those daemons.
Quoting Bob Moench <r...@cray.com>:
Moe,
The srun -vv has allowed me to get "slurm_debug" messages to
stderr. That's progress! But none of my SPANK plugin slurm_debug
calls go to the log.
By changing these config's:
SlurmctldDebug=debug
SlurmdDebug=debug
a fair amount of "debug:" stuff _is_ turned on inside
slurmctld.log. The part of the log for my run has such, but
none of _my_ plugin's calls:
[2015-04-21T12:29:30.147] debug: sched: Running job scheduler
[2015-04-21T12:29:54.170] debug: aeld sync event
[2015-04-21T12:29:54.171] debug: _start_session: Created aeld session fd 5
[2015-04-21T12:29:56.811] debug: backfill: beginning
[2015-04-21T12:29:56.811] debug: backfill: no jobs to backfill
[2015-04-21T12:29:56.881] debug: sched: Running job scheduler
[2015-04-21T12:30:56.932] debug: sched: Running job scheduler
[2015-04-21T12:31:49.000] sched: _slurm_rpc_allocate_resources
JobId=455 NodeList=nid00032 usec=451
[2015-04-21T12:31:49.002] debug: Configuration for job 455 complete
[2015-04-21T12:31:49.002] debug: gres/craynetwork: _step_test
455.4294967294 gres_bit_alloc is NULL
[2015-04-21T12:31:49.002] debug: gres/craynetwork: _step_test
455.4294967294 gres_bit_alloc is NULL
[2015-04-21T12:31:49.002] debug: gres/craynetwork: _step_test
455.0 gres_bit_alloc is NULL
[2015-04-21T12:31:49.002] debug: laying out the 4 tasks on 1
hosts nid00032 dist 2
[2015-04-21T12:31:49.002] debug: reserved ports 20003 for step 455.0
[2015-04-21T12:31:49.194] debug: gres/craynetwork: _step_alloc
gres_bit_alloc for 455.0 is NULL
[2015-04-21T12:31:49.280] _slurm_rpc_job_sbcast_cred JobId=455
NodeList=nid00032 usec=3
[2015-04-21T12:31:56.812] debug: backfill: beginning
[2015-04-21T12:31:56.812] debug: backfill: no jobs to backfill
[2015-04-21T12:31:56.984] debug: sched: Running job scheduler
[2015-04-21T12:31:58.025] _slurm_rpc_job_sbcast_cred JobId=455
NodeList=nid00032 usec=1
[2015-04-21T12:31:58.842] debug: _slurm_rpc_job_alloc_info_lite
JobId=455 NodeList=nid00032 usec=1
[2015-04-21T12:31:58.843] debug: laying out the 1 tasks on 1
hosts nid00032 dist 1
[2015-04-21T12:31:58.843] debug: reserved ports 20004 for step 455.1
[2015-04-21T12:32:04.485] job_complete: JobID=455 State=0x1
NodeCnt=1 WIFEXITED 0 WEXITSTATUS 0
[2015-04-21T12:32:04.486] job_complete: JobID=455 State=0x8003
NodeCnt=1 done
[2015-04-21T12:32:04.974] debug: freed ports 20004 for step 455.1
[2015-04-21T12:32:05.105] debug: freed ports 20003 for step 455.0
[2015-04-21T12:32:05.492] debug: sched: Running job scheduler
[2015-04-21T12:32:19.052] debug: Spawning ping agent for
nid000[20-27,33-35,40-43,48-51,56-59]
"My calls", specifically, are from slurm_spank_local_user_init().
I have not done anything to change config for "DebugFlags".
"scontrol show config" says it is (null). So that doesn't
seem to explain how the other debug entries have been enabled.
Any idea what else I am missing?
Thanks,
Bob
On Mon, 20 Apr 2015, Moe Jette wrote:
You'll need to increase the verbosity of messages for whatever is
calling the spank plugin function of interest. Perhaps increase
SlurmdDebug in slurm.conf or for the srun/salloc/sbatch commands,
add "-vv" to the execute line.
Quoting Bob Moench <r...@cray.com>:
Moe,
That all makes sense and sounds good, but I went to the
slurm.conf file and added the line:
SlurmctldDebug=debug
and did a /etc/init.d/slurm restart. (I also tried a
scontrol reconfigure.) I even rebooted.
Experimentally, my plugin does both slurm_info and slurm_debug
calls. I see the slurm_info deliver to my terminal, but not
the slurm_debug. Neither show up in slurmctld.log, where I
was expecting to see the slurm_debug.
Any idea what I am doing wrong?
Thanks,
Bob
On Fri, 17 Apr 2015, Moe Jette wrote:
Messages printed with those functions are only seen if someone
has the daemons configured to print those more verbose messages.
See the "SlurmctldDebug" parameter info in the slurm.conf man page:
http://slurm.schedmd.com/slurm.conf.html
Quoting Bob Moench <r...@cray.com>:
Hi,
Does anyone know what the various SPANK logging functions
do? They exist in spank.h with the description: "SLURM
logging functions which are exported to plugins". I have
been using slurm_info and slurm_error and they come out to
stdout/err.
What are slurm_debug, slurm_debug2, slurm_debug3? How do
they differ?
The man page and Schedmd documentation are silent on these.
Thanks,
Bob
--
Bob Moench (rwm); PE Debugger Development; 605-9034; 354-7895; SP 24227
--
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support
--
Bob Moench (rwm); PE Debugger Development; 605-9034; 354-7895; SP 24227
--
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support
--
Bob Moench (rwm); PE Debugger Development; 605-9034; 354-7895; SP 24227
--
Morris "Moe" Jette
CTO, SchedMD LLC
Commercial Slurm Development and Support
