It looks like the messages are getting through to the slurmdbd now,
but it should be processing hundreds of messages per second as the
slurmctld uploads accounting records. I'd suggest trying to find out
why the slurmdbd or mysql are running so slowly.
Quoting "Evans, Brandon" <bev...@syntheticgenomics.com>:
>
> Thanks for the quick reply.
> I did find a mysql error regarding max_packet_size. I increased
> max_packet_size to 256M
> restarted slurmctld, slurmdbd and MySQL and no longer get the packet size
> error.
>
> However, I am still seeing the 'agent queue full' message in slurmctld
>
>
> [2012-07-05T14:44:40] error: slurmdbd: agent queue is full, discarding
> request
> [2012-07-05T14:44:40] error: slurmdbd: agent queue is full, discarding
> request
> [2012-07-05T14:44:40] sched: job_complete for JobId=107544 successful
> [2012-07-05T14:44:41] error: slurmdbd: agent queue is full, discarding
> request
> [2012-07-05T14:44:41] error: slurmdbd: agent queue is full, discarding
> request
> [2012-07-05T14:44:41] sched: Allocate JobId=112609 NodeList=cpu05 #CPUs=1
>
>
> The slurmdbd.log is chugging along with output like this:
>
> [2012-07-05T14:45:52] debug2: DBD_JOB_SUSPEND: ID:279 STATE:RUNNING
> [2012-07-05T14:45:53] debug2: DBD_JOB_SUSPEND: ID:280 STATE:RUNNING
> [2012-07-05T14:45:54] debug2: DBD_JOB_SUSPEND: ID:281 STATE:RUNNING
> [2012-07-05T14:45:55] debug2: DBD_JOB_SUSPEND: ID:282 STATE:RUNNING
> [2012-07-05T14:45:56] debug2: DBD_JOB_SUSPEND: ID:2192 STATE:SUSPENDED
> [2012-07-05T14:45:56] debug2: DBD_JOB_SUSPEND: ID:2193 STATE:SUSPENDED
>
>
>
> Any other thoughts?
>
>
> Thank you,
>
> Brandon Evans
>
>
>
>
>
>
> On 7/5/12 2:03 PM, "Moe Jette" <je...@schedmd.com> wrote:
>
>>
>> Check your slurmdbd logs. For some reason it does not seem to be
>> accepting accounting records from the slurmctld, so they keep building
>> up in slurmctld's memory.
>>
>> Quoting "Evans, Brandon" <bev...@syntheticgenomics.com>:
>>
>>>
>>> I am getting these messages almost non stop on our slurm cluster:
>>>
>>> [2012-07-03T16:17:54] completing job 6585
>>> [2012-07-03T16:17:54] error: slurmdbd: agent queue is full, discarding
>>> request
>>> [2012-07-03T16:17:54] error: slurmdbd: agent queue is full, discarding
>>> request
>>> [2012-07-03T16:17:54] sched: job_complete for JobId=6585 successful
>>> [2012-07-03T16:17:54] error: slurmdbd: agent queue is full, discarding
>>> request
>>> [2012-07-03T16:17:54] error: slurmdbd: agent queue is full, discarding
>>> request
>>> [2012-07-03T16:17:54] sched: Allocate JobId=8128 NodeList=cpu24 #CPUs=1
>>> [2012-07-03T16:17:54] error: slurmdbd: agent queue is full, discarding
>>> request
>>> [2012-07-03T16:17:54] error: slurmdbd: agent queue is full, discarding
>>> request
>>> [2012-07-03T16:17:54] sched: Allocate JobId=8129 NodeList=cpu13 #CPUs=1
>>>
>>>
>>>
>>> While this is going on I am unable to stop slurmdbd using the typical
>>> 'service slurmdbd stop' command.
>>> The main reason I am concerned about these log entries is because I've
>>> had
>>> slurmctld crash twice in the past 2 days.
>>> I've observed slurmctld slowly consuming all available RAM on the box.
>>> The RAM is released when I restart slurmctld.
>>> Last night the slurm controller machine spike to +500 load average
>>> before
>>> slurmctld crash.
>>>
>>>
>>>
>>>
>>> Jobs are being sent to slurm using the slurmWrapper.c I've included in
>>> this email.
>>> We keep the job count around 2k with jobs being added every minute when
>>> less than 2k jobs are in the queue.
>>>
>>>
>>>
>>> Any ideas what might be causing the memory leak and where the slurmdbd
>>> errors are coming from?
>>>
>>> Any insite into this issue would be greatly appreciated.
>>>
>>>
>>> Cluster Details:
>>> =================
>>> slurm 2.3.4
>>> Centos 6
>>> Slurmdbd is using a MySQL 5.1.61 backend configured with 5GB innodb
>>> pool.
>>> slurm controller running on 8 core 24GB RAM blade.
>>> 32 8 core Dell blade compute cluster.
>>>
>>>
>>> slurm.conf
>>> ===============
>>>
>>> ControlMachine=slurm02
>>> ControlAddr=10.1.1.2
>>>
>>>
>>> AuthType=auth/munge
>>> CacheGroups=0
>>> CryptoType=crypto/munge
>>> EpilogSlurmctld=/usr/local/code/queue/slurm/post_exec_slurm.py
>>> MaxJobCount=20000
>>> MpiDefault=none
>>> PluginDir=/usr/lib64/slurm
>>> ProctrackType=proctrack/linuxproc
>>> ReturnToService=1
>>> SlurmctldPidFile=/var/run/slurmctld.pid
>>> SlurmctldPort=6817
>>> SlurmdPidFile=/var/run/slurmd.pid
>>> SlurmdPort=6818
>>> SlurmdSpoolDir=/var/lib/slurm/slurmctld
>>> SlurmUser=archetype
>>> StateSaveLocation=/var/lib/slurm/slurmctld
>>> SwitchType=switch/none
>>> TaskPlugin=task/none
>>> InactiveLimit=0
>>> KillWait=30
>>> MessageTimeout=60
>>> MinJobAge=60
>>> SlurmctldTimeout=120
>>> SlurmdTimeout=300
>>> Waittime=0
>>> FastSchedule=1
>>> SchedulerType=sched/backfill
>>> SchedulerPort=7321
>>> SelectType=select/cons_res
>>>
>>> SelectTypeParameters=CR_CPU_Memory
>>> SchedulerParameters=max_job_bf=15,interval=45
>>>
>>>
>>> PreemptMode=SUSPEND,GANG
>>> PreemptType=preempt/partition_prio
>>>
>>>
>>>
>>> AccountingStorageHost=localhost
>>> AccountingStorageType=accounting_storage/slurmdbd
>>> ClusterName=c01
>>> JobCompLoc=/var/log/slurm/slurm_job.log
>>> JobCompType=jobcomp/none
>>> JobAcctGatherFrequency=30
>>> JobAcctGatherType=jobacct_gather/linux
>>> SlurmctldDebug=1
>>> SlurmctldLogFile=/var/log/slurm/slurmctld.log
>>> SlurmdDebug=1
>>> SlurmdLogFile=/var/log/slurm/slurmd.log
>>>
>>>
>>>
>>>
>>>
>>>
>>> NodeName=cpu01,cpu02,cpu[04-33] Procs=8 Sockets=2 CoresPerSocket=2
>>> ThreadsPerCore=2 State=UNKNOWN
>>>
>>> NodeName=asm01,asm02 Procs=8 Sockets=2 CoresPerSocket=2 ThreadsPerCore=2
>>> State=UNKNOWN
>>>
>>> NodeName=asm03 Procs=16 Sockets=2 CoresPerSocket=4 ThreadsPerCore=2
>>> State=UNKNOWN
>>>
>>> NodeName=asm04 Procs=24 Sockets=2 CoresPerSocket=6 ThreadsPerCore=2
>>> State=UNKNOWN
>>>
>>> NodeName=asm-dev01 Procs=4 Sockets=2 CoresPerSocket=2 ThreadsPerCore=1
>>> State=UNKNOWN
>>>
>>> NodeName=admin-dev01 Procs=8 Sockets=2 CoresPerSocket=2 ThreadsPerCore=2
>>> State=UNKNOWN
>>>
>>> NodeName=slurm01,slurm02 Procs=8 Sockets=2 CoresPerSocket=2
>>> ThreadsPerCore=2 State=UNKNOWN
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> PartitionName=betest_cpuNodes
>>> Nodes=cpu01,cpu02,cpu[04-33] MaxTime=INFINITE State=UP
>>> Priority=1500 Shared=FORCE
>>>
>>>
>>>
>>>
>>> PartitionName=user_submitted_jobs Nodes=cpu01,cpu02,cpu[04-33]
>>> MaxTime=INFINITE State=UP Priority=2000 Shared=FORCE
>>>
>>> PartitionName=output_processor_batch Nodes=cpu01,cpu02,cpu[04-33]
>>> MaxNodes=2 MaxTime=INFINITE State=UP Priority=2000 Shared=FORCE
>>>
>>> PartitionName=test_exclusive Nodes=cpu01,cpu02,cpu[04-33]
>>> MaxTime=INFINITE State=UP Priority=1700 Shared=FORCE
>>>
>>> PartitionName=geneflag_batch Nodes=cpu01,cpu02,cpu[04-33]
>>> MaxTime=INFINITE State=UP Priority=1600 Shared=FORCE
>>>
>>>
>>>
>>>
>>> slurmWrapper.c
>>> ===================
>>> /*
>>> slurmWrapper.c
>>> =-=-=-=-=-=-=-
>>>
>>> Version 0.10
>>> - added support for /etc/slurm/slurm_job.env to load environmental
>>> variables
>>>
>>> Version 0.9
>>> - hard coded PYTHONPATH
>>>
>>> Version 0.8
>>> - adding extraEnvVars
>>>
>>> Version 0.7
>>> - doneDir set via export and also passed to EpilofSlurmctld
>>>
>>>
>>> */
>>>
>>> #include <Python.h>
>>> #include <signal.h>
>>> #include <unistd.h>
>>> #include <slurm/slurm.h>
>>> #include <slurm/slurm_errno.h>
>>> /*
>>> /path/to/slrum-source must be added to your include path
>>> You may need to set -DHAVE_STDBOOL_H to use env.h
>>> */
>>> #include <src/common/env.h>
>>> #include <src/common/xmalloc.h>
>>>
>>>
>>>
>>> int submitSlurmBatch (char *queueName, char *stderrFile, char *doneDir,
>>> char *extraEnvVars, char *postExecScript, char
>>> *jobName,
>>> char *outputFile, char *command)
>>> {
>>>
>>> job_desc_msg_t job_desc_msg;
>>> submit_response_msg_t *submit_response_msg_ptr;
>>> int job_id = 0;
>>>
>>>
>>> /* Verify we can write to stderrFile and stdoutFile */
>>> FILE *out = fopen(outputFile, "w");
>>> if ( out == NULL ) {
>>> fprintf(stderr, "slurmWrapper: Can not open outputFile
>>> file for write!\n");
>>> return -1;
>>> }
>>> fclose (out);
>>> FILE *err = fopen(stderrFile, "w");
>>> if ( out == NULL ) {
>>> fprintf(stderr, "slurmWrapper: Can not open stderrFile
>>> file for write!\n");
>>> return -2;
>>> }
>>> fclose (err);
>>>
>>> /*
>>> Jobs submitted to slurm need the shebang in order to
>>> execute.
>>> The doneDir envrionmental variable is set via export for
>>> usage in 'command'.
>>> */
>>> char *shebang = "#!/bin/bash\nif [ -e /etc/slurm/slurm_job.env ];
>>> then
>>> . /etc/slurm/slurm_job.env; fi\nexport SGI_SLURM_DONE_DIR=";
>>>
>>> char *formatted_command;
>>> formatted_command = ( char * ) calloc( strlen(shebang) +
>>> strlen(doneDir) + 1 + strlen(command) + 1 , sizeof(char));
>>>
>>> strcpy(formatted_command, shebang);
>>> strcat(formatted_command, doneDir);
>>> strcat(formatted_command, "\n");
>>> strcat(formatted_command, command);
>>>
>>>
>>> #ifdef SGIVERBOSE
>>> printf("VERBOSE slurmWrapper.c \n Input:\n
>>> queueName=%s\n
>>> stderrFile=%s\n doneDir=%s\n extraEnvVars=%s\n postExecScript=%s\n
>>> jobName=%s\n outputFile=%s\n command=%s\n\n",
>>> queueName, stderrFile, doneDir, extraEnvVars,
>>> postExecScript, jobName, outputFile, command);
>>> #endif
>>>
>>> /* initialize our job descriptor */
>>> slurm_init_job_desc_msg( &job_desc_msg );
>>>
>>> //job_desc_msg.script = (char * ) calloc ( strlen(
>>> formatted_command) + 1, sizeof(char) );
>>> //uint16_t cpus_per_task; /* number of processors required for each
>>> task */
>>> //uint32_t min_cpus; /* minimum number of processors required, *
>>> default=0 */
>>> //uint32_t max_cpus; /* maximum number of processors required, *
>>> default=0 */
>>> //uint32_t min_nodes; /* minimum number of nodes required by job, *
>>> default=0 */
>>> //uint32_t max_nodes; /* maximum number of nodes usable by job, *
>>> default=0 */
>>>
>>> job_desc_msg.name = jobName;
>>> job_desc_msg.partition = queueName;
>>> env_array_overwrite( &job_desc_msg.spank_job_env,
>>> "SGI_SLURM_DONE_DIR", doneDir );
>>> env_array_overwrite( &job_desc_msg.spank_job_env,
>>> "SGI_SLURM_EXTRA_VARS", extraEnvVars );
>>> env_array_overwrite( &job_desc_msg.spank_job_env,
>>> "SGI_SLURM_STDERR_FILE", stderrFile );
>>> env_array_overwrite( &job_desc_msg.spank_job_env, "PYTHONPATH",
>>> "/usr/local" );
>>> job_desc_msg.spank_job_env_size = 4;
>>> job_desc_msg.script = formatted_command;
>>> //job_desc_msg.begin_time = 0;
>>> job_desc_msg.immediate = 0;
>>> job_desc_msg.work_dir = "/tmp";
>>> job_desc_msg.user_id = getuid();
>>> job_desc_msg.group_id = getgid();
>>> job_desc_msg.std_err = stderrFile;
>>> job_desc_msg.std_out = outputFile;
>>>
>>>
>>> while ( slurm_submit_batch_job( &job_desc_msg,
>>> &submit_response_msg_ptr ) < 0 ) {
>>> static char *msg;
>>>
>>> if (errno == ESLURM_ERROR_ON_DESC_TO_RECORD_COPY)
>>> msg = "slurmWrapper: Slurm job queue full,
>>> sleeping and retrying.";
>>> else if (errno == ESLURM_NODES_BUSY) {
>>> msg = "slurmWrapper: Job step creation
>>> temporarily
>>> disabled, retrying";
>>> } else if (errno == EAGAIN) {
>>> msg = "slurmWrapper: Slurm temporarily unable to
>>> accept job, sleeping and retrying.";
>>> } else
>>> msg = NULL;
>>>
>>> if ( msg == NULL ) {
>>> //error("Batch job submission failed: %m");
>>> printf ("slurmWrapper: error: Batch job
>>> submission
>>> failed\n");
>>> exit(1);
>>> }
>>>
>>> if (errno == ESLURM_NODES_BUSY)
>>> printf("slurmWrapper: w info: %s", msg); /* Not an
>>> error,
>>> powering up nodes */
>>> else
>>> printf("slurmWrapper: w error: %s", msg);
>>>
>>> }
>>>
>>> job_id = submit_response_msg_ptr->job_id;
>>> #ifdef SGIVERBOSE
>>> printf ( "slurmWrapper: Submitted JobID: %d\n", job_id );
>>> #endif
>>>
>>> //cfree(job_desc_msg.script);
>>> cfree(formatted_command);
>>> slurm_free_submit_response_response_msg (
>>> submit_response_msg_ptr
>>> );
>>> return (job_id);
>>> }
>>>
>>>
>>>
>>> /* Python integration */
>>> #ifndef SKIPPYTHON
>>> static PyObject *wrap_submitSlurmBatch( PyObject *self, PyObject *args )
>>> {
>>> char *queueName, *stderrFile, *doneDir, *extraEnvVars,
>>> *postExecScript, *jobName, *outputFile, *command;
>>> if (!PyArg_ParseTuple(args, "ssssssss", &queueName, &stderrFile,
>>> &doneDir, &extraEnvVars, &postExecScript, &jobName, &outputFile,
>>> &command)) return NULL;
>>> int returnVal = submitSlurmBatch(queueName, stderrFile,
>>> doneDir, extraEnvVars, postExecScript, jobName, outputFile, command);
>>>
>>> return Py_BuildValue("i", returnVal);
>>> }
>>>
>>> static PyMethodDef addMethods[] =
>>> {
>>> {"submitSlurmBatch", wrap_submitSlurmBatch, METH_VARARGS},
>>> {NULL, NULL}
>>> };
>>>
>>> void initSlurmExtension(void)
>>> {
>>> (void) Py_InitModule("SlurmExtension", addMethods);
>>> }
>>> #endif
>>> /* End Python integration */
>>>
>>>
>>>
>>> int main(void) {
>>>
>>> /* submitSlurmBatch queueName stderrFile
>>> doneDir extraEnvVars postExecScript jobName outputFile
>>> command */
>>> submitSlurmBatch( "betest_cpuNodes",
>>> "/tmp/slurm_stderr.out",
>>> "/tmp/donedir", "xtraVars:1", "brawndo.script\n", "job001",
>>> "/tmp/slurm_stdout.out", "/usr/bin/stress -c 1 -t 60s" );
>>> //submitSlurmBatch( "highpri", "/tmp/slurm_stderr.out",
>>> "brawndo.script\n", "job001", "/tmp/slurm_stdout.out",
>>> "/usr/bin/env > /tmp/snarf.out" );
>>>
>>>
>>> exit(0);
>>> }
>>>
>>>
>>>
>>>
>>> Brandon Evans
>>>
>>>
>>>
>>
>
