the debug log will continue to show everything that it's doing. I
expect that once the problem starts, you will have it attempting to
re-deliver the same message repeatedly, but new messages arriving will
cause debug activity as well.
David Lang
On Wed, 6 Nov 2013 18:57:59 +0000, Leggett, Torrance I. wrote:
Ok. I have it logging as you suggested and I also have it running
with '-d -n'. Should I expect for rsyslogd debugging output to stop
at
the point where I see this behavior or will the debugging output
continue on with no discernible change?
On Nov 6, 2013, at 12:11 PM, David Lang wrote:
What is probably happening here is that you have a log message that
cannot be written out, and that is causing the output to stop.
The fact that you have to delete the queue files matches this.
looking at your ruleset, my guess is that you are getting a hostname
arriving that creates an invalid filename in some way, and the OS is
refusing to allow rsyslog to write the file.
I would suggest creating a new logfile that logs fromhost-ip and
hostname (nothing else), put this at the beginning of the config and
when a
stall happens look at the last entry in that file. If you can get a
debug
output when it's stuck, it probably will tell you what it's trying to
do at
that point.
what version are you running?
David Lang
On Wed, 6 Nov 2013 18:01:57 +0000, Leggett, Torrance I. wrote:
I have a central logging server that accepts messages and writes
them
out - //. However, recently I've had a problem
where the server stops writing out virtually all such messages and
the
main message queue fills and starts writing to the disk asssisted
queue. Once it starts doing this, the DA queue only grows until it
hits the max size or the disk fills. Restarting doesn't seem to
help
unless I remove all the files in the DA spool directory. Below is
the
server portion of the config. If you need more configs or
debugging,
just let me know what. I'm relatively new to debugging rsyslog
issues.
# cat 99-server.conf
# Switch to server ruleset
$RuleSet server
$MainMsgQueueFileName mainqueue # unique name prefix for spool
files
$MainMsgQueueType LinkedList # main queue should be a
dynamic list in memory
$MainMsgQueueSize 100000 # increase the queue size to
handle the message traffic
$MainMsgQueueHighWatermark 80000 # increase the high water
mark to write messages to disk
$MainMsgQueueLowWatermark 20000 # increase the low water mark
to stop writing to disk
$MainMsgQueueMaxDiskSpace 1g # 1gb disk space limit
$MainMsgQueueSaveOnShutdown off # save messages to disk on shutdown
$MainMsgQueueWorkerThreads 5 # spawn up to 5 threads for
queue processing
$MaxMessageSize 8k # handle larger messages if needed
$RepeatedMsgReduction off # log all messages as they come
# Load UDP module
$ModLoad imudp
$InputUDPServerBindRuleset server
$UDPServerRun 514
# Load TCP module
$ModLoad imtcp
$InputTCPServerBindRuleset server
$InputTCPServerRun 514
# Load RELP module
$ModLoad imrelp
$InputRELPServerBindRuleset server
$InputRELPServerRun 20514
# Send logs to logstash for indexing
*.* @@127.0.0.1:5544;RSYSLOG_TraditionalForwardFormat
# Templates
$Template auditFormat,"%MSG%n"
$Template radiusFormat,"%MSG%n"
$Template tsmFormat,"%MSG%n"
$Template
dynAuditLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/audit.log"
$Template
dynAuthLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/secure"
$Template
dynCronLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/cron.log"
$Template
dynDaemonLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/daemon.log"
$Template
dynDebug,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/debug"
$Template
dynHttpAccess,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_access.log"
$Template
dynHttpError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_error.log"
$Template
dynKernLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/kern.log"
$Template
dynMailLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/mail.log"
$Template
dynPuppetAgent,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-agent.log"
$Template
dynPuppetMaster,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-master.log"
$Template
dynRadiusLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/radius.log"
$Template
dynSyslog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/messages"
$Template
dynTsmInfo,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmcmd.log"
$Template
dynTsmError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmerror.log"
$Template
dynUserLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/user.log"
# First capture auditd messages from remotes
#
if $programname == 'auditd' and $syslogfacility-text == 'local6'
then
?dynAuditLog;auditFormat
# Next capture RADIUS messages from remotes
#
if $programname == 'radiusd' and $syslogfacility-text == 'local6'
then ?dynRadiusLog;radiusFormat
# Next handle any apache logs and remove them from the stream
#
if $programname == 'httpd' and $syslogfacility-text == 'local6'
then {
?dynHttpAccess
stop
}
if $programname == 'httpd' and $syslogfacility-text == 'local7'
then {
?dynHttpError
stop
}
# Next handle any nginx logs and remove them from the stream
#
if $programname == 'nginx' and $syslogfacility-text == 'local6'
then {
?dynHttpAccess
stop
}
if $programname == 'nginx' and $syslogfacility-text == 'local7'
then {
?dynHttpError
stop
}
# Next handle any puppet logs and remove them from the stream
#
if $programname == 'puppet-agent' then {
?dynPuppetAgent
stop
}
if $programname == 'puppet-master' then {
?dynPuppetMaster
stop
}
# Next handle any TSM logs and remove them from the stream
#
if $programname == 'dsmc' and $syslogfacility-text == 'local3' and
$syslogseverity-text == 'info' then ?dynTsmInfo;tsmFormat
if $programname == 'dsmserv' and $syslogfacility-text == 'local3'
and $syslogseverity-text == 'err' then ?dynTsmError;tsmFormat
# Rules
auth,authpriv.* ?dynAuthLog
*.*;
mail.none;
cron.none -?dynSyslog
cron.* ?dynCronLog
daemon.* -?dynDaemonLog
kern.* -?dynKernLog
mail.* -?dynMailLog
user.* -?dynUserLog
# Switch back to default ruleset
$RuleSet RSYSLOG_DefaultRuleset
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad
of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you
DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
