What is probably happening here is that you have a log message that
cannot be written out, and that is causing the output to stop.
The fact that you have to delete the queue files matches this.
looking at your ruleset, my guess is that you are getting a hostname
arriving that creates an invalid filename in some way, and the OS is
refusing to allow rsyslog to write the file.
I would suggest creating a new logfile that logs fromhost-ip and
hostname (nothing else), put this at the beginning of the config and
when a stall happens look at the last entry in that file. If you can get
a debug output when it's stuck, it probably will tell you what it's
trying to do at that point.
what version are you running?
David Lang
On Wed, 6 Nov 2013 18:01:57 +0000, Leggett, Torrance I. wrote:
I have a central logging server that accepts messages and writes them
out - <year>/<month>/<host>. However, recently I’ve had a problem
where the server stops writing out virtually all such messages and
the
main message queue fills and starts writing to the disk asssisted
queue. Once it starts doing this, the DA queue only grows until it
hits the max size or the disk fills. Restarting doesn’t seem to help
unless I remove all the files in the DA spool directory. Below is the
server portion of the config. If you need more configs or debugging,
just let me know what. I’m relatively new to debugging rsyslog
issues.
# cat 99-server.conf
# Switch to server ruleset
$RuleSet server
$MainMsgQueueFileName mainqueue # unique name prefix for
spool files
$MainMsgQueueType LinkedList # main queue should be a
dynamic list in memory
$MainMsgQueueSize 100000 # increase the queue size to
handle the message traffic
$MainMsgQueueHighWatermark 80000 # increase the high water
mark to write messages to disk
$MainMsgQueueLowWatermark 20000 # increase the low water mark
to stop writing to disk
$MainMsgQueueMaxDiskSpace 1g # 1gb disk space limit
$MainMsgQueueSaveOnShutdown off # save messages to disk on
shutdown
$MainMsgQueueWorkerThreads 5 # spawn up to 5 threads for
queue processing
$MaxMessageSize 8k # handle larger messages if
needed
$RepeatedMsgReduction off # log all messages as they
come
# Load UDP module
$ModLoad imudp
$InputUDPServerBindRuleset server
$UDPServerRun 514
# Load TCP module
$ModLoad imtcp
$InputTCPServerBindRuleset server
$InputTCPServerRun 514
# Load RELP module
$ModLoad imrelp
$InputRELPServerBindRuleset server
$InputRELPServerRun 20514
# Send logs to logstash for indexing
*.* @@127.0.0.1:5544;RSYSLOG_TraditionalForwardFormat
# Templates
$Template auditFormat,"%MSG%\n"
$Template radiusFormat,"%MSG%\n"
$Template tsmFormat,"%MSG%\n"
$Template
dynAuditLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/audit.log"
$Template
dynAuthLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/secure"
$Template
dynCronLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/cron.log"
$Template
dynDaemonLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/daemon.log"
$Template
dynDebug,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/debug"
$Template
dynHttpAccess,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_access.log"
$Template
dynHttpError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_error.log"
$Template
dynKernLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/kern.log"
$Template
dynMailLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/mail.log"
$Template
dynPuppetAgent,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-agent.log"
$Template
dynPuppetMaster,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-master.log"
$Template
dynRadiusLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/radius.log"
$Template
dynSyslog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/messages"
$Template
dynTsmInfo,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmcmd.log"
$Template
dynTsmError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmerror.log"
$Template
dynUserLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/user.log"
# First capture auditd messages from remotes
#
if $programname == 'auditd' and $syslogfacility-text == 'local6' then
?dynAuditLog;auditFormat
# Next capture RADIUS messages from remotes
#
if $programname == 'radiusd' and $syslogfacility-text == 'local6'
then ?dynRadiusLog;radiusFormat
# Next handle any apache logs and remove them from the stream
#
if $programname == 'httpd' and $syslogfacility-text == 'local6' then
{
?dynHttpAccess
stop
}
if $programname == 'httpd' and $syslogfacility-text == 'local7' then
{
?dynHttpError
stop
}
# Next handle any nginx logs and remove them from the stream
#
if $programname == 'nginx' and $syslogfacility-text == 'local6' then
{
?dynHttpAccess
stop
}
if $programname == 'nginx' and $syslogfacility-text == 'local7' then
{
?dynHttpError
stop
}
# Next handle any puppet logs and remove them from the stream
#
if $programname == 'puppet-agent' then {
?dynPuppetAgent
stop
}
if $programname == 'puppet-master' then {
?dynPuppetMaster
stop
}
# Next handle any TSM logs and remove them from the stream
#
if $programname == 'dsmc' and $syslogfacility-text == 'local3' and
$syslogseverity-text == 'info' then ?dynTsmInfo;tsmFormat
if $programname == 'dsmserv' and $syslogfacility-text == 'local3'
and $syslogseverity-text == 'err' then ?dynTsmError;tsmFormat
# Rules
auth,authpriv.* ?dynAuthLog
*.*;\
mail.none;\
cron.none -?dynSyslog
cron.* ?dynCronLog
daemon.* -?dynDaemonLog
kern.* -?dynKernLog
mail.* -?dynMailLog
user.* -?dynUserLog
# Switch back to default ruleset
$RuleSet RSYSLOG_DefaultRuleset
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
