I have a central logging server that accepts messages and writes them out - <year>/<month>/<host>. However, recently I’ve had a problem where the server stops writing out virtually all such messages and the main message queue fills and starts writing to the disk asssisted queue. Once it starts doing this, the DA queue only grows until it hits the max size or the disk fills. Restarting doesn’t seem to help unless I remove all the files in the DA spool directory. Below is the server portion of the config. If you need more configs or debugging, just let me know what. I’m relatively new to debugging rsyslog issues.
# cat 99-server.conf
# Switch to server ruleset
$RuleSet server
$MainMsgQueueFileName mainqueue # unique name prefix for spool files
$MainMsgQueueType LinkedList # main queue should be a dynamic list
in memory
$MainMsgQueueSize 100000 # increase the queue size to handle the
message traffic
$MainMsgQueueHighWatermark 80000 # increase the high water mark to write
messages to disk
$MainMsgQueueLowWatermark 20000 # increase the low water mark to stop
writing to disk
$MainMsgQueueMaxDiskSpace 1g # 1gb disk space limit
$MainMsgQueueSaveOnShutdown off # save messages to disk on shutdown
$MainMsgQueueWorkerThreads 5 # spawn up to 5 threads for queue
processing
$MaxMessageSize 8k # handle larger messages if needed
$RepeatedMsgReduction off # log all messages as they come
# Load UDP module
$ModLoad imudp
$InputUDPServerBindRuleset server
$UDPServerRun 514
# Load TCP module
$ModLoad imtcp
$InputTCPServerBindRuleset server
$InputTCPServerRun 514
# Load RELP module
$ModLoad imrelp
$InputRELPServerBindRuleset server
$InputRELPServerRun 20514
# Send logs to logstash for indexing
*.* @@127.0.0.1:5544;RSYSLOG_TraditionalForwardFormat
# Templates
$Template auditFormat,"%MSG%\n"
$Template radiusFormat,"%MSG%\n"
$Template tsmFormat,"%MSG%\n"
$Template dynAuditLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/audit.log"
$Template dynAuthLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/secure"
$Template dynCronLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/cron.log"
$Template dynDaemonLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/daemon.log"
$Template dynDebug,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/debug"
$Template
dynHttpAccess,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_access.log"
$Template
dynHttpError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/http_error.log"
$Template dynKernLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/kern.log"
$Template dynMailLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/mail.log"
$Template
dynPuppetAgent,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-agent.log"
$Template
dynPuppetMaster,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/puppet-master.log"
$Template dynRadiusLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/radius.log"
$Template dynSyslog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/messages"
$Template dynTsmInfo,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmcmd.log"
$Template
dynTsmError,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/dsmerror.log"
$Template dynUserLog,"/var/log/remotes/%$YEAR%/%$MONTH%/%HOSTNAME%/user.log"
# First capture auditd messages from remotes
#
if $programname == 'auditd' and $syslogfacility-text == 'local6' then
?dynAuditLog;auditFormat
# Next capture RADIUS messages from remotes
#
if $programname == 'radiusd' and $syslogfacility-text == 'local6' then
?dynRadiusLog;radiusFormat
# Next handle any apache logs and remove them from the stream
#
if $programname == 'httpd' and $syslogfacility-text == 'local6' then {
?dynHttpAccess
stop
}
if $programname == 'httpd' and $syslogfacility-text == 'local7' then {
?dynHttpError
stop
}
# Next handle any nginx logs and remove them from the stream
#
if $programname == 'nginx' and $syslogfacility-text == 'local6' then {
?dynHttpAccess
stop
}
if $programname == 'nginx' and $syslogfacility-text == 'local7' then {
?dynHttpError
stop
}
# Next handle any puppet logs and remove them from the stream
#
if $programname == 'puppet-agent' then {
?dynPuppetAgent
stop
}
if $programname == 'puppet-master' then {
?dynPuppetMaster
stop
}
# Next handle any TSM logs and remove them from the stream
#
if $programname == 'dsmc' and $syslogfacility-text == 'local3' and
$syslogseverity-text == 'info' then ?dynTsmInfo;tsmFormat
if $programname == 'dsmserv' and $syslogfacility-text == 'local3' and
$syslogseverity-text == 'err' then ?dynTsmError;tsmFormat
# Rules
auth,authpriv.* ?dynAuthLog
*.*;\
mail.none;\
cron.none -?dynSyslog
cron.* ?dynCronLog
daemon.* -?dynDaemonLog
kern.* -?dynKernLog
mail.* -?dynMailLog
user.* -?dynUserLog
# Switch back to default ruleset
$RuleSet RSYSLOG_DefaultRuleset
signature.asc
Description: Message signed with OpenPGP using GPGMail
_______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
