The spool files for my DA syslog forwarder queue have consumed all the inodes on my system and aren't being cleaned up by rsyslog:
[root@devhost ~]# df -i . Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/myvg-rootvol 971040 971040 0 100% / [root@devhost ~]# cd /var/spool/rsyslog_spool [root@devhost rsyslog_spool]# ls -l | wc -l 891185 Here's my queue index file: [root@devhost rsyslog_spool]# cat aqfwd.qi <OPB:1:qqueue:1: +iQueueSize:2:2:34: +tVars.disk.sizeOnDisk:2:5:22629: +tVars.disk.bytesRead:2:3:499: >End . <Obj:1:strm:1: +iCurrFNum:2:2:64: +pszFName:1:5:aqfwd: +iMaxFiles:2:8:10000000: +bDeleteOnClose:2:1:0: +sType:2:1:1: +tOperationsMode:2:1:2: +tOpenMode:2:3:384: +iCurrOffs:2:1:0: >End . <Obj:1:strm:1: +iCurrFNum:2:2:27: +pszFName:1:5:aqfwd: +iMaxFiles:2:8:10000000: +bDeleteOnClose:2:1:1: +sType:2:1:1: +tOperationsMode:2:1:1: +tOpenMode:2:3:384: +iCurrOffs:2:3:499: >End . [root@devhost rsyslog_spool]# Rsyslog has open file descriptors to these files: [root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep rsyslog_spool rsyslogd 1086 root 4w REG 253,0 541 522935 /var/spool/rsyslog_spool/aqfwd.00000064 rsyslogd 1086 root 5r REG 253,0 499 529874 /var/spool/rsyslog_spool/aqfwd.00000027 [root@devhost rsyslog_spool]# [root@devhost rsyslog_spool]# ls -l /var/spool/rsyslog_spool/aqfwd.00000064 /var/spool/rsyslog_spool/aqfwd.00000027 -rw-------. 1 root root 499 Feb 22 19:12 /var/spool/rsyslog_spool/aqfwd.00000027 -rw-------. 1 root root 541 Mar 11 17:26 /var/spool/rsyslog_spool/aqfwd.00000064 [root@devhost rsyslog_spool]# Here's the action queue config: # Forward messages to central log server. $ActionQueueType LinkedList $ActionQueueFileName aqfwd $ActionQueueSize 150000 $ActionQueueLowWaterMark 30000 # 20% of QueueSize $ActionQueueHighWaterMark 120000 # 80% of QueueSize $ActionQueueSaveOnShutdown on $ActionQueueDiscardSeverity 7 # Dont discard messages. $ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want to lose messages :) $ActionResumeRetryCount -1 # Keep trying *.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line that actually does forwarding I thought maybe clearing up a few inodes (and a service restart) might give rsyslog enough breathing room to start cleaning up these files but that hasn't proven to be the case. This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64. If there's any additional information needed please let me know. I still relatively new to rsyslog and I didn't write the config above so I can't answer any "why" questions about it. Thanks in advance for any help! -Sean _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.