Lots of Inodes in use means lots of files.
There is a possibility that files have been deleted, but some process still has
them open (preventing them from being really deleted), but a restart of rsyslog
means that it doesn't have them open any longer.
You just need to look at your filesystem and see where you have so many files.
ls |while read dir; echo -n "$dir "; find $dir |wc -l ; done
then cd to the directory with the most files in it and repeat until you rind
what's useing so many inodes.
David Lang
On Thu, 21 Mar 2013, EXT-Edge, Sean wrote:
Date: Thu, 21 Mar 2013 06:52:35 -0700
From: "EXT-Edge, Sean" <sean.e...@boeing.com>
Reply-To: rsyslog-users <rsyslog@lists.adiscon.com>
To: "rsyslog@lists.adiscon.com" <rsyslog@lists.adiscon.com>
Subject: [rsyslog] Having an issue with spool file cleanup after remote syslog
server comes back online.
The spool files for my DA syslog forwarder queue have consumed all the inodes
on my system and aren't being cleaned up by rsyslog:
[root@devhost ~]# df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/myvg-rootvol
971040 971040 0 100% /
[root@devhost ~]# cd /var/spool/rsyslog_spool
[root@devhost rsyslog_spool]# ls -l | wc -l
891185
Here's my queue index file:
[root@devhost rsyslog_spool]# cat aqfwd.qi
<OPB:1:qqueue:1:
+iQueueSize:2:2:34:
+tVars.disk.sizeOnDisk:2:5:22629:
+tVars.disk.bytesRead:2:3:499:
End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:64:
+pszFName:1:5:aqfwd:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:0:
+sType:2:1:1:
+tOperationsMode:2:1:2:
+tOpenMode:2:3:384:
+iCurrOffs:2:1:0:
End
.
<Obj:1:strm:1:
+iCurrFNum:2:2:27:
+pszFName:1:5:aqfwd:
+iMaxFiles:2:8:10000000:
+bDeleteOnClose:2:1:1:
+sType:2:1:1:
+tOperationsMode:2:1:1:
+tOpenMode:2:3:384:
+iCurrOffs:2:3:499:
End
.
[root@devhost rsyslog_spool]#
Rsyslog has open file descriptors to these files:
[root@devhost rsyslog_spool]# lsof -p `pgrep rsyslog` | grep rsyslog_spool
rsyslogd 1086 root 4w REG 253,0 541 522935
/var/spool/rsyslog_spool/aqfwd.00000064
rsyslogd 1086 root 5r REG 253,0 499 529874
/var/spool/rsyslog_spool/aqfwd.00000027
[root@devhost rsyslog_spool]#
[root@devhost rsyslog_spool]# ls -l /var/spool/rsyslog_spool/aqfwd.00000064
/var/spool/rsyslog_spool/aqfwd.00000027
-rw-------. 1 root root 499 Feb 22 19:12 /var/spool/rsyslog_spool/aqfwd.00000027
-rw-------. 1 root root 541 Mar 11 17:26 /var/spool/rsyslog_spool/aqfwd.00000064
[root@devhost rsyslog_spool]#
Here's the action queue config:
# Forward messages to central log server.
$ActionQueueType LinkedList
$ActionQueueFileName aqfwd
$ActionQueueSize 150000
$ActionQueueLowWaterMark 30000 # 20% of QueueSize
$ActionQueueHighWaterMark 120000 # 80% of QueueSize
$ActionQueueSaveOnShutdown on
$ActionQueueDiscardSeverity 7 # Dont discard messages.
$ActionQueueMaxFileSize 10g # Ridiculous limit, but dont want to
lose messages :)
$ActionResumeRetryCount -1 # Keep trying
*.* @@syslog-server:10514;RSYSLOG_ForwardFormat # This the line that actually
does forwarding
I thought maybe clearing up a few inodes (and a service restart) might give
rsyslog enough breathing room to start cleaning up these files but that hasn't
proven to be the case.
This is a RHEL6 machine running rsyslog-5.8.10-6.el6.x86_64.
If there's any additional information needed please let me know. I still relatively new
to rsyslog and I didn't write the config above so I can't answer any "why"
questions about it.
Thanks in advance for any help!
-Sean
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
