Hi All, I have an rsyslog-5.8.6 with patch http://git.adiscon.com/?p=rsyslog.git;a=commitdiff;h=791b16ce06d75944e338a6e5fa14c0394bde6f1d, as central log receiver accepting connections at udp/514, tcp/514 and uxsock, and feeding to a mysql backend as well as /var/log/messages. Last week I found the messages file, /var/log/messages, was empty, with the last update from rotated archive /var/log/messages.1 being:
Feb 29 16:23:39 host81 snmpd[324] Received SNMP packet(s) from UDP: [ip_230] 55109 I also observed that the working directory was holding lots of disk queue files, # ls -l /var/spool/rsyslog/mq.00000* -h -rw------- 1 root root 5.1M Feb 29 17:03 /var/spool/rsyslog/mq.00000001 -rw------- 1 root root 5.1M Feb 29 17:32 /var/spool/rsyslog/mq.00000002 ... -rw------- 1 root root 5.1M Mar 12 12:37 /var/spool/rsyslog/mq.00000786 -rw------- 1 root root 110K Mar 12 12:37 /var/spool/rsyslog/mq.00000787 with the first entry in /var/spool/rsyslog/mq.00000001 being: <Obj:1:msg:1: +iProtocolVersion:2:1:0: +iSeverity:2:2:-1: +iFacility:2:2:-1: +msgFlags:2:2:48: +ttGenTime:2:10:1330503819: +tRcvdAt:3:35:2:2012:2:29:16:23:39:131013:6:+:8:0: +tTIMESTAMP:3:35:2:2012:2:29:16:23:39:131013:6:+:8:0: +pszTAG:1:0:: +pszRawMsg:1:94:<30>1 2012-02-29T16:23:39+08:00 host81 snmpd 324 - - Connection from UDP: [ip_230]:49203 : +pszInputName:1:6:imptcp: +pszRcvFrom:1:11:ip_81: +pszRcvFromIP:1:11:ip_81: +offMSG:2:2:-1: >End So I think messages were queued rather than lost. And the head and tail of disk queue were also observed as reported by "lsof -p <pid> -nP" rsyslogd 25177 root 1w REG 8,1 112214 186663427 /var/spool/rsyslog/mq.00000787 rsyslogd 25177 root 70r REG 8,1 5243319 167918536 /var/spool/rsyslog/mq.00000001 An intuitive guess is the queue consumer was stuck for some reason, so I check the mysqld, yes it accepts connection and allows writting. I tried restarting the instance with the hope that disk queue would be consumed. It's not. And I did another restart after having commented out the ommail filtering rule as well as the debug file. To my astonishment, the queue head and tail were no longer reported in "lsof -p <pid> -nP", it looks to me that rsyslog has lost the disk queue? Any clue to debug the problem? I wish the queue could be recovered. Thanks, Kaiwang _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/
