I traced the problem upstream and looks like there are hosts that are sending bad syslog messages by inserting an extra space between source/host/ip and syslogtag. Trying to find out what these hosts are (OS, app, version etc).
On Tue, Mar 4, 2014 at 11:56 AM, Xuri Nagarin <[email protected]> wrote: > Hi David, > > On Tue, Mar 4, 2014 at 11:13 AM, David Lang <[email protected]> wrote: >> this is the debug logs, what was mentiond was to write a lot with the format >> RSYSLOG_DebugFormat >> >> *.* /var/log/somename;RSYSLOG_DebugFormat >> >> it will show you the raw log that was received, and what ends up in each of >> the default properties. > The first blob I posted *is* output by setting "RSYSLOG_DebugFormat". > It is not a debug log from the rsyslogd daemon. > > >> >> the tcpdump you are showing looks like there are two spaces between the IP >> address and the syslog tag, is that the input to rsyslog that you are >> showing? > Yes, the tcpdump shows packets coming from the Window RSyslog agent to > the Linux RSyslog server. The flow is: > Router/UnixHost (A) -> Windows RSyslog Agent (B) -> Linux RSyslog > landing server (C) -> Final RSyslog aggregator (D) > > The debug logformat logs and tcpdump are from "B". On "A", the output > format is simply set to the default "%msg%" > > Hope this clarifies what I posted. > > > Thanks, > > Xuri > > >> >> David Lang >> >> On Tue, 4 Mar 2014, Xuri Nagarin wrote: >> >>> Date: Tue, 4 Mar 2014 11:07:55 -0800 >>> From: Xuri Nagarin <[email protected]> >>> Reply-To: rsyslog-users <[email protected]> >>> To: rsyslog-users <[email protected]> >>> Subject: Re: [rsyslog] omfwd inserts extra space between source and >>> syslogtag >>> >>> >>> For "sshd" syslogtag, here is what the debug log lines look like: >>> >>> inputname: imudp rawmsg: '<87>Mar 04 18:39:18 10.0.188.226 >>> sshd[24647]: pam_unix(sshd:account): expired password for user joe >>> (password aged)' >>> msg: ' sshd[24647]: pam_unix(sshd:session): session opened for user >>> joe by (uid=0)' >>> escaped msg: ' sshd[24647]: pam_unix(sshd:session): session opened for >>> user joe by (uid=0)' >>> inputname: imudp rawmsg: '<86>Mar 04 18:39:18 10.0.188.226 >>> sshd[24647]: pam_unix(sshd:session): session opened for user joe by >>> (uid=0)' >>> msg: ' sshd[24647]: pam_unix(sshd:session): session closed for user joe' >>> escaped msg: ' sshd[24647]: pam_unix(sshd:session): session closed for >>> user joe' >>> inputname: imudp rawmsg: '<86>Mar 04 18:39:18 10.0.188.226 >>> sshd[24647]: pam_unix(sshd:session): session closed for user joe' >>> msg: ' sshd[24651]: pam_unix(sshd:account): expired password for user >>> joe (password aged)' >>> escaped msg: ' sshd[24651]: pam_unix(sshd:account): expired password >>> for user joe (password aged)' >>> inputname: imudp rawmsg: '<87>Mar 04 18:39:23 10.0.188.226 >>> sshd[24651]: pam_unix(sshd:account): expired password for user joe >>> (password aged)' >>> msg: ' sshd[24651]: pam_unix(sshd:session): session opened for user >>> joe by (uid=0)' >>> escaped msg: ' sshd[24651]: pam_unix(sshd:session): session opened for >>> user joe by (uid=0)' >>> inputname: imudp rawmsg: '<86>Mar 04 18:39:23 10.0.188.226 >>> sshd[24651]: pam_unix(sshd:session): session opened for user joe by >>> (uid=0)' >>> msg: ' sshd[24651]: pam_unix(sshd:session): session closed for user joe' >>> escaped msg: ' sshd[24651]: pam_unix(sshd:session): session closed for >>> user joe' >>> >>> Logging with RSyslog_FileFormat: >>> 2014-03-04T19:03:50+00:00 10.0.188.226 sshd[7099]: >>> pam_unix(sshd:account): expired password for user joe (password aged) >>> 2014-03-04T19:03:50+00:00 10.0.188.226 sshd[7099]: >>> pam_unix(sshd:session): session opened for user joe by (uid=0) >>> 2014-03-04T19:03:50+00:00 10.0.188.226 sshd[7099]: >>> pam_unix(sshd:session): session closed for user joe >>> >>> Looking at the tcpdump of packets coming from the Windows RSyslog >>> agent, actually looks like it is the Windows agent inserting an extra >>> space between hostname and syslogtag: >>> $ sudo tcpdump -i eth0 -nnn -c 50000 -A udp port 514 | grep sshd >>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode >>> listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes >>> ."......}.g<86>Mar 04 19:06:02 10.1.42.22 sshd[26968]: >>> pam_unix(sshd:session): session opened for user joe by (uid=0) >>> ."......r..<86>Mar 04 19:06:02 10.1.42.22 sshd[26968]: >>> pam_unix(sshd:session): session closed for user joe >>> >>> >>> >>> >>> >>> On Tue, Mar 4, 2014 at 9:08 AM, Rainer Gerhards >>> <[email protected]> wrote: >>>> >>>> If I remember correctly, underscore is not a valid character in a syslog >>>> tag, so the parser terminates the tag here. The debug format will show. >>>> >>>> Sent from phone, thus brief. >>>> Am 04.03.2014 14:07 schrieb "Radu Gheorghe" <[email protected]>: >>>> >>>>> Hi Xuri, >>>>> >>>>> Maybe others know better, but can you try RSYSLOG_DebugFormat, so see >>>>> which text lands in which variable? Then you might want to make up a >>>>> custom template as a workaround, for example to omit forwarding that >>>>> extra space. >>>>> >>>>> On Tue, Mar 4, 2014 at 3:43 AM, Xuri Nagarin <[email protected]> wrote: >>>>>> >>>>>> Running RSyslog 7.6.0-1 on RHEL 6.2 x64. >>>>>> >>>>>> I have a RSyslog Windows agent forwarding logs to this RHEL based >>>>>> Rsyslog service. >>>>>> >>>>>> I checked the incoming events using tcpdump from the Windows server to >>>>>> the Linux server and verified that incoming messages are good. But >>>>>> when the Linux servers forwards the messages again to another RSyslog >>>>>> server using omfwd/tcp, it inserts an extra space between source/host >>>>>> and the syslogtag. >>>>>> >>>>>> It also eats up an underscore char from FileMonitor generated syslog >>>>>> tags. Example, tag_audit_log ends up being "tag audit_log". >>>>>> >>>>>> I have seen this issue with 7.4 as well. >>>>>> >>>>>> I have tried using a custom template and using the default >>>>>> RSyslog_ForwardFormat template, seems to make no difference - the >>>>>> extra space appears. >>>>>> >>>>>> Any suggestions? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> - Xuri >>>>>> _______________________________________________ >>>>>> rsyslog mailing list >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>>> http://www.rsyslog.com/professional-services/ >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a >>>>>> myriad >>>>> >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>>> >>>>> >>>>> -- >>>>> Performance Monitoring * Log Analytics * Search Analytics >>>>> Solr & Elasticsearch Support * http://sematext.com/ >>>>> _______________________________________________ >>>>> rsyslog mailing list >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>>> http://www.rsyslog.com/professional-services/ >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>>> DON'T LIKE THAT. >>>>> >>>> _______________________________________________ >>>> rsyslog mailing list >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>>> http://www.rsyslog.com/professional-services/ >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you >>>> DON'T >>>> LIKE THAT. >>> >>> _______________________________________________ >>> rsyslog mailing list >>> http://lists.adiscon.net/mailman/listinfo/rsyslog >>> http://www.rsyslog.com/professional-services/ >>> What's up with rsyslog? Follow https://twitter.com/rgerhards >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >>> LIKE THAT. >>> >> _______________________________________________ >> rsyslog mailing list >> http://lists.adiscon.net/mailman/listinfo/rsyslog >> http://www.rsyslog.com/professional-services/ >> What's up with rsyslog? Follow https://twitter.com/rgerhards >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of >> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T >> LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
