> > Just a quick follow-up: I currently work with Rick on this issue. It > looks like there are actually two issues in one: > > > > 1) in v5, it can happen that an action in front of an included get's > reordered, and this seems to have happened here. This is actually the > second time in many years that I see that. I won't fix that in v5, as > it would probably lead to something like the v7 engine ;) Rick has > moved to v7. > > > > 2) omudpspoof discards messages that are larger than the network- > supported MTU. I am working to fragment them. Note that I know that > some receivers have problems with fragmented UDP, but hey, what else > shall we do? Messages over 64K can never be transmitted in a single UDP > packet, and as such 7.2.5 will now truncate them to 64K. I will > probably change v5 so that >1472 byte messages get truncated, but the > message sent instead of being discarded. > > > > Will keep you posted when there is notable news (but maybe you just > see a new release announcement if everything works out and no new > problems pop up). > > I would suggest making the default spoof template do the truncation, so > that > someone who's system can handle larger messages can set it longer.
That's rather complicated, because the template system can limit fields but not the overall string. If the consensus is that this is a bad change, I'll revert the change and keep the old behavior. I just thought I solve this hard-to-find-and-debug issue, but I don't want to invest any real time in v5 (I don't even know if I ever will do a new release...). What do you say in the light of this? Rainer > > David Lang > > > Rainer > > > >> -----Original Message----- > >> From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog- > >> boun...@lists.adiscon.com] On Behalf Of Rick Brown > >> Sent: Friday, December 14, 2012 3:24 PM > >> To: rsyslog-users > >> Subject: Re: [rsyslog] imfile and omudpspoof > >> > >> ----- Original Message ----- > >>> From: "Rainer Gerhards" <rgerha...@hq.adiscon.com> > >>> To: "rsyslog-users" <rsyslog@lists.adiscon.com> > >>> Sent: Friday, December 14, 2012 9:06:10 AM > >>> Subject: Re: [rsyslog] imfile and omudpspoof > >>> > >>>> -----Original Message----- > >>>> From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog- > >>>> boun...@lists.adiscon.com] On Behalf Of Rick Brown > >>>> Sent: Friday, December 14, 2012 3:04 PM > >>>> To: rsyslog-users > >>>> Subject: Re: [rsyslog] imfile and omudpspoof > >>>> > >>>> ----- Original Message ----- > >>>>> From: "Rainer Gerhards" <rgerha...@hq.adiscon.com> > >>>>> To: "rsyslog-users" <rsyslog@lists.adiscon.com> > >>>>> Sent: Friday, December 14, 2012 2:40:05 AM > >>>>> Subject: Re: [rsyslog] imfile and omudpspoof > >>>>> > >>>>> > >>>>> > >>>>>> -----Original Message----- > >>>>>> From: Rick Brown [mailto:rick.br...@oit.gatech.edu] > >>>>>> Sent: Thursday, December 13, 2012 7:06 PM > >>>>>> To: Rainer Gerhards > >>>>>> Subject: Re: [rsyslog] imfile and omudpspoof > >>>>>> > >>>>>> You're right, it's rather large. I gzipp'd the output and > >>>>>> posted > >>>>>> it > >>>>>> at xxxx > >>>>> > >>>>> Mhhh... That log does not include anything from imfile. Did you > >>>>> accidently capture the wrong system? > >>>>> > >>>> > >>>> I captured from the syslog server in the middle. Let me try to > be > >>>> clearer in stating the problem. > >>>> > >>>> Client A scrapes log files with imfile and sends them to my syslog > >>>> server. The syslog server is configured to forward the messages > to > >>>> a > >>>> SIEM using imudpspoof. The syslog server successfully writes the > >>>> logs > >>>> to disk, however not all messages are being spoofed and sent on. > >>>> The > >>>> person who runs the SIEM noted the difference and I then observed > >>>> tcpdumps on the syslog server. Watching in two terminals: > >>>> > >>>> tcpdump host A and host syslog > >>>> > >>>> and: > >>>> tcpdump host syslog and host SIEM > >>>> > >>>> I see many more messages in the top terminal than the bottom. On > >>>> first > >>>> glance, it looked like none of the messages generated by imfile on > >>>> client A were making it to the SIEM, but now that I've stared at > it > >>>> long enough, I do occassionally see a couple go by. If I > restarted > >>>> rsyslog while watching the tcpdumps, I see a burst of the imfile > >>>> messages get spoofed, then they stop for a while, then a few will > >>>> trickle by. > >>> > >>> OK - I misunderstood the initial description. So let's continue > with > >>> that debug log file. Can you tell me which logs lines are > >>> imfile-originated (and not delivered) and which one are not? > >>> > >>> Rainer > >> > >> Ahh, good. All of the imfile messages are set to facility news and > we > >> no longer actually run news servers, so it's a safe bet that any > news > >> messages is generated from imfile. I've been particularly > >> concentrating on a host cas1, who's ip ends in .165.151, as it > >> generates quite a bit of application logs that I scrape with imfile. > >> And the spoof target address ends in .165.95. > >> > >>>> > >>>> > >>>>> From what I have seen, it may also happen that the debug log info > >>>>> is > >>>>> not sufficient. Just treat this as advance warning ;) > >>>>> > >>>>> We can probably get around a limitation by creating an additional > >>>>> log > >>>>> file, so that I can see all message properties. Please add > >>>>> > >>>>> *.* /var/log/rsyslog-props.log;RSYSLOG_DebugFormat > >>>>> > >>>>> To your config (the file name is obviously irrelevant, but the > >>>>> template is important!). > >>>>> > >>>>> Make sure that the debug log and the property file are from the > >>>>> same > >>>>> run. > >>>>> > >>>>> Rainer > >>>> > >>>> > >>>> Can do. I'll go look into this now. > >>>> > >>>> > >>>>>> I see that some of the imfile generated messages are making it > >>>>>> through > >>>>>> to the SIEM from time to time, but the vast majority are being > >>>>>> dropped. > >>>>>> I have to wonder if I'm overflowing a buffer in the output > >>>>>> modules. > >>>>>> > >>>>>> Thanks for taking a look! > >>>>>> > >>>>>> ----- Original Message ----- > >>>>>>> From: "Rainer Gerhards" <rgerha...@hq.adiscon.com> > >>>>>>> To: "rsyslog-users" <rsyslog@lists.adiscon.com> > >>>>>>> Sent: Thursday, December 13, 2012 10:05:02 AM > >>>>>>> Subject: Re: [rsyslog] imfile and omudpspoof > >>>>>>> > >>>>>>> Can you post a (sufficiently large or complete) debug log > >>>>>>> that > >>>>>>> exposes the problem? The list will probably reject it, so it > >>>>>>> is > >>>>>>> best > >>>>>>> to put it on something like pastbin. > >>>>>>> > >>>>>>> Rainer > >>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog- > >>>>>>>> boun...@lists.adiscon.com] On Behalf Of Rick Brown > >>>>>>>> Sent: Thursday, December 13, 2012 4:01 PM > >>>>>>>> To: rsyslog-users > >>>>>>>> Subject: Re: [rsyslog] imfile and omudpspoof > >>>>>>>> > >>>>>>>>>> On Tue, 11 Dec 2012, Rick Brown wrote: > >>>>>>>>>> > >>>>>>>>>>> I use imfile to gather application logs such as > >>>>>>>>>>> apache, > >>>>>>>>>>> tomcat, > >>>>>>>>>>> php, etc. and send those on to my syslog server along > >>>>>>>>>>> with > >>>>>>>>>>> the > >>>>>>>>>>> client machines normal syslog traffic. My syslog > >>>>>>>>>>> server > >>>>>>>>>>> then > >>>>>>>>>>> dutifully writes all the messages locally and > >>>>>>>>>>> additionally > >>>>>>>>>>> forwards the messages on to a SIEM product via > >>>>>>>>>>> omudpspoof. > >>>>>>>>>>> > >>>>>>>>>>> Watching packet captures, I can see some messages are > >>>>>>>>>>> being > >>>>>>>>>>> spoofed > >>>>>>>>>>> and sent on to the SIEM, but some are not. At first > >>>>>>>>>>> glance, > >>>>>>>>>>> it > >>>>>>>>>>> appears that all regular syslog messages that are > >>>>>>>>>>> generated > >>>>>>>>>>> on > >>>>>>>>>>> the > >>>>>>>>>>> client are being spoofed and sent on to the SIEM, but > >>>>>>>>>>> most, > >>>>>>>>>>> if > >>>>>>>>>>> not > >>>>>>>>>>> all messages generated via imfile on the client are > >>>>>>>>>>> not > >>>>>>>>>>> being > >>>>>>>>>>> spoofed and sent on to the SIEM at all. > >>>>>>>>>>> > >>>>>>>>>>> I've tried the standard > >>>>>>>>>>> *.* :omudspoof: > >>>>>>>>>>> > >>>>>>>>>>> as well as > >>>>>>>>>>> $template spooftemplate,"%fromhost-ip% %rawmsg%" > >>>>>>>>>>> *.* :omudpspoof:;spooftemplate > >>>>>>>>>>> > >>>>>>>>>>> and > >>>>>>>>>>> $template spooftemplate,"%rawmsg%" > >>>>>>>>>>> *.* :omudpspoof:;spooftemplate > >>>>>>>>>>> > >>>>>>>>>>> All with the same effect. Am I missing something > >>>>>>>>>>> here? > >>>>>>>>>>> Is > >>>>>>>>>>> anyone > >>>>>>>>>>> else doing similar, or seen similar behavior? > >>>>>>>>>>> > >>>>>>>>>>> For the record, I'm running a patched version of > >>>>>>>>>>> 5.8.11. > >>>>>>>>>>> The > >>>>>>>>>>> patch, > >>>>>>>>>>> now that I'm reading it again, was to protect against > >>>>>>>>>>> calling > >>>>>>>>>>> more > >>>>>>>>>>> than one instance of libnet code in the omudpspoof > >>>>>>>>>>> module. > >>>>>>>>>> > >>>>>>>>>> the udpspoof module grabs the first field from the > >>>>>>>>>> template > >>>>>>>>>> and > >>>>>>>>>> uses > >>>>>>>>>> that as the > >>>>>>>>>> IP address to spoof from. What is the fromhost-ip when > >>>>>>>>>> you > >>>>>>>>>> get > >>>>>>>>>> the > >>>>>>>>>> files from > >>>>>>>>>> imfile? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Ahh, you may be onto something there.. the messages > >>>>>>>>> coming > >>>>>>>>> into my > >>>>>>>>> syslog server have just the hostname in it, not FQDN, and > >>>>>>>>> not > >>>>>>>>> IP > >>>>>>>>> address. The few that I see leaving for the SIEM have > >>>>>>>>> the > >>>>>>>>> IP > >>>>>>>>> address in that field of the message. Now to figure out > >>>>>>>>> why > >>>>>>>>> they're different. > >>>>>>>> > >>>>>>>> Hrm. Upon inspection, all logs are coming in with just > >>>>>>>> the > >>>>>>>> hostname, > >>>>>>>> so that's a dead end. All normal syslogs (like cron.info, > >>>>>>>> kernel.notice, daemon.debug, etc.) I can see are being > >>>>>>>> spoofed > >>>>>>>> and > >>>>>>>> sent > >>>>>>>> on to the SIEM, but the logs generated by imfile (all > >>>>>>>> configured as > >>>>>>>> news.info) from client machines get logged on the syslog > >>>>>>>> server, > >>>>>>>> but > >>>>>>>> aren't being spoofed and sent on to the SIEM. > >>>> > >>>> > >>>> > >>>> -- > >>>> Rick Brown > >>>> Office of Information Technology > >>>> Georgia Institute of Technology > >> ___________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > >>> POST if you DON'T LIKE THAT. > >>> > >> > >> -- > >> Rick Brown > >> Office of Information Technology > >> Georgia Institute of Technology > >> > >> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >> myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT > POST > >> if you DON'T LIKE THAT. > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if you DON'T LIKE THAT. _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
