On Mon, 17 Dec 2012, Rainer Gerhards wrote:
Just a quick follow-up: I currently work with Rick on this issue. It looks like
there are actually two issues in one:
1) in v5, it can happen that an action in front of an included get's reordered,
and this seems to have happened here. This is actually the second time in many
years that I see that. I won't fix that in v5, as it would probably lead to
something like the v7 engine ;) Rick has moved to v7.
2) omudpspoof discards messages that are larger than the network-supported MTU. I
am working to fragment them. Note that I know that some receivers have problems
with fragmented UDP, but hey, what else shall we do? Messages over 64K can never
be transmitted in a single UDP packet, and as such 7.2.5 will now truncate them to
64K. I will probably change v5 so that >1472 byte messages get truncated, but
the message sent instead of being discarded.
Will keep you posted when there is notable news (but maybe you just see a new
release announcement if everything works out and no new problems pop up).
I would suggest making the default spoof template do the truncation, so that
someone who's system can handle larger messages can set it longer.
David Lang
Rainer
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-
boun...@lists.adiscon.com] On Behalf Of Rick Brown
Sent: Friday, December 14, 2012 3:24 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
----- Original Message -----
From: "Rainer Gerhards" <rgerha...@hq.adiscon.com>
To: "rsyslog-users" <rsyslog@lists.adiscon.com>
Sent: Friday, December 14, 2012 9:06:10 AM
Subject: Re: [rsyslog] imfile and omudpspoof
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-
boun...@lists.adiscon.com] On Behalf Of Rick Brown
Sent: Friday, December 14, 2012 3:04 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
----- Original Message -----
From: "Rainer Gerhards" <rgerha...@hq.adiscon.com>
To: "rsyslog-users" <rsyslog@lists.adiscon.com>
Sent: Friday, December 14, 2012 2:40:05 AM
Subject: Re: [rsyslog] imfile and omudpspoof
-----Original Message-----
From: Rick Brown [mailto:rick.br...@oit.gatech.edu]
Sent: Thursday, December 13, 2012 7:06 PM
To: Rainer Gerhards
Subject: Re: [rsyslog] imfile and omudpspoof
You're right, it's rather large. I gzipp'd the output and
posted
it
at xxxx
Mhhh... That log does not include anything from imfile. Did you
accidently capture the wrong system?
I captured from the syslog server in the middle. Let me try to be
clearer in stating the problem.
Client A scrapes log files with imfile and sends them to my syslog
server. The syslog server is configured to forward the messages to
a
SIEM using imudpspoof. The syslog server successfully writes the
logs
to disk, however not all messages are being spoofed and sent on.
The
person who runs the SIEM noted the difference and I then observed
tcpdumps on the syslog server. Watching in two terminals:
tcpdump host A and host syslog
and:
tcpdump host syslog and host SIEM
I see many more messages in the top terminal than the bottom. On
first
glance, it looked like none of the messages generated by imfile on
client A were making it to the SIEM, but now that I've stared at it
long enough, I do occassionally see a couple go by. If I restarted
rsyslog while watching the tcpdumps, I see a burst of the imfile
messages get spoofed, then they stop for a while, then a few will
trickle by.
OK - I misunderstood the initial description. So let's continue with
that debug log file. Can you tell me which logs lines are
imfile-originated (and not delivered) and which one are not?
Rainer
Ahh, good. All of the imfile messages are set to facility news and we
no longer actually run news servers, so it's a safe bet that any news
messages is generated from imfile. I've been particularly
concentrating on a host cas1, who's ip ends in .165.151, as it
generates quite a bit of application logs that I scrape with imfile.
And the spoof target address ends in .165.95.
From what I have seen, it may also happen that the debug log info
is
not sufficient. Just treat this as advance warning ;)
We can probably get around a limitation by creating an additional
log
file, so that I can see all message properties. Please add
*.* /var/log/rsyslog-props.log;RSYSLOG_DebugFormat
To your config (the file name is obviously irrelevant, but the
template is important!).
Make sure that the debug log and the property file are from the
same
run.
Rainer
Can do. I'll go look into this now.
I see that some of the imfile generated messages are making it
through
to the SIEM from time to time, but the vast majority are being
dropped.
I have to wonder if I'm overflowing a buffer in the output
modules.
Thanks for taking a look!
----- Original Message -----
From: "Rainer Gerhards" <rgerha...@hq.adiscon.com>
To: "rsyslog-users" <rsyslog@lists.adiscon.com>
Sent: Thursday, December 13, 2012 10:05:02 AM
Subject: Re: [rsyslog] imfile and omudpspoof
Can you post a (sufficiently large or complete) debug log
that
exposes the problem? The list will probably reject it, so it
is
best
to put it on something like pastbin.
Rainer
-----Original Message-----
From: rsyslog-boun...@lists.adiscon.com [mailto:rsyslog-
boun...@lists.adiscon.com] On Behalf Of Rick Brown
Sent: Thursday, December 13, 2012 4:01 PM
To: rsyslog-users
Subject: Re: [rsyslog] imfile and omudpspoof
On Tue, 11 Dec 2012, Rick Brown wrote:
I use imfile to gather application logs such as
apache,
tomcat,
php, etc. and send those on to my syslog server along
with
the
client machines normal syslog traffic. My syslog
server
then
dutifully writes all the messages locally and
additionally
forwards the messages on to a SIEM product via
omudpspoof.
Watching packet captures, I can see some messages are
being
spoofed
and sent on to the SIEM, but some are not. At first
glance,
it
appears that all regular syslog messages that are
generated
on
the
client are being spoofed and sent on to the SIEM, but
most,
if
not
all messages generated via imfile on the client are
not
being
spoofed and sent on to the SIEM at all.
I've tried the standard
*.* :omudspoof:
as well as
$template spooftemplate,"%fromhost-ip% %rawmsg%"
*.* :omudpspoof:;spooftemplate
and
$template spooftemplate,"%rawmsg%"
*.* :omudpspoof:;spooftemplate
All with the same effect. Am I missing something
here?
Is
anyone
else doing similar, or seen similar behavior?
For the record, I'm running a patched version of
5.8.11.
The
patch,
now that I'm reading it again, was to protect against
calling
more
than one instance of libnet code in the omudpspoof
module.
the udpspoof module grabs the first field from the
template
and
uses
that as the
IP address to spoof from. What is the fromhost-ip when
you
get
the
files from
imfile?
Ahh, you may be onto something there.. the messages
coming
into my
syslog server have just the hostname in it, not FQDN, and
not
IP
address. The few that I see leaving for the SIEM have
the
IP
address in that field of the message. Now to figure out
why
they're different.
Hrm. Upon inspection, all logs are coming in with just
the
hostname,
so that's a dead end. All normal syslogs (like cron.info,
kernel.notice, daemon.debug, etc.) I can see are being
spoofed
and
sent
on to the SIEM, but the logs generated by imfile (all
configured as
news.info) from client machines get logged on the syslog
server,
but
aren't being spoofed and sent on to the SIEM.
--
Rick Brown
Office of Information Technology
Georgia Institute of Technology
___________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT
POST if you DON'T LIKE THAT.
--
Rick Brown
Office of Information Technology
Georgia Institute of Technology
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a
myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST
if you DON'T LIKE THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
