Michael Richards wrote:
In my case, I had a lookup table to determine the log type based on a json
list of 500 or so different sending ips. Although parsed by default chain,
the work has to be largely redone to get valid properties.
In this case, I think mmnormalize is the way to go.
the default parsing is fast, and mmnormalize is fast, so you should not have
that much of a problem
200G of logs a day is an average of ~10k logs/sec (yes peak will be higher).
This is not extreme for syslog. It does get up into the range where you will
want to watch performance, but mmnormalize reparsing it should not be an issue.
mmnormalize is very efficient with lots of parsing rules. You don't need to
identify the type of rule and invoke the appropriate parser, you can put all the
rules for all parsing in one ruleset (assuming the rules are specific enough to
not have two that would parse the same message). At one prior job, I had a
parsing ruleset of 1400 patterns and the difference in performance between a
message that matched the first rule in the ruleset and one that matched the last
rule in the ruleset was only 30% (I did this testing after seeing something
where a 30 line regex based ruleset was several thousand % slower on the last
line than the first line)
There is a parser module that takes a mmnormalize ruleset and uses that to parse
the ruleset.
With the initial parsing, you define a stack of parsers, and each one checks the
message to see if it matches the pattern that it knows how to parse, if it
matches, it finalizes the parsing and other parsers aren't checked, if it
doesn't match, the next parser gets invoked.
But this can only look at the incoming message, not other things like a table
lookup.
but if you can identify the messages by their content efficiently, this will
work.
The pmcisco module that I wrote is actually a bit of a cheat, it doesn't fully
parse the message, it just looks to see if it has the messageid instead of the
expected syslog header data, if it finds it, it removes it and lets the message
fall through to the next parser (it saved me having to duplicate all the normal
parser functionality)
you currently are identifying the messages by their source ip, but can you
identify them by content?
David Lang
_______________________________________________
rsyslog mailing list
https://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com/professional-services/
What's up with rsyslog? Follow https://twitter.com/rgerhards
NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of
sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE
THAT.
