On Mon, 11 Oct 2010, Rainer Gerhards wrote:
I have just written another post on the normalization library. It looks like
the design tends to favor a split into two libraries:
http://blog.gerhards.net/2010/10/splitting-up-normalization-library.html
this seems like a good idea.
there is a definate need for a good, efficient parsing tool that can be
used for high volume sites. There are a lot of tools that heavily use
regex matching, but those tend to collapse at high volumes.
you can create your own parser with lex, yacc, bison, or flex, but th work
needed to create the input config file for these (with their specific
syntax) is daunting.
a tool that could take it's configuration in something that looks very
similar to log lines (with some sort of syntax to show the variable part),
that would then compile into something very effient like the tools above
would be very useful for a lot of different tools.
this may just need to be a configuration generator for the tools listed
above that can take the list of annotated lines and create the appropriate
config file to build the parser. If this can accept regex lines and
then compile them down to a parser tree it would be wonderful.
so once there is a high performance parser to pull the data apart, then
the question is what to do with it.
some people will want to write it to various places, others will want to
make decisions based on what is matched.
for those who are wanting to write the normalized output to various
places, a plugin structure like rsyslog has (with the ability to format
the messages based on the various properties that are discovered) is very
appealing, and it may make a lot of sense to see what can be done to
re-use that work. If so, there will need to be a 'format string' that
creates the output with all the properties that are known tagged, but
without including ones that didn't have any matches in this log message.
for those who are wanting to then implement logic based on what it gets,
thing get much more interesting. I suspect that the thing to do here will
be to make the event normalization engine be something that can be a
library included in other programs (in various languages), something so
that you can have the config file be something along the lines of
documentation (hopefully including a sample raw line)
line-to-match
function to call when matched
there are a log of programs out there written to do good and interesting
stuff with lines that it receives, if there was an ability to replace
their sequential 'does it match rule 1, does it match rule 2' logic with a
more efficient parser it would be a huge win.
I don't think you are wanting to tackle that portion of the task.
David Lang
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Rainer Gerhards
Sent: Monday, October 11, 2010 9:01 AM
To: rsyslog-users
Subject: Re: [rsyslog] liblognorm vs. libeventnorm
I would like to add as an argument pro liblognorm, that many people
probably
better understand what "log normalization" is whereas "event
normalization"
may sound strange. In that sense, liblognorm may be a better name.
Feedback
is appreciated.
Rainer
-----Original Message-----
From: [email protected] [mailto:rsyslog-
[email protected]] On Behalf Of Rainer Gerhards
Sent: Sunday, October 10, 2010 11:53 AM
To: rsyslog-users
Subject: [rsyslog] liblognorm vs. libeventnorm
Hi all,
I think I'll start with the libeventnorm name for the normalizing
library
instead of liblognorm. Reason here:
http://blog.gerhards.net/2010/10/liblognorm-or-libeventnorm.html
Further name suggestions or arguments are very welcome!
Rainer
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
_______________________________________________
rsyslog mailing list
http://lists.adiscon.net/mailman/listinfo/rsyslog
http://www.rsyslog.com
