2012/10/18 Risto Vaarandi <[email protected]>:
> On 10/18/2012 03:20 AM, [email protected] wrote:
>> There is a lot of activity nowdays around 'structured' logging, with a lot
>> of the effort going into having systems generate logs in the JSON format
>> (see the stuff from systemd, project lumberjack, etc). Some of this is
>> going to be showing up in the next release of Fedora where the kernel logs
>> are going to be 'structured'
>>
>> to a large extent, these logs can be handled just like all other logs in
>> SEC, but I was wondering if there has been any thought into possible ways
>> to take advantage of this structure?
>>
>> The answer may be to do all this with perl snippets, but this could be
>> standard enought o make it worthwhile implementing more direct support.
>>
>> I'm thinking that it would be nice if there was a way to detect a
>> structured log and have it parsed once into a (probably hash based) data
>> structure, and then have the ability for all the normal SEC variable use
>> be able to refer to the elements of that structure.
>
> This is a good question. A lot depends on how the structured message is
> actually written down, and if this is always done in the same way. In
> the web, I have seen several slightly different formats. Also, in some
> cases structured messages are single-line entries, while in other cases
> they could cover many lines. The latter approach renders most UNIX tools
> (like grep, wc, sed, and many others) essentially useless, since when
> doing searches from command line through pipelines, you are only hitting
> fragments of relevant messages.
>
> Assuming the logging is done in single-line fashion which allows for
> backward compatibility, it is possible to do some sort of automatic
> parsing for json style logs. However, there could still be some issues
> -- on one hand, there is a structure which has been defined by the
> logging application, but in a number of cases you would still need to do
> some matching for such log (e.g., in the case of apache logs, there
> could be a need for matching some parts of the requested URL). So it is
> the question how well the defined structure actually meets your needs
> and whether you need to look inside the fields for discovering
> additional data. Also, while in principle $+{name} variables can be used
> for json input fields, the fields can easily be full sub-records with
> their own structure structure, which complicates the issue further.
> Also, it is not clear how field delimiters are escaped inside field
> values, and whether the removal of escaping characters needs to be done.
>
> I've felt for some time that SEC would need some sort of simple
> interface for recognizing input events with a structure, but such
> interface should not be implemented as a SEC patterns, but rather as a
> processing routine for specific input source. And if the json logging
> would be done in multi-line way, this would essentially be the only
> reasonable option.
>
> with kind regards,
> risto
>
>>
>> thoughts?
>>
>> David Lang
>>
...what about having a separate pattern type for parsing purposes,
which would create a pattern cache entry? Since the last version, SEC
has the ability of matching multi-line messages even if several input
files are involved. I am thinking about having a rule like
type=Parse
ptype=PerlFuncN
pattern=sub { ... ; return %hash }
varmap=event1
The %hash associative array would contain key-value pairs, which would
correspond to json data structure, but would also set match variables
$+{key}. This approach would be suitable for any other structured log
message format, such as IETF syslog. It would also take advantage of
the newest SEC features which were added to the 2.6.2 version. The
only change to the code would be the handling of returned hash table
(currently, SEC is accepting lists and scalars as an output from
PerlFunc).
I was also thinking about adding json support directly, but...
recently, there have been several log format standardization efforts,
but none of the formats has become the default one. And I feel that
soon new formats will be proposed, which will eventually make event
logs into something viewable only with some graphical tool :)
Any thoughts?
with kind regards,
risto
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
