On Fri, 19 Oct 2012, Risto Vaarandi wrote:
> On 10/18/2012 03:20 AM, [email protected] wrote:
>> There is a lot of activity nowdays around 'structured' logging, with a lot
>> of the effort going into having systems generate logs in the JSON format
>> (see the stuff from systemd, project lumberjack, etc). Some of this is
>> going to be showing up in the next release of Fedora where the kernel logs
>> are going to be 'structured'
>>
>> to a large extent, these logs can be handled just like all other logs in
>> SEC, but I was wondering if there has been any thought into possible ways
>> to take advantage of this structure?
>>
>> The answer may be to do all this with perl snippets, but this could be
>> standard enought o make it worthwhile implementing more direct support.
>>
>> I'm thinking that it would be nice if there was a way to detect a
>> structured log and have it parsed once into a (probably hash based) data
>> structure, and then have the ability for all the normal SEC variable use
>> be able to refer to the elements of that structure.
>>
>> thoughts?
>>
>> David Lang
>>
>
> David,
>
> I've checked some json messages and the following subroutine appears to do
> basic parsing for these messages (including the case where a field points to
> another record). Keyword-value pairs are stored into a return hash, and if
> keyword was found in a substructure, the keyword will get the parent keyword
> for the prefix.
>
> For example, if the structure is
> {"source": { "ip":"127.0.0.1", "host":"myhost"}, "dest":"myhost2" }
> the hash will contain the following data:
> source_ip = 127.0.0.1
> source_host = myhost
> dest = myhost2
I was thinking more along the lines of
$hash{source}{ip} = 127.0.0.1
$hash{source}{host} = myhost
$hash{dest} = myhost2
but your approach will work. I'm not particularly happy with using
underscore as the level separator (I am too likely to use it in a variable
name), could we use . or would that break other things?
the question is if it's possible to use this result in other parts of SEC
(outside of perl snippets) such as when constructing the desc field.
in another message you refer to %+ variables, that may be a feature of SEC
that does this (which I'm not familiar with). Is there a good explination
on using this anywhere?
more background.
The JSON structured logging that I am talking about is line-based so that
it will continue to work with syslog transports that don't know anything
about it. It is possible to do most of the work by just ignoring that it's
structured and use the normal regex etcmatching, but I'm thinking that if
we can do things right, we should be able to make it so that it's faster
and easier to config by taking advantage of the structure when it exists.
The two types of structured data that I think it makes sense to deal with
are the JSON logs that i was pointing out, and trying to turn logs that
have lots of name=value pairs into %+name variables.
David Lang
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
