On 10/18/2012 03:20 AM, [email protected] wrote:
> There is a lot of activity nowdays around 'structured' logging, with a lot
> of the effort going into having systems generate logs in the JSON format
> (see the stuff from systemd, project lumberjack, etc). Some of this is
> going to be showing up in the next release of Fedora where the kernel logs
> are going to be 'structured'
>
> to a large extent, these logs can be handled just like all other logs in
> SEC, but I was wondering if there has been any thought into possible ways
> to take advantage of this structure?
>
> The answer may be to do all this with perl snippets, but this could be
> standard enought o make it worthwhile implementing more direct support.
>
> I'm thinking that it would be nice if there was a way to detect a
> structured log and have it parsed once into a (probably hash based) data
> structure, and then have the ability for all the normal SEC variable use
> be able to refer to the elements of that structure.

This is a good question. A lot depends on how the structured message is 
actually written down, and if this is always done in the same way. In 
the web, I have seen several slightly different formats. Also, in some 
cases structured messages are single-line entries, while in other cases 
they could cover many lines. The latter approach renders most UNIX tools 
(like grep, wc, sed, and many others) essentially useless, since when 
doing searches from command line through pipelines, you are only hitting 
fragments of relevant messages.

Assuming the logging is done in single-line fashion which allows for 
backward compatibility, it is possible to do some sort of automatic 
parsing for json style logs. However, there could still be some issues 
-- on one hand, there is a structure which has been defined by the 
logging application, but in a number of cases you would still need to do 
some matching for such log (e.g., in the case of apache logs, there 
could be a need for matching some parts of the requested URL). So it is 
the question how well the defined structure actually meets your needs 
and whether you need to look inside the fields for discovering 
additional data. Also, while in principle $+{name} variables can be used 
for json input fields, the fields can easily be full sub-records with 
their own structure structure, which complicates the issue further. 
Also, it is not clear how field delimiters are escaped inside field 
values, and whether the removal of escaping characters needs to be done.

I've felt for some time that SEC would need some sort of simple 
interface for recognizing input events with a structure, but such 
interface should not be implemented as a SEC patterns, but rather as a 
processing routine for specific input source. And if the json logging 
would be done in multi-line way, this would essentially be the only 
reasonable option.

with kind regards,
risto

>
> thoughts?
>
> David Lang
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Simple-evcorr-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
>


------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users

Reply via email to