2012/10/20 <[email protected]>:
> On Fri, 19 Oct 2012, Risto Vaarandi wrote:
>
>> On 10/18/2012 03:20 AM, [email protected] wrote:
>>> There is a lot of activity nowdays around 'structured' logging, with a lot
>>> of the effort going into having systems generate logs in the JSON format
>>> (see the stuff from systemd, project lumberjack, etc). Some of this is
>>> going to be showing up in the next release of Fedora where the kernel logs
>>> are going to be 'structured'
>>>
>>> to a large extent, these logs can be handled just like all other logs in
>>> SEC, but I was wondering if there has been any thought into possible ways
>>> to take advantage of this structure?
>>>
>>> The answer may be to do all this with perl snippets, but this could be
>>> standard enought o make it worthwhile implementing more direct support.
>>>
>>> I'm thinking that it would be nice if there was a way to detect a
>>> structured log and have it parsed once into a (probably hash based) data
>>> structure, and then have the ability for all the normal SEC variable use
>>> be able to refer to the elements of that structure.
>>>
>>> thoughts?
>>>
>>> David Lang
>>>
>>
>> David,
>>
>> I've checked some json messages and the following subroutine appears to do
>> basic parsing for these messages (including the case where a field points to
>> another record). Keyword-value pairs are stored into a return hash, and if
>> keyword was found in a substructure, the keyword will get the parent keyword
>> for the prefix.
>>
>> For example, if the structure is
>> {"source": { "ip":"127.0.0.1", "host":"myhost"}, "dest":"myhost2" }
>> the hash will contain the following data:
>> source_ip = 127.0.0.1
>> source_host = myhost
>> dest = myhost2
>
> I was thinking more along the lines of
>
> $hash{source}{ip} = 127.0.0.1
> $hash{source}{host} = myhost
> $hash{dest} = myhost2
>
> but your approach will work. I'm not particularly happy with using
> underscore as the level separator (I am too likely to use it in a variable
> name), could we use . or would that break other things?
>
> the question is if it's possible to use this result in other parts of SEC
> (outside of perl snippets) such as when constructing the desc field.
>
> in another message you refer to %+ variables, that may be a feature of SEC
> that does this (which I'm not familiar with). Is there a good explination
> on using this anywhere?
OK, actually I meant named match variables which can be referred with $+{name}.
They can be created from more recent Perl regular expression language
versions with the (?<name>regexp) constructs, and they coexist with
traditional $1, $2, ... match variables. For example, if you have a
regular expression
Connection from (?<ip>[\d.]+)
and a message "Connection from 10.1.1.1" appears, then in addition to
match variable $1 another variable $+{ip} gets created, and both hold
the value of 10.1.1.1
SEC supports $+{name} variables in rule definition, and also allows
for creating new variables with 'varmap' statements (e.g. varmap=
user=2) will create variable $+{user} and set it to $2. In the SEC man
page, there is a discussion on named match variables under a section
which describes pattern types.
In the current version of SEC, named match variables can be created
from regular expressions (provided the Perl version is relatively
recent), but not from PerlFunc patterns. However, JSON structure is
too complex to be parsed with one regular expression only.
My yesterday's post was suggesting that PerlFunc pattern should be
elaborated, so that you could return hash tables from it. In the
current version, PerlFunc return values can be either scalars or
lists, and in both cases return value(s) are assigned to $1, $2, ...
match variables. If scalar is returned, only $1 is set, and if list is
returned with N elements, $1,...,$N are set. However, if there is a
way for returning a hash, hash keys could be treated as variable
names, and corresponding hash values as variable values. The function
I posted yesterday was a quick example how the PerlFunc pattern could
look like.
As for the naming convention -- Perl has a requirement that match
variable names can consist of alphanumerals and underscores.
Currently, similar sanity check has been implemented for SEC named
match variables. However, it is always possible to change the list of
allowed characters. Now my question -- how strict is the keyword
definition for JSON and is the set of legal characters clearly
defined, or can any printable characters be freely used?
>
> more background.
>
> The JSON structured logging that I am talking about is line-based so that
> it will continue to work with syslog transports that don't know anything
> about it.
That's good news, because multiline format would cause huge problems
not only for many shell scripts, but also for syslog transmission
itself. Although not strictly part of RFCs, syslog-ng and rsyslog
support widely used TCP based transmission modes, where newline is
used as a delimiter between messages.
> It is possible to do most of the work by just ignoring that it's
> structured and use the normal regex etcmatching, but I'm thinking that if
> we can do things right, we should be able to make it so that it's faster
> and easier to config by taking advantage of the structure when it exists.
>
What the example from yesterday did, was a recognition of field names
and values without explicitly writing a regexp for parsing several
specific fields. Created match variables can be stored in match cache,
so that the results of parsing are reusable across many rules.
> The two types of structured data that I think it makes sense to deal with
> are the JSON logs that i was pointing out, and trying to turn logs that
> have lots of name=value pairs into %+name variables.
For both of these types, the PerlFunc pattern would allow for defining
a common parsing scheme for name=value pairs. However, for such
already parsed messages any additional matching has to be done through
context expressions. Although it is possible to use custom perl
functions for any comparisons (such as verifying that $+{user} equals
to some constant), maybe it could be worthwhile to implement some
common checks as operators?
with kind regards,
risto
>
> David Lang
>
> ------------------------------------------------------------------------------
> Everyone hates slow websites. So do we.
> Make your web apps faster with AppDynamics
> Download AppDynamics Lite for free today:
> http://p.sf.net/sfu/appdyn_sfd2d_oct
> _______________________________________________
> Simple-evcorr-users mailing list
> [email protected]
> https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
