On 10/18/2012 03:20 AM, [email protected] wrote:
> There is a lot of activity nowdays around 'structured' logging, with a lot
> of the effort going into having systems generate logs in the JSON format
> (see the stuff from systemd, project lumberjack, etc). Some of this is
> going to be showing up in the next release of Fedora where the kernel logs
> are going to be 'structured'
>
> to a large extent, these logs can be handled just like all other logs in
> SEC, but I was wondering if there has been any thought into possible ways
> to take advantage of this structure?
>
> The answer may be to do all this with perl snippets, but this could be
> standard enought o make it worthwhile implementing more direct support.
>
> I'm thinking that it would be nice if there was a way to detect a
> structured log and have it parsed once into a (probably hash based) data
> structure, and then have the ability for all the normal SEC variable use
> be able to refer to the elements of that structure.
>
> thoughts?
>
> David Lang
>
David,
I've checked some json messages and the following subroutine appears to
do basic parsing for these messages (including the case where a field
points to another record). Keyword-value pairs are stored into a return
hash, and if keyword was found in a substructure, the keyword will get
the parent keyword for the prefix.
For example, if the structure is
{"source": { "ip":"127.0.0.1", "host":"myhost"}, "dest":"myhost2" }
the hash will contain the following data:
source_ip = 127.0.0.1
source_host = myhost
dest = myhost2
If this subroutine is given for a PerlFunc pattern, and SEC would have
an additional capability of checking if the return value is a hash
reference, named match variables can be created from the hash content.
Of course, there is a convention that variable names can contain
alphanumerals and underscores only, so this issue has to be handled
somehow. Also, the below parsing function does not remove \ from the
front of an escaped character, but this extra processing can be easily done.
with kind regards,
risto
sub json_parse {
my($line) = $_[0];
my(%hash, @prefix, $prefix, $key, $val, $i);
%hash = ();
@prefix = ();
$prefix = "";
$i = 0;
for (;;) {
if ($line =~ /\G\s*\{\s*/gc) {
print "Got opening brace\n";
if (defined($key)) { push @prefix, $key; $key = undef; }
$prefix = join("_", @prefix);
++$i;
}
elsif ($line =~ /\G\s*\}/gc) {
print "Got closing brace\n";
if (scalar(@prefix)) { pop @prefix; }
$prefix = join("_", @prefix);
--$i;
if ($i < 0) { return 0; } # } without previous {
}
elsif ($line =~ /\G\s*,\s*/gc) {
print "Got separating comma\n";
$val = undef;
}
elsif (!defined($key) && $line =~ /\G\s*"((?:\\.|[^"])++)"\s*:\s*/gc) {
if (defined($val)) { return 0; } # separating comma missing
$key = $1;
print "Got key $key\n";
}
elsif (defined($key) && $line =~ /\G\s*"((?:\\.|[^"])++)"/gc) {
$val = $1;
print "Got value $val\n";
if (length($prefix) > 0) { $hash{$prefix . "_" . $key} = $val; }
else { $hash{$key} = $val; }
$key = undef;
}
else { last; }
}
# keyword without value or unbalanced braces
if (defined($key) || $i != 0) { return 0; }
return \%hash;
}
------------------------------------------------------------------------------
Everyone hates slow websites. So do we.
Make your web apps faster with AppDynamics
Download AppDynamics Lite for free today:
http://p.sf.net/sfu/appdyn_sfd2d_oct
_______________________________________________
Simple-evcorr-users mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/simple-evcorr-users
