Hi, Thank you for the welcome! The concepts behind Metron are nice, very nice. Finally network log / data analysis is possible using different approaches. It has been a few years since I used tools such as Suricata, Bro, Snort, OSSEC, PF_RING. But the integration of some of those at scale... nice possibilities! In our environment we only have some krb issues, maybe related to our specific choices, but I am sure we will be able to solve those....
I think our use cases can perfectly work with the defaults. We probably need to get the necessary experience. Both in parsing, using Stellar, writing Elasticsearch templates, Spark.... So basically If I set the timestamp field correctly, we can correlate and profile with other events around the same timestamp, right? We could also write a 3th party parser , almost a copy of the CEF parser and modify it specifically for this device (and contribute back...)? But if it is only about the timestamp, probably only the fieldTransformation is necessary. I suppose it also by design that the ASA parser only does some 'basic' parsing? Currently there is no interpretation of access-list, hit counts, etcetera. But I thought it was better using a Java parser than grok patterns if possible, because of optimalisations....? Pieter On Mon, Dec 17, 2018 at 5:49 PM Simon Elliston Ball < si...@simonellistonball.com> wrote: > Hi Pieter, > > Welcome to the Metron community. > > The logic used by the CEF parser right now is to populate the timestamp > field as follows: > 1. if the rt field exists, use that. > 2. if there is a field matching the syslogTime pattern (either the old > pattern of 5424) > 3. If neither are present, it uses current system time (as a last resort) > > You can use stellar fieldTransformations after the fact to change the > timestamp field to map to anything you like, including any parsed field. > > The field type is ignored in custom fields from CEF, since type is not a > required construct in the Metron JSON object, so we just take the > corresponding label field and map the value to that, since the Metron > schema is more flexible than the CEF schema. I can see there may be edge > cases where this could create a collision, and would be interested to hear > about your use case around this, and any problems that approach might > cause, but it seems to have worked for most people so far. > > Simon > > > On Mon, 17 Dec 2018 at 15:38, Pieter Baele <pieter.ba...@gmail.com> wrote: > >> Hi, >> >> For correlation & profiling the presense of a correct timestamp / >> eventtime is important, >> what to do with a device implementing CEF output, but not properly >> providing the rt field? >> Also syslogTime is not parsed by the CEF parser. >> >> There is another field present, how can I assure this field is taken as >> timestamp for further analysis and ingest in ES and HDFS? >> >> Also, is it not necessary to have the type of a field set during (CEF) >> parsing (maybe I am missing something there) >> >> Sincerely >> Pieter >> >> >> >> > > -- > -- > simon elliston ball > @sireb >
