I’ll put some thoughts in METRON-1453, unless we want a discuss thread On July 20, 2018 at 10:32:48, Casey Stella (ceste...@gmail.com) wrote:
So, I would really love to see METRON-1453 go in, because I'd love to decouple syslog parsing (very common) from generic grok. On Fri, Jul 20, 2018 at 10:26 AM Otto Fowler <ottobackwa...@gmail.com> wrote: > Metron does not have a generic Syslog Parser. > > Nifi has Syslog parsing ( either Records or standard Processor ), in two > modes. > > ParseSyslog is the original, where regex’s are used to parse the syslog > RFC3164 and RFC5424, but only extracts the common fields ( so the > ‘additional info’ like program id, message id, structured data in 5424 is > in the MSG ). I have recently added a record reader for that method as well > ( Nifi PR#2900 <https://github.com/apache/nifi/pull/2900>). > > Syslog5424Reader(records) and ParseSyslog5424 are new and instead of using > regexes they use a new library simple-syslog–5424 > <https://github.com/palindromicity/simple-syslog-5424> I wrote that > parses RFC5424 messages completely ( note properly formatted RFC 5424 > messages ) see Nifi PR#2805 <https://github.com/apache/nifi/pull/2805> > and Nifi PR#2816 <https://github.com/apache/nifi/pull/2816> using an > antlr grammar. > > You should be able to pick the manner best for you and parse that out in > Nifi if you choose. > > Metron parses syslog as required in specific parsers that have messages > assumed to be embedded in syslog. > > What I have been talking about in METRON–1453 > <https://issues.apache.org/jira/browse/METRON-1453> and other places is > separating out the syslog from the parser, such that the parsers don’t need > to know that the message is delivered embedded in syslog. > > The new parser chaining work would give us an avenue to this, and as you > can see here MetronPR#1099 > <https://github.com/apache/metron/pull/1099#issuecomment-405701948> I > have put that case forward. > > If that hits, I think that we’d be able to : 1. parse plain syslog to > metron 2. parse plain syslog as a transform and then have less complicated, > more specific parsers for the msg part. > > We may end up having syslog parsers and transforms at the end of this. > > In the mean time, if you wish to parse plain syslog in Metron, you will > have to use grok, which doesn’t get structured data. > > If you want the complete 5424 set of data, then you can open a jira for > creating a parser using simple-syslog–5424. > > > > On July 20, 2018 at 04:23:36, Farrukh Naveed Anjum ( > anjum.farr...@gmail.com) wrote: > > Hi, > > I am trying to index the Syslog using CEF Parser with Nifi. > > It does not give any error though, transport data to kafa without indexing > it. It keepg giving FAILED in Spout. > > I believe indexing Syslog are most basic usecase for all. But metron fails > to do it with each in standard format. > > I tried bro for it. But even it keeps giving PARSER Error. > > Any help ? Fast will be apperciated. > > > > > -- > With Regards > Farrukh Naveed Anjum > >
