Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Otto Fowler 
I’ll put some thoughts in METRON-1453, unless we want a discuss thread

On July 20, 2018 at 10:32:48, Casey Stella (ceste...@gmail.com) wrote:

So, I would really love to see METRON-1453 go in, because I'd love to
decouple syslog parsing (very common) from generic grok.

On Fri, Jul 20, 2018 at 10:26 AM Otto Fowler 
wrote:

> Metron does not have a generic Syslog Parser.
>
> Nifi has Syslog parsing ( either Records or standard Processor ), in two
> modes.
>
> ParseSyslog is the original, where regex’s are used to parse the syslog
> RFC3164 and RFC5424, but only extracts the common fields ( so the
> ‘additional info’ like program id, message id, structured data in 5424 is
> in the MSG ). I have recently added a record reader for that method as well
> ( Nifi PR#2900 ).
>
> Syslog5424Reader(records) and ParseSyslog5424 are new and instead of using
> regexes they use a new library simple-syslog–5424
>  I wrote that
> parses RFC5424 messages completely ( note properly formatted RFC 5424
> messages ) see Nifi PR#2805 
> and Nifi PR#2816  using an
> antlr grammar.
>
> You should be able to pick the manner best for you and parse that out in
> Nifi if you choose.
>
> Metron parses syslog as required in specific parsers that have messages
> assumed to be embedded in syslog.
>
> What I have been talking about in METRON–1453
>  and other places is
> separating out the syslog from the parser, such that the parsers don’t need
> to know that the message is delivered embedded in syslog.
>
> The new parser chaining work would give us an avenue to this, and as you
> can see here MetronPR#1099
>  I
> have put that case forward.
>
> If that hits, I think that we’d be able to : 1. parse plain syslog to
> metron 2. parse plain syslog as a transform and then have less complicated,
> more specific parsers for the msg part.
>
> We may end up having syslog parsers and transforms at the end of this.
>
> In the mean time, if you wish to parse plain syslog in Metron, you will
> have to use grok, which doesn’t get structured data.
>
> If you want the complete 5424 set of data, then you can open a jira for
> creating a parser using simple-syslog–5424.
>
>
>
> On July 20, 2018 at 04:23:36, Farrukh Naveed Anjum (
> anjum.farr...@gmail.com) wrote:
>
> Hi,
>
> I am trying to index the Syslog using CEF Parser with Nifi.
>
> It does not give any error though, transport data to kafa without indexing
> it. It keepg giving FAILED in Spout.
>
> I believe indexing Syslog are most basic usecase for all. But metron fails
> to do it with each in standard format.
>
> I tried bro for it. But even it keeps giving PARSER Error.
>
> Any help ? Fast will be apperciated.
>
>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
>

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Casey Stella 
So, I would really love to see METRON-1453 go in, because I'd love to
decouple syslog parsing (very common) from generic grok.

On Fri, Jul 20, 2018 at 10:26 AM Otto Fowler 
wrote:

> Metron does not have a generic Syslog Parser.
>
> Nifi has Syslog parsing ( either Records or standard Processor ), in two
> modes.
>
> ParseSyslog is the original, where regex’s are used to parse the syslog
> RFC3164 and RFC5424, but only extracts the common fields ( so the
> ‘additional info’ like program id, message id, structured data in 5424 is
> in the MSG ). I have recently added a record reader for that method as well
> ( Nifi PR#2900 ).
>
> Syslog5424Reader(records) and ParseSyslog5424 are new and instead of using
> regexes they use a new library simple-syslog–5424
>  I wrote that
> parses RFC5424 messages completely ( note properly formatted RFC 5424
> messages ) see Nifi PR#2805 
> and Nifi PR#2816  using an
> antlr grammar.
>
> You should be able to pick the manner best for you and parse that out in
> Nifi if you choose.
>
> Metron parses syslog as required in specific parsers that have messages
> assumed to be embedded in syslog.
>
> What I have been talking about in METRON–1453
>  and other places is
> separating out the syslog from the parser, such that the parsers don’t need
> to know that the message is delivered embedded in syslog.
>
> The new parser chaining work would give us an avenue to this, and as you
> can see here MetronPR#1099
>  I
> have put that case forward.
>
> If that hits, I think that we’d be able to : 1. parse plain syslog to
> metron 2. parse plain syslog as a transform and then have less complicated,
> more specific parsers for the msg part.
>
> We may end up having syslog parsers and transforms at the end of this.
>
> In the mean time, if you wish to parse plain syslog in Metron, you will
> have to use grok, which doesn’t get structured data.
>
> If you want the complete 5424 set of data, then you can open a jira for
> creating a parser using simple-syslog–5424.
>
>
>
>
> On July 20, 2018 at 04:23:36, Farrukh Naveed Anjum (
> anjum.farr...@gmail.com) wrote:
>
> Hi,
>
> I am trying to index the Syslog using CEF Parser with Nifi.
>
> It does not give any error though, transport data to kafa without indexing
> it. It keepg giving FAILED in Spout.
>
> I believe indexing Syslog are most basic usecase for all. But metron fails
> to do it with each in standard format.
>
> I tried bro for it. But even it keeps giving PARSER Error.
>
> Any help ? Fast will be apperciated.
>
>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
>

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Otto Fowler 
Metron does not have a generic Syslog Parser.

Nifi has Syslog parsing ( either Records or standard Processor ), in two
modes.

ParseSyslog is the original, where regex’s are used to parse the syslog
RFC3164 and RFC5424, but only extracts the common fields ( so the
‘additional info’ like program id, message id, structured data in 5424 is
in the MSG ). I have recently added a record reader for that method as well
( Nifi PR#2900 ).

Syslog5424Reader(records) and ParseSyslog5424 are new and instead of using
regexes they use a new library simple-syslog–5424
 I wrote that parses
RFC5424 messages completely ( note properly formatted RFC 5424 messages )
see Nifi PR#2805  and Nifi PR#2816
 using an antlr grammar.

You should be able to pick the manner best for you and parse that out in
Nifi if you choose.

Metron parses syslog as required in specific parsers that have messages
assumed to be embedded in syslog.

What I have been talking about in METRON–1453
 and other places is
separating out the syslog from the parser, such that the parsers don’t need
to know that the message is delivered embedded in syslog.

The new parser chaining work would give us an avenue to this, and as you
can see here MetronPR#1099
 I have
put that case forward.

If that hits, I think that we’d be able to : 1. parse plain syslog to
metron 2. parse plain syslog as a transform and then have less complicated,
more specific parsers for the msg part.

We may end up having syslog parsers and transforms at the end of this.

In the mean time, if you wish to parse plain syslog in Metron, you will
have to use grok, which doesn’t get structured data.

If you want the complete 5424 set of data, then you can open a jira for
creating a parser using simple-syslog–5424.




On July 20, 2018 at 04:23:36, Farrukh Naveed Anjum (anjum.farr...@gmail.com)
wrote:

Hi,

I am trying to index the Syslog using CEF Parser with Nifi.

It does not give any error though, transport data to kafa without indexing
it. It keepg giving FAILED in Spout.

I believe indexing Syslog are most basic usecase for all. But metron fails
to do it with each in standard format.

I tried bro for it. But even it keeps giving PARSER Error.

Any help ? Fast will be apperciated.




--
With Regards
Farrukh Naveed Anjum

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Casey Stella 
I just want to pile in here and recommend taking a look at the parser
chaining use-case, which is a walk-through of pulling in firewall logs over
syslog using grok (
https://github.com/apache/metron/tree/master/use-cases/parser_chaining).
Unfortunately this is in master and yet in a release, but it will show you
how to use grok to parse syslogs containing some other format inside.

Casey

On Fri, Jul 20, 2018 at 5:34 AM Simon Elliston Ball <
si...@simonellistonball.com> wrote:

> What you need to do is NOT ParseCEF in NiFi. Metron should handle be CEF
> parsing.
>
> Just use NiFi to do the listen syslog (no need to parse in NiFi) then
> SplitText to get one line of CEF per kafka message (if your syslog is
> batching, this may not be necessary. Set up a sensor in Metron using the
> CEF parser and you should be fine.
>
> Simon
>
>
> On 20 Jul 2018, at 09:39, Srikanth Nagarajan 
> wrote:
>
> Hi Farrukh,
>
> You can try using the Grok Parser and search for regular expression
> pattern for your log.  You can customize the regex to meet your needs.
>
>
> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
>
> Look at Step-5 on how to create a regex for grok parser. Grok parser
> also allows to validate the fields.
>
> Good luck !
>
> Thanks
> Srikanth
>
> On July 20, 2018 at 4:23 AM Farrukh Naveed Anjum 
> wrote:
>
> Hi,
>
> I am trying to index the Syslog using CEF Parser with Nifi.
>
> It does not give any error though, transport data to kafa without indexing
> it. It keepg giving FAILED in Spout.
>
> I believe indexing Syslog are most basic usecase for all. But metron fails
> to do it with each in standard format.
>
> I tried bro for it. But even it keeps giving PARSER Error.
>
> Any help ? Fast will be apperciated.
>
>
>
>
> --
> With Regards
> Farrukh Naveed Anjum
>
>
> __
>
> *Srikanth Nagarajan *
> *Principal*
>
> *Gandiva Networks Inc*
>
> *732.690.1884* Mobile
>
> s...@gandivanetworks.com
>
> www.gandivanetworks.com
>
> Please consider the environment before printing this. NOTICE: The
> information contained in this e-mail message is intended for addressee(s)
> only. If you have received this message in error please notify the sender.
>
>

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Simon Elliston Ball 
What you need to do is NOT ParseCEF in NiFi. Metron should handle be CEF 
parsing. 

Just use NiFi to do the listen syslog (no need to parse in NiFi) then SplitText 
to get one line of CEF per kafka message (if your syslog is batching, this may 
not be necessary. Set up a sensor in Metron using the CEF parser and you should 
be fine. 

Simon 


> On 20 Jul 2018, at 09:39, Srikanth Nagarajan  wrote:
> 
> Hi Farrukh,
> 
> You can try using the Grok Parser and search for regular expression pattern 
> for your log.  You can customize the regex to meet your needs.   
> 
> https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+Telemetry
> 
> Look at Step-5 on how to create a regex for grok parser. Grok parser also 
> allows to validate the fields.
> 
> Good luck !
> 
> Thanks
> Srikanth
> 
>> On July 20, 2018 at 4:23 AM Farrukh Naveed Anjum  
>> wrote: 
>> 
>> Hi,
>> 
>> I am trying to index the Syslog using CEF Parser with Nifi.
>> 
>> It does not give any error though, transport data to kafa without indexing 
>> it. It keepg giving FAILED in Spout.
>> 
>> I believe indexing Syslog are most basic usecase for all. But metron fails 
>> to do it with each in standard format.
>> 
>> I tried bro for it. But even it keeps giving PARSER Error.
>> 
>> Any help ? Fast will be apperciated.
>> 
>> 
>> 
>> 
>> -- 
>> With Regards 
>> Farrukh Naveed Anjum
> 
> __
> 
> Srikanth Nagarajan 
> Principal
> 
> Gandiva Networks Inc
> 
> 732.690.1884 Mobile
> 
> s...@gandivanetworks.com
> 
> www.gandivanetworks.com
> 
> Please consider the environment before printing this. NOTICE: The information 
> contained in this e-mail message is intended for addressee(s) only. If you 
> have received this message in error please notify the sender.

Re: CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Srikanth Nagarajan 

Hi Farrukh,You can try using the Grok Parser and search for regular _expression_ pattern for your log.  You can customize the regex to meet your needs.   https://cwiki.apache.org/confluence/display/METRON/2016/04/25/Metron+Tutorial+-+Fundamentals+Part+1%3A+Creating+a+New+TelemetryLook at Step-5 on how to create a regex for grok parser.     Grok parser also allows to validate the fields.Good luck !ThanksSrikanthOn July 20, 2018 at 4:23 AM Farrukh Naveed Anjum  wrote:  Hi,I am trying to index the Syslog using CEF Parser with Nifi.It does not give any error though, transport data to kafa without indexing it. It keepg giving FAILED in Spout.I believe indexing Syslog are most basic usecase for all. But metron fails to do it with each in standard format.I tried bro for it. But even it keeps giving PARSER Error.Any help ? Fast will be apperciated.-- With Regards Farrukh Naveed Anjum__Srikanth Nagarajan PrincipalGandiva Networks Inc732.690.1884 Mobiles...@gandivanetworks.com www.gandivanetworks.com Please consider the environment before printing this. NOTICE: The information contained in this e-mail message is intended for addressee(s) only. If you have received this message in error please notify the sender.

CEF Parser not Indexing data via Nifi (SysLogs)

2018-07-20 Thread Farrukh Naveed Anjum 
Hi,

I am trying to index the Syslog using CEF Parser with Nifi.

It does not give any error though, transport data to kafa without indexing
it. It keepg giving FAILED in Spout.

I believe indexing Syslog are most basic usecase for all. But metron fails
to do it with each in standard format.

I tried bro for it. But even it keeps giving PARSER Error.

Any help ? Fast will be apperciated.




-- 
With Regards
Farrukh Naveed Anjum