I think the primary limitation for parser contributors is that you need a fair amount of good data to work with. Building a parser from a small amount of test data will probably miss some corner cases that one would see "in the wild." I don't think there are any right or wrong answers here though. If you have access to good Infoblox data, for example, then start there.
I would also suggest that you try building a parser with Grok expressions first. You can do quite a bit with grok expressions. Get a feel for what you can do with those, what the limitations are, and then transition into Java parsers only as you see a need. Really glad that you are joining the community. I look forward to your contributions. On Fri, Apr 7, 2017 at 10:33 AM, Mark de Rijk <me.der...@gmail.com> wrote: > Ali, > > Coding wise I am still getting my footing on and doing a Java course > online. > So building in unit tests I am afraid is a bit further away from me. > For the parsers: > > - Infoblox Syslog > - Cisco IOS Syslog > - Unix Syslog > - Alcatel Syslog > - Sendmail Syslog > - Blackberry Enterprise Server File > > I have also developed file and DB based connectors but as said I need to > figure it out how to actually develop parsers for the Metron platform > specifically. > > Is there a list of most requested devices documented somewhere so I can > focus my efforts? > > Cheers, > Mark > > > On Fri, Apr 7, 2017 at 2:56 PM, Ali Nazemian <alinazem...@gmail.com> > wrote: > > > Mark, > > > > Can you please specify the Parsers you are familiar with? We are > > prioritizing the parser implementation, so your effort can help us to > > target them faster. > > > > Cheers, > > Ali > > > > On Fri, Apr 7, 2017 at 11:41 PM, Mark De Rijk <me.der...@gmail.com> > wrote: > > > > > I will review them this afternoon. Thanks for that. > > > > > > Sent from my iPhone > > > > > > > On 7 Apr 2017, at 14:37, Ali Nazemian <alinazem...@gmail.com> wrote: > > > > > > > > Mark, > > > > > > > > Have you seen the following pages? > > > > > > > > https://cwiki.apache.org/confluence/display/METRON/ > > > Development+Guidelines > > > > https://cwiki.apache.org/confluence/display/METRON/ > Metron+Development+ > > > Environment+Setup+Instructions > > > > https://cwiki.apache.org/confluence/display/METRON/ > Community+Resources > > > > > > > > > > > >> On Fri, Apr 7, 2017 at 11:20 PM, Mark de Rijk <me.der...@gmail.com> > > > wrote: > > > >> > > > >> To clarify I have written a lot of parsers for ArcSight over the > years > > > and > > > >> I would like to start contributing by developing parsers for the > > Metron > > > >> project. > > > >> Is there any documentation that will help me get started so I can > > start > > > >> cranking them out? > > > >> This is the first open source project I am looking to contribute to > so > > > >> forgive me If I am asking stupid questions. > > > >> > > > >> > > > >> > > > >> On Fri, Apr 7, 2017 at 2:13 PM, Otto Fowler < > ottobackwa...@gmail.com> > > > >> wrote: > > > >> > > > >>> I also believe that grok parsers can be added through configuration > > > only, > > > >>> without having to > > > >>> compile a parser. > > > >>> > > > >>> You can add a parser configuration targeting the basic grok parser > > and > > > >> just > > > >>> provide the grok > > > >>> parser rules. > > > >>> > > > >>> > > > >>> Just as a heads up, I’m currently working on the parsers to allow > for > > > >>> writing and maintaining parsers > > > >>> outside the metron code tree, including providing a maven > archetype. > > > >> This > > > >>> will allow you to create parsers > > > >>> without having to maintain a fork etc. > > > >>> > > > >>> Keep an eye out for METRON-258 as a PR on the list. > > > >>> > > > >>> > > > >>> > > > >>> On April 7, 2017 at 08:54:35, Justin Leet (justinjl...@gmail.com) > > > wrote: > > > >>> > > > >>> My understanding of Grok vs Java is to provide a tradeoff for ease > of > > > >>> implementation vs performance (plus Java can also handle parsing > that > > > >> would > > > >>> be too complicated for Grok. > > > >>> > > > >>> Grok is less performant and handles less complex parsing, but it's > > easy > > > >> to > > > >>> get things going and potentially maintained without writing and > > > compiling > > > >>> Java. > > > >>> > > > >>> The Java implementation will be better for performance and can > handle > > > >> more > > > >>> complicated parsing Grok can't. > > > >>> > > > >>> I believe the preference has generally been for Grok parsers if > > > >>> appropriate, otherwise Java parsers. > > > >>> > > > >>> Justin > > > >>> > > > >>> On Fri, Apr 7, 2017 at 8:09 AM, Ali Nazemian < > alinazem...@gmail.com> > > > >>> wrote: > > > >>> > > > >>>> Hi Mark, > > > >>>> > > > >>>> Yeah, that would be great. Can you please specify which devices > you > > > >> have > > > >>>> developed so far? > > > >>>> > > > >>>> Cheers, > > > >>>> Ali > > > >>>> > > > >>>> On Fri, Apr 7, 2017 at 4:10 PM, Mark De Rijk <me.der...@gmail.com > > > > > >>> wrote: > > > >>>> > > > >>>>> Dear all, > > > >>>>> > > > >>>>> I am a heavy arcsight user and I have written quite a few parsers > > > >> over > > > >>>>> time. > > > >>>>> I am new to contributing to open source projects however. > > > >>>>> @Ali, would you like to cooperate on development of some parsers? > > > >>>>> > > > >>>>> Kind Regards, > > > >>>>> Mark de Rijk > > > >>>>> > > > >>>>> > > > >>>>>> On 7 Apr 2017, at 04:30, Ali Nazemian <alinazem...@gmail.com> > > > >> wrote: > > > >>>>>> > > > >>>>>> Hi all, > > > >>>>>> > > > >>>>>> We are going to develop some parsers and have some contribution > to > > > >>> the > > > >>>>>> community as a start point. I was wondering what the reason is > > > >> behind > > > >>>>>> choosing Grok statements for some of the implementations and > Java > > > >>> regex > > > >>>>> for > > > >>>>>> other ones? Is there any policy for that? Probably it would be > > > >> better > > > >>>> to > > > >>>>>> have the Java regex implementation due to performance concerns. > > > >>>> However, > > > >>>>> I > > > >>>>>> am sure there is a reason that some of them have been > implemented > > > >>> with > > > >>>>>> using Grok statements. > > > >>>>>> > > > >>>>>> Regards, > > > >>>>>> Ali > > > >>>>> > > > >>>> > > > >>>> > > > >>>> > > > >>>> -- > > > >>>> A.Nazemian > > > >>>> > > > >>> > > > >> > > > > > > > > > > > > > > > > -- > > > > A.Nazemian > > > > > > > > > > > -- > > A.Nazemian > > >
