As long as this doesn't break backwards compatibility I think I am ok with this approach. I think this is a great idea. We would probably need to follow up with an Ambari view that can allow you to list and deploy/upgrade/delete various parsers
10.03.2017, 15:16, "Casey Stella" <ceste...@gmail.com>: > Ok, so, after some thought about this, I am in agreement over Nar. I do > want to make sure that on the roadmap we retrofit stellar to accept Nar > plugins and build an archetype for it. We should have a single strategy > for plugins. NOt saying it has to be part of the same PR, but it needs to > be associated and a follow-on task IMO. > > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> Also a Nar can depend on ‘one’ other nar, which is interesting >> >> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com) >> wrote: >> >> The isolation is just a ‘extra’ in the parser case. >> >> The parts of Nar that *are* more pertinent: >> >> * supporting a deployment artifact with just a jar, or a tar.gz with a jar >> and jar dependencies in it >> * taking a ‘package’ and deploying it for loading ( which will upgrade the >> deployed part if it is newer ) >> * setting up the classloader hierarchy between the ‘system’ and provided >> things, and the dependencies of the individual plugin >> >> On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote: >> >> Why would we need classpath isolation here in the case of the parser? >> >> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> I *would* use the classloader part, extending it with VFS. >>> >>> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote: >>> >>> I'm a bit worried about copying and pasting from the NiFi project their >>> nar infrastructure. That seems..unclean to me and since we're not using >>> the classloader part of nar for this, does it make more sense to just use >>> jar? >>> >>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <ottobackwa...@gmail.com> >>> wrote: >>> >>>> Compared to how much time vagrant up takes now, you won’t even notice it >>>> ;) >>>> >>>> That is definitely an option. I guess what I want to work out is if we >>>> are going to want to >>>> go to NAR, why not just go to NAR. >>>> >>>> In the end, the customer for this - like Jon Zeolla, isn’t going to care >>>> about the intermediate step, >>>> he wants the archetype that builds the ‘metron parser plugin’. >>>> >>>> Which is why I hesitate to put out an archetype that is going to >>>> obsolete so soon. >>>> >>>> Does that make sense? >>>> >>>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote: >>>> >>>> I'm a little concerned about this increasing the size and length of the >>>> build due to the repeated shading. Should we figure out a way to deploy >>>> jars with provided dependencies on metron-parser-common as suggested in >>>> the >>>> previous JIRAs first? >>>> >>>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfo...@hortonworks.com> >>>> wrote: >>>> >>>> > It sounds like: >>>> > - This is a self-contained chunk of work, that can be tested, reviewed, >>>> > and committed on its own, then the other ideas you propose can follow >>>> it. >>>> > - It crosses a lot of lines, and restructures a lot of code, so will >>>> “rot” >>>> > fairly quickly as other people make commits, so if possible you should >>>> get >>>> > a PR out there and we should work through it as soon as possible. >>>> > Are those both true? >>>> > >>>> > How do other people feel about grouping a given sensor’s parser, >>>> enricher, >>>> > indexing logic all together? It seems to have multiple advantages are >>>> > there also disadvantages? >>>> > >>>> > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: >>>> > >>>> > As previously discussed here, I have been working on side loading of >>>> > parsers. The goals of this work are: >>>> > * Make it possible of developers to create, maintain and deploy parsers >>>> > outside of the Metron code tree and not have to fork >>>> > * Create maven archetype support for developers of parsers >>>> > * Introduce a parser ‘lifecycle’ to support multiple instances and >>>> > configurations, states of being installed, under configuration, and >>>> > deployed >>>> > etc. >>>> > >>>> > I would like to have some discussion based on where I am after rebasing >>>> > onto METRON-671 which revamps deployment to be totally ambari based. >>>> > >>>> > >>>> > Parsers as components: >>>> > >>>> > I have all the parsers broken out into individual packages/rpms/jars. >>>> > What I have done is taken metron-parsers and broken it out to: >>>> > >>>> > * metron-parsers-common >>>> > * This has all the base classes and interfaces, common testing >>>> > components >>>> > etc >>>> > * metron-parser-base >>>> > * This has the Grok, CSV, and JsonMap parsers and support >>>> > * metron-parser-X >>>> > * A module per parser type which we currently have in the system >>>> > * Each parser has all the indexing, enrichment and parser >>>> > configurations >>>> > for that parser in its package >>>> > >>>> > I will go into packaging and deployment issues in another email. >>>> > >>>> > I have this all working: >>>> > * the parsers are built >>>> > * the parsers are tested >>>> > * the parsers are integrated into the deployment build such that >>>> > vagrant up >>>> > just works as previously in full and quick dev >>>> > * maven component of rpm docker >>>> > * the metron.spec file >>>> > * ambari installation >>>> > * zookeeper configuration deployment >>>> > * the ambari parser service code >>>> > * the Rest interface works >>>> > * see all installed parser configurations etc >>>> > >>>> > >>>> > So this part of the work, is I think ready for a PR and review/next >>>> > steps >>>> > on it’s own. >>>> > >>>> > I think that it sets up the components and is a base for building out >>>> > the >>>> > rest of the functionality we want. >>>> > >>>> > >>>> > ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
