Ok, so, after some thought about this, I am in agreement over Nar. I do want to make sure that on the roadmap we retrofit stellar to accept Nar plugins and build an archetype for it. We should have a single strategy for plugins. NOt saying it has to be part of the same PR, but it needs to be associated and a follow-on task IMO.
On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwa...@gmail.com> wrote: > Also a Nar can depend on ‘one’ other nar, which is interesting > > > On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com) > wrote: > > The isolation is just a ‘extra’ in the parser case. > > The parts of Nar that *are* more pertinent: > > * supporting a deployment artifact with just a jar, or a tar.gz with a jar > and jar dependencies in it > * taking a ‘package’ and deploying it for loading ( which will upgrade the > deployed part if it is newer ) > * setting up the classloader hierarchy between the ‘system’ and provided > things, and the dependencies of the individual plugin > > > > On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) wrote: > > Why would we need classpath isolation here in the case of the parser? > > On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > >> I *would* use the classloader part, extending it with VFS. >> >> >> >> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) wrote: >> >> I'm a bit worried about copying and pasting from the NiFi project their >> nar infrastructure. That seems..unclean to me and since we're not using >> the classloader part of nar for this, does it make more sense to just use >> jar? >> >> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler <ottobackwa...@gmail.com> >> wrote: >> >>> Compared to how much time vagrant up takes now, you won’t even notice it >>> ;) >>> >>> That is definitely an option. I guess what I want to work out is if we >>> are going to want to >>> go to NAR, why not just go to NAR. >>> >>> In the end, the customer for this - like Jon Zeolla, isn’t going to care >>> about the intermediate step, >>> he wants the archetype that builds the ‘metron parser plugin’. >>> >>> Which is why I hesitate to put out an archetype that is going to >>> obsolete so soon. >>> >>> Does that make sense? >>> >>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) wrote: >>> >>> I'm a little concerned about this increasing the size and length of the >>> build due to the repeated shading. Should we figure out a way to deploy >>> jars with provided dependencies on metron-parser-common as suggested in >>> the >>> previous JIRAs first? >>> >>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfo...@hortonworks.com> >>> wrote: >>> >>> > It sounds like: >>> > - This is a self-contained chunk of work, that can be tested, reviewed, >>> > and committed on its own, then the other ideas you propose can follow >>> it. >>> > - It crosses a lot of lines, and restructures a lot of code, so will >>> “rot” >>> > fairly quickly as other people make commits, so if possible you should >>> get >>> > a PR out there and we should work through it as soon as possible. >>> > Are those both true? >>> > >>> > How do other people feel about grouping a given sensor’s parser, >>> enricher, >>> > indexing logic all together? It seems to have multiple advantages are >>> > there also disadvantages? >>> > >>> > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: >>> > >>> > As previously discussed here, I have been working on side loading of >>> > parsers. The goals of this work are: >>> > * Make it possible of developers to create, maintain and deploy parsers >>> > outside of the Metron code tree and not have to fork >>> > * Create maven archetype support for developers of parsers >>> > * Introduce a parser ‘lifecycle’ to support multiple instances and >>> > configurations, states of being installed, under configuration, and >>> > deployed >>> > etc. >>> > >>> > I would like to have some discussion based on where I am after rebasing >>> > onto METRON-671 which revamps deployment to be totally ambari based. >>> > >>> > >>> > Parsers as components: >>> > >>> > I have all the parsers broken out into individual packages/rpms/jars. >>> > What I have done is taken metron-parsers and broken it out to: >>> > >>> > * metron-parsers-common >>> > * This has all the base classes and interfaces, common testing >>> > components >>> > etc >>> > * metron-parser-base >>> > * This has the Grok, CSV, and JsonMap parsers and support >>> > * metron-parser-X >>> > * A module per parser type which we currently have in the system >>> > * Each parser has all the indexing, enrichment and parser >>> > configurations >>> > for that parser in its package >>> > >>> > I will go into packaging and deployment issues in another email. >>> > >>> > I have this all working: >>> > * the parsers are built >>> > * the parsers are tested >>> > * the parsers are integrated into the deployment build such that >>> > vagrant up >>> > just works as previously in full and quick dev >>> > * maven component of rpm docker >>> > * the metron.spec file >>> > * ambari installation >>> > * zookeeper configuration deployment >>> > * the ambari parser service code >>> > * the Rest interface works >>> > * see all installed parser configurations etc >>> > >>> > >>> > So this part of the work, is I think ready for a PR and review/next >>> > steps >>> > on it’s own. >>> > >>> > I think that it sets up the components and is a base for building out >>> > the >>> > rest of the functionality we want. >>> > >>> > >>> > >>> >>> >> >
