Gotcha. Makes sense. Unrelated, but something I've had in the back of my head for a while... I think we should try to loosely define what roles would be using Ambari vs Metron UI vs CLI. For example, I'm thinking there may be a difference between the cluster admin and the Metron admin in an organization. With that, I think the idea of using the Metron UI to load parser packages makes a lot of sense.
On Tue, Mar 14, 2017 at 10:39 AM, Otto Fowler <ottobackwa...@gmail.com> wrote: > The RPM’s are only a requirement for installation by ambari. > > We could drop in Nars to the system location ( hdfs in the end ). > I imagine us doing that through the UI. > > Nar’s *are* automatically unpacked at execution time. > > Nar’s are unpackaged into a ‘working’ area ( if required ), that working > area is the class path that is > loaded. > > > > > On March 14, 2017 at 09:57:14, Kyle Richardson (kylerichards...@gmail.com) > wrote: > > Solid work, Otto. I'm excited to see us move in this direction. It's an > important step to making Metron a more user friendly platform. > > I agree with Matt. I think a PR for this piece is needed sooner rather > than > later so you don't continually have to rework it as we make more commits > into master. > > One question/concern/future thought, is there anyway we can get away from > needing the RPMs for these extensible components? Ideally, I'd like to be > able to dump my parser package file (jar, nar, etc) into a directory and > have it automagically unpacked. > > -Kyle > > On Mon, Mar 13, 2017 at 10:10 AM, Otto Fowler <ottobackwa...@gmail.com> > wrote: > > > James, > > Can you clarify your concerns about backward compatibility? > > > > > > > > On March 12, 2017 at 23:48:17, James Sirota (jsir...@apache.org) wrote: > > > > As long as this doesn't break backwards compatibility I think I am ok > with > > this approach. I think this is a great idea. We would probably need to > > follow up with an Ambari view that can allow you to list and > > deploy/upgrade/delete various parsers > > > > 10.03.2017, 15:16, "Casey Stella" <ceste...@gmail.com>: > > > Ok, so, after some thought about this, I am in agreement over Nar. I > do > > > want to make sure that on the roadmap we retrofit stellar to accept > Nar > > > plugins and build an archetype for it. We should have a single > strategy > > > for plugins. NOt saying it has to be part of the same PR, but it needs > to > > > be associated and a follow-on task IMO. > > > > > > On Fri, Mar 10, 2017 at 4:06 PM, Otto Fowler <ottobackwa...@gmail.com> > > > > wrote: > > > > > >> Also a Nar can depend on ‘one’ other nar, which is interesting > > >> > > >> On March 10, 2017 at 16:02:18, Otto Fowler (ottobackwa...@gmail.com) > > >> wrote: > > >> > > >> The isolation is just a ‘extra’ in the parser case. > > >> > > >> The parts of Nar that *are* more pertinent: > > >> > > >> * supporting a deployment artifact with just a jar, or a tar.gz with > a > > jar > > >> and jar dependencies in it > > >> * taking a ‘package’ and deploying it for loading ( which will > upgrade > > the > > >> deployed part if it is newer ) > > >> * setting up the classloader hierarchy between the ‘system’ and > > provided > > >> things, and the dependencies of the individual plugin > > >> > > >> On March 10, 2017 at 15:56:08, Casey Stella (ceste...@gmail.com) > > wrote: > > >> > > >> Why would we need classpath isolation here in the case of the parser? > > >> > > >> On Fri, Mar 10, 2017 at 3:55 PM, Otto Fowler <ottobackwa...@gmail.com> > > > >> wrote: > > >> > > >>> I *would* use the classloader part, extending it with VFS. > > >>> > > >>> On March 10, 2017 at 15:53:05, Casey Stella (ceste...@gmail.com) > > wrote: > > >>> > > >>> I'm a bit worried about copying and pasting from the NiFi project > > their > > >>> nar infrastructure. That seems..unclean to me and since we're not > > using > > >>> the classloader part of nar for this, does it make more sense to > just > > use > > >>> jar? > > >>> > > >>> On Fri, Mar 10, 2017 at 3:50 PM, Otto Fowler < > ottobackwa...@gmail.com > > > > > >>> wrote: > > >>> > > >>>> Compared to how much time vagrant up takes now, you won’t even > notice > > it > > >>>> ;) > > >>>> > > >>>> That is definitely an option. I guess what I want to work out is if > > we > > >>>> are going to want to > > >>>> go to NAR, why not just go to NAR. > > >>>> > > >>>> In the end, the customer for this - like Jon Zeolla, isn’t going to > > care > > >>>> about the intermediate step, > > >>>> he wants the archetype that builds the ‘metron parser plugin’. > > >>>> > > >>>> Which is why I hesitate to put out an archetype that is going to > > >>>> obsolete so soon. > > >>>> > > >>>> Does that make sense? > > >>>> > > >>>> On March 10, 2017 at 14:50:55, Casey Stella (ceste...@gmail.com) > > wrote: > > >>>> > > >>>> I'm a little concerned about this increasing the size and length of > > the > > >>>> build due to the repeated shading. Should we figure out a way to > > deploy > > >>>> jars with provided dependencies on metron-parser-common as > suggested > > in > > >>>> the > > >>>> previous JIRAs first? > > >>>> > > >>>> On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfo...@hortonworks.com> > > > >>>> wrote: > > >>>> > > >>>> > It sounds like: > > >>>> > - This is a self-contained chunk of work, that can be tested, > > reviewed, > > >>>> > and committed on its own, then the other ideas you propose can > > follow > > >>>> it. > > >>>> > - It crosses a lot of lines, and restructures a lot of code, so > > will > > >>>> “rot” > > >>>> > fairly quickly as other people make commits, so if possible you > > should > > >>>> get > > >>>> > a PR out there and we should work through it as soon as possible. > > >>>> > Are those both true? > > >>>> > > > >>>> > How do other people feel about grouping a given sensor’s parser, > > >>>> enricher, > > >>>> > indexing logic all together? It seems to have multiple advantages > > are > > >>>> > there also disadvantages? > > >>>> > > > >>>> > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwa...@gmail.com> > > wrote: > > >>>> > > > >>>> > As previously discussed here, I have been working on side loading > > of > > >>>> > parsers. The goals of this work are: > > >>>> > * Make it possible of developers to create, maintain and deploy > > parsers > > >>>> > outside of the Metron code tree and not have to fork > > >>>> > * Create maven archetype support for developers of parsers > > >>>> > * Introduce a parser ‘lifecycle’ to support multiple instances > and > > >>>> > configurations, states of being installed, under configuration, > and > > >>>> > deployed > > >>>> > etc. > > >>>> > > > >>>> > I would like to have some discussion based on where I am after > > rebasing > > >>>> > onto METRON-671 which revamps deployment to be totally ambari > > based. > > >>>> > > > >>>> > > > >>>> > Parsers as components: > > >>>> > > > >>>> > I have all the parsers broken out into individual > > packages/rpms/jars. > > >>>> > What I have done is taken metron-parsers and broken it out to: > > >>>> > > > >>>> > * metron-parsers-common > > >>>> > * This has all the base classes and interfaces, common testing > > >>>> > components > > >>>> > etc > > >>>> > * metron-parser-base > > >>>> > * This has the Grok, CSV, and JsonMap parsers and support > > >>>> > * metron-parser-X > > >>>> > * A module per parser type which we currently have in the system > > >>>> > * Each parser has all the indexing, enrichment and parser > > >>>> > configurations > > >>>> > for that parser in its package > > >>>> > > > >>>> > I will go into packaging and deployment issues in another email. > > >>>> > > > >>>> > I have this all working: > > >>>> > * the parsers are built > > >>>> > * the parsers are tested > > >>>> > * the parsers are integrated into the deployment build such that > > >>>> > vagrant up > > >>>> > just works as previously in full and quick dev > > >>>> > * maven component of rpm docker > > >>>> > * the metron.spec file > > >>>> > * ambari installation > > >>>> > * zookeeper configuration deployment > > >>>> > * the ambari parser service code > > >>>> > * the Rest interface works > > >>>> > * see all installed parser configurations etc > > >>>> > > > >>>> > > > >>>> > So this part of the work, is I think ready for a PR and > review/next > > >>>> > steps > > >>>> > on it’s own. > > >>>> > > > >>>> > I think that it sets up the components and is a base for building > > out > > >>>> > the > > >>>> > rest of the functionality we want. > > >>>> > > > >>>> > > > >>>> > > > > > ------------------- > > Thank you, > > > > James Sirota > > PPMC- Apache Metron (Incubating) > > jsirota AT apache DOT org > > > >
