Just to be clear, I didn’t group the logic of the indexing, enrichments and parser, I grouped the configuration. The idea is that a 3rd party developer would like to produce a single product, that is self contained.
Yes those are both true, if we agree that this is a good way to go of course. On March 10, 2017 at 14:31:46, Matt Foley (mfo...@hortonworks.com) wrote: It sounds like: - This is a self-contained chunk of work, that can be tested, reviewed, and committed on its own, then the other ideas you propose can follow it. - It crosses a lot of lines, and restructures a lot of code, so will “rot” fairly quickly as other people make commits, so if possible you should get a PR out there and we should work through it as soon as possible. Are those both true? How do other people feel about grouping a given sensor’s parser, enricher, indexing logic all together? It seems to have multiple advantages are there also disadvantages? On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: As previously discussed here, I have been working on side loading of parsers. The goals of this work are: * Make it possible of developers to create, maintain and deploy parsers outside of the Metron code tree and not have to fork * Create maven archetype support for developers of parsers * Introduce a parser ‘lifecycle’ to support multiple instances and configurations, states of being installed, under configuration, and deployed etc. I would like to have some discussion based on where I am after rebasing onto METRON-671 which revamps deployment to be totally ambari based. Parsers as components: I have all the parsers broken out into individual packages/rpms/jars. What I have done is taken metron-parsers and broken it out to: * metron-parsers-common * This has all the base classes and interfaces, common testing components etc * metron-parser-base * This has the Grok, CSV, and JsonMap parsers and support * metron-parser-X * A module per parser type which we currently have in the system * Each parser has all the indexing, enrichment and parser configurations for that parser in its package I will go into packaging and deployment issues in another email. I have this all working: * the parsers are built * the parsers are tested * the parsers are integrated into the deployment build such that vagrant up just works as previously in full and quick dev * maven component of rpm docker * the metron.spec file * ambari installation * zookeeper configuration deployment * the ambari parser service code * the Rest interface works * see all installed parser configurations etc So this part of the work, is I think ready for a PR and review/next steps on it’s own. I think that it sets up the components and is a base for building out the rest of the functionality we want.
