I'm a little concerned about this increasing the size and length of the build due to the repeated shading. Should we figure out a way to deploy jars with provided dependencies on metron-parser-common as suggested in the previous JIRAs first?
On Fri, Mar 10, 2017 at 2:31 PM, Matt Foley <mfo...@hortonworks.com> wrote: > It sounds like: > - This is a self-contained chunk of work, that can be tested, reviewed, > and committed on its own, then the other ideas you propose can follow it. > - It crosses a lot of lines, and restructures a lot of code, so will “rot” > fairly quickly as other people make commits, so if possible you should get > a PR out there and we should work through it as soon as possible. > Are those both true? > > How do other people feel about grouping a given sensor’s parser, enricher, > indexing logic all together? It seems to have multiple advantages are > there also disadvantages? > > On 3/10/17, 6:31 AM, "Otto Fowler" <ottobackwa...@gmail.com> wrote: > > As previously discussed here, I have been working on side loading of > parsers. The goals of this work are: > * Make it possible of developers to create, maintain and deploy parsers > outside of the Metron code tree and not have to fork > * Create maven archetype support for developers of parsers > * Introduce a parser ‘lifecycle’ to support multiple instances and > configurations, states of being installed, under configuration, and > deployed > etc. > > I would like to have some discussion based on where I am after rebasing > onto METRON-671 which revamps deployment to be totally ambari based. > > > Parsers as components: > > I have all the parsers broken out into individual packages/rpms/jars. > What I have done is taken metron-parsers and broken it out to: > > * metron-parsers-common > * This has all the base classes and interfaces, common testing > components > etc > * metron-parser-base > * This has the Grok, CSV, and JsonMap parsers and support > * metron-parser-X > * A module per parser type which we currently have in the system > * Each parser has all the indexing, enrichment and parser > configurations > for that parser in its package > > I will go into packaging and deployment issues in another email. > > I have this all working: > * the parsers are built > * the parsers are tested > * the parsers are integrated into the deployment build such that > vagrant up > just works as previously in full and quick dev > * maven component of rpm docker > * the metron.spec file > * ambari installation > * zookeeper configuration deployment > * the ambari parser service code > * the Rest interface works > * see all installed parser configurations etc > > > So this part of the work, is I think ready for a PR and review/next > steps > on it’s own. > > I think that it sets up the components and is a base for building out > the > rest of the functionality we want. > > >
