OK, Current status
Complete:
* Parsers broken out into common, base ( raw types - csv, json, grok ) and
module for each type
* Ansible modified to install new parsers, adding new parser should just be
adding a new name to the array
* Parsers install into /usr/metron/ver/telemetry/{NAME}/ with contents of
the archive/dir structure
* Maven Archetype for creating metron parsers
* Archetype includes ansible playbook and roles for deploying the produced
parser into an existing metron system
* Archetype has simple ( and not satisfactory ) sample implementation
* Archetype uses input variables to name parser, classes, and configure the
ansible scripts, such that if you stick with the name you don’t have to make
modifications for the parser in the project.
* First pass README documentation of the archetype, playbook, roles,
parsers common, parsers base and each parser ( will need jiras to document
parsers )
* Script to run playbook from archetype produced project and deploy parser
into an existing vagrant instance - such that if you do quick or full-dev
you can deploy your parser into it
The bad:
* the archetype version you give must be the same as the metron version,
i’m not sure how I want to plumb that through
* the deployment to monit doesn’t end with the parser start or monitored, I
don’t know how to add the service correctly apparently
* no real data, so parser doesn’t deploy and then show in elasticsearch etc
* This breaks the RPM + ambari deployment - i’m starting to look at that, I
hate to hard code each jar, but I don’t know much about rpm specs ( help! )
and I’d like to do what I do in ansible, then I have to look at ambari and
the comands
The Ugly ( IE why this isn’t a pr with don’t merge yet )
* I cannot build in travis. Just leaving in the install -DskipTests fails.
Here is a raw log of a branch with only the build enabled, no tests :
https://s3.amazonaws.com/archive.travis-ci.org/jobs/205506160/log.txt. I
think it has to do with the shading/reporting. That is where it seems to
stall out (10 minutes without report ). I could honestly use another set or
sets of eyes
with some experience on how to get the parser pom’s correct
dependency-wise. This is my present concern. I can, as usually build
locally.
* I have not been able to deploy the new stuff to a cluster, only local
vagrant.. and resource wise it is never great to start with, so it still
needs some shaking out
So - the big things keeping this away from a pr:
* fixing the travis stuff
* not regressing the rpm / ambari stuff
* design review and feedback / iterations
If anyone has any ideas or time, I’d be happy for them.
https://github.com/ottobackwards/incubator-metron/tree/METRON-258
https://github.com/ottobackwards/incubator-metron/tree/METRON-258/metron-maven-archetypes
https://github.com/ottobackwards/incubator-metron/tree/METRON-258/metron-maven-archetypes/metron-maven-parser-archetype
https://github.com/ottobackwards/incubator-metron/tree/METRON-258/metron-maven-archetypes/metron-maven-parser-archetype/src/main/resources/archetype-resources/metron-parser-deployment
https://github.com/ottobackwards/incubator-metron/tree/METRON-258/metron-maven-archetypes/metron-maven-parser-archetype/src/main/resources/archetype-resources/metron-parser-deployment/roles
https://github.com/ottobackwards/incubator-metron/tree/METRON-258/metron-platform/metron-parsers
https://github.com/ottobackwards/incubator-metron/blob/METRON-258/metron-maven-archetypes/metron-maven-parser-archetype/src/main/resources/archetype-resources/metron-parser-deployment/scripts/deploy_parsers_to_vagrant.sh
etc etc
Once we get this going, we can start talking about some next step ideas
On February 20, 2017 at 14:26:12, Otto Fowler (ottobackwa...@gmail.com)
wrote:
More thoughts
(1) We should do a treatment for each area
(2) We can use the telemetry stuff as an incubator, itself to be replaced
with something better that is developed after
(3) That is a nice idea - ‘live packaging’ ( i’m getting the TM and a
website as we speak )
(4) sure, but we may need to think through the idea that an existing
mechanism my provide some of that and we piggy back on it, but that can be
a goal.
Having everything in one package, with a defined deployment and state
system will make that possible.
On February 20, 2017 at 13:58:08, Nick Allen (n...@nickallen.org) wrote:
Your mention of a "package mechanism" sparked some half-baked ideas on my
part.
Be forewarned these are probably tangents from your immediate goal (sorry),
but maybe these ideas might help shape how you want to take this forward.
(1)
We should consider that eventually each "function" of Metron
should be extensible
. Not just parsers, but enrichment, triage, indexing, profil
ing
, or maas. Ideally we could cover each of these
functional areas
with the same mechanism.
(2)
We would want packages to cover different
kinds of
deployable bits; code (a new parser class), configuration (a triage rule
set), and also external actions (like deploying an Elasticsearch index
template or creating a Kafka topic).
(3)
I'd also love to be able to export
"
packages
"
from a live system. For example, I setup a test Metron environment and
validate it. I can then export a package from the test environment and
import it into production.
(4)
Could a package mechanism also help us
provide
a
clean
, automated
upgrade path?
First, I export a package from my system running Metron version N. Then I
import that package into a separate system running version N+1. Importing
these packages gives us a hook where we can do upgrade-y stuff, like modify
the configs or do whatever needs done to upgrade
. If I want a package that's native to version N+1, then I just export
the package from the system running version N+1
.
On Feb 17, 2017 4:54 PM, "Otto Fowler" <ottobackwa...@gmail.com> wrote:
> The ability for implementors and developers building on the project to
> ‘side load’, that is to build, maintain, and install, telemetry sources
> into the system without having to actually develop within METRON itself is
> very important.
>
> If done properly it gives developers and easier and more manageable
> proposition for extending METRON to suit their needs in what may be the
> most common extension case. It also may reduce the necessity to create and
> maintain forks of METRON.
>
> I would like to put forward a proposal on a way to move this forward, and
> ask the community for feedback and assistance in reaching an acceptable
> approach and raising the issues that I have surely missed.
>
> Conceptually what I would like to propose is the following:
>
> * What is currently metron-parsers should be broken apart such that each
> parser is it’s own individual component
> * Each of these components should be completely self contained ( or
produce
> a self contained package )
> * These packages will include the shaded jar for the parser, default
> configurations for the parser and enrichment, default elasticsearch
> template, and a default log-rotate script
> * These packages will be deployed to disk in a new library directory under
> metron
> * Zookeeper should have a new telemetry or source area where all
> ‘installed’ sources exist
> * This area would host the default configurations, rules, templates, and
> scripts and metadata
> * Installed sources can be instantiated as named instances
> * Instantiating an instance will move the default configurations to what
is
> currently the enrichment and parser areas for the instance name
> * It will also deploy the elasticsearch template for the instance
> name
> * It will deploy the log-rotate scripts
> * Installed and instantiated sources can be ‘redeployed’ from disk to
> upgrade
> * Installed sources are available for selection in ambari
> * question on post selection configuration, but we have that problem
> already
> * Instantiation is exposed through REST
> * the UI can install a new package
> * the UI can allow a workflow to edit the configurations and templates
> before finalizing
> * are there three states here? Installed | Edited | Instantiated
> ?
> * the UI can edit existing and redeploy
> * possibly re-deploy ES template after adding fields or account for fields
> added by enrichment…. manually or automatically?
> * a script can be made to instantiate a ‘base’ parser ( json, grok, csv )
> with only configuration
> * The installation and instantiation should be exposed through the Stellar
> management console
> * Starting a topology will now start the parser’s shaded jar found through
> the parser type ( which may need to added to the configurations ) and the
> library
> * A Maven Archetype should be created for a parser | telemetry source
> project that allows the proper setup of a development project outside the
> METRON source tree
> * should be published
> * should have a useful default set
>
> So the developer’s workflow:
>
> * Create a new project from the archetype outside of the metron tree
> * edit the configurations, templates, rules etc in the project
> * code or modify the sample
> * build
> * run the installer script or the ui to upload/deploy the package
> * use the console or ui to create an instance
>
> QUESTIONS:
> * it seems strange to have this as ‘parsers’ when conceptually parsers are
> a part of the whole, should we introduce something like ‘source’ that is
> all of it?
> * should configurations etc be in ZK or on disk? or HDFS? or All of the
> above?
> * did you read this far? good!
> * I am sure that after hitting send I will think of 10 things that are
> missing from this
>
> I have started a POC of this, and thus far have created
> metron-parsers-common and started breaking out metron-parser-asa.
> I will continue to work through some of this here
> https://github.com/ottobackwards/incubator-metron/tree/METRON-258
>
> Again, thank you for your time and feedback.
>
