Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[David Lyle]

That's correct. Quick dev will be built using the MPack, but after that,
Quick Dev will remove the Metron components and reinstall them using REST
calls to Ambari, so pieces of the MPack that was used to create the Quick
Dev image would be executed. RPMs are executed in all cases.

The instructions (that will need a bit of modification because of a
name-change) are here:
https://github.com/apache/incubator-metron/blob/master/metron-deployment/packer-build/README.md

-D...


On Thu, Mar 2, 2017 at 3:02 PM, Casey Stella  wrote:

> Just to clarify, your 1 and 2, which you're working on, will give us the
> ability with full-dev (not quick-dev) to exercise the RPMs and management
> pack on the non-sensor code (i.e. the current state of the management
> pack).  As far as I'm concerned, this is huge.  This ensures we have an
> easy vehicle to ensure PRs work with the management pack.  I think a couple
> things should be ensured going forward:
>
>- People whose PR have changes that affect the management pack, should
>   - notify the reviewers that it was tested on full-dev
>   - They should regenerate quickdev to ensure things aren't broken.
>   Dave, can you remind us all where the instructions are for that?
>
>
> On Thu, Mar 2, 2017 at 9:47 AM, David Lyle  wrote:
>
> > Just wanted to update this thread:
> >
> > I've been diligently working to the plan we discussed above:
> >
> > *1) Refactor existing Ansible deployment to use the Ambari MPack to
> install
> > metron-common, metron-enrichments and metron-parsers. *
> > *2) Regenerate quick-dev to leverage the change.*
> > 3) Create rpm packages for all deployed components that don't currently
> > have them.
> >  - Sensor probes
> >  - Sensor stubs
> > 4) Create MPack service defs for the RPMs in (2).
> > 5) Refactor existing Ansible deployment to use the Ambari MPack to
> install
> > all services.
> > 6) Regenerate quick-dev to leverage the change.
> > 7) Plan iteration 2 to see if there are other opportunities to reduce our
> > use of Ansible.
> >
> > I've completed #1 and will have #2 completed shortly, that will close out
> > METRON-671.  If we're all still good with that direction, once I finish
> > 671, I'd like to cut JIRAs for #3 and #4.
> >
> > Thoughts?
> >
> > -D...
> >
> >
> > On Thu, Jan 19, 2017 at 9:00 AM, David Lyle 
> wrote:
> >
> > > For the first increment, I am planning on using the Ambari Metron
> > topology
> > > service that currently exists in the MPack. Handling the side loading
> > isn't
> > > in the scope of this proposal. We'll need to design that separately.
> > >
> > > -D...
> > >
> > >
> > > On Thu, Jan 19, 2017 at 8:48 AM, Otto Fowler 
> > > wrote:
> > >
> > >> So - my prototype was adding a new parser type and completely
> > integrating
> > >> it with the install, all the way down to monit templates.
> > >> All of that work was configuration work in ansible.
> > >>
> > >> I think I’m asking about changes to something that wasn’t documented
> > >> anyways, as I think you are pointing out by reference, so I’ll just
> say
> > >> that this change has an effect on side loading.   You will be building
> > an
> > >> Ambari Metron Topology service, probably from some template -
> hopefully
> > >> using maven archetypes or something the whole way.
> > >>
> > >>
> > >> On January 19, 2017 at 05:56:42, David Lyle (dlyle65...@gmail.com)
> > wrote:
> > >>
> > >> Looks like my replies were only going to Otto, sorry about that- I'll
> > >> gather them here:
> > >>
> > >> "What does this do for Monit?"
> > >>
> > >> Monit would be deprecated for components under Ambari management.
> > >>
> > >> "How would this effect deploying new parsers or parsers not shipped?
> > >> When I prototyped this I added a monit entry."
> > >>
> > >> Hard to say having not seen Otto's prototype, but I suspect no effect.
> > >>
> > >> Fwiw, there is a jira about sideloading that is meant for deploying
> > >> custom
> > >> parsers. Last I looked, the management was still up for design.
> > >>
> > >> I"ve opened https://issues.apache.org/jira/browse/METRON-667 to track
> > >> this
> > >> work. I'd like to get started. Thoughts?
> > >>
> > >> -D...
> > >>
> > >>
> > >> On Wed, Jan 18, 2017 at 1:28 PM, David Lyle 
> > >> wrote:
> > >>
> > >> > Hard to say having not seen your prototype, but I suspect no effect.
> > >> >
> > >> > Fwiw, there is a jira about sideloading that is meant for deploying
> > >> custom
> > >> > parsers. Last I looked, the management was still up for design.
> > >> >
> > >> > -D...
> > >> >
> > >> > On Wed, Jan 18, 2017 at 13:07 Otto Fowler 
> > >> wrote:
> > >> >
> > >> >> How would this effect deploying new parsers or parsers not shipped?
> > >> >> When I prototyped this I added a monit entry.
> > >> >>
> > >> >>
> > >> >> On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com)
> 

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Casey Stella]

Just to clarify, your 1 and 2, which you're working on, will give us the
ability with full-dev (not quick-dev) to exercise the RPMs and management
pack on the non-sensor code (i.e. the current state of the management
pack).  As far as I'm concerned, this is huge.  This ensures we have an
easy vehicle to ensure PRs work with the management pack.  I think a couple
things should be ensured going forward:

   - People whose PR have changes that affect the management pack, should
  - notify the reviewers that it was tested on full-dev
  - They should regenerate quickdev to ensure things aren't broken.
  Dave, can you remind us all where the instructions are for that?


On Thu, Mar 2, 2017 at 9:47 AM, David Lyle  wrote:

> Just wanted to update this thread:
>
> I've been diligently working to the plan we discussed above:
>
> *1) Refactor existing Ansible deployment to use the Ambari MPack to install
> metron-common, metron-enrichments and metron-parsers. *
> *2) Regenerate quick-dev to leverage the change.*
> 3) Create rpm packages for all deployed components that don't currently
> have them.
>  - Sensor probes
>  - Sensor stubs
> 4) Create MPack service defs for the RPMs in (2).
> 5) Refactor existing Ansible deployment to use the Ambari MPack to install
> all services.
> 6) Regenerate quick-dev to leverage the change.
> 7) Plan iteration 2 to see if there are other opportunities to reduce our
> use of Ansible.
>
> I've completed #1 and will have #2 completed shortly, that will close out
> METRON-671.  If we're all still good with that direction, once I finish
> 671, I'd like to cut JIRAs for #3 and #4.
>
> Thoughts?
>
> -D...
>
>
> On Thu, Jan 19, 2017 at 9:00 AM, David Lyle  wrote:
>
> > For the first increment, I am planning on using the Ambari Metron
> topology
> > service that currently exists in the MPack. Handling the side loading
> isn't
> > in the scope of this proposal. We'll need to design that separately.
> >
> > -D...
> >
> >
> > On Thu, Jan 19, 2017 at 8:48 AM, Otto Fowler 
> > wrote:
> >
> >> So - my prototype was adding a new parser type and completely
> integrating
> >> it with the install, all the way down to monit templates.
> >> All of that work was configuration work in ansible.
> >>
> >> I think I’m asking about changes to something that wasn’t documented
> >> anyways, as I think you are pointing out by reference, so I’ll just say
> >> that this change has an effect on side loading.   You will be building
> an
> >> Ambari Metron Topology service, probably from some template - hopefully
> >> using maven archetypes or something the whole way.
> >>
> >>
> >> On January 19, 2017 at 05:56:42, David Lyle (dlyle65...@gmail.com)
> wrote:
> >>
> >> Looks like my replies were only going to Otto, sorry about that- I'll
> >> gather them here:
> >>
> >> "What does this do for Monit?"
> >>
> >> Monit would be deprecated for components under Ambari management.
> >>
> >> "How would this effect deploying new parsers or parsers not shipped?
> >> When I prototyped this I added a monit entry."
> >>
> >> Hard to say having not seen Otto's prototype, but I suspect no effect.
> >>
> >> Fwiw, there is a jira about sideloading that is meant for deploying
> >> custom
> >> parsers. Last I looked, the management was still up for design.
> >>
> >> I"ve opened https://issues.apache.org/jira/browse/METRON-667 to track
> >> this
> >> work. I'd like to get started. Thoughts?
> >>
> >> -D...
> >>
> >>
> >> On Wed, Jan 18, 2017 at 1:28 PM, David Lyle 
> >> wrote:
> >>
> >> > Hard to say having not seen your prototype, but I suspect no effect.
> >> >
> >> > Fwiw, there is a jira about sideloading that is meant for deploying
> >> custom
> >> > parsers. Last I looked, the management was still up for design.
> >> >
> >> > -D...
> >> >
> >> > On Wed, Jan 18, 2017 at 13:07 Otto Fowler 
> >> wrote:
> >> >
> >> >> How would this effect deploying new parsers or parsers not shipped?
> >> >> When I prototyped this I added a monit entry.
> >> >>
> >> >>
> >> >> On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com)
> >> wrote:
> >> >>
> >> >> In our "Dev Guide and Committer Review Guide additions" discussion,
> we
> >> had
> >> >>
> >> >>
> >> >> a bit of a side discussion about reducing reliance (perhaps to zero)
> >> on
> >> >>
> >> >>
> >> >> Ansible for our installation.
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> It seemed there was consensus around that idea (if not, please let me
> >> >>
> >> >>
> >> >> know), so I propose the following steps to get there:
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> 1) Refactor existing Ansible deployment to use the Ambari MPack to
> >> install
> >> >>
> >> >>
> >> >> metron-common, metron-enrichments and metron-parsers.
> >> >>
> >> >>
> >> >> 2) Regenerate quick-dev to leverage the change.
> >> >>
> >> >>
> >> >> 3) Create rpm packages for 

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[David Lyle]

Just wanted to update this thread:

I've been diligently working to the plan we discussed above:

*1) Refactor existing Ansible deployment to use the Ambari MPack to install
metron-common, metron-enrichments and metron-parsers. *
*2) Regenerate quick-dev to leverage the change.*
3) Create rpm packages for all deployed components that don't currently
have them.
 - Sensor probes
 - Sensor stubs
4) Create MPack service defs for the RPMs in (2).
5) Refactor existing Ansible deployment to use the Ambari MPack to install
all services.
6) Regenerate quick-dev to leverage the change.
7) Plan iteration 2 to see if there are other opportunities to reduce our
use of Ansible.

I've completed #1 and will have #2 completed shortly, that will close out
METRON-671.  If we're all still good with that direction, once I finish
671, I'd like to cut JIRAs for #3 and #4.

Thoughts?

-D...


On Thu, Jan 19, 2017 at 9:00 AM, David Lyle  wrote:

> For the first increment, I am planning on using the Ambari Metron topology
> service that currently exists in the MPack. Handling the side loading isn't
> in the scope of this proposal. We'll need to design that separately.
>
> -D...
>
>
> On Thu, Jan 19, 2017 at 8:48 AM, Otto Fowler 
> wrote:
>
>> So - my prototype was adding a new parser type and completely integrating
>> it with the install, all the way down to monit templates.
>> All of that work was configuration work in ansible.
>>
>> I think I’m asking about changes to something that wasn’t documented
>> anyways, as I think you are pointing out by reference, so I’ll just say
>> that this change has an effect on side loading.   You will be building an
>> Ambari Metron Topology service, probably from some template - hopefully
>> using maven archetypes or something the whole way.
>>
>>
>> On January 19, 2017 at 05:56:42, David Lyle (dlyle65...@gmail.com) wrote:
>>
>> Looks like my replies were only going to Otto, sorry about that- I'll
>> gather them here:
>>
>> "What does this do for Monit?"
>>
>> Monit would be deprecated for components under Ambari management.
>>
>> "How would this effect deploying new parsers or parsers not shipped?
>> When I prototyped this I added a monit entry."
>>
>> Hard to say having not seen Otto's prototype, but I suspect no effect.
>>
>> Fwiw, there is a jira about sideloading that is meant for deploying
>> custom
>> parsers. Last I looked, the management was still up for design.
>>
>> I"ve opened https://issues.apache.org/jira/browse/METRON-667 to track
>> this
>> work. I'd like to get started. Thoughts?
>>
>> -D...
>>
>>
>> On Wed, Jan 18, 2017 at 1:28 PM, David Lyle 
>> wrote:
>>
>> > Hard to say having not seen your prototype, but I suspect no effect.
>> >
>> > Fwiw, there is a jira about sideloading that is meant for deploying
>> custom
>> > parsers. Last I looked, the management was still up for design.
>> >
>> > -D...
>> >
>> > On Wed, Jan 18, 2017 at 13:07 Otto Fowler 
>> wrote:
>> >
>> >> How would this effect deploying new parsers or parsers not shipped?
>> >> When I prototyped this I added a monit entry.
>> >>
>> >>
>> >> On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com)
>> wrote:
>> >>
>> >> In our "Dev Guide and Committer Review Guide additions" discussion, we
>> had
>> >>
>> >>
>> >> a bit of a side discussion about reducing reliance (perhaps to zero)
>> on
>> >>
>> >>
>> >> Ansible for our installation.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> It seemed there was consensus around that idea (if not, please let me
>> >>
>> >>
>> >> know), so I propose the following steps to get there:
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> 1) Refactor existing Ansible deployment to use the Ambari MPack to
>> install
>> >>
>> >>
>> >> metron-common, metron-enrichments and metron-parsers.
>> >>
>> >>
>> >> 2) Regenerate quick-dev to leverage the change.
>> >>
>> >>
>> >> 3) Create rpm packages for all deployed components that don't
>> currently
>> >>
>> >>
>> >> have them.
>> >>
>> >>
>> >> - Sensor probes
>> >>
>> >>
>> >> - Sensor stubs
>> >>
>> >>
>> >> 4) Create MPack service defs for the RPMs in (2).
>> >>
>> >>
>> >> 5) Refactor existing Ansible deployment to use the Ambari MPack to
>> install
>> >>
>> >>
>> >> all services.
>> >>
>> >>
>> >> 6) Regenerate quick-dev to leverage the change.
>> >>
>> >>
>> >> 7) Plan iteration 2 to see if there are other opportunities to reduce
>> our
>> >>
>> >>
>> >> use of Ansible.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> One note: if we decide to go this direction, it'd be helpful if,
>> during
>> >> the
>> >>
>> >>
>> >> transition, we stopped adding additional Ansible deployment code.
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Thoughts?
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Thanks,
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -David...
>> >>
>> >>
>> >>
>>
>>
>

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[David Lyle]

For the first increment, I am planning on using the Ambari Metron topology
service that currently exists in the MPack. Handling the side loading isn't
in the scope of this proposal. We'll need to design that separately.

-D...


On Thu, Jan 19, 2017 at 8:48 AM, Otto Fowler 
wrote:

> So - my prototype was adding a new parser type and completely integrating
> it with the install, all the way down to monit templates.
> All of that work was configuration work in ansible.
>
> I think I’m asking about changes to something that wasn’t documented
> anyways, as I think you are pointing out by reference, so I’ll just say
> that this change has an effect on side loading.   You will be building an
> Ambari Metron Topology service, probably from some template - hopefully
> using maven archetypes or something the whole way.
>
>
> On January 19, 2017 at 05:56:42, David Lyle (dlyle65...@gmail.com) wrote:
>
> Looks like my replies were only going to Otto, sorry about that- I'll
> gather them here:
>
> "What does this do for Monit?"
>
> Monit would be deprecated for components under Ambari management.
>
> "How would this effect deploying new parsers or parsers not shipped?
> When I prototyped this I added a monit entry."
>
> Hard to say having not seen Otto's prototype, but I suspect no effect.
>
> Fwiw, there is a jira about sideloading that is meant for deploying custom
> parsers. Last I looked, the management was still up for design.
>
> I"ve opened https://issues.apache.org/jira/browse/METRON-667 to track
> this
> work. I'd like to get started. Thoughts?
>
> -D...
>
>
> On Wed, Jan 18, 2017 at 1:28 PM, David Lyle  wrote:
>
> > Hard to say having not seen your prototype, but I suspect no effect.
> >
> > Fwiw, there is a jira about sideloading that is meant for deploying
> custom
> > parsers. Last I looked, the management was still up for design.
> >
> > -D...
> >
> > On Wed, Jan 18, 2017 at 13:07 Otto Fowler 
> wrote:
> >
> >> How would this effect deploying new parsers or parsers not shipped?
> >> When I prototyped this I added a monit entry.
> >>
> >>
> >> On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com)
> wrote:
> >>
> >> In our "Dev Guide and Committer Review Guide additions" discussion, we
> had
> >>
> >>
> >> a bit of a side discussion about reducing reliance (perhaps to zero) on
> >>
> >>
> >> Ansible for our installation.
> >>
> >>
> >>
> >>
> >>
> >> It seemed there was consensus around that idea (if not, please let me
> >>
> >>
> >> know), so I propose the following steps to get there:
> >>
> >>
> >>
> >>
> >>
> >> 1) Refactor existing Ansible deployment to use the Ambari MPack to
> install
> >>
> >>
> >> metron-common, metron-enrichments and metron-parsers.
> >>
> >>
> >> 2) Regenerate quick-dev to leverage the change.
> >>
> >>
> >> 3) Create rpm packages for all deployed components that don't currently
> >>
> >>
> >> have them.
> >>
> >>
> >> - Sensor probes
> >>
> >>
> >> - Sensor stubs
> >>
> >>
> >> 4) Create MPack service defs for the RPMs in (2).
> >>
> >>
> >> 5) Refactor existing Ansible deployment to use the Ambari MPack to
> install
> >>
> >>
> >> all services.
> >>
> >>
> >> 6) Regenerate quick-dev to leverage the change.
> >>
> >>
> >> 7) Plan iteration 2 to see if there are other opportunities to reduce
> our
> >>
> >>
> >> use of Ansible.
> >>
> >>
> >>
> >>
> >>
> >> One note: if we decide to go this direction, it'd be helpful if, during
> >> the
> >>
> >>
> >> transition, we stopped adding additional Ansible deployment code.
> >>
> >>
> >>
> >>
> >>
> >> Thoughts?
> >>
> >>
> >>
> >>
> >>
> >> Thanks,
> >>
> >>
> >>
> >>
> >>
> >> -David...
> >>
> >>
> >>
>
>

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Otto Fowler]

So - my prototype was adding a new parser type and completely integrating
it with the install, all the way down to monit templates.
All of that work was configuration work in ansible.

I think I’m asking about changes to something that wasn’t documented
anyways, as I think you are pointing out by reference, so I’ll just say
that this change has an effect on side loading.   You will be building an
Ambari Metron Topology service, probably from some template - hopefully
using maven archetypes or something the whole way.


On January 19, 2017 at 05:56:42, David Lyle (dlyle65...@gmail.com) wrote:

Looks like my replies were only going to Otto, sorry about that- I'll
gather them here:

"What does this do for Monit?"

Monit would be deprecated for components under Ambari management.

"How would this effect deploying new parsers or parsers not shipped?
When I prototyped this I added a monit entry."

Hard to say having not seen Otto's prototype, but I suspect no effect.

Fwiw, there is a jira about sideloading that is meant for deploying custom
parsers. Last I looked, the management was still up for design.

I"ve opened https://issues.apache.org/jira/browse/METRON-667 to track this
work. I'd like to get started. Thoughts?

-D...


On Wed, Jan 18, 2017 at 1:28 PM, David Lyle  wrote:

> Hard to say having not seen your prototype, but I suspect no effect.
>
> Fwiw, there is a jira about sideloading that is meant for deploying
custom
> parsers. Last I looked, the management was still up for design.
>
> -D...
>
> On Wed, Jan 18, 2017 at 13:07 Otto Fowler 
wrote:
>
>> How would this effect deploying new parsers or parsers not shipped?
>> When I prototyped this I added a monit entry.
>>
>>
>> On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com)
wrote:
>>
>> In our "Dev Guide and Committer Review Guide additions" discussion, we
had
>>
>>
>> a bit of a side discussion about reducing reliance (perhaps to zero) on
>>
>>
>> Ansible for our installation.
>>
>>
>>
>>
>>
>> It seemed there was consensus around that idea (if not, please let me
>>
>>
>> know), so I propose the following steps to get there:
>>
>>
>>
>>
>>
>> 1) Refactor existing Ansible deployment to use the Ambari MPack to
install
>>
>>
>> metron-common, metron-enrichments and metron-parsers.
>>
>>
>> 2) Regenerate quick-dev to leverage the change.
>>
>>
>> 3) Create rpm packages for all deployed components that don't currently
>>
>>
>> have them.
>>
>>
>> - Sensor probes
>>
>>
>> - Sensor stubs
>>
>>
>> 4) Create MPack service defs for the RPMs in (2).
>>
>>
>> 5) Refactor existing Ansible deployment to use the Ambari MPack to
install
>>
>>
>> all services.
>>
>>
>> 6) Regenerate quick-dev to leverage the change.
>>
>>
>> 7) Plan iteration 2 to see if there are other opportunities to reduce
our
>>
>>
>> use of Ansible.
>>
>>
>>
>>
>>
>> One note: if we decide to go this direction, it'd be helpful if, during
>> the
>>
>>
>> transition, we stopped adding additional Ansible deployment code.
>>
>>
>>
>>
>>
>> Thoughts?
>>
>>
>>
>>
>>
>> Thanks,
>>
>>
>>
>>
>>
>> -David...
>>
>>
>>

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Otto Fowler]

How would this effect deploying new parsers or parsers not shipped?
When I prototyped this I added a monit entry.


On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com) wrote:

In our "Dev Guide and Committer Review Guide additions" discussion, we had
a bit of a side discussion about reducing reliance (perhaps to zero) on
Ansible for our installation.

It seemed there was consensus around that idea (if not, please let me
know), so I propose the following steps to get there:

1) Refactor existing Ansible deployment to use the Ambari MPack to install
metron-common, metron-enrichments and metron-parsers.
2) Regenerate quick-dev to leverage the change.
3) Create rpm packages for all deployed components that don't currently
have them.
- Sensor probes
- Sensor stubs
4) Create MPack service defs for the RPMs in (2).
5) Refactor existing Ansible deployment to use the Ambari MPack to install
all services.
6) Regenerate quick-dev to leverage the change.
7) Plan iteration 2 to see if there are other opportunities to reduce our
use of Ansible.

One note: if we decide to go this direction, it'd be helpful if, during the
transition, we stopped adding additional Ansible deployment code.

Thoughts?

Thanks,

-David...

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Otto Fowler]

What does this do for Monit?


On January 17, 2017 at 10:34:32, David Lyle (dlyle65...@gmail.com) wrote:

In our "Dev Guide and Committer Review Guide additions" discussion, we had
a bit of a side discussion about reducing reliance (perhaps to zero) on
Ansible for our installation.

It seemed there was consensus around that idea (if not, please let me
know), so I propose the following steps to get there:

1) Refactor existing Ansible deployment to use the Ambari MPack to install
metron-common, metron-enrichments and metron-parsers.
2) Regenerate quick-dev to leverage the change.
3) Create rpm packages for all deployed components that don't currently
have them.
- Sensor probes
- Sensor stubs
4) Create MPack service defs for the RPMs in (2).
5) Refactor existing Ansible deployment to use the Ambari MPack to install
all services.
6) Regenerate quick-dev to leverage the change.
7) Plan iteration 2 to see if there are other opportunities to reduce our
use of Ansible.

One note: if we decide to go this direction, it'd be helpful if, during the
transition, we stopped adding additional Ansible deployment code.

Thoughts?

Thanks,

-David...

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Matt Foley]

+1, great you’re addressing this!

On 1/17/17, 7:34 AM, "David Lyle"  wrote:

In our "Dev Guide and Committer Review Guide additions" discussion, we had
a bit of a side discussion about reducing reliance (perhaps to zero) on
Ansible for our installation.

It seemed there was consensus around that idea (if not, please let me
know), so I propose the following steps to get there:

1) Refactor existing Ansible deployment to use the Ambari MPack to install
metron-common, metron-enrichments and metron-parsers.
2) Regenerate quick-dev to leverage the change.
3) Create rpm packages for all deployed components that don't currently
have them.
 - Sensor probes
 - Sensor stubs
4) Create MPack service defs for the RPMs in (2).
5) Refactor existing Ansible deployment to use the Ambari MPack to install
all services.
6) Regenerate quick-dev to leverage the change.
7) Plan iteration 2 to see if there are other opportunities to reduce our
use of Ansible.

One note: if we decide to go this direction, it'd be helpful if, during the
transition, we stopped adding additional Ansible deployment code.

Thoughts?

Thanks,

-David...

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[David Lyle]

Ambari exposes templating via configs and parameter classes. The current
configuration screens for Metron Enrichment is a pretty good example of
that, they fill in templates under the covers.

-D...


On Tue, Jan 17, 2017 at 11:55 AM, Nick Allen  wrote:

> Plan sounds solid.
>
> I've been curious how to replicate the templating that Ansible gives us. Do
> we have a plan for that or is that something we have to figure out?
>
> On Tue, Jan 17, 2017 at 10:34 AM, David Lyle  wrote:
>
> > In our "Dev Guide and Committer Review Guide additions" discussion, we
> had
> > a bit of a side discussion about reducing reliance (perhaps to zero) on
> > Ansible for our installation.
> >
> > It seemed there was consensus around that idea (if not, please let me
> > know), so I propose the following steps to get there:
> >
> > 1) Refactor existing Ansible deployment to use the Ambari MPack to
> install
> > metron-common, metron-enrichments and metron-parsers.
> > 2) Regenerate quick-dev to leverage the change.
> > 3) Create rpm packages for all deployed components that don't currently
> > have them.
> >  - Sensor probes
> >  - Sensor stubs
> > 4) Create MPack service defs for the RPMs in (2).
> > 5) Refactor existing Ansible deployment to use the Ambari MPack to
> install
> > all services.
> > 6) Regenerate quick-dev to leverage the change.
> > 7) Plan iteration 2 to see if there are other opportunities to reduce our
> > use of Ansible.
> >
> > One note: if we decide to go this direction, it'd be helpful if, during
> the
> > transition, we stopped adding additional Ansible deployment code.
> >
> > Thoughts?
> >
> > Thanks,
> >
> > -David...
> >
>
>
>
> --
> Nick Allen 
>

Re: [PROPOSAL] Reduce Reliance on Ansible for Deployment

[Nick Allen]

Plan sounds solid.

I've been curious how to replicate the templating that Ansible gives us. Do
we have a plan for that or is that something we have to figure out?

On Tue, Jan 17, 2017 at 10:34 AM, David Lyle  wrote:

> In our "Dev Guide and Committer Review Guide additions" discussion, we had
> a bit of a side discussion about reducing reliance (perhaps to zero) on
> Ansible for our installation.
>
> It seemed there was consensus around that idea (if not, please let me
> know), so I propose the following steps to get there:
>
> 1) Refactor existing Ansible deployment to use the Ambari MPack to install
> metron-common, metron-enrichments and metron-parsers.
> 2) Regenerate quick-dev to leverage the change.
> 3) Create rpm packages for all deployed components that don't currently
> have them.
>  - Sensor probes
>  - Sensor stubs
> 4) Create MPack service defs for the RPMs in (2).
> 5) Refactor existing Ansible deployment to use the Ambari MPack to install
> all services.
> 6) Regenerate quick-dev to leverage the change.
> 7) Plan iteration 2 to see if there are other opportunities to reduce our
> use of Ansible.
>
> One note: if we decide to go this direction, it'd be helpful if, during the
> transition, we stopped adding additional Ansible deployment code.
>
> Thoughts?
>
> Thanks,
>
> -David...
>



-- 
Nick Allen