Re: [DISCUSS] The bro kafka plugin

[zeo...@gmail.com]

Ok, great.

I agree, I definitely want to hear from Nick on the topic.  My team is
currently looking into enhancing the plugin as well to potentially allow
sending to multiple clusters, investigating some issues we see when our bro
cluster is under load, turn it into a package, etc.

The work you just did was on our to do list as well so I'm very excited to
see it come through.

Jon

On Thu, Mar 30, 2017, 11:16 PM Casey Stella  wrote:

I *think* it's possible.  People do ask for mirrors of directories from
time to time (see https://issues.apache.org/jira/browse/INFRA-7060).  If we
think this is a good idea, we can pose it to INFRA as a request.  I'd love
to see us be able to use the bro packaging infrastructure and get more
visibility for the plugin.

I'd be particularly interested in Nick's opinion on this, though.

On Thu, Mar 30, 2017 at 11:12 PM, zeo...@gmail.com  wrote:

> You can version packages -
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> versioning
>
> I agree that having a separate repo provided by Apache would be optimal, I
> just don't know the process for that or if it was even reasonable to
> suggest.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:01 PM Casey Stella  wrote:
>
> > Looking at the bro packages, it appears that bro is expecting things to
> be
> > its own git repository.  I wonder if we could either request INFRA
> provide
> > another repo for the bro-kafka plugin and integrate it into metron as a
> git
> > submodule *or* if we could request INFRA to create a github mirror of
the
> > metron-sensors/bro-kafka-plugin directory.  I'm not sure how viable
> either
> > of those options are, frankly.
> >
> > One thing that I didn't see is how do you specify a particular release
of
> > the plugin that you want to install?  For us, we'd want to release the
> > plugin along with the product.  I didn't quite see how you'd push
> releases
> > for bro plugins.
> >
> > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella 
> wrote:
> >
> > > So, I do agree with the concern.  Is there a way to host the package
> > > within Metron?  I definitely would like to see the modifications at
> > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us
> to
> > > host the plugin.
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > >> Today I was taking a look at METRON-812
> > >> , which made me
> > recall
> > >> some conversations from a while back regarding where the bro kafka
> > plugin
> > >> should ultimately live, and how to update it.
> > >>
> > >> Back in METRON-348 
> I
> > >> brought up the fact that some important changes
> > >>  > >> 5348da0a5043a8353b4a0a8>
> > >> were made to the externally hosted version of the kafka plugin, and
> were
> > >> never introduced to Metron's hosted version (i.e. the one we use
> > >>  > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > >> in vagrant when bro is installed).  The conversation went down the
> route
> > >> of
> > >> discussing whether or not the bro kafka plugin code should continue
to
> > >> live
> > >> in Metron in the first place.  Now, with METRON-812, I see us further
> > >> muddying the waters of where to go for the right plugin, as our
> version
> > is
> > >> still missing the public changes but adds some very important new
> > >> functionality.
> > >>
> > >> I'd like to bring up the idea of using bro's packages
> > >>  framework, released in late 2016
> > >> 
> > >> (additional
> > >> documentation here <
> > http://bro-package-manager.readthedocs.io/en/stable/
> > >> >),
> > >> as a potential place for this to be hosted/referenced.  This is a
> simple
> > >> and supported method (funded by Mozilla
> > >>  > >> e-support-first-awards-made/>)
> > >> to install and uninstall bro scripts, plugins, etc., and it also
> allows
> > us
> > >> to continue to have enough control over updates to the plugin so that
> it
> > >> will not slow down Metron development by having it as a dependency
> > >> (resolving both of Casey's concerns noted here
> > >>  > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > >> and I think this solution is supported by Nick's comments here
> > >>  > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > >> as
> > >> well).
> >

Re: [DISCUSS] The bro kafka plugin

[Casey Stella]

I *think* it's possible.  People do ask for mirrors of directories from
time to time (see https://issues.apache.org/jira/browse/INFRA-7060).  If we
think this is a good idea, we can pose it to INFRA as a request.  I'd love
to see us be able to use the bro packaging infrastructure and get more
visibility for the plugin.

I'd be particularly interested in Nick's opinion on this, though.

On Thu, Mar 30, 2017 at 11:12 PM, zeo...@gmail.com  wrote:

> You can version packages -
> http://bro-package-manager.readthedocs.io/en/stable/package.html#package-
> versioning
>
> I agree that having a separate repo provided by Apache would be optimal, I
> just don't know the process for that or if it was even reasonable to
> suggest.
>
> Jon
>
> On Thu, Mar 30, 2017, 11:01 PM Casey Stella  wrote:
>
> > Looking at the bro packages, it appears that bro is expecting things to
> be
> > its own git repository.  I wonder if we could either request INFRA
> provide
> > another repo for the bro-kafka plugin and integrate it into metron as a
> git
> > submodule *or* if we could request INFRA to create a github mirror of the
> > metron-sensors/bro-kafka-plugin directory.  I'm not sure how viable
> either
> > of those options are, frankly.
> >
> > One thing that I didn't see is how do you specify a particular release of
> > the plugin that you want to install?  For us, we'd want to release the
> > plugin along with the product.  I didn't quite see how you'd push
> releases
> > for bro plugins.
> >
> > On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella 
> wrote:
> >
> > > So, I do agree with the concern.  Is there a way to host the package
> > > within Metron?  I definitely would like to see the modifications at
> > > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us
> to
> > > host the plugin.
> > >
> > > Thoughts?
> > >
> > >
> > > On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com 
> > > wrote:
> > >
> > >> Today I was taking a look at METRON-812
> > >> , which made me
> > recall
> > >> some conversations from a while back regarding where the bro kafka
> > plugin
> > >> should ultimately live, and how to update it.
> > >>
> > >> Back in METRON-348 
> I
> > >> brought up the fact that some important changes
> > >>  > >> 5348da0a5043a8353b4a0a8>
> > >> were made to the externally hosted version of the kafka plugin, and
> were
> > >> never introduced to Metron's hosted version (i.e. the one we use
> > >>  > >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> > >> in vagrant when bro is installed).  The conversation went down the
> route
> > >> of
> > >> discussing whether or not the bro kafka plugin code should continue to
> > >> live
> > >> in Metron in the first place.  Now, with METRON-812, I see us further
> > >> muddying the waters of where to go for the right plugin, as our
> version
> > is
> > >> still missing the public changes but adds some very important new
> > >> functionality.
> > >>
> > >> I'd like to bring up the idea of using bro's packages
> > >>  framework, released in late 2016
> > >> 
> > >> (additional
> > >> documentation here <
> > http://bro-package-manager.readthedocs.io/en/stable/
> > >> >),
> > >> as a potential place for this to be hosted/referenced.  This is a
> simple
> > >> and supported method (funded by Mozilla
> > >>  > >> e-support-first-awards-made/>)
> > >> to install and uninstall bro scripts, plugins, etc., and it also
> allows
> > us
> > >> to continue to have enough control over updates to the plugin so that
> it
> > >> will not slow down Metron development by having it as a dependency
> > >> (resolving both of Casey's concerns noted here
> > >>  > >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391865>,
> > >> and I think this solution is supported by Nick's comments here
> > >>  > >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> > >> issuetabpanels:comment-tabpanel#comment-15391872>
> > >> as
> > >> well).
> > >>
> > >> The only thing I'm not sure about is where to host the plugin itself -
> > my
> > >> first thought would be Nick's github ,
> > as
> > >> he
> > >> really kicked off this effort, but maybe we can think of something
> > better.
> > >>
> > >> Is this approach of interest to anybody?  It is extremely simple to
> put
> > >> together - I was able to throw one together
> > >> 

[GitHub] incubator-metron issue #501: METRON-812: Make the bro-kafka plugin work with...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/501
  
Nope.

On Thu, Mar 30, 2017, 11:03 PM Casey Stella 
wrote:

> Thanks for the discussion @JonZeolla  Just
> for clarity, do you have issues with this work going in independent of the
> final fate of the bro-kafka plugin?
>
> —
> You are receiving this because you were mentioned.
>
>
> Reply to this email directly, view it on GitHub
> 
,
> or mute the thread
> 

> .
>
-- 

Jon



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/497
  
Just a comment, METRON-797 and METRON-793 are both in master as of the time 
of this comment, so these docs can be done on current master.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] The bro kafka plugin

[zeo...@gmail.com]

You can version packages -
http://bro-package-manager.readthedocs.io/en/stable/package.html#package-versioning

I agree that having a separate repo provided by Apache would be optimal, I
just don't know the process for that or if it was even reasonable to
suggest.

Jon

On Thu, Mar 30, 2017, 11:01 PM Casey Stella  wrote:

> Looking at the bro packages, it appears that bro is expecting things to be
> its own git repository.  I wonder if we could either request INFRA provide
> another repo for the bro-kafka plugin and integrate it into metron as a git
> submodule *or* if we could request INFRA to create a github mirror of the
> metron-sensors/bro-kafka-plugin directory.  I'm not sure how viable either
> of those options are, frankly.
>
> One thing that I didn't see is how do you specify a particular release of
> the plugin that you want to install?  For us, we'd want to release the
> plugin along with the product.  I didn't quite see how you'd push releases
> for bro plugins.
>
> On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella  wrote:
>
> > So, I do agree with the concern.  Is there a way to host the package
> > within Metron?  I definitely would like to see the modifications at
> > https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> > 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us to
> > host the plugin.
> >
> > Thoughts?
> >
> >
> > On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com 
> > wrote:
> >
> >> Today I was taking a look at METRON-812
> >> , which made me
> recall
> >> some conversations from a while back regarding where the bro kafka
> plugin
> >> should ultimately live, and how to update it.
> >>
> >> Back in METRON-348  I
> >> brought up the fact that some important changes
> >>  >> 5348da0a5043a8353b4a0a8>
> >> were made to the externally hosted version of the kafka plugin, and were
> >> never introduced to Metron's hosted version (i.e. the one we use
> >>  >> on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> >> in vagrant when bro is installed).  The conversation went down the route
> >> of
> >> discussing whether or not the bro kafka plugin code should continue to
> >> live
> >> in Metron in the first place.  Now, with METRON-812, I see us further
> >> muddying the waters of where to go for the right plugin, as our version
> is
> >> still missing the public changes but adds some very important new
> >> functionality.
> >>
> >> I'd like to bring up the idea of using bro's packages
> >>  framework, released in late 2016
> >> 
> >> (additional
> >> documentation here <
> http://bro-package-manager.readthedocs.io/en/stable/
> >> >),
> >> as a potential place for this to be hosted/referenced.  This is a simple
> >> and supported method (funded by Mozilla
> >>  >> e-support-first-awards-made/>)
> >> to install and uninstall bro scripts, plugins, etc., and it also allows
> us
> >> to continue to have enough control over updates to the plugin so that it
> >> will not slow down Metron development by having it as a dependency
> >> (resolving both of Casey's concerns noted here
> >>  >> mentId=15391865&page=com.atlassian.jira.plugin.system.
> >> issuetabpanels:comment-tabpanel#comment-15391865>,
> >> and I think this solution is supported by Nick's comments here
> >>  >> mentId=15391872&page=com.atlassian.jira.plugin.system.
> >> issuetabpanels:comment-tabpanel#comment-15391872>
> >> as
> >> well).
> >>
> >> The only thing I'm not sure about is where to host the plugin itself -
> my
> >> first thought would be Nick's github ,
> as
> >> he
> >> really kicked off this effort, but maybe we can think of something
> better.
> >>
> >> Is this approach of interest to anybody?  It is extremely simple to put
> >> together - I was able to throw one together
> >> 
> and
> >> get it working with a fresh bro 2.5 install when attending the brocon
> talk
> >>  >> bro-packagemanager>
> >>  (recording , slides
> >> ) that introduced
> >> this
> >> to me in the first place.
> >>
> >> Jon
> >> --
> >>
> >> Jon
> >>
> >
> >
>
-- 

Jon

[GitHub] incubator-metron pull request #491: METRON-773: Intermittent unit test error...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/491


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #501: METRON-812: Make the bro-kafka plugin work with...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/501
  
Thanks for the discussion @JonZeolla  Just for clarity, do you have issues 
with this work going in independent of the final fate of the bro-kafka plugin?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] The bro kafka plugin

[Casey Stella]

Looking at the bro packages, it appears that bro is expecting things to be
its own git repository.  I wonder if we could either request INFRA provide
another repo for the bro-kafka plugin and integrate it into metron as a git
submodule *or* if we could request INFRA to create a github mirror of the
metron-sensors/bro-kafka-plugin directory.  I'm not sure how viable either
of those options are, frankly.

One thing that I didn't see is how do you specify a particular release of
the plugin that you want to install?  For us, we'd want to release the
plugin along with the product.  I didn't quite see how you'd push releases
for bro plugins.

On Thu, Mar 30, 2017 at 10:49 PM, Casey Stella  wrote:

> So, I do agree with the concern.  Is there a way to host the package
> within Metron?  I definitely would like to see the modifications at
> https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db
> 065348da0a5043a8353b4a0a8 brought back into Metron and I'd love for us to
> host the plugin.
>
> Thoughts?
>
>
> On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com 
> wrote:
>
>> Today I was taking a look at METRON-812
>> , which made me recall
>> some conversations from a while back regarding where the bro kafka plugin
>> should ultimately live, and how to update it.
>>
>> Back in METRON-348  I
>> brought up the fact that some important changes
>> > 5348da0a5043a8353b4a0a8>
>> were made to the externally hosted version of the kafka plugin, and were
>> never introduced to Metron's hosted version (i.e. the one we use
>> > on-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
>> in vagrant when bro is installed).  The conversation went down the route
>> of
>> discussing whether or not the bro kafka plugin code should continue to
>> live
>> in Metron in the first place.  Now, with METRON-812, I see us further
>> muddying the waters of where to go for the right plugin, as our version is
>> still missing the public changes but adds some very important new
>> functionality.
>>
>> I'd like to bring up the idea of using bro's packages
>>  framework, released in late 2016
>> 
>> (additional
>> documentation here > >),
>> as a potential place for this to be hosted/referenced.  This is a simple
>> and supported method (funded by Mozilla
>> > e-support-first-awards-made/>)
>> to install and uninstall bro scripts, plugins, etc., and it also allows us
>> to continue to have enough control over updates to the plugin so that it
>> will not slow down Metron development by having it as a dependency
>> (resolving both of Casey's concerns noted here
>> > mentId=15391865&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15391865>,
>> and I think this solution is supported by Nick's comments here
>> > mentId=15391872&page=com.atlassian.jira.plugin.system.
>> issuetabpanels:comment-tabpanel#comment-15391872>
>> as
>> well).
>>
>> The only thing I'm not sure about is where to host the plugin itself - my
>> first thought would be Nick's github , as
>> he
>> really kicked off this effort, but maybe we can think of something better.
>>
>> Is this approach of interest to anybody?  It is extremely simple to put
>> together - I was able to throw one together
>>  and
>> get it working with a fresh bro 2.5 install when attending the brocon talk
>> > bro-packagemanager>
>>  (recording , slides
>> ) that introduced
>> this
>> to me in the first place.
>>
>> Jon
>> --
>>
>> Jon
>>
>
>

Re: [DISCUSS] The bro kafka plugin

[Casey Stella]

So, I do agree with the concern.  Is there a way to host the package within
Metron?  I definitely would like to see the modifications at
https://github.com/bro/bro-plugins/commit/b9f1f35415cb0db065348da0a5043a
8353b4a0a8 brought back into Metron and I'd love for us to host the plugin.


Thoughts?


On Thu, Mar 30, 2017 at 9:09 PM, zeo...@gmail.com  wrote:

> Today I was taking a look at METRON-812
> , which made me recall
> some conversations from a while back regarding where the bro kafka plugin
> should ultimately live, and how to update it.
>
> Back in METRON-348  I
> brought up the fact that some important changes
>  8353b4a0a8>
> were made to the externally hosted version of the kafka plugin, and were
> never introduced to Metron's hosted version (i.e. the one we use
>  metron-deployment/roles/bro/tasks/bro-plugin-kafka.yml>
> in vagrant when bro is installed).  The conversation went down the route of
> discussing whether or not the bro kafka plugin code should continue to live
> in Metron in the first place.  Now, with METRON-812, I see us further
> muddying the waters of where to go for the right plugin, as our version is
> still missing the public changes but adds some very important new
> functionality.
>
> I'd like to bring up the idea of using bro's packages
>  framework, released in late 2016
> 
> (additional
> documentation here  >),
> as a potential place for this to be hosted/referenced.  This is a simple
> and supported method (funded by Mozilla
>  source-support-first-awards-made/>)
> to install and uninstall bro scripts, plugins, etc., and it also allows us
> to continue to have enough control over updates to the plugin so that it
> will not slow down Metron development by having it as a dependency
> (resolving both of Casey's concerns noted here
>  focusedCommentId=15391865&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15391865>,
> and I think this solution is supported by Nick's comments here
>  focusedCommentId=15391872&page=com.atlassian.jira.
> plugin.system.issuetabpanels:comment-tabpanel#comment-15391872>
> as
> well).
>
> The only thing I'm not sure about is where to host the plugin itself - my
> first thought would be Nick's github , as
> he
> really kicked off this effort, but maybe we can think of something better.
>
> Is this approach of interest to anybody?  It is extremely simple to put
> together - I was able to throw one together
>  and
> get it working with a fresh bro 2.5 install when attending the brocon talk
>  packagemanager>
>  (recording , slides
> ) that introduced this
> to me in the first place.
>
> Jon
> --
>
> Jon
>

[GitHub] incubator-metron issue #501: METRON-812: Make the bro-kafka plugin work with...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/incubator-metron/pull/501
  
I just shot an email out to the dev mailing list titled "[DISCUSS] The bro 
kafka plugin" to discuss some thoughts broader than this individual PR, but 
related to this work.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[DISCUSS] The bro kafka plugin

[zeo...@gmail.com]

Today I was taking a look at METRON-812
, which made me recall
some conversations from a while back regarding where the bro kafka plugin
should ultimately live, and how to update it.

Back in METRON-348  I
brought up the fact that some important changes

were made to the externally hosted version of the kafka plugin, and were
never introduced to Metron's hosted version (i.e. the one we use

in vagrant when bro is installed).  The conversation went down the route of
discussing whether or not the bro kafka plugin code should continue to live
in Metron in the first place.  Now, with METRON-812, I see us further
muddying the waters of where to go for the right plugin, as our version is
still missing the public changes but adds some very important new
functionality.

I'd like to bring up the idea of using bro's packages
 framework, released in late 2016
 (additional
documentation here ),
as a potential place for this to be hosted/referenced.  This is a simple
and supported method (funded by Mozilla
)
to install and uninstall bro scripts, plugins, etc., and it also allows us
to continue to have enough control over updates to the plugin so that it
will not slow down Metron development by having it as a dependency
(resolving both of Casey's concerns noted here
,
and I think this solution is supported by Nick's comments here

as
well).

The only thing I'm not sure about is where to host the plugin itself - my
first thought would be Nick's github , as he
really kicked off this effort, but maybe we can think of something better.

Is this approach of interest to anybody?  It is extremely simple to put
together - I was able to throw one together
 and
get it working with a fresh bro 2.5 install when attending the brocon talk

 (recording , slides
) that introduced this
to me in the first place.

Jon
-- 

Jon

Re: [GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

[Matt Foley]

Okay, try this: 
https://github.com/mattf-horton/incubator-metron/blob/METRON-804/metron-deployment/vagrant/Kerberos-setup.md

I wasn’t able to build a PR to your branch, seems there’s a non-ff in the way 
the previous patch was merged.  Anyway, if you just grab that file and diff 
against yours, you’ll see the change is small.

Items 7, 8, and 20 needed to be fixed.  The problem is that “a-b-c” paragraphs 
aren’t actually list-items, as MD only knows roman numerals for list numbering. 
 Since they are paragraphs, the codeblocks and images under them should be at 
the SAME indent level, and separated by explicit blank line.

This works in both Github-MD and doxia-markdown.  It looks slightly better in 
doxia because in Github the “a-b-c” paragraphs are exdented a little.  If you 
hate it we can try a couple other things, but I thought this was close enough.

Cheers,
--Matt

On 3/30/17, 2:40 PM, "Matt Foley"  wrote:

That’s weird.  Mine looks fine: 
https://github.com/mattf-horton/incubator-metron/blob/METRON-804-notes/METRON-804-mf.tiff

But the tooling was exactly that of 
https://github.com/mattf-horton/incubator-metron/tree/METRON-804/site-book/bin

What additional changes did you make?

Oh, I just looked in github, and it’s broken there!  How ironic.
On your side, is it broken in Github or in the site-book? 
--Matt

On 3/30/17, 11:15 AM, "mmiklavc"  wrote:

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/497
  
@mattf-horton Thanks again for the patch! I made a couple more 
minor tweaks to get the images and indentation correct for the nested lists. 
I'm unable to get a nested list code block to format correctly, however. It's 
not bad, but it's just not quite right. If anyone has any suggestions, please 
chime in.

![image](https://cloud.githubusercontent.com/assets/658443/24519468/7f928ea6-1542-11e7-80c6-0070a1810f5e.png)



---
If your project is set up for it, you can reply to this email and have 
your
reply appear on GitHub as well. If your project does not have this 
feature
enabled and wishes so, or if the feature is enabled but not working, 
please
contact infrastructure at infrastruct...@apache.org or file a JIRA 
ticket
with INFRA.
---

[GitHub] incubator-metron issue #501: METRON-812: Make the bro-kafka plugin work with...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/501
  
# Testing Plan
## Preliminaries

* Spin up the full-dev environment via `vagrant up` from 
`metron-deployment/vagrant/full-dev-platform`
* Stop the "Metron" service in ambari and put it in maintenance mode.
* Stop the sensor stubs 
  * `service sensor-stubs stop`
  * Make sure the bro sensor stub is dead via `for i in $(ps -ef | grep 
start-bro-stub | awk '{print $2}');do kill -9 $i;done`
* Install sensors without stubs by running the following from 
`metron-deployment/vagrant/full-dev-platform`:
```
vagrant --ansible-tags="sensors,bro,pycapa" 
--ansible-skip-tags="solr,flume,snort,yaf,sensor-test-mode" provision
```
* Set some environment variables to indicate `METRON_HOME`:
```
export METRON_HOME=/usr/metron/0.3.1
export HDP_HOME=/usr/hdp/current
``` 

## Non Kerberized Environment

Ensure that you can see bro data flowing with the base configuration by
running the console consumer and ensuring bro data flows through:
```
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server 
node1:6667 --zookeeper node1:2181 --topic bro
```

## Kerberized Environment

* Follow steps 1-10, 14-18 of the kerberization instructions for full-dev

[here](https://github.com/mmiklavc/incubator-metron/blob/9ef9d5d97ca654c9120cae5e40eddfe69d6420a8/metron-deployment/vagrant/Kerberos-setup.md).
* Create a new topic called `b_k`
```
export KERB_USER=metron;
${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOOKEEPER:2181 
--create --topic b_k --partitions 1 --replication-factor 1
${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=node1:2181 --add --allow-principal User:${KERB_USER} --topic 
b_k;
```
* Edit `~/consumer.config` to contain the following:
```
group.id=b_k_grp
```
* Edit `/usr/local/bro/share/bro/site/local.bro` to configure the 
bro-kafka-plugin with the following (at the end of the document):
```
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "b_k";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "node1:6667"
   , ["security.protocol"] = "SASL_PLAINTEXT"
   , ["sasl.kerberos.keytab"] = 
"/etc/security/keytabs/metron.headless.keytab"
   , ["sasl.kerberos.principal"] = 
"met...@example.com"
   );
```
* Redeploy bro via `/usr/local/bro/bin/broctl deploy`
* Listen for messages on the `b_k` topic via
```
${HDP_HOME}/kafka-broker/bin/kafka-console-consumer.sh --bootstrap-server 
node1:6667 --zookeeper node1:2181 --security-protocol SASL_PLAINTEXT  --topic 
b_k --new-consumer --consumer.config ~/consumer.config
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #495: METRON-797: Pass security.protocol and e...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/495


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #501: METRON-812: Make the bro-kafka plugin wo...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/incubator-metron/pull/501

METRON-812: Make the bro-kafka plugin work with kerberos

## Contributor Comments
The bro-kafka plugin does not currently support kerberos. This JIRA should
* make the version of librdkafka supported 0.9.4
* ensure the plugin can write to a kerberized kafka
* provide instructions on how to configure the plugin to write to a 
kerberized kafka



## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron bro_kafka

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/501.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #501






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

[Matt Foley]

That’s weird.  Mine looks fine: 
https://github.com/mattf-horton/incubator-metron/blob/METRON-804-notes/METRON-804-mf.tiff

But the tooling was exactly that of 
https://github.com/mattf-horton/incubator-metron/tree/METRON-804/site-book/bin

What additional changes did you make?

Oh, I just looked in github, and it’s broken there!  How ironic.
On your side, is it broken in Github or in the site-book? 
--Matt

On 3/30/17, 11:15 AM, "mmiklavc"  wrote:

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/497
  
@mattf-horton Thanks again for the patch! I made a couple more minor 
tweaks to get the images and indentation correct for the nested lists. I'm 
unable to get a nested list code block to format correctly, however. It's not 
bad, but it's just not quite right. If anyone has any suggestions, please chime 
in.

![image](https://cloud.githubusercontent.com/assets/658443/24519468/7f928ea6-1542-11e7-80c6-0070a1810f5e.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #500: METRON-795: Install Metron REST with Amb...

[merrimanr]

GitHub user merrimanr opened a pull request:

https://github.com/apache/incubator-metron/pull/500

METRON-795: Install Metron REST with Ambari MPack

## Contributor Comments
This PR adds the metron-rest module to the Ambari MPack.  This can be 
tested by building the rpms and mpack and spinning up full-dev-platform.  Once 
the ansible script has finished, navigate to "Ambari > Services > Metron" and 
there should be an additional "Metron REST" component in the Summary.  The 
"Quick Links" tab will have a link to the REST Swagger UI that should be 
functional if the service is running.  All the Ambari lifecycle commands should 
work.  I also tested starting/stopping the service with the "service 
metron-rest start/stop" commands and verified Ambari correctly reports status.

I ran into an issue with rpmlint complaining about missing chkconfig calls:

```
The package contains an init script but doesn't contain a %post with a call 
to
chkconfig
```

I believe this is benign because Ambari managements the REST application 
lifecycle but I can't figure out how to suppress them yet.  Also considering 
adding a couple lines to the metron-rest README that references the MPack 
install path.  Credentials for the REST app are "user/password" (will also add 
that to the metron-rest README).

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/merrimanr/incubator-metron METRON-795

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/500.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #500


commit 033988cbbfa4b5b061221d4e6d9d255aa3bc9596
Author: merrimanr 
Date:   2017-03-24T00:45:38Z

added metron-rest to rpm

commit 5bd738b2681f2f4b2ff1b6e620ee047c679c7c7a
Author: merrimanr 
Date:   2017-03-24T01:06:50Z

Fixed Provides

commit 2b1c88832254e4509e1bcd7e91c8ccb01f731a26
Author: merrimanr 
Date:   2017-03-30T19:04:51Z

Initial commit for METRON-795

commit 3fd0a5e6e7db2c40dde9e45f3968

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/497#discussion_r108982409
  
--- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
@@ -0,0 +1,238 @@
+# Setting Up Kerberos in Vagrant Full Dev
+**Note:** These are manual instructions for Kerberizing Metron Storm 
topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor 
connections, or MAAS.
+
+1. Build full dev and ssh into the machine
+```
+cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+vagrant up
+vagrant ssh
+```
+
+2. Export env vars
+```
+# execute as root
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.3.1"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+```
+
+3. Stop all topologies - we will  restart them again once Kerberos has 
been enabled.
+```
+for topology in bro snort enrichment indexing; do storm kill $topology; 
done
+```
+
+4. Setup Kerberos
+```
+# Note: if you copy/paste this full set of commands, the kdb5_util command 
will not run as expected, so run the commands individually to ensure they all 
execute
+yum -y install krb5-server krb5-libs krb5-workstation
+sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+# This step takes a moment. It creates the kerberos database.
+kdb5_util create -s
+/etc/rc.d/init.d/krb5kdc start
+/etc/rc.d/init.d/kadmin start
+chkconfig krb5kdc on
+chkconfig kadmin on
+```
+
+5. Setup the admin and metron user principals. You'll kinit as the metron 
user when running topologies. Make sure to remember the passwords.
+```
+kadmin.local -q "addprinc admin/admin"
+kadmin.local -q "addprinc metron"
+```
+
+6. Create the metron user HDFS home directory
+```
+sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+```
+
+7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the 
submitting user. Add the following properties to custom storm-site. In the 
Storm config section in Ambari, choose “Add Property” under custom 
storm-site. In the dialog window, choose the “bulk property add mode” 
toggle button and add the below values.
+```

+topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']

+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+```
+
+![custom storm-site](readme-images/ambari-storm-site.png)
+
+![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
+
+8. Kerberize the cluster via Ambari. More detailed documentation can be 
found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+   1. For this exercise, choose existing MIT KDC (this is what we setup 
and installed in the previous steps.)
+  ![enable keberos](readme-images/enable-kerberos.png)
+  ![enable keberos get 
started](readme-images/enable-kerberos-started.png)
+   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin 
principal will end up as admin/ad...@example.com when testing the KDC. Use the 
password you entered during the step for adding the admin principal.
+  ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
+   3. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+  ![enable keberos 
configure](readme-images/custom-storm-site-final.png)
+
+9. Setup Metron keytab
+```
+kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
+cp metron.headless.keytab /etc/security/keytabs && \
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+```
+
+10. Kinit with the metron user
+```
+kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+```
+
+11. First create any additional Kafka topics you will need. We need to 
create the topics before adding the required ACLs. The current full dev 
installation will deploy bro, snort, enrichments, and indexing only. e.g.
+```
+${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOO

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/497#discussion_r108997211
  
--- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
@@ -0,0 +1,238 @@
+# Setting Up Kerberos in Vagrant Full Dev
+**Note:** These are manual instructions for Kerberizing Metron Storm 
topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor 
connections, or MAAS.
+
+1. Build full dev and ssh into the machine
+```
+cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+vagrant up
+vagrant ssh
+```
+
+2. Export env vars
+```
+# execute as root
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.3.1"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+```
+
+3. Stop all topologies - we will  restart them again once Kerberos has 
been enabled.
+```
+for topology in bro snort enrichment indexing; do storm kill $topology; 
done
+```
+
+4. Setup Kerberos
+```
+# Note: if you copy/paste this full set of commands, the kdb5_util command 
will not run as expected, so run the commands individually to ensure they all 
execute
+yum -y install krb5-server krb5-libs krb5-workstation
+sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+# This step takes a moment. It creates the kerberos database.
+kdb5_util create -s
+/etc/rc.d/init.d/krb5kdc start
+/etc/rc.d/init.d/kadmin start
+chkconfig krb5kdc on
+chkconfig kadmin on
+```
+
+5. Setup the admin and metron user principals. You'll kinit as the metron 
user when running topologies. Make sure to remember the passwords.
+```
+kadmin.local -q "addprinc admin/admin"
+kadmin.local -q "addprinc metron"
+```
+
+6. Create the metron user HDFS home directory
+```
+sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+```
+
+7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the 
submitting user. Add the following properties to custom storm-site. In the 
Storm config section in Ambari, choose “Add Property” under custom 
storm-site. In the dialog window, choose the “bulk property add mode” 
toggle button and add the below values.
+```

+topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']

+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+```
+
+![custom storm-site](readme-images/ambari-storm-site.png)
+
+![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
+
+8. Kerberize the cluster via Ambari. More detailed documentation can be 
found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+   1. For this exercise, choose existing MIT KDC (this is what we setup 
and installed in the previous steps.)
+  ![enable keberos](readme-images/enable-kerberos.png)
+  ![enable keberos get 
started](readme-images/enable-kerberos-started.png)
+   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin 
principal will end up as admin/ad...@example.com when testing the KDC. Use the 
password you entered during the step for adding the admin principal.
+  ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
+   3. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+  ![enable keberos 
configure](readme-images/custom-storm-site-final.png)
+
+9. Setup Metron keytab
+```
+kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
+cp metron.headless.keytab /etc/security/keytabs && \
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+```
+
+10. Kinit with the metron user
+```
+kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+```
+
+11. First create any additional Kafka topics you will need. We need to 
create the topics before adding the required ACLs. The current full dev 
installation will deploy bro, snort, enrichments, and indexing only. e.g.
+```
+${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOO

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/497#discussion_r108999605
  
--- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
@@ -0,0 +1,238 @@
+# Setting Up Kerberos in Vagrant Full Dev
+**Note:** These are manual instructions for Kerberizing Metron Storm 
topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor 
connections, or MAAS.
+
+1. Build full dev and ssh into the machine
+```
+cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+vagrant up
+vagrant ssh
+```
+
+2. Export env vars
+```
+# execute as root
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.3.1"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+```
+
+3. Stop all topologies - we will  restart them again once Kerberos has 
been enabled.
+```
+for topology in bro snort enrichment indexing; do storm kill $topology; 
done
+```
+
+4. Setup Kerberos
+```
+# Note: if you copy/paste this full set of commands, the kdb5_util command 
will not run as expected, so run the commands individually to ensure they all 
execute
+yum -y install krb5-server krb5-libs krb5-workstation
+sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+# This step takes a moment. It creates the kerberos database.
+kdb5_util create -s
+/etc/rc.d/init.d/krb5kdc start
+/etc/rc.d/init.d/kadmin start
+chkconfig krb5kdc on
+chkconfig kadmin on
+```
+
+5. Setup the admin and metron user principals. You'll kinit as the metron 
user when running topologies. Make sure to remember the passwords.
+```
+kadmin.local -q "addprinc admin/admin"
+kadmin.local -q "addprinc metron"
+```
+
+6. Create the metron user HDFS home directory
+```
+sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+```
+
+7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the 
submitting user. Add the following properties to custom storm-site. In the 
Storm config section in Ambari, choose “Add Property” under custom 
storm-site. In the dialog window, choose the “bulk property add mode” 
toggle button and add the below values.
+```

+topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']

+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+```
+
+![custom storm-site](readme-images/ambari-storm-site.png)
+
+![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
+
+8. Kerberize the cluster via Ambari. More detailed documentation can be 
found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+   1. For this exercise, choose existing MIT KDC (this is what we setup 
and installed in the previous steps.)
+  ![enable keberos](readme-images/enable-kerberos.png)
+  ![enable keberos get 
started](readme-images/enable-kerberos-started.png)
+   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin 
principal will end up as admin/ad...@example.com when testing the KDC. Use the 
password you entered during the step for adding the admin principal.
+  ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
+   3. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+  ![enable keberos 
configure](readme-images/custom-storm-site-final.png)
+
+9. Setup Metron keytab
+```
+kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
+cp metron.headless.keytab /etc/security/keytabs && \
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+```
+
+10. Kinit with the metron user
+```
+kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+```
+
+11. First create any additional Kafka topics you will need. We need to 
create the topics before adding the required ACLs. The current full dev 
installation will deploy bro, snort, enrichments, and indexing only. e.g.
+```
+${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOO

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/497#discussion_r108983684
  
--- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
@@ -0,0 +1,238 @@
+# Setting Up Kerberos in Vagrant Full Dev
+**Note:** These are manual instructions for Kerberizing Metron Storm 
topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor 
connections, or MAAS.
+
+1. Build full dev and ssh into the machine
+```
+cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+vagrant up
+vagrant ssh
+```
+
+2. Export env vars
+```
+# execute as root
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.3.1"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+```
+
+3. Stop all topologies - we will  restart them again once Kerberos has 
been enabled.
+```
+for topology in bro snort enrichment indexing; do storm kill $topology; 
done
+```
+
+4. Setup Kerberos
+```
+# Note: if you copy/paste this full set of commands, the kdb5_util command 
will not run as expected, so run the commands individually to ensure they all 
execute
+yum -y install krb5-server krb5-libs krb5-workstation
+sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+# This step takes a moment. It creates the kerberos database.
+kdb5_util create -s
+/etc/rc.d/init.d/krb5kdc start
+/etc/rc.d/init.d/kadmin start
+chkconfig krb5kdc on
+chkconfig kadmin on
+```
+
+5. Setup the admin and metron user principals. You'll kinit as the metron 
user when running topologies. Make sure to remember the passwords.
+```
+kadmin.local -q "addprinc admin/admin"
+kadmin.local -q "addprinc metron"
+```
+
+6. Create the metron user HDFS home directory
+```
+sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+```
+
+7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the 
submitting user. Add the following properties to custom storm-site. In the 
Storm config section in Ambari, choose “Add Property” under custom 
storm-site. In the dialog window, choose the “bulk property add mode” 
toggle button and add the below values.
+```

+topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']

+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+```
+
+![custom storm-site](readme-images/ambari-storm-site.png)
+
+![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
+
+8. Kerberize the cluster via Ambari. More detailed documentation can be 
found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+   1. For this exercise, choose existing MIT KDC (this is what we setup 
and installed in the previous steps.)
+  ![enable keberos](readme-images/enable-kerberos.png)
+  ![enable keberos get 
started](readme-images/enable-kerberos-started.png)
+   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin 
principal will end up as admin/ad...@example.com when testing the KDC. Use the 
password you entered during the step for adding the admin principal.
+  ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
+   3. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+  ![enable keberos 
configure](readme-images/custom-storm-site-final.png)
+
+9. Setup Metron keytab
+```
+kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
+cp metron.headless.keytab /etc/security/keytabs && \
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+```
+
+10. Kinit with the metron user
+```
+kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+```
+
+11. First create any additional Kafka topics you will need. We need to 
create the topics before adding the required ACLs. The current full dev 
installation will deploy bro, snort, enrichments, and indexing only. e.g.
+```
+${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOO

[GitHub] incubator-metron pull request #497: METRON-804: Create a document to describ...

[anandsubbu]

Github user anandsubbu commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/497#discussion_r108999637
  
--- Diff: metron-deployment/vagrant/KERBEROS_SETUP.md ---
@@ -0,0 +1,238 @@
+# Setting Up Kerberos in Vagrant Full Dev
+**Note:** These are manual instructions for Kerberizing Metron Storm 
topologies from Kafka to Kafka. This does not cover the Ambari MPack, sensor 
connections, or MAAS.
+
+1. Build full dev and ssh into the machine
+```
+cd incubator-metron/metron-deployment/vagrant/full-dev-platform
+vagrant up
+vagrant ssh
+```
+
+2. Export env vars
+```
+# execute as root
+sudo su -
+export ZOOKEEPER=node1
+export BROKERLIST=node1
+export HDP_HOME="/usr/hdp/current"
+export METRON_VERSION="0.3.1"
+export METRON_HOME="/usr/metron/${METRON_VERSION}"
+```
+
+3. Stop all topologies - we will  restart them again once Kerberos has 
been enabled.
+```
+for topology in bro snort enrichment indexing; do storm kill $topology; 
done
+```
+
+4. Setup Kerberos
+```
+# Note: if you copy/paste this full set of commands, the kdb5_util command 
will not run as expected, so run the commands individually to ensure they all 
execute
+yum -y install krb5-server krb5-libs krb5-workstation
+sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
+cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
+# This step takes a moment. It creates the kerberos database.
+kdb5_util create -s
+/etc/rc.d/init.d/krb5kdc start
+/etc/rc.d/init.d/kadmin start
+chkconfig krb5kdc on
+chkconfig kadmin on
+```
+
+5. Setup the admin and metron user principals. You'll kinit as the metron 
user when running topologies. Make sure to remember the passwords.
+```
+kadmin.local -q "addprinc admin/admin"
+kadmin.local -q "addprinc metron"
+```
+
+6. Create the metron user HDFS home directory
+```
+sudo -u hdfs hdfs dfs -mkdir /user/metron && \
+sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron && \
+sudo -u hdfs hdfs dfs -chmod 770 /user/metron
+```
+
+7. In Ambari, setup Storm to run with Kerberos and run worker jobs as the 
submitting user. Add the following properties to custom storm-site. In the 
Storm config section in Ambari, choose “Add Property” under custom 
storm-site. In the dialog window, choose the “bulk property add mode” 
toggle button and add the below values.
+```

+topology.auto-credentials=['org.apache.storm.security.auth.kerberos.AutoTGT']

+nimbus.credential.renewers.classes=['org.apache.storm.security.auth.kerberos.AutoTGT']
+supervisor.run.worker.as.user=true
+```
+
+![custom storm-site](readme-images/ambari-storm-site.png)
+
+![custom storm-site 
properties](readme-images/ambari-storm-site-properties.png)
+
+8. Kerberize the cluster via Ambari. More detailed documentation can be 
found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
+   1. For this exercise, choose existing MIT KDC (this is what we setup 
and installed in the previous steps.)
+  ![enable keberos](readme-images/enable-kerberos.png)
+  ![enable keberos get 
started](readme-images/enable-kerberos-started.png)
+   2. Setup Kerberos configuration. Realm is EXAMPLE.COM. The admin 
principal will end up as admin/ad...@example.com when testing the KDC. Use the 
password you entered during the step for adding the admin principal.
+  ![enable keberos 
configure](readme-images/enable-kerberos-configure-kerberos.png)
+   3. Click through to “Start and Test Services.” Let the cluster spin 
up, but don't worry about starting up Metron via Ambari - we're going to run 
the parsers manually against the rest of the Hadoop cluster Kerberized. The 
wizard will fail at starting Metron, but this is OK. Click “continue.” When 
you’re finished, the custom storm-site should look similar to the following:
+  ![enable keberos 
configure](readme-images/custom-storm-site-final.png)
+
+9. Setup Metron keytab
+```
+kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com" && \
+cp metron.headless.keytab /etc/security/keytabs && \
+chown metron:hadoop /etc/security/keytabs/metron.headless.keytab && \
+chmod 440 /etc/security/keytabs/metron.headless.keytab
+```
+
+10. Kinit with the metron user
+```
+kinit -kt /etc/security/keytabs/metron.headless.keytab met...@example.com
+```
+
+11. First create any additional Kafka topics you will need. We need to 
create the topics before adding the required ACLs. The current full dev 
installation will deploy bro, snort, enrichments, and indexing only. e.g.
+```
+${HDP_HOME}/kafka-broker/bin/kafka-topics.sh --zookeeper $ZOO

[GitHub] incubator-metron issue #497: METRON-804: Create a document to describe kerbe...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/497
  
@mattf-horton Thanks again for the patch! I made a couple more minor tweaks 
to get the images and indentation correct for the nested lists. I'm unable to 
get a nested list code block to format correctly, however. It's not bad, but 
it's just not quite right. If anyone has any suggestions, please chime in.

![image](https://cloud.githubusercontent.com/assets/658443/24519468/7f928ea6-1542-11e7-80c6-0070a1810f5e.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---