Actually, METRON-797 (https://github.com/apache/incubator-metron/pull/490) is inspired by that work, Dave. Specifically here are the differences:
- The mpack work is not included - It's based on 793, so it uses storm-kafka-client rather than storm-kafka - It adds a flag when starting the parsers to pass the security protocol and sets up the writers and the spout automatically - The core extension points (flux properties) are set up like in that PR, to make the follow-on work easier What's left is to pull the great work on the mpack out of there and get it working as a follow-on PR, which I created METRON-799 ( https://issues.apache.org/jira/browse/METRON-799) to accomplish. Casey On Sat, Mar 25, 2017 at 10:11 AM, David Lyle <dlyle65...@gmail.com> wrote: > Sounds good to me. A few of us did some initial exploration on that topic a > week or so back. The branch is here: > https://github.com/dlyle65535/incubator-metron/tree/kerb-testing?files=1 > > It contains a prototype quality version of everything you've identified > except METRON-793 and the probes. > > If it's a good starting place, it'd probably be short work to get it to a > PR-able form. > > Thoughts? > > -D... > > On Fri, Mar 24, 2017 at 23:03 Casey Stella <ceste...@gmail.com> wrote: > > > Hi All, > > > > I'd like to talk and start to formulate a plan around supporting running > > Metron on a kerberized cluster. This is a big bundle of work and seems > > dauntingly nebulous, so I wanted to have a chat and get a firm direction. > > When initially contemplating the issue, it's apparent that there are a > few > > snags that need to be thought through: > > > > - The kafka spout (storm-kafka) from apache does not support > interacting > > with kerberized kafka > > - We need to ensure the kafka writers are passing along the security > > protocol > > - We need to ensure that the credentials are auto-renewed > > > > Since I knew this was necessary, I wanted to get over this initial > barrier > > and submitted a couple of PRs associated with the following JIRAs that > are > > in review now: > > > > - METRON-793: Migrate to storm-kafka-client, a kafka spout which > > supports kerberized kafka > > - METRON-797: Pass along security protocols and set up autorenewal > > > > Between these, we get what I believe is the enrichment, profiler and > parser > > topologies and interactions with hbase and hdfs from them working along > > with the MR jobs. We still lack a few things: > > > > - A document describing the process of kerberizing a vagrant > environment > > to enable testing > > - Kerberos support for the librdkafka-based sensors (bro and fastcapa) > > - Kerberos support for the sensors that use the console-producer > (snort > > and yaf) > > - This should be as easy as ensuring to pass the right params to > the > > console producer, but still, we need to adjust the sensor stubs > > to make the > > right call. > > - Kerberos support for the mpack > > - Kerberos support for the REST API > > - Kerberos support for the pycapa > > > > I'm tracking these as JIRAs with the label of "kerberos" (see > > > > https://issues.apache.org/jira/browse/METRON-802?jql= > labels%20%3D%20kerberos%20AND%20project%20%3D%20Metron > > ) > > > > Please chime in if I missed something; I'll add it to the list. > > > > Best, > > > > Casey > > >
