Sounds good to me. A few of us did some initial exploration on that topic a week or so back. The branch is here: https://github.com/dlyle65535/incubator-metron/tree/kerb-testing?files=1
It contains a prototype quality version of everything you've identified except METRON-793 and the probes. If it's a good starting place, it'd probably be short work to get it to a PR-able form. Thoughts? -D... On Fri, Mar 24, 2017 at 23:03 Casey Stella <[email protected]> wrote: > Hi All, > > I'd like to talk and start to formulate a plan around supporting running > Metron on a kerberized cluster. This is a big bundle of work and seems > dauntingly nebulous, so I wanted to have a chat and get a firm direction. > When initially contemplating the issue, it's apparent that there are a few > snags that need to be thought through: > > - The kafka spout (storm-kafka) from apache does not support interacting > with kerberized kafka > - We need to ensure the kafka writers are passing along the security > protocol > - We need to ensure that the credentials are auto-renewed > > Since I knew this was necessary, I wanted to get over this initial barrier > and submitted a couple of PRs associated with the following JIRAs that are > in review now: > > - METRON-793: Migrate to storm-kafka-client, a kafka spout which > supports kerberized kafka > - METRON-797: Pass along security protocols and set up autorenewal > > Between these, we get what I believe is the enrichment, profiler and parser > topologies and interactions with hbase and hdfs from them working along > with the MR jobs. We still lack a few things: > > - A document describing the process of kerberizing a vagrant environment > to enable testing > - Kerberos support for the librdkafka-based sensors (bro and fastcapa) > - Kerberos support for the sensors that use the console-producer (snort > and yaf) > - This should be as easy as ensuring to pass the right params to the > console producer, but still, we need to adjust the sensor stubs > to make the > right call. > - Kerberos support for the mpack > - Kerberos support for the REST API > - Kerberos support for the pycapa > > I'm tracking these as JIRAs with the label of "kerberos" (see > > https://issues.apache.org/jira/browse/METRON-802?jql=labels%20%3D%20kerberos%20AND%20project%20%3D%20Metron > ) > > Please chime in if I missed something; I'll add it to the list. > > Best, > > Casey >
