Hi All, I'd like to talk and start to formulate a plan around supporting running Metron on a kerberized cluster. This is a big bundle of work and seems dauntingly nebulous, so I wanted to have a chat and get a firm direction. When initially contemplating the issue, it's apparent that there are a few snags that need to be thought through:
- The kafka spout (storm-kafka) from apache does not support interacting with kerberized kafka - We need to ensure the kafka writers are passing along the security protocol - We need to ensure that the credentials are auto-renewed Since I knew this was necessary, I wanted to get over this initial barrier and submitted a couple of PRs associated with the following JIRAs that are in review now: - METRON-793: Migrate to storm-kafka-client, a kafka spout which supports kerberized kafka - METRON-797: Pass along security protocols and set up autorenewal Between these, we get what I believe is the enrichment, profiler and parser topologies and interactions with hbase and hdfs from them working along with the MR jobs. We still lack a few things: - A document describing the process of kerberizing a vagrant environment to enable testing - Kerberos support for the librdkafka-based sensors (bro and fastcapa) - Kerberos support for the sensors that use the console-producer (snort and yaf) - This should be as easy as ensuring to pass the right params to the console producer, but still, we need to adjust the sensor stubs to make the right call. - Kerberos support for the mpack - Kerberos support for the REST API - Kerberos support for the pycapa I'm tracking these as JIRAs with the label of "kerberos" (see https://issues.apache.org/jira/browse/METRON-802?jql=labels%20%3D%20kerberos%20AND%20project%20%3D%20Metron ) Please chime in if I missed something; I'll add it to the list. Best, Casey