Hi All,
I'd like to talk and start to formulate a plan around supporting running
Metron on a kerberized cluster. This is a big bundle of work and seems
dauntingly nebulous, so I wanted to have a chat and get a firm direction.
When initially contemplating the issue, it's apparent that there are a few
snags that need to be thought through:
- The kafka spout (storm-kafka) from apache does not support interacting
with kerberized kafka
- We need to ensure the kafka writers are passing along the security
protocol
- We need to ensure that the credentials are auto-renewed
Since I knew this was necessary, I wanted to get over this initial barrier
and submitted a couple of PRs associated with the following JIRAs that are
in review now:
- METRON-793: Migrate to storm-kafka-client, a kafka spout which
supports kerberized kafka
- METRON-797: Pass along security protocols and set up autorenewal
Between these, we get what I believe is the enrichment, profiler and parser
topologies and interactions with hbase and hdfs from them working along
with the MR jobs. We still lack a few things:
- A document describing the process of kerberizing a vagrant environment
to enable testing
- Kerberos support for the librdkafka-based sensors (bro and fastcapa)
- Kerberos support for the sensors that use the console-producer (snort
and yaf)
- This should be as easy as ensuring to pass the right params to the
console producer, but still, we need to adjust the sensor stubs
to make the
right call.
- Kerberos support for the mpack
- Kerberos support for the REST API
- Kerberos support for the pycapa
I'm tracking these as JIRAs with the label of "kerberos" (see
https://issues.apache.org/jira/browse/METRON-802?jql=labels%20%3D%20kerberos%20AND%20project%20%3D%20Metron
)
Please chime in if I missed something; I'll add it to the list.
Best,
Casey