Github user cestella commented on a diff in the pull request:

    https://github.com/apache/incubator-metron/pull/501#discussion_r109178739
  
    --- Diff: metron-sensors/bro-plugin-kafka/README.md ---
    @@ -0,0 +1,160 @@
    +Bro Logging Output to Kafka
    +===========================
    +
    +A Bro log writer that sends logging output to Kafka.  This provides a 
convenient
    +means for tools in the Hadoop ecosystem, such as Storm, Spark, and others, 
to
    +process the data generated by Bro.
    +
    +Installation
    +------------
    +
    +Install librdkafka (https://github.com/edenhill/librdkafka), a native 
client
    +library for Kafka.  This plugin has been tested against the latest release 
of
    +librdkafka, which at the time of this writing is v0.9.4.  In order to 
support interacting
    +with a kerberized kafka, you will need libsasl2 installed
    +
    +    # curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz 
| tar xvz
    +    # cd librdkafka-0.9.4/
    +    # ./configure --enable-sasl=true
    +    # make
    +    # sudo make install
    +
    +Then compile this Bro plugin using the following commands.
    +
    +    # ./configure --bro-dist=$BRO_SRC
    +    # make
    +    # sudo make install
    +
    +Run the following command to ensure that the plugin was installed 
successfully.
    +
    +    # bro -N Bro::Kafka
    +    Bro::Kafka - Writes logs to Kafka (dynamic, version 0.1)
    +
    +Activation
    +----------
    +
    +The easiest way to enable Kafka output is to load the plugin's
    +``logs-to-kafka.bro`` script.  If you are using BroControl, the following 
lines
    +added to local.bro will activate it.
    +
    +```
    +@load Bro/Kafka/logs-to-kafka.bro
    +redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
    +redef Kafka::topic_name = "bro";
    +redef Kafka::kafka_conf = table(
    +    ["metadata.broker.list"] = "localhost:9092"
    +);
    +```
    +
    +This example will send all HTTP, DNS, and Conn logs to a Kafka broker 
running on
    +the localhost to a topic called ``bro``. Any configuration value accepted 
by
    +librdkafka can be added to the ``kafka_conf`` configuration table.
    +
    +Settings
    +--------
    +
    +### ``kafka_conf``
    +
    +The global configuration settings for Kafka.  These values are passed 
through
    +directly to librdkafka.  Any valid librdkafka settings can be defined in 
this
    +table.  The full set of valid librdkafka settings are available
    
+[here](https://github.com/edenhill/librdkafka/blob/v0.9.4/CONFIGURATION.md).
    +
    +```
    +redef Kafka::kafka_conf = table(
    +    ["metadata.broker.list"] = "localhost:9092",
    +    ["client.id"] = "bro"
    +);
    +```
    +
    +### ``topic_name``
    +
    +The name of the topic in Kafka where all Bro logs will be sent to.
    +
    +```
    +redef Kafka::topic_name = "bro";
    +```
    +
    +### ``max_wait_on_shutdown``
    +
    +The maximum number of milliseconds that the plugin will wait for any 
backlog of
    +queued messages to be sent to Kafka before forced shutdown.
    +
    +```
    +redef Kafka::max_wait_on_shutdown = 3000;
    +```
    +
    +### ``tag_json``
    +
    +If true, a log stream identifier is appended to each JSON-formatted 
message. For
    +example, a Conn::LOG message will look like ``{ 'conn' : { ... }}``.
    +
    +```
    +redef Kafka::tag_json = T;
    +```
    +
    +### ``debug``
    +
    +A comma separated list of debug contexts in librdkafka which you want to
    +enable.  The available contexts are:
    +* generic
    +* broker
    +* topic
    +* metadata
    +* queue
    +* msg
    +* protocol
    +* cgrp
    +* security
    +* fetch
    +* feature
    +* all  
    +
    +Kerberos
    +--------
    +
    +This plugin supports producing messages from a kerberized kafka.  There
    +are a couple of prerequisites and a couple of settings to set.  
    +
    +### SASL
    +If you are using SASL as a security protocol for kafka, then you must have
    +libsasl or libsasl2 installed.  You can tell if sasl is enabled by
    +running the following from the directory in which you have build
    +librdkafka:
    +```
    +examples/rdkafka_example -X builtin.features
    +builtin.features = gzip,snappy,ssl,sasl,regex
    +```
    +
    +### Producer Config
    +
    +As stated above, you can configure the producer kafka configs in
    +`${BRO_HOME}/share/bro/site/local.bro`.  There are a few configs
    +necessary to set, which are described
    
+[here](https://github.com/edenhill/librdkafka/wiki/Using-SASL-with-librdkafka).
    +For an environment where the following is true:
    +* The broker is `node1:6667`
    +* This kafka is using `SASL_PLAINTEXT` as the security protocol
    +* The keytab used is the `metron` keytab
    +* The service principal for `metron` is `[email protected]`
    +
    +The kafka topic `bro` has been given permission for the `metron` user to
    +write:
    +```
    +# login using the metron user 
    +kinit -kt /etc/security/keytabs/metron.headless.keytab [email protected]
    +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=node1:2181 --add --allow-principal User:metron --topic bro
    --- End diff --
    
    I was using the same terminology in the instructions for kerberization, 
which isn't what we want here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---

Reply via email to