Nick Allen created BIT-1543:
-------------------------------
Summary: Kafka Logger - Writes Bro Logs to Kafka
Key: BIT-1543
URL: https://bro-tracker.atlassian.net/browse/BIT-1543
Project: Bro Issue Tracker
Issue Type: Problem
Components: Bro
Reporter: Nick Allen
As part of the Apache Metron project, we needed a way to send Bro logs to
Kafka. From my research it seems like this is a common request. I'd rather give
this code back to the Bro community than maintain it as part of Apache Metron.
This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as
simple as adding the following Bro script.
{{
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "localhost:9092"
);
}}
This plugin has the following features.
* The user can specify a subset of all logs that should be sent to kafka. For
example, to only send conn, http, and dns logs, specify the following.
{{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
}}
* Full configurability of Kafka connectivity. Any configuration setting
accepted by the librdkafka library can be passed to the plugin to tune how the
logs are sent to Kafka.
{{redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "localhost:9092",
["client.id"] = "bro"
);
}}
* The plugin will wait a configurable period of time (for example, 3 seconds)
after shutdown to attempt to send any queued messages to Kafka.
{{redef Kafka::max_wait_on_shutdown = 3000;
}}
* There are two message formats to choose from. By default, the standard Bro
JSON format is used. There is an alternative 'tagged JSON' format that is
provided by the plugin. Currently, all messages are sent to a single Bro topic.
This 'tagged JSON' format helps a Kafka consumer distinguish which log stream
the message originated from. This format prepends the log stream identifier to
the JSON message.
{{{'conn': { ... }}
{'http': { ... }}
{'dns': { ... }}}}
To enable this alternative format, simply specify the following.
{{redef Kafka::tag_json = T;}}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-010#72000)
_______________________________________________
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
