[ https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nick Allen updated BIT-1543: ---------------------------- Status: In Progress (was: Open) > Kafka Logger - Writes Bro Logs to Kafka > --------------------------------------- > > Key: BIT-1543 > URL: https://bro-tracker.atlassian.net/browse/BIT-1543 > Project: Bro Issue Tracker > Issue Type: Problem > Components: Bro > Reporter: Nick Allen > > As part of the Apache Metron project, we needed a way to send Bro logs to > Kafka. From my research it seems like this is a common request. I'd rather > give this code back to the Bro community than maintain it as part of Apache > Metron. > This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as > simple as adding the following Bro script. > {{ > @load Bro/Kafka/logs-to-kafka.bro > redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > redef Kafka::topic_name = "bro"; > redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092" > ); > }} > This plugin has the following features. > * The user can specify a subset of all logs that should be sent to kafka. For > example, to only send conn, http, and dns logs, specify the following. > {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG); > }} > * Full configurability of Kafka connectivity. Any configuration setting > accepted by the librdkafka library can be passed to the plugin to tune how > the logs are sent to Kafka. > {{redef Kafka::kafka_conf = table( > ["metadata.broker.list"] = "localhost:9092", > ["client.id"] = "bro" > ); > }} > * The plugin will wait a configurable period of time (for example, 3 seconds) > after shutdown to attempt to send any queued messages to Kafka. > {{redef Kafka::max_wait_on_shutdown = 3000; > }} > * There are two message formats to choose from. By default, the standard Bro > JSON format is used. There is an alternative 'tagged JSON' format that is > provided by the plugin. Currently, all messages are sent to a single Bro > topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log > stream the message originated from. This format prepends the log stream > identifier to the JSON message. > {{{'conn': { ... }} > {'http': { ... }} > {'dns': { ... }}}} > To enable this alternative format, simply specify the following. > {{redef Kafka::tag_json = T;}} -- This message was sent by Atlassian JIRA (v7.2.0-OD-03-010#72000) _______________________________________________ bro-dev mailing list bro-dev@bro.org http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev