[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka
[
https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Sommer updated BIT-1543:
--
Resolution: Merged (was: Fixed)
Status: Closed (was: Merge Request)
> Kafka Logger - Writes Bro Logs to Kafka
> ---
>
> Key: BIT-1543
> URL: https://bro-tracker.atlassian.net/browse/BIT-1543
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
>Reporter: Nick Allen
>Assignee: Robin Sommer
>
> As part of the Apache Metron project, we needed a way to send Bro logs to
> Kafka. From my research it seems like this is a common request. I'd rather
> give this code back to the Bro community than maintain it as part of Apache
> Metron.
> This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as
> simple as adding the following Bro script.
> {{
> @load Bro/Kafka/logs-to-kafka.bro
> redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> redef Kafka::topic_name = "bro";
> redef Kafka::kafka_conf = table(
> ["metadata.broker.list"] = "localhost:9092"
> );
> }}
> This plugin has the following features.
> * The user can specify a subset of all logs that should be sent to kafka. For
> example, to only send conn, http, and dns logs, specify the following.
> {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> }}
> * Full configurability of Kafka connectivity. Any configuration setting
> accepted by the librdkafka library can be passed to the plugin to tune how
> the logs are sent to Kafka.
> {{redef Kafka::kafka_conf = table(
>["metadata.broker.list"] = "localhost:9092",
>["client.id"] = "bro"
> );
> }}
> * The plugin will wait a configurable period of time (for example, 3 seconds)
> after shutdown to attempt to send any queued messages to Kafka.
> {{redef Kafka::max_wait_on_shutdown = 3000;
> }}
> * There are two message formats to choose from. By default, the standard Bro
> JSON format is used. There is an alternative 'tagged JSON' format that is
> provided by the plugin. Currently, all messages are sent to a single Bro
> topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log
> stream the message originated from. This format prepends the log stream
> identifier to the JSON message.
> {{{'conn': { ... }}
> {'http': { ... }}
> {'dns': { ...
> To enable this alternative format, simply specify the following.
> {{redef Kafka::tag_json = T;}}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-014#72000)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka
[
https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Sommer reassigned BIT-1543:
-
Assignee: Robin Sommer (was: Seth Hall)
> Kafka Logger - Writes Bro Logs to Kafka
> ---
>
> Key: BIT-1543
> URL: https://bro-tracker.atlassian.net/browse/BIT-1543
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
>Reporter: Nick Allen
>Assignee: Robin Sommer
>
> As part of the Apache Metron project, we needed a way to send Bro logs to
> Kafka. From my research it seems like this is a common request. I'd rather
> give this code back to the Bro community than maintain it as part of Apache
> Metron.
> This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as
> simple as adding the following Bro script.
> {{
> @load Bro/Kafka/logs-to-kafka.bro
> redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> redef Kafka::topic_name = "bro";
> redef Kafka::kafka_conf = table(
> ["metadata.broker.list"] = "localhost:9092"
> );
> }}
> This plugin has the following features.
> * The user can specify a subset of all logs that should be sent to kafka. For
> example, to only send conn, http, and dns logs, specify the following.
> {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> }}
> * Full configurability of Kafka connectivity. Any configuration setting
> accepted by the librdkafka library can be passed to the plugin to tune how
> the logs are sent to Kafka.
> {{redef Kafka::kafka_conf = table(
>["metadata.broker.list"] = "localhost:9092",
>["client.id"] = "bro"
> );
> }}
> * The plugin will wait a configurable period of time (for example, 3 seconds)
> after shutdown to attempt to send any queued messages to Kafka.
> {{redef Kafka::max_wait_on_shutdown = 3000;
> }}
> * There are two message formats to choose from. By default, the standard Bro
> JSON format is used. There is an alternative 'tagged JSON' format that is
> provided by the plugin. Currently, all messages are sent to a single Bro
> topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log
> stream the message originated from. This format prepends the log stream
> identifier to the JSON message.
> {{{'conn': { ... }}
> {'http': { ... }}
> {'dns': { ...
> To enable this alternative format, simply specify the following.
> {{redef Kafka::tag_json = T;}}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-012#72000)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka
[
https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robin Sommer reassigned BIT-1543:
-
Assignee: Seth Hall
> Kafka Logger - Writes Bro Logs to Kafka
> ---
>
> Key: BIT-1543
> URL: https://bro-tracker.atlassian.net/browse/BIT-1543
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
>Reporter: Nick Allen
>Assignee: Seth Hall
>
> As part of the Apache Metron project, we needed a way to send Bro logs to
> Kafka. From my research it seems like this is a common request. I'd rather
> give this code back to the Bro community than maintain it as part of Apache
> Metron.
> This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as
> simple as adding the following Bro script.
> {{
> @load Bro/Kafka/logs-to-kafka.bro
> redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> redef Kafka::topic_name = "bro";
> redef Kafka::kafka_conf = table(
> ["metadata.broker.list"] = "localhost:9092"
> );
> }}
> This plugin has the following features.
> * The user can specify a subset of all logs that should be sent to kafka. For
> example, to only send conn, http, and dns logs, specify the following.
> {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> }}
> * Full configurability of Kafka connectivity. Any configuration setting
> accepted by the librdkafka library can be passed to the plugin to tune how
> the logs are sent to Kafka.
> {{redef Kafka::kafka_conf = table(
>["metadata.broker.list"] = "localhost:9092",
>["client.id"] = "bro"
> );
> }}
> * The plugin will wait a configurable period of time (for example, 3 seconds)
> after shutdown to attempt to send any queued messages to Kafka.
> {{redef Kafka::max_wait_on_shutdown = 3000;
> }}
> * There are two message formats to choose from. By default, the standard Bro
> JSON format is used. There is an alternative 'tagged JSON' format that is
> provided by the plugin. Currently, all messages are sent to a single Bro
> topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log
> stream the message originated from. This format prepends the log stream
> identifier to the JSON message.
> {{{'conn': { ... }}
> {'http': { ... }}
> {'dns': { ...
> To enable this alternative format, simply specify the following.
> {{redef Kafka::tag_json = T;}}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-010#72000)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka
[
https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated BIT-1543:
Status: Merge Request (was: In Progress)
> Kafka Logger - Writes Bro Logs to Kafka
> ---
>
> Key: BIT-1543
> URL: https://bro-tracker.atlassian.net/browse/BIT-1543
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
>Reporter: Nick Allen
>
> As part of the Apache Metron project, we needed a way to send Bro logs to
> Kafka. From my research it seems like this is a common request. I'd rather
> give this code back to the Bro community than maintain it as part of Apache
> Metron.
> This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as
> simple as adding the following Bro script.
> {{
> @load Bro/Kafka/logs-to-kafka.bro
> redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> redef Kafka::topic_name = "bro";
> redef Kafka::kafka_conf = table(
> ["metadata.broker.list"] = "localhost:9092"
> );
> }}
> This plugin has the following features.
> * The user can specify a subset of all logs that should be sent to kafka. For
> example, to only send conn, http, and dns logs, specify the following.
> {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> }}
> * Full configurability of Kafka connectivity. Any configuration setting
> accepted by the librdkafka library can be passed to the plugin to tune how
> the logs are sent to Kafka.
> {{redef Kafka::kafka_conf = table(
>["metadata.broker.list"] = "localhost:9092",
>["client.id"] = "bro"
> );
> }}
> * The plugin will wait a configurable period of time (for example, 3 seconds)
> after shutdown to attempt to send any queued messages to Kafka.
> {{redef Kafka::max_wait_on_shutdown = 3000;
> }}
> * There are two message formats to choose from. By default, the standard Bro
> JSON format is used. There is an alternative 'tagged JSON' format that is
> provided by the plugin. Currently, all messages are sent to a single Bro
> topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log
> stream the message originated from. This format prepends the log stream
> identifier to the JSON message.
> {{{'conn': { ... }}
> {'http': { ... }}
> {'dns': { ...
> To enable this alternative format, simply specify the following.
> {{redef Kafka::tag_json = T;}}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-010#72000)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
[Bro-Dev] [JIRA] (BIT-1543) Kafka Logger - Writes Bro Logs to Kafka
[
https://bro-tracker.atlassian.net/browse/BIT-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen updated BIT-1543:
Status: In Progress (was: Open)
> Kafka Logger - Writes Bro Logs to Kafka
> ---
>
> Key: BIT-1543
> URL: https://bro-tracker.atlassian.net/browse/BIT-1543
> Project: Bro Issue Tracker
> Issue Type: Problem
> Components: Bro
>Reporter: Nick Allen
>
> As part of the Apache Metron project, we needed a way to send Bro logs to
> Kafka. From my research it seems like this is a common request. I'd rather
> give this code back to the Bro community than maintain it as part of Apache
> Metron.
> This Bro plugin logs all Bro output to Kafka. Configuring this plugin is as
> simple as adding the following Bro script.
> {{
> @load Bro/Kafka/logs-to-kafka.bro
> redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> redef Kafka::topic_name = "bro";
> redef Kafka::kafka_conf = table(
> ["metadata.broker.list"] = "localhost:9092"
> );
> }}
> This plugin has the following features.
> * The user can specify a subset of all logs that should be sent to kafka. For
> example, to only send conn, http, and dns logs, specify the following.
> {{redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
> }}
> * Full configurability of Kafka connectivity. Any configuration setting
> accepted by the librdkafka library can be passed to the plugin to tune how
> the logs are sent to Kafka.
> {{redef Kafka::kafka_conf = table(
>["metadata.broker.list"] = "localhost:9092",
>["client.id"] = "bro"
> );
> }}
> * The plugin will wait a configurable period of time (for example, 3 seconds)
> after shutdown to attempt to send any queued messages to Kafka.
> {{redef Kafka::max_wait_on_shutdown = 3000;
> }}
> * There are two message formats to choose from. By default, the standard Bro
> JSON format is used. There is an alternative 'tagged JSON' format that is
> provided by the plugin. Currently, all messages are sent to a single Bro
> topic. This 'tagged JSON' format helps a Kafka consumer distinguish which log
> stream the message originated from. This format prepends the log stream
> identifier to the JSON message.
> {{{'conn': { ... }}
> {'http': { ... }}
> {'dns': { ...
> To enable this alternative format, simply specify the following.
> {{redef Kafka::tag_json = T;}}
--
This message was sent by Atlassian JIRA
(v7.2.0-OD-03-010#72000)
___
bro-dev mailing list
bro-dev@bro.org
http://mailman.icsi.berkeley.edu/mailman/listinfo/bro-dev
