[
https://issues.apache.org/jira/browse/METRON-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133056#comment-15133056
]
ASF GitHub Bot commented on METRON-25:
--------------------------------------
Github user nickwallen commented on the pull request:
https://github.com/apache/incubator-metron/pull/17#issuecomment-180056871
The values in `bro-plugin-kafka/scripts/init.bro` are merely defaults.
They could even be completely removed from there. I just find them useful so
that a user doesn't have to define all of the configuration values all of the
time. For example a user will rarely want to change `max_wait_on_delivery`.
The way to configure the kafka broker and topic name as a user of this
plugin is described in the README. You define these in your
`.../site/local.bro` script so that it looks something like the following:
```
@load Metron/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(Conn::LOG, HTTP::LOG, DNS::LOG);
redef Kafka::kafka_broker_list = "localhost:9092";
redef Kafka::topic_name = "bro";
```
As an example, you can see how the Ansible scripts configure these values
in `deployment/roles/bro/tasks/bro-plugin-kafka.yml`.
```
- name: Configure bro plugin
lineinfile:
dest: /usr/local/bro/share/bro/site/local.bro
line: "{{ item }}"
with_items:
- "@load Metron/Kafka/logs-to-kafka.bro"
- "redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);"
- "redef Kafka::kafka_broker_list = \"{{ kafka_broker_url }}\";"
- "redef Kafka::topic_name = \"{{ bro_topic }}\";"
```
Good eye. Does that make sense?
> Create Bro Plugin to Send Logs Directly to Kafka
> ------------------------------------------------
>
> Key: METRON-25
> URL: https://issues.apache.org/jira/browse/METRON-25
> Project: Metron
> Issue Type: Improvement
> Reporter: Nick Allen
> Priority: Critical
> Original Estimate: 48h
> Remaining Estimate: 48h
>
> Create a Bro plugin that will consume the logs produced by Bro and send them
> directly to a Kafka topic. The types of logs to send should be configurable,
> so that only a subset of them are published to Kafka. For example, I may
> only want DNS::LOG and HTTP::LOG sent to Kafka. This should not interfere
> with the existing file based logging which is useful for diagnostics and
> troubleshooting.
> The alternative solution to creating this Bro plugin is to use some means of
> tailing the log files that are generated by Bro. Each stream in Bro is logged
> to a separate file, so you'd have to tail each of these files independently.
> Tailing log files like this is problematic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
