Repository: metron-bro-plugin-kafka Updated Branches: refs/heads/master 4db999e82 -> b145bf01f
METRON-1329 Simplify metron-bro-plugin-kafka package loading (JonZeolla) closes apache/metron-bro-plugin-kafka#4 Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/b145bf01 Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/b145bf01 Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/b145bf01 Branch: refs/heads/master Commit: b145bf01f888591cbdf39ea663b34ae02254e0ea Parents: 4db999e Author: JonZeolla <zeo...@gmail.com> Authored: Thu Dec 7 15:10:41 2017 -0500 Committer: jonzeolla <jonzeo...@apache.org> Committed: Thu Dec 7 15:10:41 2017 -0500 ---------------------------------------------------------------------- README.md | 8 ++--- bro-pkg.meta | 2 +- scripts/Apache/Kafka/__load__.bro | 21 ++++++++++++++ scripts/Apache/Kafka/logs-to-kafka.bro | 36 +++++++++++++++++++++++ scripts/Bro/Kafka/__load__.bro | 21 -------------- scripts/Bro/Kafka/logs-to-kafka.bro | 45 ----------------------------- scripts/init.bro | 1 + 7 files changed, 63 insertions(+), 71 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index ea8d7da..e5d61e6 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,7 @@ The goal in this example is to send all HTTP and DNS records to a Kafka topic na * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. ``` -@load metron-bro-plugin-kafka/Bro/Kafka +@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::kafka_conf = table( @@ -60,7 +60,7 @@ It is also possible to send each log stream to a uniquely named topic. The goal * Each log writer accepts a separate configuration table. ``` -@load metron-bro-plugin-kafka/Bro/Kafka +@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::topic_name = ""; redef Kafka::tag_json = T; @@ -98,7 +98,7 @@ You may want to configure bro to filter log messages with certain characteristic * If the log message contains a 128 byte long source or destination IP address, the log is not sent to kafka. ``` -@load metron-bro-plugin-kafka/Bro/Kafka +@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; @@ -237,7 +237,7 @@ ${KAFKA_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.Si The following is how the `${BRO_HOME}/share/bro/site/local.bro` looks: ``` -@load metron-bro-plugin-kafka/Bro/Kafka +@load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/bro-pkg.meta ---------------------------------------------------------------------- diff --git a/bro-pkg.meta b/bro-pkg.meta index a31fdaf..fbed55e 100644 --- a/bro-pkg.meta +++ b/bro-pkg.meta @@ -1,7 +1,7 @@ [package] description = A Bro log writer plugin that sends logging output to Kafka. tags = log writer, bro plugin, kafka -script_dir = scripts +script_dir = build/scripts/Apache/Kafka build_command = ./configure --bro-dist=%(bro_dist)s --with-librdkafka=%(LIBRDKAFKA_ROOT)s && make test_command = ( cd tests && btest -d ) plugin_dir = build http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/scripts/Apache/Kafka/__load__.bro ---------------------------------------------------------------------- diff --git a/scripts/Apache/Kafka/__load__.bro b/scripts/Apache/Kafka/__load__.bro new file mode 100644 index 0000000..2a1efa8 --- /dev/null +++ b/scripts/Apache/Kafka/__load__.bro @@ -0,0 +1,21 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This is loaded when a user activates the plugin. Include scripts here that should be +# loaded automatically at that point. +# + +@load ./logs-to-kafka.bro http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/scripts/Apache/Kafka/logs-to-kafka.bro ---------------------------------------------------------------------- diff --git a/scripts/Apache/Kafka/logs-to-kafka.bro b/scripts/Apache/Kafka/logs-to-kafka.bro new file mode 100644 index 0000000..8f12932 --- /dev/null +++ b/scripts/Apache/Kafka/logs-to-kafka.bro @@ -0,0 +1,36 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +##! load this script to enable log output to kafka + +module Kafka; + +event bro_init() &priority=-5 +{ + for (stream_id in Log::active_streams) + { + if (stream_id in Kafka::logs_to_send) + { + local filter: Log::Filter = [ + $name = fmt("kafka-%s", stream_id), + $writer = Log::WRITER_KAFKAWRITER, + $config = table(["stream_id"] = fmt("%s", stream_id)) + ]; + + Log::add_filter(stream_id, filter); + } + } +} http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/scripts/Bro/Kafka/__load__.bro ---------------------------------------------------------------------- diff --git a/scripts/Bro/Kafka/__load__.bro b/scripts/Bro/Kafka/__load__.bro deleted file mode 100644 index 2a1efa8..0000000 --- a/scripts/Bro/Kafka/__load__.bro +++ /dev/null @@ -1,21 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -# This is loaded when a user activates the plugin. Include scripts here that should be -# loaded automatically at that point. -# - -@load ./logs-to-kafka.bro http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/scripts/Bro/Kafka/logs-to-kafka.bro ---------------------------------------------------------------------- diff --git a/scripts/Bro/Kafka/logs-to-kafka.bro b/scripts/Bro/Kafka/logs-to-kafka.bro deleted file mode 100644 index d62e03f..0000000 --- a/scripts/Bro/Kafka/logs-to-kafka.bro +++ /dev/null @@ -1,45 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -##! load this script to enable log output to kafka - -module Kafka; - -export { - ## - ## which log streams should be sent to kafka? - ## example: - ## redef Kafka::logs_to_send = set(Conn::Log, HTTP::LOG, DNS::LOG); - ## - const logs_to_send: set[Log::ID] &redef; -} - -event bro_init() &priority=-5 -{ - for (stream_id in Log::active_streams) - { - if (stream_id in Kafka::logs_to_send) - { - local filter: Log::Filter = [ - $name = fmt("kafka-%s", stream_id), - $writer = Log::WRITER_KAFKAWRITER, - $config = table(["stream_id"] = fmt("%s", stream_id)) - ]; - - Log::add_filter(stream_id, filter); - } - } -} http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/b145bf01/scripts/init.bro ---------------------------------------------------------------------- diff --git a/scripts/init.bro b/scripts/init.bro index 1f06c2d..65fb9e7 100644 --- a/scripts/init.bro +++ b/scripts/init.bro @@ -18,6 +18,7 @@ module Kafka; export { + const logs_to_send: set[Log::ID] &redef; const topic_name: string = "bro" &redef; const max_wait_on_shutdown: count = 3000 &redef; const tag_json: bool = F &redef;
