Okay, I figured it out. There was a mismatch in my install bro (yum installed),
the source (git cloned) and the plugin version. I removed everything and them
compiled both zeek and the plugin from source and the issue seems to have gone.
I can run the test command I get the following output.
# zeek -N Apache::Kafka
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
However, now I can't seem to get alerts/logs to Kafka. Here's the config I'm
using in /usr/local/zeek/share/zeek/site/local.zeek
#This doesn't work in the new version anymore.
#@load packages/metron-bro-plugin-kafka/Apache/Kafka
#Tried added this line to ensure all packages are automatically loaded.
#@load packages
#Then tried loading the specific module
#@load metron-bro-plugin-kafka
#And then I eventually removed the three previous load lines
redef Kafka::send_all_active_logs = T;
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table(
["metadata.broker.list"] = "mysecrethost:6667",
["client.id"] = "bro"
);
Even when I have the `@loads` disabled, I still see the script being loaded
(see logs below).
To start, I did the following:
zeekctl> deploy
zeekctl> restart --clean
zeekctl> start
I can see the following in startup logs:
starting ...
starting zeek ...
[ZeekControl] > diag
[zeek]
No core file found.
Zeek 2.6-558
Linux 3.10.0-957.21.3.el7.x86_64
Zeek plugins:
Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0)
==== No reporter.log
==== stderr.log
listening on em1
==== stdout.log
max memory size (kbytes, -m) unlimited
data seg size (kbytes, -d) unlimited
virtual memory (kbytes, -v) unlimited
core file size (blocks, -c) unlimited
==== .cmdline
-i em1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek
local.zeek zeekctl zeekctl/standalone zeekctl/auto
==== .env_vars
PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/openssl/bin:/opt/apache-maven-3.3.9/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/zeek/bin
ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site
CLUSTER_NODE=
==== .status
RUNNING [net_run]
==== No prof.log
==== packet_filter.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path packet_filter
#open 2019-07-03-19-36-56
#fields ts node filter init success
#types time string string bool bool
1562175416.590048 zeek ip or not ip T T
==== loaded_scripts.log
#separator \x09
#set_separator ,
#empty_field (empty)
#unset_field -
#path loaded_scripts
#open 2019-07-03-19-36-56
#fields name
#types string
/usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/__load__.zeek
/usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/kafka.bif.zeek
/usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/__load__.bro
/usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/init.bro
it starts up fine, no error messages. running "diag" in zeekctl just gives a
long list of plugins that were loaded.
If I tail logs in I see new connection logs being added. However, I dont see
any messages in Kafka console consumer. What am I missing? How do I go about
debugging this?
Thank you for your help and assistance.
Best regards,
Sanket
________________________________
From: zeo...@gmail.com <zeo...@gmail.com>
Sent: Tuesday, July 2, 2019 11:46 AM
To: user@metron.apache.org
Subject: Re: metron-bro-plugin-kafka error
Did you install it manual or with bro-pkg/zkg? I believe bro-pkg was renamed
to zkg as of their 2.0 release but I haven't used it in a little while. Any
more details regarding the installation process, or versions of software in use
may be helpful
Jon Zeolla
On Tue, Jul 2, 2019, 12:26 AM Sanket Sharma
<sanket.sha...@dukstra.com<mailto:sanket.sha...@dukstra.com>> wrote:
Hi,
I’m trying to configure Metron bro plugin by following instructions here:
https://github.com/apache/metron-bro-plugin-kafka
I’m able to build/install the plugin successfully but when I test it using the
command:
$ bro -N Apache::Kafka
I get the following error:
fatal error in /opt/bro/share/bro/base/init-bare.bro, line 1: cannot load
plugin library
/opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/APACHE-KAFKA.linux-x86_64.so<http://APACHE-KAFKA.linux-x86_64.so>:
/opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/APACHE-KAFKA.linux-x86_64.so<http://APACHE-KAFKA.linux-x86_64.so>:
undefined symbol: bro_version_2_6_558_plugin_7
Not sure what am I missing? Any help would be greatly appreciated.
Best regards,
Sanket
