Re: metron-bro-plugin-kafka error
Please start a new thread On December 5, 2019 at 02:07:53, Farrukh Naveed Anjum ( anjum.farr...@gmail.com) wrote: I am not receiving data from Bro to Kafka # @load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(SSH::LOG, RDP::LOG, KRB::LOG, SSL::LOG, DHCP::LOG, Cluster::LOG, Syslog::LOG, SNMP::LOG, Reporter::LOG, DNP3::LOG, RADIUS::LOG, Tunnel::LOG, Conn::LOG, HTTP::LOG, DNS::LOG, Software::LOG, Intel::LOG, Notice::LOG, Signatures::LOG); redef Kafka::send_all_active_logs = T; redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:6667", ["client.id"] = "bro" ); Commented out line as per your recommendation. Still not getting any data in Kafka Topic Any suggestions ? On Thu, Jul 4, 2019 at 5:08 PM zeo...@gmail.com wrote: > If you had the all active logs set to true it should send everything. > What is the latest commit of the version of plugin are you running? I see > it's 0.3 but since that hasn't been "released" (tagged) I'm assuming you > are installing from master? > > Jon Zeolla > > On Wed, Jul 3, 2019, 5:57 PM Sanket Sharma > wrote: > >> Seems like all I had to do was to specify the exact logs that I wanted to >> export. All working now. >> >> >> >> Thanks for the help @Jon Zeolla >> >> >> >> >> >> Best regards, >> >> Sanket >> >> >> >> >> >> *From:* Sanket Sharma >> *Reply-To:* "user@metron.apache.org" >> *Date:* Wednesday, 03 July 2019 at 19:47 >> *To:* "user@metron.apache.org" >> *Subject:* Re: metron-bro-plugin-kafka error >> >> >> >> Okay, I figured it out. There was a mismatch in my install bro (yum >> installed), the source (git cloned) and the plugin version. I removed >> everything and them compiled both zeek and the plugin from source and the >> issue seems to have gone. I can run the test command I get the following >> output. >> >> >> >> # zeek -N Apache::Kafka >> >> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) >> >> >> >> However, now I can't seem to get alerts/logs to Kafka. Here's the config >> I'm using in /usr/local/zeek/share/zeek/site/local.zeek >> >> >> >> #This doesn't work in the new version anymore. >> >> #@load packages/metron-bro-plugin-kafka/Apache/Kafka >> >> >> >> #Tried added this line to ensure all packages are automatically loaded. >> >> #@load packages >> >> >> >> #Then tried loading the specific module >> >> #@load metron-bro-plugin-kafka >> >> #And then I eventually removed the three previous load lines >> >> >> >> redef Kafka::send_all_active_logs = T; >> >> redef Kafka::tag_json = T; >> >> redef Kafka::kafka_conf = table( >> >> ["metadata.broker.list"] = "mysecrethost:6667", >> >> ["client.id"] = "bro" >> >> ); >> >> >> >> Even when I have the `@loads` disabled, I still see the script being >> loaded (see logs below). >> >> >> >> To start, I did the following: >> >> >> >> zeekctl> deploy >> >> zeekctl> restart --clean >> >> zeekctl> start >> >> >> >> I can see the following in startup logs: >> >> >> >> starting ... >> >> starting zeek ... >> >> [ZeekControl] > diag >> >> [zeek] >> >> >> >> No core file found. >> >> >> >> Zeek 2.6-558 >> >> Linux 3.10.0-957.21.3.el7.x86_64 >> >> >> >> Zeek plugins: >> >> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) >> >> >> >> No reporter.log >> >> >> >> stderr.log >> >> listening on em1 >> >> >> >> >> >> stdout.log >> >> max memory size (kbytes, -m) unlimited >> >> data seg size (kbytes, -d) unlimited >> >> virtual memory (kbytes, -v) unlimited >> >> core file size (blocks, -c) unlimited >> >> >> >> .cmdline >> >> -i em1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p >> zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto >> >> >> >> .env_vars >> >> >> PATH=/usr/lo
Re: metron-bro-plugin-kafka error
I am not receiving data from Bro to Kafka # @load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(SSH::LOG, RDP::LOG, KRB::LOG, SSL::LOG, DHCP::LOG, Cluster::LOG, Syslog::LOG, SNMP::LOG, Reporter::LOG, DNP3::LOG, RADIUS::LOG, Tunnel::LOG, Conn::LOG, HTTP::LOG, DNS::LOG, Software::LOG, Intel::LOG, Notice::LOG, Signatures::LOG); redef Kafka::send_all_active_logs = T; redef Kafka::topic_name = "bro"; redef Kafka::tag_json = T; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "localhost:6667", ["client.id"] = "bro" ); Commented out line as per your recommendation. Still not getting any data in Kafka Topic Any suggestions ? On Thu, Jul 4, 2019 at 5:08 PM zeo...@gmail.com wrote: > If you had the all active logs set to true it should send everything. > What is the latest commit of the version of plugin are you running? I see > it's 0.3 but since that hasn't been "released" (tagged) I'm assuming you > are installing from master? > > Jon Zeolla > > On Wed, Jul 3, 2019, 5:57 PM Sanket Sharma > wrote: > >> Seems like all I had to do was to specify the exact logs that I wanted to >> export. All working now. >> >> >> >> Thanks for the help @Jon Zeolla >> >> >> >> >> >> Best regards, >> >> Sanket >> >> >> >> >> >> *From: *Sanket Sharma >> *Reply-To: *"user@metron.apache.org" >> *Date: *Wednesday, 03 July 2019 at 19:47 >> *To: *"user@metron.apache.org" >> *Subject: *Re: metron-bro-plugin-kafka error >> >> >> >> Okay, I figured it out. There was a mismatch in my install bro (yum >> installed), the source (git cloned) and the plugin version. I removed >> everything and them compiled both zeek and the plugin from source and the >> issue seems to have gone. I can run the test command I get the following >> output. >> >> >> >> # zeek -N Apache::Kafka >> >> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) >> >> >> >> However, now I can't seem to get alerts/logs to Kafka. Here's the config >> I'm using in /usr/local/zeek/share/zeek/site/local.zeek >> >> >> >> #This doesn't work in the new version anymore. >> >> #@load packages/metron-bro-plugin-kafka/Apache/Kafka >> >> >> >> #Tried added this line to ensure all packages are automatically loaded. >> >> #@load packages >> >> >> >> #Then tried loading the specific module >> >> #@load metron-bro-plugin-kafka >> >> #And then I eventually removed the three previous load lines >> >> >> >> redef Kafka::send_all_active_logs = T; >> >> redef Kafka::tag_json = T; >> >> redef Kafka::kafka_conf = table( >> >> ["metadata.broker.list"] = "mysecrethost:6667", >> >> ["client.id"] = "bro" >> >> ); >> >> >> >> Even when I have the `@loads` disabled, I still see the script being >> loaded (see logs below). >> >> >> >> To start, I did the following: >> >> >> >> zeekctl> deploy >> >> zeekctl> restart --clean >> >> zeekctl> start >> >> >> >> I can see the following in startup logs: >> >> >> >> starting ... >> >> starting zeek ... >> >> [ZeekControl] > diag >> >> [zeek] >> >> >> >> No core file found. >> >> >> >> Zeek 2.6-558 >> >> Linux 3.10.0-957.21.3.el7.x86_64 >> >> >> >> Zeek plugins: >> >> Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) >> >> >> >> No reporter.log >> >> >> >> stderr.log >> >> listening on em1 >> >> >> >> >> >> stdout.log >> >> max memory size (kbytes, -m) unlimited >> >> data seg size (kbytes, -d) unlimited >> >> virtual memory (kbytes, -v) unlimited >> >> core file size (blocks, -c) unlimited >> >> >> >> .cmdline >> >> -i em1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p >> zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto >> >> >> >> .env_vars >> >> >> PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/openssl/bin:/opt/apache-maven-3.3.9/bin:/sbin:/bin:/usr/sbi
Re: metron-bro-plugin-kafka error
If you had the all active logs set to true it should send everything. What is the latest commit of the version of plugin are you running? I see it's 0.3 but since that hasn't been "released" (tagged) I'm assuming you are installing from master? Jon Zeolla On Wed, Jul 3, 2019, 5:57 PM Sanket Sharma wrote: > Seems like all I had to do was to specify the exact logs that I wanted to > export. All working now. > > > > Thanks for the help @Jon Zeolla > > > > > > Best regards, > > Sanket > > > > > > *From: *Sanket Sharma > *Reply-To: *"user@metron.apache.org" > *Date: *Wednesday, 03 July 2019 at 19:47 > *To: *"user@metron.apache.org" > *Subject: *Re: metron-bro-plugin-kafka error > > > > Okay, I figured it out. There was a mismatch in my install bro (yum > installed), the source (git cloned) and the plugin version. I removed > everything and them compiled both zeek and the plugin from source and the > issue seems to have gone. I can run the test command I get the following > output. > > > > # zeek -N Apache::Kafka > > Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) > > > > However, now I can't seem to get alerts/logs to Kafka. Here's the config > I'm using in /usr/local/zeek/share/zeek/site/local.zeek > > > > #This doesn't work in the new version anymore. > > #@load packages/metron-bro-plugin-kafka/Apache/Kafka > > > > #Tried added this line to ensure all packages are automatically loaded. > > #@load packages > > > > #Then tried loading the specific module > > #@load metron-bro-plugin-kafka > > #And then I eventually removed the three previous load lines > > > > redef Kafka::send_all_active_logs = T; > > redef Kafka::tag_json = T; > > redef Kafka::kafka_conf = table( > > ["metadata.broker.list"] = "mysecrethost:6667", > > ["client.id"] = "bro" > > ); > > > > Even when I have the `@loads` disabled, I still see the script being > loaded (see logs below). > > > > To start, I did the following: > > > > zeekctl> deploy > > zeekctl> restart --clean > > zeekctl> start > > > > I can see the following in startup logs: > > > > starting ... > > starting zeek ... > > [ZeekControl] > diag > > [zeek] > > > > No core file found. > > > > Zeek 2.6-558 > > Linux 3.10.0-957.21.3.el7.x86_64 > > > > Zeek plugins: > > Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) > > > > No reporter.log > > > > stderr.log > > listening on em1 > > > > > > stdout.log > > max memory size (kbytes, -m) unlimited > > data seg size (kbytes, -d) unlimited > > virtual memory (kbytes, -v) unlimited > > core file size (blocks, -c) unlimited > > > > .cmdline > > -i em1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p > zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto > > > > .env_vars > > > PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/openssl/bin:/opt/apache-maven-3.3.9/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/zeek/bin > > > ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site > > CLUSTER_NODE= > > > > .status > > RUNNING [net_run] > > > > No prof.log > > > > packet_filter.log > > #separator \x09 > > #set_separator , > > #empty_field(empty) > > #unset_field- > > #path packet_filter > > #open 2019-07-03-19-36-56 > > #fields ts nodefilter initsuccess > > #types timestring string boolbool > > 1562175416.590048 zeekip or not ipT T > > > > loaded_scripts.log > > #separator \x09 > > #set_separator , > > #empty_field(empty) > > #unset_field- > > #path loaded_scripts > > #open 2019-07-03-19-36-56 > > #fields name > > #types string > > /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/__load__.zeek > > /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/kafka.bif.zeek > > /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/__load__.bro > > /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/init.bro > > > > > > it starts up fine, no error messages. running "diag" in
Re: metron-bro-plugin-kafka error
Okay, I figured it out. There was a mismatch in my install bro (yum installed), the source (git cloned) and the plugin version. I removed everything and them compiled both zeek and the plugin from source and the issue seems to have gone. I can run the test command I get the following output. # zeek -N Apache::Kafka Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) However, now I can't seem to get alerts/logs to Kafka. Here's the config I'm using in /usr/local/zeek/share/zeek/site/local.zeek #This doesn't work in the new version anymore. #@load packages/metron-bro-plugin-kafka/Apache/Kafka #Tried added this line to ensure all packages are automatically loaded. #@load packages #Then tried loading the specific module #@load metron-bro-plugin-kafka #And then I eventually removed the three previous load lines redef Kafka::send_all_active_logs = T; redef Kafka::tag_json = T; redef Kafka::kafka_conf = table( ["metadata.broker.list"] = "mysecrethost:6667", ["client.id"] = "bro" ); Even when I have the `@loads` disabled, I still see the script being loaded (see logs below). To start, I did the following: zeekctl> deploy zeekctl> restart --clean zeekctl> start I can see the following in startup logs: starting ... starting zeek ... [ZeekControl] > diag [zeek] No core file found. Zeek 2.6-558 Linux 3.10.0-957.21.3.el7.x86_64 Zeek plugins: Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3.0) No reporter.log stderr.log listening on em1 stdout.log max memory size (kbytes, -m) unlimited data seg size (kbytes, -d) unlimited virtual memory (kbytes, -v) unlimited core file size (blocks, -c) unlimited .cmdline -i em1 -U .status -p zeekctl -p zeekctl-live -p standalone -p local -p zeek local.zeek zeekctl zeekctl/standalone zeekctl/auto .env_vars PATH=/usr/local/zeek/bin:/usr/local/zeek/share/zeekctl/scripts:/usr/local/openssl/bin:/opt/apache-maven-3.3.9/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/zeek/bin ZEEKPATH=/usr/local/zeek/spool/installed-scripts-do-not-touch/site::/usr/local/zeek/spool/installed-scripts-do-not-touch/auto:/usr/local/zeek/share/zeek:/usr/local/zeek/share/zeek/policy:/usr/local/zeek/share/zeek/site CLUSTER_NODE= .status RUNNING [net_run] No prof.log packet_filter.log #separator \x09 #set_separator , #empty_field(empty) #unset_field- #path packet_filter #open 2019-07-03-19-36-56 #fields ts nodefilter initsuccess #types timestring string boolbool 1562175416.590048 zeekip or not ipT T loaded_scripts.log #separator \x09 #set_separator , #empty_field(empty) #unset_field- #path loaded_scripts #open 2019-07-03-19-36-56 #fields name #types string /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/__load__.zeek /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/lib/bif/kafka.bif.zeek /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/__load__.bro /usr/local/zeek/lib/zeek/plugins/APACHE_KAFKA/scripts/init.bro it starts up fine, no error messages. running "diag" in zeekctl just gives a long list of plugins that were loaded. If I tail logs in I see new connection logs being added. However, I dont see any messages in Kafka console consumer. What am I missing? How do I go about debugging this? Thank you for your help and assistance. Best regards, Sanket From: zeo...@gmail.com Sent: Tuesday, July 2, 2019 11:46 AM To: user@metron.apache.org Subject: Re: metron-bro-plugin-kafka error Did you install it manual or with bro-pkg/zkg? I believe bro-pkg was renamed to zkg as of their 2.0 release but I haven't used it in a little while. Any more details regarding the installation process, or versions of software in use may be helpful Jon Zeolla On Tue, Jul 2, 2019, 12:26 AM Sanket Sharma mailto:sanket.sha...@dukstra.com>> wrote: Hi, I’m trying to configure Metron bro plugin by following instructions here: https://github.com/apache/metron-bro-plugin-kafka I’m able to build/install the plugin successfully but when I test it using the command: $ bro -N Apache::Kafka I get the following error: fatal error in /opt/bro/share/bro/base/init-bare.bro, line 1: cannot load plugin library /opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/APACHE-KAFKA.linux-x86_64.so<http://APACHE-KAFKA.linux-x86_64.so>: /opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/APACHE-KAFKA.linux-x86_64.so<http://APACHE-KAFKA.linux-x86_64.so>: undefined symbol: bro_version_2_6_558_plugin_7 Not sure what am I missing? Any help would be greatly appreciated. Best regards, Sanket
Re: metron-bro-plugin-kafka error
Did you install it manual or with bro-pkg/zkg? I believe bro-pkg was renamed to zkg as of their 2.0 release but I haven't used it in a little while. Any more details regarding the installation process, or versions of software in use may be helpful Jon Zeolla On Tue, Jul 2, 2019, 12:26 AM Sanket Sharma wrote: > Hi, > > > > I’m trying to configure Metron bro plugin by following instructions here: > https://github.com/apache/metron-bro-plugin-kafka > > > > > > I’m able to build/install the plugin successfully but when I test it using > the command: > > > > $ bro -N Apache::Kafka > > > > > > I get the following error: > > > > fatal error in /opt/bro/share/bro/base/init-bare.bro, line 1: cannot load > plugin library /opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/ > APACHE-KAFKA.linux-x86_64.so: /opt/bro/lib/bro/plugins/APACHE_KAFKA//lib/ > APACHE-KAFKA.linux-x86_64.so: undefined symbol: > bro_version_2_6_558_plugin_7 > > > > Not sure what am I missing? Any help would be greatly appreciated. > > > > > > Best regards, > > Sanket > > > > >
