GitHub user JonZeolla opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/3
METRON-813: Migrate metron-bro-plugin-kafka to be a bro package This should turn this repo into a bro package containing a bro plugin. # Testing The below testing plan assumes CentOS/RHEL, but can be tweaked to work on most mainstream linux distros. 1. Install [Kafka 0.10.0.1](https://kafka.apache.org/0101/documentation.html#quickstart), [Zookeeper 3.4.6](https://zookeeper.apache.org/doc/r3.4.6/zookeeperStarted.html) (The same versions from [HDP 2.5.5](https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.5/bk_release-notes/content/ch01s01.html)), and any package dependancies for testing. ``` # cd # yum -y install java screen # wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.6/zookeeper-3.4.6.tar.gz{,.sha1} # if [[ "$(sha1sum zookeeper-3.4.6.tar.gz)" == "$(cat zookeeper-3.4.6.tar.gz.sha1)" ]]; then tar -xvf zookeeper-3.4.6.tar.gz; else echo "sha1 sums do not match"; fi # cd zookeeper-3.4.6 # cp conf/zoo_sample.cfg conf/zoo.cfg # bin/zkServer.sh start # cd # wget https://mirrors.sonic.net/apache/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz # wget https://dist.apache.org/repos/dist/release/kafka/0.10.0.1/kafka_2.10-0.10.0.1.tgz.md5 # # Compare MD5s using md5sum # tar -xvf kafka_2.10-0.10.0.1.tgz # cd kafka_2.10-0.10.0.1 # bin/kafka-server-start.sh config/server.properties & # bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic bro ``` 1. [Install bro 2.5.2](https://www.bro.org/sphinx/install/install.html) and [bro-pkg](http://bro-package-manager.readthedocs.io/en/stable/quickstart.html#installation). Make sure you are running at least bro 2.5 and bro-pkg 1.2.0, and configure bro-pkg properly. ``` # export PATH=$PATH:/usr/local/bro/bin # bro --version bro version 2.5.2 # bro-pkg --version bro-pkg 1.2.2 # bro-pkg autoconfig ``` 1. Create a working directory and pull in this PR (selfishly using my branch of `checkout-pr` from [metron-commit-stuff](https://github.com/jonzeolla/metron-commit-stuff/tree/support-bro-plugin) to test some recent updates) ``` # git clone https://github.com/jonzeolla/metron-commit-stuff ~/metron-commit-stuff # cd ~/metron-commit-stuff # git checkout support-bro-plugin # cd # ~/metron-commit-stuff/checkout-pr 3 Please select a repository: 1) metron 2) metron-bro-plugin-kafka Selection [metron]: bro <snip> ``` 1. Install the package, and all of its dependancies, from the PR branch. Ensure it passes its unit tests. ``` # # Install librdkafka by following ONLY instruction 1 [here](https://github.com/apache/metron-bro-plugin-kafka#installation) # cd ~/metron-bro-plugin-kafka-pr3/ # bro-pkg install . ``` 1. Configure the plugin. ``` cat << EOF >> /usr/local/bro/share/bro/site/local.bro # Activate metron-bro-plugin-kafka @load metron-bro-plugin-kafka-pr3/Bro/Kafka # Configure metron-bro-plugin-kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG); EOF ``` 1. Run bro manually while monitoring kafka to confirm things are working. ``` # mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic # wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap # wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap # export PATH=$PATH:~/kafka_2.11-0.10.1.0/bin # screen # kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro # # Ctrl+A c to make a new screen window # cd ~/brotmp/example-traffic # bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C # # Use Ctrl+A n to cycle through screen sessions for validation. To run another test, on your second window, do # cd ~/brotmp/nitroba # bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-813 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/3.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3 ---- commit f21e51f4f91452d66b644b1c041e9a3ae3b39bd7 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-11-07T12:12:53Z METRON-1303: Reorganize the metron-bro-plugin-kafka commit c2f8b2c347f647076c1d0ba17dad5b3794d7957d Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-11-07T12:22:39Z Fix broken link commit 28594e75e423166eb5ca6e6df7a3f7ed2deeecd7 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-11-14T12:39:04Z METRON-813: Migrate metron-bro-plugin-kafka to be a bro package commit 22bbb7e1da6c3b207f9b92d9e959ae31d3fe83a0 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-11-14T18:55:02Z Fix load statements commit 84993a9fb55d274b63b22474b1758903bce64f28 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-11-15T13:01:27Z Merge branch 'master' of https://github.com/apache/metron-bro-plugin-kafka into METRON-813 ---- ---