[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054412#comment-16054412 ]
ASF GitHub Bot commented on METRON-508: --------------------------------------- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin # Look at the storm logs (The "failed to parse" errors for ip_src_addr and ip_dst_addr are expected, and should be addressed as a part of METRON-939) tail -f /var/log/storm/workers-artifacts/indexing-*/*/worker.log | grep -i "org.elasticsearch.index.mapper.MapperParsingException: failed to parse" # You may want to evaluate worker.log for other errors, but the prior command is helpful to cut through some of the failed indexing of IPv6 addresses ``` 1. Run bro against some public pcaps. ``` # In the first of your three terminals # These are kept separate so that the flat file log output won't stomp the prior ones, for ingest validation mkdir -p ~/brotmp/nitroba ~/brotmp/example-traffic ~/brotmp/ssh ~/brotmp/ftp ~/brotmp/radius wget https://www.bro.org/static/traces/exercise-traffic.pcap -O ~/brotmp/example-traffic/exercise-traffic.pcap wget http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap -O ~/brotmp/nitroba/nitroba.pcap wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap wget https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true -O ~/brotmp/ftp/ftp.pcap wget https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true -O ~/brotmp/radius/radius_localhost.pcapng cd ~/brotmp/example-traffic bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/nitroba bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/ssh bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/ftp bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C cd ~/brotmp/radius editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C ``` 1. Validate that terminals 2 and 3 don't have any errors that you don't expect. 1. Verify proper indexing in ES and availability in kibana. ``` # Check around and make sure things look okay declare -a exists notexists; for protocol in http dns conn dpd dhcp ftp ssh ssl smtp radius weird files notice software known_certs x509 known_devices; do if [[ $(curl -s -XGET "node1:9200/bro*/_search?q=protocol:${protocol}" | jq '.hits.hits') == '[]' ]]; then notexists+=" ${protocol}"; else exists+=" ${protocol}"; fi; done; if [ ${#notexists[@]} -ne 0 ]; then echo -e "\n\n\033[0mThe following do exist in ES: ${exists[@]}\n\033[0;31mThe following do NOT exist in ES: ${notexists[@]}\033[0m"; else echo 'All of the log types are in ES! Success!'; fi; unset exists notexists # Check Kibana. For example: http://node1:5000/app/kibana#/visualize/create?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-15y,mode:quick,to:now))&_a=(filters:!(),linked:!f,query:(query_string:(analyze_wildcard:!t,query:'*')),uiState:(),vis:(aggs:!((id:'3',params:(field:protocol,orderBy:'2',size:20),schema:segment,type:terms),(id:'2',schema:metric,type:count)),type:histogram))&indexPattern=bro*&type=histogram # OPTIONAL testing # Run `/usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head` and look around curl -XGET node1:9200/_cat/indices # First column should be all green curl -XGET "node1:9200/bro*/_count" # Check the count of entries in the bro index, you can re-run bro against specific PCAPs and watch this increase, etc. ``` ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? #### Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron METRON-508 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/586.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #586 ---- commit 28990c61fb249c286f6eaac09be33e529a9dd7f6 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-05-12T13:31:32Z METRON-508 Expand Elasticsearch templates to support the standard bro logs commit 04a17479ff2903b0755ce3ada0c4425b387b3c1e Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-05-14T15:42:46Z First pass at updating the integration tests commit 314eb285f40e6de82bb64db032d60fc461fcefec Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-05-16T20:53:07Z Add Known Devices support (leverages DHCP client IDs) commit a6e7b8fbe8e1723a8ab57f4283e7c93f3d7d5080 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-05-16T20:53:38Z Fix failed to parse [trans_id] error in ES (Numeric value (X) out of range of int) commit 3efad3494599007c9507bd21db6b7585ad002d0c Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-05-20T02:36:10Z Comment change commit 121ec28df0e2ed933210b0737e002420d54f9f17 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-05-31T00:44:00Z Brief Multiline transformation commit cbfad879ab227ff6c780585f9113cd0d356b75ce Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-06-01T13:34:10Z Semicolons are hard commit a1384c0561ac3605150bb59be801a4d4efcb2f21 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-06-01T15:39:56Z I wish I had more time to work on this commit ee84084d164a7b0a5cf69d600dae786007ef9ffe Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-06-01T18:20:44Z Add more multiline commit 9776cb266bced38837e38b250c65177c2839ce7f Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-06-19T14:31:27Z Finish multiline work commit bc9c82654f4aabe1f04ef5eaf066290da22ce0ba Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-06-19T14:57:37Z Merge branch 'master' of https://github.com/jonzeolla/metron into METRON-508 commit 7e761480c9749d67acfa7de538f54eee96dcba05 Author: Jon Zeolla <zeo...@gmail.com> Date: 2017-06-19T15:32:37Z Fix bro test - missing rawMessageMap ---- > Expand Elasticsearch templates to support the standard bro logs > --------------------------------------------------------------- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task > Reporter: Jon Zeolla > Assignee: Jon Zeolla > Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)