[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16074620#comment-16074620 ] ASF GitHub Bot commented on METRON-508: --- Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16071475#comment-16071475
]
ASF GitHub Bot commented on METRON-508:
---
Github user JonZeolla commented on the issue:
https://github.com/apache/metron/pull/586
I spotted a duplicate in taking a scan of the template, so I ran `grep '":
{'
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
| sort | uniq -c | grep -v 1` to make sure that was the only one. I fixed it
and pushed things up, along with other minor tweaks - Travis was successful so
I plan to merge soon. I also put some thoughts for a follow-on PR in
[METRON-1010](https://issues.apache.org/jira/browse/METRON-1010).
> Expand Elasticsearch templates to support the standard bro logs
> ---
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
> Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn,
> HTTP, and DNS. We should provide additional templates so that an
> out-of-the-box bro install can send all of its logs into Metron and they will
> get probably indexed in elasticsearch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068760#comment-16068760 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 Last try before I merge in #624 and make it a dependancy. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068679#comment-16068679 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068678#comment-16068678 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068568#comment-16068568 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068567#comment-16068567 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16068381#comment-16068381 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067150#comment-16067150 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067151#comment-16067151 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067039#comment-16067039 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16067038#comment-16067038 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066955#comment-16066955 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065303#comment-16065303 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16057739#comment-16057739 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 @nickwallen So, I'm not entirely done with the documentation but I pushed it out for a quick, general review. In doing this, I noticed that some new default-on fields were added with the release of 2.5/2.5.1 (for example, `server_appdata` in [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) was added with 2.5). For now, I'm going to ignore those. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056245#comment-16056245
]
ASF GitHub Bot commented on METRON-508:
---
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r123058316
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
---
@@ -238,6 +238,538 @@
"qtype_name": {
"type": "string",
"index": "not_analyzed"
+},
+"analyzer": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"failure_reason": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"user": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"password": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"command": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"arg": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"mime_type": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"file_size": {
+ "type": "long"
+},
+"reply_code": {
+ "type": "integer"
+},
+"reply_msg": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"data_channel:passive": {
+ "type": "boolean"
+},
+"data_channel:orig_h": {
+ "type": "ip"
+},
+"data_channel:resp_h": {
+ "type": "ip"
+},
+"data_channel:resp_p": {
+ "type": "integer"
+},
+"cwd": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"passive": {
+ "type": "boolean"
+},
+"capture_password": {
+ "type": "boolean"
+},
+"fuid": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"conn_uids": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"source": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"depth": {
+ "type": "integer"
+},
+"analyzers": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"filename": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"duration": {
+ "type": "float"
+},
+"local_orig": {
+ "type": "boolean"
+},
+"is_orig": {
+ "type": "boolean"
+},
+"seen_bytes": {
+ "type": "long"
+},
+"total_bytes": {
+ "type": "long"
+},
+"missing_bytes": {
+ "type": "long"
+},
+"overflow_bytes": {
+ "type": "long"
+},
+"timedout": {
+ "type": "boolean"
+},
+"parent_fuid": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"md5": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"sha1": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"sha256": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"port_num": {
+ "type": "integer"
+},
+"subject": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"issuer_subject": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"serial": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"helo": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"mailfrom": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"rcptto": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"date": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"from": {
+ "type": "string",
+ "analyzer": "simple"
+
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056241#comment-16056241 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056240#comment-16056240 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056224#comment-16056224
]
ASF GitHub Bot commented on METRON-508:
---
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r123055525
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
---
@@ -238,6 +238,538 @@
"qtype_name": {
"type": "string",
"index": "not_analyzed"
+},
+"analyzer": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"failure_reason": {
--- End diff --
I'm sorry if this is left field but. What if, we provide more than one
'example' template in the bro parser extension, and not try to come up with the
perfect configuration?
Also - if and when 777 hits and we move on to 'create an instance of a
parser - clone and edit configurations' they can use those samples to do so.
> Expand Elasticsearch templates to support the standard bro logs
> ---
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
> Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn,
> HTTP, and DNS. We should provide additional templates so that an
> out-of-the-box bro install can send all of its logs into Metron and they will
> get probably indexed in elasticsearch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056124#comment-16056124 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056122#comment-16056122
]
ASF GitHub Bot commented on METRON-508:
---
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r123040458
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
---
@@ -238,6 +238,538 @@
"qtype_name": {
"type": "string",
"index": "not_analyzed"
+},
+"analyzer": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"failure_reason": {
--- End diff --
Right, I considered both options, and implemented option 2 at one point,
but I removed the comments because of the field name collisions (i.e. two
separate bro logs with an overlapping field name). While reading through the
template, it was confusing that a given section wouldn't contain all of the
fields for a specific log, because they were addressed earlier in the template
under another log's section.
I would prefer to merge this in as-is, and address the collision problem
separately (at least, that was my intent). The first true solution that comes
to mind is to put the individual bro logs into distinct indexes, but then we
would need to change anywhere in Metron that touches bro data. I would prefer
to do that after METRON-939 (#619), if it gets merged.
> Expand Elasticsearch templates to support the standard bro logs
> ---
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
> Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn,
> HTTP, and DNS. We should provide additional templates so that an
> out-of-the-box bro install can send all of its logs into Metron and they will
> get probably indexed in elasticsearch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056123#comment-16056123 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056102#comment-16056102
]
ASF GitHub Bot commented on METRON-508:
---
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r123037274
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
---
@@ -238,6 +238,538 @@
"qtype_name": {
"type": "string",
"index": "not_analyzed"
+},
+"analyzer": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"failure_reason": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"user": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"password": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"command": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"arg": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"mime_type": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"file_size": {
+ "type": "long"
+},
+"reply_code": {
+ "type": "integer"
+},
+"reply_msg": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"data_channel:passive": {
+ "type": "boolean"
+},
+"data_channel:orig_h": {
+ "type": "ip"
+},
+"data_channel:resp_h": {
+ "type": "ip"
+},
+"data_channel:resp_p": {
+ "type": "integer"
+},
+"cwd": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"passive": {
+ "type": "boolean"
+},
+"capture_password": {
+ "type": "boolean"
+},
+"fuid": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"conn_uids": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"source": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"depth": {
+ "type": "integer"
+},
+"analyzers": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"filename": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"duration": {
+ "type": "float"
+},
+"local_orig": {
+ "type": "boolean"
+},
+"is_orig": {
+ "type": "boolean"
+},
+"seen_bytes": {
+ "type": "long"
+},
+"total_bytes": {
+ "type": "long"
+},
+"missing_bytes": {
+ "type": "long"
+},
+"overflow_bytes": {
+ "type": "long"
+},
+"timedout": {
+ "type": "boolean"
+},
+"parent_fuid": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"md5": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"sha1": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"sha256": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"port_num": {
+ "type": "integer"
+},
+"subject": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"issuer_subject": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"serial": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"helo": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"mailfrom": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"rcptto": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"date": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"from": {
+ "type": "string",
+ "analyzer": "simple"
+
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055973#comment-16055973
]
ASF GitHub Bot commented on METRON-508:
---
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r123014881
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
---
@@ -238,6 +238,538 @@
"qtype_name": {
"type": "string",
"index": "not_analyzed"
+},
+"analyzer": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"failure_reason": {
--- End diff --
It would be handy to know which fields in the template belong to which Bro
log source (DPD vs Conn). This might make it easier for users (and us) to
maintain or customize the template going forward. I can think of two possible
ways to do this, but maybe there are better ways.
**Option 1:** We can use multiple templates that get [merged into a single
final
template](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html#multiple-templates).
This would provide a nice, clean separation between the different log
sources. But it might take more effort than it is worth.
**Option 2:** We can embed `/* c-style block comments */` in the template
itself. We could throw a comment above each set of fields that pertain to a
given source. We could also maintain the handy links that you embedded in the
PR description.
```
/*
* DPD
*
https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info
*/
"failure_reason": {
..
```
> Expand Elasticsearch templates to support the standard bro logs
> ---
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
> Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn,
> HTTP, and DNS. We should provide additional templates so that an
> out-of-the-box bro install can send all of its logs into Metron and they will
> get probably indexed in elasticsearch.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055974#comment-16055974
]
ASF GitHub Bot commented on METRON-508:
---
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r123012618
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
---
@@ -238,6 +238,538 @@
"qtype_name": {
"type": "string",
"index": "not_analyzed"
+},
+"analyzer": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"failure_reason": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"user": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"password": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"command": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"arg": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"mime_type": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"file_size": {
+ "type": "long"
+},
+"reply_code": {
+ "type": "integer"
+},
+"reply_msg": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"data_channel:passive": {
+ "type": "boolean"
+},
+"data_channel:orig_h": {
+ "type": "ip"
+},
+"data_channel:resp_h": {
+ "type": "ip"
+},
+"data_channel:resp_p": {
+ "type": "integer"
+},
+"cwd": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"passive": {
+ "type": "boolean"
+},
+"capture_password": {
+ "type": "boolean"
+},
+"fuid": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"conn_uids": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"source": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"depth": {
+ "type": "integer"
+},
+"analyzers": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"filename": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"duration": {
+ "type": "float"
+},
+"local_orig": {
+ "type": "boolean"
+},
+"is_orig": {
+ "type": "boolean"
+},
+"seen_bytes": {
+ "type": "long"
+},
+"total_bytes": {
+ "type": "long"
+},
+"missing_bytes": {
+ "type": "long"
+},
+"overflow_bytes": {
+ "type": "long"
+},
+"timedout": {
+ "type": "boolean"
+},
+"parent_fuid": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"md5": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"sha1": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"sha256": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"port_num": {
+ "type": "integer"
+},
+"subject": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"issuer_subject": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"serial": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"helo": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"mailfrom": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"rcptto": {
+ "type": "string",
+ "analyzer": "simple"
+},
+"date": {
+ "type": "string",
+ "index": "not_analyzed"
+},
+"from": {
+ "type": "string",
+ "analyzer": "simple"
+
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16055013#comment-16055013 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054411#comment-16054411 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16054412#comment-16054412 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla reopened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) - [DevicesInfo](https://www.bro.org/sphinx/scripts/policy/misc/known-devices.bro.html#type-Known::DevicesInfo) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/metron ~/metron-508/metron cd ~/metron-508/metron git remote add jonzeolla https://github.com/jonzeolla/metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, Known::DEVICES_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro' /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su -
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030436#comment-16030436 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 Thanks. I haven't used org.adrianwalker.multilinestring.Multiline before, took a quick shot with two of them, I'll do the rest when I have more time. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030389#comment-16030389 ] ASF GitHub Bot commented on METRON-508: --- Github user justinleet commented on the issue: https://github.com/apache/metron/pull/586 I did leave one comment about refactoring tests a bit, but I'm +1 regardless of if that change happens or not. Spun it up in full dev and was able to get all the types to show up in the histogram and things looked good all around. Thanks for the great contribution! > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[
https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030384#comment-16030384
]
ASF GitHub Bot commented on METRON-508:
---
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/586#discussion_r119243250
--- Diff:
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/bro/BasicBroParserTest.java
---
@@ -68,27 +68,29 @@ public void testDecimalFormatAssumptions() {
}
public void testUnwrappedBroMessage() throws ParseException {
-String rawMessage =
"{\"timestamp\":1449511228.474,\"uid\":\"CFgSLp4HgsGqXnNjZi\",\"source_ip\":\"104.130.172.191\",\"source_port\":33893,\"dest_ip\":\"69.20.0.164\",\"dest_port\":53,\"proto\":\"udp\",\"trans_id\":3514,\"rcode\":3,\"rcode_name\":\"NXDOMAIN\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":false,\"Z\":0,\"rejected\":false,\"sensor\":\"cloudbro\",\"type\":\"dns\"}";
+String rawMessage =
"{\"ts\":1449511228.474,\"uid\":\"CFgSLp4HgsGqXnNjZi\",\"id.orig_h\":\"104.130.172.191\",\"id.orig_p\":33893,\"id.resp_h\":\"69.20.0.164\",\"id.resp_p\":53,\"proto\":\"udp\",\"trans_id\":3514,\"rcode\":3,\"rcode_name\":\"NXDOMAIN\",\"AA\":false,\"TC\":false,\"RD\":false,\"RA\":false,\"Z\":0,\"rejected\":false,\"sensor\":\"cloudbro\",\"type\":\"dns\"}";
--- End diff --
I'd like to see these pulled out and using `@Multiline` for readability.
Given that it's not currently in this format, I'm not opposed to just leaving
it, but it does make it easier to understand at a glance. Given that these all
get touched anyway right now, it seems like the perfect time to refactor it.
Example:
```
/**
* {
* "ts":1449511228.474,
* "uid":"CFgSLp4HgsGqXnNjZi",
* "id.orig_h":"104.130.172.191",
* "id.orig_p":33893,
* "id.resp_h":"69.20.0.164",
* "id.resp_p":53,
* "proto":"udp",
* "trans_id":3514,
* "rcode":3,
* "rcode_name":"NXDOMAIN",
* "AA":false,
* "TC":false,
* "RD":false,
* "RA":false,
* "Z":0,
* "rejected":false,
* "sensor":"cloudbro",
* "type":"dns"
* }
*/
@Multiline
public static String unwrappedBroMessage;
public void testUnwrappedBroMessage() throws ParseException {
JSONObject rawJson =
(JSONObject)jsonParser.parse(unwrappedBroMessage);
JSONObject broJson =
broParser.parse(unwrappedBroMessage.getBytes()).get(0);
String expectedBroTimestamp = "1449511228.474";
Assert.assertEquals(broJson.get("bro_timestamp"),
expectedBroTimestamp);
String expectedTimestamp = "1449511228474";
Assert.assertEquals(broJson.get("timestamp").toString(),
expectedTimestamp);
Assert.assertEquals(broJson.get("ip_src_addr").toString(),
rawJson.get("id.orig_h").toString());
Assert.assertEquals(broJson.get("ip_dst_addr").toString(),
rawJson.get("id.resp_h").toString());
Assert.assertEquals(broJson.get("ip_src_port"),
rawJson.get("id.orig_p"));
Assert.assertEquals(broJson.get("ip_dst_port"),
rawJson.get("id.resp_p"));
Assert.assertEquals(broJson.get("uid").toString(),
rawJson.get("uid").toString());
Assert.assertEquals(broJson.get("trans_id").toString(),
rawJson.get("trans_id").toString());
Assert.assertEquals(broJson.get("sensor").toString(),
rawJson.get("sensor").toString());
Assert.assertEquals(broJson.get("type").toString(),
rawJson.get("type").toString());
Assert.assertEquals(broJson.get("rcode").toString(),
rawJson.get("rcode").toString());
Assert.assertEquals(broJson.get("rcode_name").toString(),
rawJson.get("rcode_name").toString());
Assert.assertTrue(broJson.get("original_string").toString().startsWith("DNS"));
}
```
> Expand Elasticsearch templates to support the standard bro logs
> ---
>
> Key: METRON-508
> URL: https://issues.apache.org/jira/browse/METRON-508
> Project: Metron
> Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Minor
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> The current elasticsearch templates do not support any logs other than Conn,
> HTTP, and DNS. We should provide additional templates so that an
> out-of-the-box bro install can send all of its logs into Metron and they will
> get probably indexed in elasticsearch.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025427#comment-16025427 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 Bump > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010523#comment-16010523 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 I updated the instructions to reflect the repo name change; hopefully it should work but I won't have a chance to test it out for a couple of days. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009329#comment-16009329 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 Didn't I do that? > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009322#comment-16009322 ] ASF GitHub Bot commented on METRON-508: --- Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/586 Still need to review, but this is a great addition Jon. What pairs nicely with this is to expand the unit tests for the Bro parser to test the other log types that you include in this PR. Right now only HTTP and DNS are tested there. We can tackle separately if you like. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16009015#comment-16009015 ] ASF GitHub Bot commented on METRON-508: --- GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/586 METRON-508 Expand Elasticsearch templates to support the standard bro logs ## Contributor Comments This PR makes it easier for someone with an existing bro install to send some of their log files into Metron, based off of a combination of the [bro documentation](https://www.bro.org/sphinx/script-reference/log-files.html) and a fresh install of bro 2.5. There are future plans to expand on this via [METRON-518](https://issues.apache.org/jira/browse/METRON-518) and [METRON-908](https://issues.apache.org/jira/browse/METRON-908). Specifically, this attempts to provide initial support the default-on fields of the following logs: - [Conn](https://www.bro.org/sphinx/scripts/base/protocols/conn/main.bro.html#type-Conn::Info) - [DPD](https://www.bro.org/sphinx-git/scripts/base/frameworks/dpd/main.bro.html#type-DPD::Info) - [FTP](https://www.bro.org/sphinx/scripts/base/protocols/ftp/info.bro.html#type-FTP::Info) - [Files](https://www.bro.org/sphinx/scripts/base/frameworks/files/main.bro.html#type-Files::Info) - [Known::CertsInfo](https://www.bro.org/sphinx/scripts/policy/protocols/ssl/known-certs.bro.html#type-Known::CertsInfo) - [SMTP](https://www.bro.org/sphinx/scripts/base/protocols/smtp/main.bro.html#type-SMTP::Info) - [SSL](https://www.bro.org/sphinx/scripts/base/protocols/ssl/main.bro.html#type-SSL::Info) - [Weird](https://www.bro.org/sphinx/scripts/base/frameworks/notice/weird.bro.html#type-Weird::Info) - [Notice](https://www.bro.org/sphinx/scripts/base/frameworks/notice/main.bro.html#type-Notice::Info) - [DHCP](https://www.bro.org/sphinx/scripts/base/protocols/dhcp/main.bro.html#type-DHCP::Info) - [SSH](https://www.bro.org/sphinx/scripts/base/protocols/ssh/main.bro.html#type-SSH::Info) - [Software](https://www.bro.org/sphinx/scripts/base/frameworks/software/main.bro.html#type-Software::Info) - [Radius](https://www.bro.org/sphinx/scripts/base/protocols/radius/main.bro.html#type-RADIUS::Info) - [X509](https://www.bro.org/sphinx/scripts/base/files/x509/main.bro.html#type-X509::Info) ## Testing 1. Create a working directory and pull in this PR ``` mkdir ~/metron-508 git clone https://github.com/apache/incubator-metron ~/metron-508/incubator-metron cd ~/metron-508/incubator-metron git remote add jonzeolla https://github.com/jonzeolla/incubator-metron git pull jonzeolla METRON-508 ``` 1. Modify [this](https://github.com/JonZeolla/incubator-metron/blob/METRON-508/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L20) to remove `sensors,` (to spin up the real sensors). ``` sed -i '' "s/ansibleSkipTags=.*/ansibleSkipTags=\'quick_dev\'/" metron-deployment/vagrant/full-dev-platform/Vagrantfile ``` 1. Start up full-dev. ``` cd metron-deployment/vagrant/full-dev-platform vagrant up ``` 1. Set up the environment in full-dev. ``` vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin service monit stop && service sensor-stubs stop bro && broctl stop yum -y install jq wireshark ``` 1. Configure kafka in local.bro. ``` sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, DHCP::LOG, FTP::LOG, SSH::LOG, SSL::LOG, SMTP::LOG, RADIUS::LOG, Weird::LOG, Files::LOG, Notice::LOG, Software::LOG, Known::CERTS_LOG, X509::LOG);/' /usr/local/bro/share/bro/site/local.bro echo "redef Kafka::debug = \"all\";" >> /usr/local/bro/share/bro/site/local.bro echo "redef Known::cert_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro echo "redef Software::asset_tracking = ALL_HOSTS;" >> /usr/local/bro/share/bro/site/local.bro ``` 1. Monitor the bro kafka topic ``` # Open a new terminal cd ~/metron-508/incubator-metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro ``` 1. Monitor the storm logs. ``` # Open a new terminal cd ~/metron-508/incubator-metron/metron-deployment/vagrant/full-dev-platform vagrant ssh sudo su - export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin # Look at the storm logs (The "failed to parse" errors for ip_src_addr and ip_dst_addr are expected, and
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15992876#comment-15992876 ] Jon Zeolla commented on METRON-508: --- I'm in the process of writing all of that up and testing my changes. I don't expect to update, for instance, the sensor-stubs to push new data in, but I will take a look at testing the different logs against the parser. Hopefully I will get this out today. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989024#comment-15989024 ] Jon Zeolla commented on METRON-508: --- We should also improve the tokenization to be more sane. For instance, the addition of things like https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pathhierarchy-tokenizer.html (forward or reverse, depending on the field) would be very helpful. > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
