[GitHub] metron issue #586: METRON-508 Expand Elasticsearch templates to support the ...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/586 Didn't I do that? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #560: METRON-912: Metron vagrant setup steps no longe...
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/560 I gave `brew cask install docker` another shot this morning and got it running no problem. Updated the instructions per discussion with @nickwallen. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #564: Metron-905 Fix square-bracket behavior a...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/564#discussion_r115301858 --- Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/ELASTICSEARCH/2.3.3/configuration/elastic-site.xml --- @@ -189,7 +189,12 @@ network_host -"_lo:ipv4_","_eth0:ipv4_" -Network interface(s) ES will bind to within each node. Confirm names via ifconfig. Should have entries for primary external and loopback interfaces, with :ipv4 annotation and quote marks around each entry: "_lo:ipv4_","_eth0:ipv4_" +[ _local_ ] +Network interface(s) ES will bind to within each node. Use [ _local_ ] for single-node install; change to [ _local_, _site_ ] (including square brackets) for typical cluster install. See https://www.elastic.co/guide/en/elasticsearch/reference/2.3/modules-network.html for ES documentation. + + +network_publish_host --- End diff -- Shouldn't we just change it to be optional, if it's actually optional? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #560: METRON-912: Metron vagrant setup steps no longe...
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/560 Ok, sounds good. The biggest part of this is that I have not been able to find anybody who got Metron working on vagrant after only installing docker via `brew cask install docker`. I spent some time on it and got relatively nowhere, only fixing it by installing Docker via their dmg, which properly starts the docker daemon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #555: METRON-902 ES improperly indexes Bro logs
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/555 bump --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #560: METRON-912: Metron vagrant setup steps n...
Github user JonZeolla closed the pull request at: https://github.com/apache/incubator-metron/pull/560 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #555: METRON-902 ES improperly indexes Bro logs
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/555 Ok, this is ready for review. It essentially re-implements METRON-403, METRON-510, and METRON-517. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/547#discussion_r113778897 --- Diff: metron-sensors/bro-plugin-kafka/README.md --- @@ -94,6 +95,52 @@ event bro_init() } ``` +### Example 3 + +As documented in [METRON-285](https://issues.apache.org/jira/browse/METRON-285) and [METRON-286](https://issues.apache.org/jira/browse/METRON-286), various components in Metron do not currently support IPv6. Because of this, you may not want to send bro logs that contain IPv6 source or destination IPs into Metron. In this example, we are assuming a somewhat standard bro configuration for sending logs into a Metron cluster, such that: + * Each type of bro log is sent to the `bro` topic, but is tagged with the appropriate log type (such as `http`, `dns`, or `conn`). This is done by setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving it unset), and by setting `tag_json` to true. + * The Kafka writer is set appropriately to send logs to the `bro` Kafka topic being used in your Metron cluster. This requires that your `kafka_conf` and `$config` tables are appropriately configured. + --- End diff -- I took a stab at being more concise. Take a look and let me know, I think I like it more this way than before so thanks for the critique. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/547#discussion_r113762122 --- Diff: metron-sensors/bro-plugin-kafka/README.md --- @@ -94,6 +95,60 @@ event bro_init() } ``` +### Example 3 + +As documented in [METRON-285](https://issues.apache.org/jira/browse/METRON-285) and [METRON-286](https://issues.apache.org/jira/browse/METRON-286), various components in Metron do not currently support IPv6. Because of this, you may not want to send bro logs that contain IPv6 source or destination IPs into Metron. In this example, we are assuming a somewhat standard bro configuration for sending logs into a Metron cluster, such that: + * Each type of bro log is sent to the `bro` topic, but is tagged with the appropriate log type (such as `http`, `dns`, or `conn`). This is done by setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving it unset), and by setting `tag_json` to true. + * The Kafka writer is set appropriately to send logs to the `bro` Kafka topic being used in your Metron cluster. This requires that your `kafka_conf` and `$config` tables are appropriately configured. + +``` +@load Bro/Kafka/logs-to-kafka.bro +redef Kafka::topic_name = "bro"; --- End diff -- I'm obviously missing something. `logs_to_send` is not `topic_name`? In my example `logs_to_send` is not set. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/547#discussion_r113745031 --- Diff: metron-sensors/bro-plugin-kafka/README.md --- @@ -94,6 +95,60 @@ event bro_init() } ``` +### Example 3 + +As documented in [METRON-285](https://issues.apache.org/jira/browse/METRON-285) and [METRON-286](https://issues.apache.org/jira/browse/METRON-286), various components in Metron do not currently support IPv6. Because of this, you may not want to send bro logs that contain IPv6 source or destination IPs into Metron. In this example, we are assuming a somewhat standard bro configuration for sending logs into a Metron cluster, such that: + * Each type of bro log is sent to the `bro` topic, but is tagged with the appropriate log type (such as `http`, `dns`, or `conn`). This is done by setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving it unset), and by setting `tag_json` to true. + * The Kafka writer is set appropriately to send logs to the `bro` Kafka topic being used in your Metron cluster. This requires that your `kafka_conf` and `$config` tables are appropriately configured. + +``` +@load Bro/Kafka/logs-to-kafka.bro +redef Kafka::topic_name = "bro"; --- End diff -- Not sure I follow. If `Kafka::logs_to_send` is empty why would `logs-to-kafka.bro` make its own filters? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/547#discussion_r113576968 --- Diff: metron-sensors/bro-plugin-kafka/README.md --- @@ -94,6 +95,52 @@ event bro_init() } ``` +### Example 3 + +As documented in [METRON-285](https://issues.apache.org/jira/browse/METRON-285) and [METRON-286](https://issues.apache.org/jira/browse/METRON-286), various components in Metron do not currently support IPv6. Because of this, you may not want to send bro logs that contain IPv6 source or destination IPs into Metron. In this example, we are assuming a somewhat standard bro configuration for sending logs into a Metron cluster, such that: + * Each type of bro log is sent to the `bro` topic, but is tagged with the appropriate log type (such as `http`, `dns`, or `conn`). This is done by setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving it unset), and by setting `tag_json` to true. + * The Kafka writer is set appropriately to send logs to the `bro` Kafka topic being used in your Metron cluster. This requires that your `kafka_conf` and `$config` tables are appropriately configured. + --- End diff -- My goal was just to be explicit. I can take another stab at it tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/547#discussion_r113538317 --- Diff: metron-sensors/bro-plugin-kafka/README.md --- @@ -94,6 +95,52 @@ event bro_init() } ``` +### Example 3 + +As documented in [METRON-285](https://issues.apache.org/jira/browse/METRON-285) and [METRON-286](https://issues.apache.org/jira/browse/METRON-286), various components in Metron do not currently support IPv6. Because of this, you may not want to send bro logs that contain IPv6 source or destination IPs into Metron. In this example, we are assuming a somewhat standard bro configuration for sending logs into a Metron cluster, such that: + * Each type of bro log is sent to the `bro` topic, but is tagged with the appropriate log type (such as `http`, `dns`, or `conn`). This is done by setting `topic_name` to `bro`, setting `$path` to an empty string (or leaving it unset), and by setting `tag_json` to true. + * The Kafka writer is set appropriately to send logs to the `bro` Kafka topic being used in your Metron cluster. This requires that your `kafka_conf` and `$config` tables are appropriately configured. + --- End diff -- Yes. Do you think it's too wordy? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #531: METRON-854 create dhcp dump parser
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/531 I would love to see Metron have a solution for both approaches - ingesting DHCP server logs, as well as DHCP observations based on network traffic. Like @ottobackwards mentioned, not everyone can get the right infrastructure/viewpoint on their network to run something like Bro and get the DHCP traffic to their sensors to be processed. I have definitely sent more than just DNS and HTTP from Bro to Metron and it has been properly ingested, but to date I haven't done DHCP. Like @simonellistonball and @nickwallen mentioned, both the parser and the kafka plugin are setup to handle new bro logs quite well, and a while back I worked on updating Metron's support for more Bro sources via [METRON-508](https://github.com/JonZeolla/incubator-metron/commit/736cc39525f9f08f6e781faea2610e893327e74c). I just never had a chance to test it, so I haven't yet opened a PR. Once #545 and #547 get merged into master, and I'm able to finish [METRON-813](https://issues.apache.org/jira/browse/METRON-813), I would be happy to work on anything related to Bro and DHCP logs at scale, including finishing up METRON-508. I have two hardware bro environments and my larger one currently sees about 7 million DHCP observations/day and sends ~30,000 messages per second into Metron. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron pull request #547: METRON-858 bro-plugin-kafka is throwing ...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/incubator-metron/pull/547
METRON-858 bro-plugin-kafka is throwing segfaults
## Contributor Comments
This PR is a follow-on of #545. Please DO NOT MERGE until the outstanding
items are all completed.
Outstanding items:
- [ ] Thoroughly test example 3
- [ ] Test at scale
The primary change here resolves a thread safety issue that is only seen
when under load. It has been reported in numerous places, but I've seen it
best documented [here](https://github.com/bro/bro-plugins/issues/43).
### Testing
The following steps can be used to validate the PR. (Mostly extracted from
METRON-883's testing steps)
1. Create a working directory.
```
mkdir metron-858
cd metron-858
```
1. Launch a CentOS host.
```
vagrant init bento/centos-6.7
vagrant up
vagrant ssh
```
1. Install some dependencies.
```
sudo su -
yum -y install epel-release
yum -y install "@Development tools" java-1.8.0-openjdk cmake
libpcap-devel openssl-devel python-devel
```
1. Create a new `HDP.repo` Yum repository; this will allow us to install
Kafka.
```
cat << EOF > /etc/yum.repos.d/HDP.repo
[HDP-2.5]
name=HDP-2.5
baseurl=http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.3.0
path=/
enabled=1
gpgcheck=0
EOF
```
1. Install and start Kafka.
```
yum -y install kafka
export PATH=$PATH:/usr/hdp/current/kafka-broker/bin
zookeeper-server start
kafka start
```
1. Install Librdkafka 0.9.4.
```
wget https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz -O -
| tar -xz
cd librdkafka-0.9.4/
./configure --prefix=/usr
make
make install
```
1. Add Librdkafka to our default load path.
```
echo "/usr/lib" >> /etc/ld.so.conf.d/bro-plugin.conf
ldconfig -v
```
1. Build and install Bro.
```
yum -y install cmake libpcap-devel openssl-devel python-devel
wget https://www.bro.org/downloads/release/bro-2.4.1.tar.gz -O
~/bro-2.4.1.tar.gz
tar -xzf ~/bro-2.4.1.tar.gz -C ~
cd ~/bro-2.4.1
./configure --prefix=/usr
make
make install
```
1. Fetch the code from this PR.
```
git clone https://github.com/apache/incubator-metron ~/incubator-metron
cd ~/incubator-metron
git pull origin pull/XXX/head
```
1. Install the Bro Plugin.
```
cd metron-sensors/bro-plugin-kafka
./configure --bro-dist=/root/bro-2.4.1
--install-root=/usr/lib/bro/plugins/ --with-librdkafka=/usr
make
make install
```
1. Modify your `/usr/share/bro/site/local.bro`:
```
cat << EOF >> /usr/share/bro/site/local.bro
@load Bro/Kafka/logs-to-kafka.bro
redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG);
redef Kafka::topic_name = "bro";
redef Kafka::tag_json = T;
redef Kafka::kafka_conf = table( ["metadata.broker.list"] =
"localhost:9092" );
EOF
```
1. Create a virtual interface called `tap0` to listen on.
```
yum install -y tunctl
tunctl -p
ifconfig tap0 10.0.0.1 up
ip link set tap0 promisc on
```
1. Configure Bro to listen on virtual interface.
```
sed -i 's/eth0/tap0/g' /usr/etc/node.cfg
```
1. Create a Kafka topic called `bro`.
```
kafka-topics.sh --zookeeper localhost:2181 --create --topic bro
--partitions 1 --replication-factor 1
```
1. Make sure the Bro changes are installed and start Bro.
```
broctl deploy
```
1. Grab an example pcap file and replay some packet data through `tap0`.
Keep this running in a separate session.
```
yum -y install tcpreplay
wget
https://github.com/apache/incubator-metron/raw/master/metron-deployment/roles/sensor-test-mode/files/example.pcap
tcpreplay -i tap0 --loop=0 --stats=5 example.pcap
```
1. Ensure that data is hitting the `bro` topic in Kafka.
```
# kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
--from-beginning
OpenJDK 64-Bit Server VM warning: If the number of processors is
expected to increase from one, then you should configure the number of parallel
GC threads appropriately using -XX:ParallelGCThreads=N
{metadata.broker.list=localhost:9092, request.timeout.ms=3,
client.id=console-consumer-99442, security.protocol=PLAINTEXT}
{"dns":
{"ts":149314
[GitHub] incubator-metron pull request #543: METRON-857 Ability to completely build p...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/incubator-metron/pull/543#discussion_r112808520 --- Diff: metron-deployment/packaging/docker/ansible-docker/Dockerfile --- @@ -21,10 +21,10 @@ RUN yum install -y tar RUN yum install -y wget RUN yum groupinstall -y "Development tools" RUN yum install -y zlib-dev openssl-devel sqlite-devel bzip2-devel libffi-devel -RUN wget https://www.python.org/ftp/python/2.7.10/Python-2.7.10.tgz -O /usr/src/Python-2.7.10.tgz +RUN wget https://www.python.org/ftp/python/2.7.12/Python-2.7.12.tgz -O /usr/src/Python-2.7.12.tgz --- End diff -- I thought we were pinning to 2.7.11 generally. I know it's a bit pedantic, as the differences are minimal, but perhaps we should just stay with a single version across all of our builds? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #539: METRON-867: In the event that we graduate, remo...
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/539 Am I missing something? I thought we were approved for graduation as of last night. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-metron issue #510: METRON-821 Minor fixes in full dev kerberos set...
Github user JonZeolla commented on the issue: https://github.com/apache/incubator-metron/pull/510 Validated this with @mmiklavc today, an issue I found was confirmed as unrelated and I will open a separate JIRA for it soon, if nobody else beats me to it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
