[GitHub] metron issue #1292: METRON-1925 Provide Verbose View of Profile Results in R...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1292 This should get added to the README. ---
[GitHub] metron-bro-plugin-kafka issue #12: METRON-1817: Remove bro package dependenc...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/12 This is now ready for review, but if #21 gets in first we should update some of the new files in that to use bro 2.6 instead of 2.5.5. I also would like to get a 0.3 release out that works in bro 2.5.x before merging this and forcing 2.6, which was very recently released. ---
[GitHub] metron pull request #:
Github user JonZeolla commented on the pull request: https://github.com/apache/metron/commit/ba9ddda6d742a0df96e24a1573f44482020c25fb#commitcomment-31583772 In metron-contrib/metron-docker/compose/kafkazk/Dockerfile: In metron-contrib/metron-docker/compose/kafkazk/Dockerfile on line 35: https://github.com/apache/metron/commit/ba9ddda6d742a0df96e24a1573f44482020c25fb#commitcomment-31576319 ---
[GitHub] metron pull request #1291: METRON-1922: Escaping incorrectly handled in curr...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1291#discussion_r239492672
--- Diff:
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/common/shell/cli/StellarShellTest.java
---
@@ -100,6 +100,12 @@ public void testExecuteStellar() throws Exception {
assertEquals("4", stdout());
}
+ @Test
+ public void testBacklashInStrings() throws Exception {
--- End diff --
`s/Backlash/Backslash/`
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238848648
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker
machine."
+ echo "--leave-running Do not stop containers after
script. The cleanup_containers.sh script should be run when done."
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+ if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
--- End diff --
I don't think this was done.
---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 @ottobackwards for permissions I would do something like 0755 ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238844866
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker
machine."
+ echo "--leave-running Do not stop containers after
script. The cleanup_containers.sh script should be run when done."
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+ if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
--- End diff --
Shouldn't this depend on the container tag specified earlier?
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238844401
--- Diff: docker/scripts/cleanup_containers.sh ---
@@ -0,0 +1,30 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
--- End diff --
Shouldn't this depend on the container tag specified earlier?
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238844199
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker
machine."
+ echo "--leave-running Do not stop containers after
script. The cleanup_containers.sh script should be run when done."
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+ if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
+ fi
+
+ if [[ "$RAN_KAFKA_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=kafka
+ fi
+
+ if [[ "$RAN_ZK_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=zookeeper
+ fi
+
+ if [[ "$CREATED_NETWORK_FLAG" = true ]]; then
+"${SCRIPT_DIR}"/destroy_docker_network.sh --network-name=bro-network
+ fi
+}
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --skip-docker-build
+ #
+ --skip-docker-build)
+ SKIP_REBUILD_BRO=true
+ shift # past argument
+ ;;
+
+ #
+ # LEAVE_RUNNING
+ #
+ # --leave-running
+ #
+--leave-running)
+LEAVE_RUNNING=true
+shift # past argument
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+ esac
+done
+EXTRA_ARGS="$@"
+echo "Running with "
+echo "SKIP_REBUILD_BRO = $SKIP_REBUILD_BRO"
+echo "==="
+
+# create the network
+bash "${SCRIPT_DIR}"/create_docker_network.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+ shutdown
+ exit ${rc}
+else
+ CREATED_NETWORK_FLAG=true
+fi
+
+
+
+# run the zookeeper container
+bash "${SCRIPT_DIR}"/run_zookeeper_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+ shutdown
+ exit ${rc}
+else
+ RAN_ZK_CONTAINER=true
+fi
+
+# run the kafka container
+bash "${SCRIPT_DIR}"/run_kafka_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+ shutdown
+ exit ${rc}
+else
+ RAN_KAFKA_CONTAINER=true
+fi
+
+#build the bro container
+if [[ "$SKIP_REBUILD_BRO" = false ]] ; then
+ bash "${SCRIPT_DIR}"/build_container.sh \
+--container-directory="${CONTAINER_DIR}" \
+--container-name=bro-docker-container:latest
+
+ rc=$?; if [[ ${rc} != 0 ]]; then
+shutdown
+exit ${rc}
+ else
+CREATED_BRO_CONTAINER=true
+ fi
+fi
+
+
+#run the bro container
+#and optionally the passed script _IN_ the container
+bash "${SCRIPT_DIR}"/run_bro_container.sh
--container-path="${CONTAINER_DIR}" \
+ --container-name=bro-docker-container:latest \
--- End diff --
We should include metron in the container tag.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238844073
--- Diff: docker/example_script.sh ---
@@ -0,0 +1,177 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+CREATED_NETWORK_FLAG=false
+RAN_ZK_CONTAINER=false
+RAN_KAFKA_CONTAINER=false
+CREATED_BRO_CONTAINER=false
+RAN_BRO_CONTAINER=false
+
+SKIP_REBUILD_BRO=false
+LEAVE_RUNNING=false
+
+SCRIPT_DIR=./scripts
+CONTAINER_DIR=./containers/bro-localbuild-container
+CONTAINER_NAME=
+LOG_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && cd logs
&& pwd )"
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-docker-build Skip build of bro docker
machine."
+ echo "--leave-running Do not stop containers after
script. The cleanup_containers.sh script should be run when done."
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+function shutdown {
+
+ if [[ "$RAN_BRO_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=bro
+ fi
+
+ if [[ "$RAN_KAFKA_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=kafka
+ fi
+
+ if [[ "$RAN_ZK_CONTAINER" = true ]]; then
+"${SCRIPT_DIR}"/stop_container.sh --container-name=zookeeper
+ fi
+
+ if [[ "$CREATED_NETWORK_FLAG" = true ]]; then
+"${SCRIPT_DIR}"/destroy_docker_network.sh --network-name=bro-network
+ fi
+}
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --skip-docker-build
+ #
+ --skip-docker-build)
+ SKIP_REBUILD_BRO=true
+ shift # past argument
+ ;;
+
+ #
+ # LEAVE_RUNNING
+ #
+ # --leave-running
+ #
+--leave-running)
+LEAVE_RUNNING=true
+shift # past argument
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+ esac
+done
+EXTRA_ARGS="$@"
+echo "Running with "
+echo "SKIP_REBUILD_BRO = $SKIP_REBUILD_BRO"
+echo "==="
+
+# create the network
+bash "${SCRIPT_DIR}"/create_docker_network.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+ shutdown
+ exit ${rc}
+else
+ CREATED_NETWORK_FLAG=true
+fi
+
+
+
+# run the zookeeper container
+bash "${SCRIPT_DIR}"/run_zookeeper_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+ shutdown
+ exit ${rc}
+else
+ RAN_ZK_CONTAINER=true
+fi
+
+# run the kafka container
+bash "${SCRIPT_DIR}"/run_kafka_container.sh --network-name=bro-network
+rc=$?; if [[ ${rc} != 0 ]]; then
+ shutdown
+ exit ${rc}
+else
+ RAN_KAFKA_CONTAINER=true
+fi
+
+#build the bro container
+if [[ "$SKIP_REBUILD_BRO" = false ]] ; then
+ bash "${SCRIPT_DIR}"/build_container.sh \
+--container-directory="${CONTAINER_DIR}" \
+--container-name=bro-docker-container:latest
--- End diff --
We should include metron in the container tag.
---
[GitHub] metron-bro-plugin-kafka issue #21: METRON-1911 [WIP] Docker setup for testin...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/21 Is there a reason the scripts in `scripts/` have inconsistent permissions? Should `example_script.sh` have execute, or maybe even just call it `run_test.sh` off the bat? ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238825786
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
--- End diff --
Per our chat, I think we're on the same page - let's just kill the `data`
dir instead
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238824961 --- Diff: docker/run_test.sh.template --- @@ -0,0 +1,199 @@ +#!/usr/bin/env bash --- End diff -- Hmm ok, as long as it's outlined in the future README.md that makes sense ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238822684
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
--- End diff --
github.com/apache/metron-bro-plugin-kafka, since you've added the pcaps
under docker/data
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238822338 --- Diff: docker/in_docker_scripts/wait-for-it.sh --- @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# Use this script to test if a given TCP host/port are available + +cmdname=$(basename $0) --- End diff -- Agreed, since it isn't ours. ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238821774 --- Diff: docker/scripts/process_data_docker.sh --- @@ -0,0 +1,29 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +echo "exectuting proces_data_dir.sh in the bro docker container" --- End diff -- "executing process_data_dir.sh" ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794511 --- Diff: docker/logs/bro-test-Tue_Dec__4_12:10:42_EST_2018.log --- @@ -0,0 +1,2 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238797889
--- Diff: docker/scripts/create_docker_network.sh ---
@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--network-name [REQUIRED] The docker network
name"
+ echo "-h/--help Usage information."
+ echo " "
+}
+
+NETWORK_NAME=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # NETWORK_NAME
+ #
+ #
+ #
+--network-name=*)
+NETWORK_NAME="${i#*=}"
+shift # past argument=value
+ ;;
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$NETWORK_NAME" ]]; then
+ echo "NETWORK_NAME must be passed"
+ exit 1
+fi
+
+docker network create "${NETWORK_NAME}"
+rc=$?; if [[ ${rc} != 0 ]]; then
+exit ${rc};
--- End diff --
No need for `;`, also should indent
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798965
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+ wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ${DATA_PATH}/nitroba/nitroba.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then
+ wget https://www.bro.org/static/traces/ssh.pcap -O
${DATA_PATH}/ssh/ssh.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ftp/ftp.pcap ]]; then
+ wget
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ${DATA_PATH}/ftp/ftp.pcap
+fi
+if [[ ! -f ${DATA_PATH}/radius/radius_localhost.pcapng ]]; then
+ wget
https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
-O ${DATA_PATH}/radius/radius_localhost.pcapng
--- End diff --
We should include this as a pcap, not pcapng. Run `editcap -F libpcap
radius_localhost.pcapng radius_localhost.pcap` on the pcap, then replace it in
the repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794488 --- Diff: docker/logs/bro-test-Tue_Dec__4_12:12:39_EST_2018.log --- @@ -0,0 +1,3 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238796266
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
--- End diff --
Should we also capture the rc and exit if != 0?
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798480
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+ wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ${DATA_PATH}/nitroba/nitroba.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then
+ wget https://www.bro.org/static/traces/ssh.pcap -O
${DATA_PATH}/ssh/ssh.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ftp/ftp.pcap ]]; then
+ wget
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ${DATA_PATH}/ftp/ftp.pcap
+fi
+if [[ ! -f ${DATA_PATH}/radius/radius_localhost.pcapng ]]; then
+ wget
https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
-O ${DATA_PATH}/radius/radius_localhost.pcapng
+fi
+if [[ ! -f ${DATA_PATH}/rfb/rfb.pcap ]]; then
+ wget
https://github.com/kholia/my-pcaps/blob/master/VNC/07-vnc-openwall-3.7.pcap?raw=true
-O ${DATA_PATH}/rfb/rfb.pcap
--- End diff --
Repoint to our repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238795456
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
--- End diff --
Just for clarity, would you mind changing this to `./code`?
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238795569
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro -N Apache::Kafka | tee v
--- End diff --
`s/v/"${RUN_LOG_PATH}"/`
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798495
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+ wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ${DATA_PATH}/nitroba/nitroba.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then
+ wget https://www.bro.org/static/traces/ssh.pcap -O
${DATA_PATH}/ssh/ssh.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ftp/ftp.pcap ]]; then
+ wget
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ${DATA_PATH}/ftp/ftp.pcap
+fi
+if [[ ! -f ${DATA_PATH}/radius/radius_localhost.pcapng ]]; then
+ wget
https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
-O ${DATA_PATH}/radius/radius_localhost.pcapng
--- End diff --
Repoint to our repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238797994
--- Diff: docker/scripts/destroy_docker_network.sh ---
@@ -0,0 +1,71 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--network-name [REQUIRED] The docker network
name"
+ echo "-h/--help Usage information."
+ echo " "
+}
+
+NETWORK_NAME=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # NETWORK_NAME
+ #
+ #
+ #
+--network-name=*)
+NETWORK_NAME="${i#*=}"
+shift # past argument=value
+ ;;
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$NETWORK_NAME" ]]; then
+ echo "NETWORK_NAME must be passed"
+ exit 1
+fi
+
+docker network rm "${NETWORK_NAME}"
+rc=$?; if [[ ${rc} != 0 ]]; then
+exit ${rc};
--- End diff --
No need for `;`, also should indent
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798567
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+ wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ${DATA_PATH}/nitroba/nitroba.pcap
--- End diff --
Repoint to our repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798518
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+ wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ${DATA_PATH}/nitroba/nitroba.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then
+ wget https://www.bro.org/static/traces/ssh.pcap -O
${DATA_PATH}/ssh/ssh.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ftp/ftp.pcap ]]; then
+ wget
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ${DATA_PATH}/ftp/ftp.pcap
--- End diff --
Repoint to our repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798536
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
+fi
+if [[ ! -f ${DATA_PATH}/nitroba/nitroba.pcap ]]; then
+ wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ${DATA_PATH}/nitroba/nitroba.pcap
+fi
+if [[ ! -f ${DATA_PATH}/ssh/ssh.pcap ]]; then
+ wget https://www.bro.org/static/traces/ssh.pcap -O
${DATA_PATH}/ssh/ssh.pcap
--- End diff --
Repoint to our repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238798456
--- Diff: docker/scripts/download_sample_pcaps.sh ---
@@ -0,0 +1,101 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--data-paththe pcap data path"
+ echo "-h/--help Usage information."
+ echo " "
+ echo " "
+}
+
+DATA_PATH=
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # DATA_PATH
+ #
+ #
+ --data-path=*)
+ DATA_PATH="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+if [[ -z "$DATA_PATH" ]]; then
+ echo "DATA_PATH must be passed"
+ exit 1
+fi
+
+echo "Running with "
+echo "DATA_PATH = $DATA_PATH"
+echo "==="
+
+for folder in nitroba example-traffic ssh ftp radius rfb; do
+ if [[ ! -d ${DATA_PATH}/${folder} ]]; then
+mkdir -p ${DATA_PATH}/${folder}
+ fi
+done
+
+if [[ ! -f ${DATA_PATH}/example-traffic/exercise-traffic.pcap ]]; then
+ wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
${DATA_PATH}/example-traffic/exercise-traffic.pcap
--- End diff --
Repoint to our repo.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238797481 --- Diff: docker/scripts/consume_bro_kafka.sh --- @@ -0,0 +1,22 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch +docker run --rm --network bro-network ches/kafka \ +kafka-console-consumer.sh --topic bro --from-beginning --bootstrap-server kafka:9092 --- End diff -- Probably indent to make it more clear it's a multiline command. ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794358 --- Diff: docker/logs/bro-test-Tue_Dec__4_11:58:05_EST_2018.log --- @@ -0,0 +1,2 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794286 --- Diff: docker/logs/bro-test-Tue_Dec__4_11:52:34_EST_2018.log --- @@ -0,0 +1,2 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238793284 --- Diff: docker/in_docker_scripts/process_data_dir.sh --- @@ -0,0 +1,32 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +shopt -s nocasematch + +cd /root || exit 1 +echo "" >> $RUN_LOG_PATH 2>&1 + +if [ ! -d /root/data ] ; then + echo "DATA_PATH has not been set and mapped" >> $RUN_LOG_PATH 2&>1 + exit 1 +fi + +ls /root/data --- End diff -- Is this needed? ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794457 --- Diff: docker/logs/bro-test-Tue_Dec__4_13:26:18_EST_2018.log --- @@ -0,0 +1,8 @@ +Running unit tests for "/root/code" --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238796550
--- Diff: docker/in_docker_scripts/wait-for-it.sh ---
@@ -0,0 +1,161 @@
+#!/usr/bin/env bash
+# Use this script to test if a given TCP host/port are available
+
+cmdname=$(basename $0)
--- End diff --
Suggest `cmdname="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null &&
pwd )"` as a general good practice. Probably no specific need if this works,
since it's a somewhat closed environment.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794421 --- Diff: docker/logs/bro-test-Tue_Dec__4_13:36:33_EST_2018.log --- @@ -0,0 +1,8 @@ +Running unit tests for "/root/code" --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794676 --- Diff: docker/in_docker_scripts/wait-for-it.sh --- @@ -0,0 +1,161 @@ +#!/usr/bin/env bash +# Use this script to test if a given TCP host/port are available --- End diff -- Is this from https://github.com/vishnubob/wait-for-it ? ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r23879 --- Diff: docker/logs/bro-test-Tue_Dec__4_13:28:12_EST_2018.log --- @@ -0,0 +1,7 @@ +Running unit tests for "/root/code" --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794530 --- Diff: docker/logs/bro-test-Tue_Dec__4_12:03:04_EST_2018.log --- @@ -0,0 +1,4 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794306 --- Diff: docker/logs/bro-test-Tue_Dec__4_11:54:46_EST_2018.log --- @@ -0,0 +1,2 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238793117
--- Diff: docker/in_docker_scripts/build_bro_plugin.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+cd /root || exit 1
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro-pkg install code --force | tee "${RUN_LOG_PATH}"
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+bro -N Apache::Kafka | tee v
+echo "" >> "${RUN_LOG_PATH}" 2>&1
+
+echo "@load packages" >> /usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG, Conn::LOG,
DPD::LOG, FTP::LOG, Files::LOG, Known::CERTS_LOG, SMTP::LOG, SSL::LOG,
Weird::LOG, Notice::LOG, DHCP::LOG, SSH::LOG, Software::LOG, RADIUS::LOG,
X509::LOG, Known::DEVICES_LOG, RFB::LOG, Stats::LOG, CaptureLoss::LOG,
SIP::LOG);/'" >> /usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::topic_name = \"bro\";" >>
/usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::tag_json = T;" >>
/usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::kafka_conf = table([\"metadata.broker.list\"] =
\"kafka:9092\");" >> /usr/local/bro/share/bro/site/local.bro
+echo "redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);" >>
/usr/local/bro/share/bro/site/local.bro
+echo "redef Known::cert_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro
+echo "redef Software::asset_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro
+sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro'
/usr/local/bro/share/bro/site/local.bro
--- End diff --
We should probably break this up. Lines 37-39 are just to make sure that
bro will generate a larger set of output logs. It's just a good initial
configuration of bro, whereas the other lines are specific to loading the
plugin package.
---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794331 --- Diff: docker/logs/bro-test-Tue_Dec__4_12:01:46_EST_2018.log --- @@ -0,0 +1,2 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238795001 --- Diff: docker/run_test.sh.template --- @@ -0,0 +1,199 @@ +#!/usr/bin/env bash --- End diff -- Do we need this file? ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238794481 --- Diff: docker/logs/bro-test-Tue_Dec__4_13:18:12_EST_2018.log --- @@ -0,0 +1,2 @@ +Apache::Kafka - Writes logs to Kafka (dynamic, version 0.3) --- End diff -- Remove ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238430863 --- Diff: docker/in_docker_scripts/build_bro.sh --- @@ -0,0 +1,31 @@ +#!/usr/bin/env bash --- End diff -- Shouldn't we rename the script, since this is building the plugin not bro. ---
[GitHub] metron-bro-plugin-kafka pull request #21: METRON-1911 [WIP] Docker setup for...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/21#discussion_r238429098
--- Diff: docker/containers/bro-localbuild-container/Dockerfile ---
@@ -0,0 +1,49 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+FROM centos:7
+WORKDIR /root
+
+# install development tools
+RUN yum -y groupinstall "Development Tools"
+RUN yum -y install cmake make gcc gcc-c++ flex bison libpcap libpcap-devel
openssl-devel python-devel swig zlib-devel perl cyrus-sasl cyrus-sasl-devel
cyrus-sasl-gssapi git
+
+# install bro
+RUN curl -L https://www.bro.org/downloads/bro-2.5.5.tar.gz | tar xvz
+WORKDIR bro-2.5.5/
+RUN ./configure
+RUN make
+RUN make install
+ENV PATH="${PATH}:/usr/local/bro/bin"
+
+# install pip
+RUN yum -y update
+RUN yum -y install epel-release
+RUN yum -y install python-pip
+RUN pip install --upgrade pip
+
+# install bro-pkg
+RUN pip install bro-pkg
+ENV PATH="${PATH}:/usr/bin"
+RUN bro-pkg autoconfig
+
+# install librdkafka
+RUN curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz |
tar xvz
+WORKDIR librdkafka-0.9.4/
--- End diff --
This and above should be 0.11.5
---
[GitHub] metron issue #1287: METRON-1915: Disable vagrant hostmanager in the developm...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1287 Specifically my issue is that when I attempt to connect to a kafka broker as a part of the bro plugin via `node1:6667` it fails because it's not listening at the IP that hostmanager inserts into `/etc/hosts`. ---
[GitHub] metron-bro-plugin-kafka issue #20: METRON-1910: bro plugin segfaults on src/...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/20 I wish I had more time to work on this, but it may be a while before I could revisit. What I found was that if any of the `delete`s are duplicated it segfaults on exit. From some commit messages it seems that in bro, the destructor is called from the main thread, but `DoFinish` is called from a child thread, so we should keep this cleanup in `DoFinish`. ---
[GitHub] metron pull request #1287: METORN-1915: Disable vagrant hostmanager in the d...
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/1287 METORN-1915: Disable vagrant hostmanager in the development guest ## Contributor Comments This makes node1 resolve to 127.0.0.1 inside of the VM, instead of 192.168.66.121 by not modifying /etc/hosts inside the guest. One example of this is that kafka only listens on 127.0.0.1:6667. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron METRON-1915 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1287.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1287 commit 01ea8b686398a90c9dacd91349eba5f4c612307c Author: Jon Zeolla Date: 2018-11-30T04:52:04Z METORN-1915: Disable vagrant hostmanager in the development guest ---
[GitHub] metron-bro-plugin-kafka pull request #20: METRON-1910: bro plugin segfaults ...
GitHub user JonZeolla opened a pull request:
https://github.com/apache/metron-bro-plugin-kafka/pull/20
METRON-1910: bro plugin segfaults on src/KafkaWriter.cc:72
## Contributor Comments
Bro was segfaulting on the KafkaWriter.cc destructor because it was
attempting to destroy things that didn't exist.
### Testing
Spin up apache/metron full-dev with `vagrant
--ansible-skip-tags="sensor-stubs,solr" up` and then run bro against some pcap,
such as:
```
vagrant ssh
sudo su -
export PATH=$PATH:/usr/local/bro/bin:/usr/hdp/current/kafka-broker/bin
service pcap-replay restart
broctl stop || broctl stop # Handle bro crashes on shutdown
service snort-producer stop
Service snortd stop
Service yaf stop
yum -y install wireshark
sed -i 's/redef Kafka::logs_to_send = .*/redef Kafka::logs_to_send =
set(HTTP::LOG, DNS::LOG, Conn::LOG, DPD::LOG, FTP::LOG, Files::LOG,
Known::CERTS_LOG, SMTP::LOG, SSL::LOG, Weird::LOG, Notice::LOG, DHCP::LOG,
SSH::LOG, Software::LOG, RADIUS::LOG, X509::LOG, Known::DEVICES_LOG, RFB::LOG,
Stats::LOG, CaptureLoss::LOG, SIP::LOG);/'
/usr/local/bro/share/bro/site/local.bro
echo "redef Kafka::logs_to_exclude = set(Conn::LOG, DHCP::LOG);" >>
/usr/local/bro/share/bro/site/local.bro
echo "redef Known::cert_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro
echo "redef Software::asset_tracking = ALL_HOSTS;" >>
/usr/local/bro/share/bro/site/local.bro
sed -i '86 a @load policy/protocols/dhcp/known-devices-and-hostnames.bro'
/usr/local/bro/share/bro/site/local.bro
screen
kafka-console-consumer.sh --zookeeper localhost:2181 --topic bro
# New screen window
for folder in nitroba example-traffic ssh ftp radius rfb; do mkdir -p
~/brotmp/${folder}; done
wget https://www.bro.org/static/traces/exercise-traffic.pcap -O
~/brotmp/example-traffic/exercise-traffic.pcap
wget
http://downloads.digitalcorpora.org/corpora/network-packet-dumps/2008-nitroba/nitroba.pcap
-O ~/brotmp/nitroba/nitroba.pcap
wget https://www.bro.org/static/traces/ssh.pcap -O ~/brotmp/ssh/ssh.pcap
wget
https://github.com/markofu/pcaps/blob/master/PracticalPacketAnalysis/ppa-capture-files/ftp.pcap?raw=true
-O ~/brotmp/ftp/ftp.pcap
wget
https://github.com/EmpowerSecurityAcademy/wireshark/blob/master/radius_localhost.pcapng?raw=true
-O ~/brotmp/radius/radius_localhost.pcapng
wget
https://github.com/kholia/my-pcaps/blob/master/VNC/07-vnc-openwall-3.7.pcap?raw=true
-O ~/brotmp/rfb/rfb.pcap
cd ~/brotmp/example-traffic
bro -r exercise-traffic.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/nitroba
bro -r nitroba.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ssh
bro -r ssh.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/ftp
bro -r ftp.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/radius
editcap -F libpcap radius_localhost.pcapng radius_localhost.pcap
bro -r radius_localhost.pcap /usr/local/bro/share/bro/site/local.bro -C
cd ~/brotmp/rfb
bro -r rfb.pcap /usr/local/bro/share/bro/site/local.bro -C
```
Notice that after each pcap is done processing it doesn't segfault.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron's Bro kafka writer
plugin.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [X] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [X] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been
executed via:
```
bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH
```
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Apache
[GitHub] metron issue #1283: METRON-1909: Remove http filter from release utils chang...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1283 +1 by inspection, the current approach works well enough for me ---
[GitHub] metron pull request #1282: METRON-1881: Update the bro plugin used in metron...
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/1282 METRON-1881: Update the bro plugin used in metron development ## Contributor Comments Once the metron-bro-plugin-kafka v0.3 is released, this will move full-dev to use it. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron METRON-1881 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1282.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1282 commit 19c33089ce6a61a783fdd43f5d4c0fda1fce5a57 Author: Jon Zeolla Date: 2018-11-18T12:41:45Z METRON-1881: Update the bro plugin used in metron development ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1261 @mmiklavc probably at some point. I assume it would require some cleanup for various scripts at the same time. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236335016
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag'
${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
Hmm, I successfully executed it locally. Did you do it on the awk? It
should only get applied to the grep.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236267508
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
--- End diff --
I would have thought this would only provide back the value, filtering out
the option.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236253176 --- Diff: metron-deployment/development/centos6_docker_build/README.md --- @@ -0,0 +1,89 @@ + +Metron on CentOS 6 Built in Docker +== + +This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6. +It utilizes Vagrant for the virtual machine, and Docker for the build and deployment. Therefore lessens the burden on the user to have the correct versions of the build and deployment tools in order to try Metron. + +Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. --- End diff -- I'd suggest at most 2 spaces between sentences, if not one. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236255405
--- Diff:
metron-deployment/development/centos6_docker_build/ansible/inventory/group_vars/all
---
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# only need to build the RPMs for CentOS
+metron_build_packages_cmd: "shell cd {{ metron_build_dir
}}/metron-deployment && mvn clean package -DskipTests -Pbuild-rpms"
--- End diff --
I see this is in our current ubuntu and centos `all` files, but what is
shell, is this an ansible thing?
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251472
--- Diff:
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
--- End diff --
```
In run_docker_only.sh line 23:
echo ${ANSIBLE_PATH}
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236257483 --- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh --- @@ -0,0 +1,46 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +# +# This script runs IN the docker container +# + +cd /root/metron || exit 1 + +# make sure we have the right c++ tools +source /opt/rh/devtoolset-6/enable + +# give the option to skip building metron, in case they have already done so +#read -p " build metron? [yN] " -n 1 -r +#echo +#if [[ $REPLY =~ ^[Yy]$ ]]; then +# USE TAGS +#fi --- End diff -- We should clean up lines 30-35. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236249643
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag'
${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
--- End diff --
`s/"^\s/"^\\s/`
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236250814 --- Diff: metron-deployment/development/centos6_docker_build/docker_run_ansible.sh --- @@ -0,0 +1,46 @@ +#!/usr/bin/env bash + +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +# +# This script runs IN the docker container +# + +cd /root/metron || exit 1 + +# make sure we have the right c++ tools +source /opt/rh/devtoolset-6/enable --- End diff -- Consider adding `# shellcheck disable=SC1091` prior. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236250204
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
+LOG_DATE=${DATE// /_}
+LOGNAME="metron-build-${LOG_DATE}.log"
+
+# get the node1 ip address so we can add it to the docker hosts
+NODE1_IP=$(awk '/^\s*hosts/{flag=1; next} /}]/{flag=0} flag'
${VAGRANT_PATH}/Vagrantfile | grep "^\s*ip:" | awk -F'"' '{print $2}')
+if [[ -z "${NODE1_IP}" ]]; then exit 1; fi
+echo "Using NODE1 IP ${NODE1_IP}"
+
+echo "===Running Docker==="
+docker run -it \
+ -v ${VAGRANT_PATH}/../../..:/root/metron \
+ -v ~/.m2:/root/.m2 \
+ -v ${VAGRANT_PATH}:/root/vagrant \
+ -v ${ANSIBLE_PATH}:/root/ansible_config \
+ -v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
+ -v ${VAGRANT_PATH}/logs:/root/logs \
+ -e ANSIBLE_CONFIG='/root/ansible_config/ansible.cfg' \
+ -e ANSIBLE_LOG_PATH="/root/logs/${LOGNAME}" \
+
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236256548
--- Diff:
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
--- End diff --
Is this still useful?
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236252613 --- Diff: metron-deployment/development/centos6_docker_build/README.md --- @@ -0,0 +1,89 @@ + +Metron on CentOS 6 Built in Docker +== + +This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6. +It utilizes Vagrant for the virtual machine, and Docker for the build and deployment. Therefore lessens the burden on the user to have the correct versions of the build and deployment tools in order to try Metron. + +Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. + +Getting Started +--- + +### Prerequisites + +The computer used to deploy Apache Metron will need to have the following components installed. + + - [Docker](https://www.docker.com/community-edition) + - [Vagrant](https://www.vagrantup.com) 2.0+ + - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) + - [Virtualbox](https://virtualbox.org) 5.0+ + +Running the following script can help validate whether you have all the prerequisites installed and running correctly. + + ``` + metron-deployment/scripts/platform-info.sh + ``` + + How do I install these on MacOS? + +Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. + +1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/). + +1. Run the following command in a terminal to install all of the required tools. + +``` +brew cask install vagrant virtualbox docker +vagrant plugin install vagrant-hostmanager +open /Applications/Docker.app +``` + +### Deploy Metron + +1. Ensure that the Docker service is running. + +1. Deploy Metron + + ```bash +cd metron-deployment/development/centos6_docker +./build_and_run.sh -h + ``` + ```bash +usage: ./build_and_run.sh +--skip-vagrant-up skip vagrant up +--force-docker-buildforce build docker machine +--skip-tags='tag,tag2,tag3' the ansible skip tags +-h/--help Usage information. + +example: to skip vagrant up and force docker build with two tags +build_and_run.sh -skip-vagrant-up --force-docker-build --skip-tags='solr,sensors' --- End diff -- `s/-skip-vagrant-up/--skip-vagrant-up/` ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236251967 --- Diff: metron-deployment/development/docker/README.md --- @@ -45,12 +45,4 @@ This provides a Docker Container containing all of the prerequisites required to ``` cd /root/metron mvn clean package -DskipTests -``` --- End diff -- This should be reinstated. ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1261#discussion_r236252426 --- Diff: metron-deployment/development/centos6_docker_build/README.md --- @@ -0,0 +1,89 @@ + +Metron on CentOS 6 Built in Docker +== + +This project fully automates the provisioning and deployment of Apache Metron and all necessary prerequisites on a single, virtualized host running CentOS 6. +It utilizes Vagrant for the virtual machine, and Docker for the build and deployment. Therefore lessens the burden on the user to have the correct versions of the build and deployment tools in order to try Metron. + +Metron is composed of many components and installing all of these on a single host, especially a virtualized one, will greatly stress the resources of the host. The host will require at least 8 GB of RAM and a fair amount of patience. It is highly recommended that you shut down all unnecessary services. + +Getting Started +--- + +### Prerequisites + +The computer used to deploy Apache Metron will need to have the following components installed. + + - [Docker](https://www.docker.com/community-edition) + - [Vagrant](https://www.vagrantup.com) 2.0+ + - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) + - [Virtualbox](https://virtualbox.org) 5.0+ + +Running the following script can help validate whether you have all the prerequisites installed and running correctly. + + ``` + metron-deployment/scripts/platform-info.sh + ``` + + How do I install these on MacOS? + +Any platform that supports these tools is suitable, but the following instructions cover installation on macOS. The easiest means of installing these tools on a Mac is to use the excellent [Homebrew](http://brew.sh/) project. + +1. Install Homebrew by following the instructions at [Homebrew](http://brew.sh/). + +1. Run the following command in a terminal to install all of the required tools. + +``` +brew cask install vagrant virtualbox docker +vagrant plugin install vagrant-hostmanager +open /Applications/Docker.app +``` + +### Deploy Metron + +1. Ensure that the Docker service is running. + +1. Deploy Metron + + ```bash +cd metron-deployment/development/centos6_docker --- End diff -- `s/centos6_docker/centos6_docker_build/` ---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236249808
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
+ echo "Error: unknown option: $UNKNOWN_OPTION"
+ help
+ ;;
+ esac
+done
+
+echo "Running with "
+echo "SKIP_VAGRANT_UP= $SKIP_VAGRANT_UP"
+echo "FORCE_DOCKER_BUILD = $FORCE_DOCKER_BUILD"
+echo "SKIP_TAGS = $A_SKIP_TAGS"
+echo "==="
+
+if [[ "$SKIP_VAGRANT_UP" = false ]]; then
+ vagrant up
+ rc=$?; if [[ $rc != 0 ]]; then exit $rc; fi
+fi
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+VAGRANT_KEY_PATH=${VAGRANT_PATH}/.vagrant/machines/node1/virtualbox
+
+# move over to the docker area
+cd ../docker || exit 1
+
+# Give the option to not build the docker container, which can take some
time and not be necessary
+if [[ "$FORCE_DOCKER_BUILD" = true ]]; then
+ echo "docker build"
+ docker build -t metron-build-docker:latest .
+fi
+
+if [[ ! -d ~/.m2 ]]; then
+ mkdir ~/.m2
+fi
+
+DATE=`date`
--- End diff --
Shellcheck prefers `$(date)` ¯\_(ã)_/¯
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236254476
--- Diff:
metron-deployment/development/centos6_docker_build/ansible/inventory/group_vars/all
---
@@ -0,0 +1,86 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# only need to build the RPMs for CentOS
+metron_build_packages_cmd: "shell cd {{ metron_build_dir
}}/metron-deployment && mvn clean package -DskipTests -Pbuild-rpms"
+
+# ambari
+ambari_host: "{{ groups.ambari_master[0] }}"
+hdp_host_group: "{{ groups.ambari_slave }}"
+ambari_port: 8080
+ambari_user: admin
+ambari_password: admin
+cluster_type: single_node_vm
+ambari_server_mem: 512
+java_home: /usr/jdk64/jdk1.8.0_77
+
+# hbase
+pcap_hbase_table: pcap
+tracker_hbase_table: access_tracker
+threatintel_hbase_table: threatintel
+enrichment_hbase_table: enrichment
+
+# metron
+metron_version: 0.6.1
--- End diff --
Just bringing awareness that this may get changed to `0.7.0` prior to the
next release. It's possible this could get missed depending on the order of
operations.
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251269
--- Diff:
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
--- End diff --
Shellcheck prefers `$(pwd)` ¯_(ã)_/¯
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236251078
--- Diff:
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
--- End diff --
From shellcheck
```
In run_docker_only.sh line 26:
echo ${VAGRANT_KEY_PATH}
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236247445
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
+ echo " "
+}
+
+SKIP_VAGRANT_UP=false
+FORCE_DOCKER_BUILD=false
+A_SKIP_TAGS="sensors,solr"
+
+# handle command line options
+for i in "$@"; do
+ case $i in
+ #
+ # SKIP_VAGRANT_UP
+ #
+ #
+ --skip-vagrant-up)
+ SKIP_VAGRANT_UP=true
+ shift # past argument
+ ;;
+
+ #
+ # FORCE_DOCKER_BUILD
+ #
+ # --force-docker-build
+ #
+ --force-docker-build)
+ FORCE_DOCKER_BUILD=true
+ shift # past argument
+ ;;
+
+ #
+ # SKIP_TAGS
+ #
+ # --skip-tags='foo,bar'
+ #
+ --skip-tags=*)
+ A_SKIP_TAGS="${i#*=}"
+ shift # past argument=value
+ ;;
+
+ #
+ # -h/--help
+ #
+ -h|--help)
+ help
+ exit 0
+ shift # past argument with no value
+ ;;
+
+ #
+ # Unknown option
+ #
+ *)
+ UNKNOWN_OPTION="${i#*=}"
--- End diff --
Why `#*=`?
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236250996
--- Diff:
metron-deployment/development/centos6_docker_build/run_docker_only.sh ---
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+VAGRANT_PATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
+echo "setting the ansible configuration path"
+ANSIBLE_PATH=${VAGRANT_PATH}/ansible
+echo ${ANSIBLE_PATH}
+echo "setting the ssh key"
+VAGRANT_KEY_PATH=`pwd`/.vagrant/machines/node1/virtualbox
+echo ${VAGRANT_KEY_PATH}
+
+# move over to the docker area
+cd ../docker || exit 1
+pwd
+
+echo "===Running Docker==="
+docker run -it \
+ -v ${VAGRANT_PATH}/../../..:/root/metron \
+ -v ~/.m2:/root/.m2 \
+ -v ${VAGRANT_PATH}:/root/vagrant \
+ -v ${ANSIBLE_PATH}:/root/ansible_config \
+ -v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
--- End diff --
From shellcheck.
```
In run_docker_only.sh line 34:
-v ${VAGRANT_PATH}/../../..:/root/metron \
^-- SC2086: Double quote to prevent globbing and word splitting.
In run_docker_only.sh line 36:
-v ${VAGRANT_PATH}:/root/vagrant \
^-- SC2086: Double quote to prevent globbing and word splitting.
In run_docker_only.sh line 37:
-v ${ANSIBLE_PATH}:/root/ansible_config \
^-- SC2086: Double quote to prevent globbing and word splitting.
In run_docker_only.sh line 38:
-v ${VAGRANT_KEY_PATH}:/root/vagrant_key \
^-- SC2086: Double quote to prevent globbing and word splitting.
```
---
[GitHub] metron pull request #1261: METRON-1860 [WIP] new developer option for ansibl...
Github user JonZeolla commented on a diff in the pull request:
https://github.com/apache/metron/pull/1261#discussion_r236253505
--- Diff:
metron-deployment/development/centos6_docker_build/build_and_run.sh ---
@@ -0,0 +1,144 @@
+#!/usr/bin/env bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+shopt -s nocasematch
+
+function help {
+ echo " "
+ echo "usage: ${0}"
+ echo "--skip-vagrant-up skip vagrant up"
+ echo "--force-docker-buildforce build docker machine"
+ echo "--skip-tags='tag,tag2,tag3' the ansible skip tags"
+ echo "-h/--help Usage information."
+ echo " "
+ echo "example: to skip vagrant up and force docker build with two tags"
+ echo " build_and_run.sh -skip-vagrant-up --force-docker-build
--skip-tags='solr,sensors'"
--- End diff --
`s/-skip-vagrant-up/--skip-vagrant-up/`
---
[GitHub] metron pull request #1279: METRON-1893 Syslog RFC-3164 parser
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron/pull/1279#discussion_r236054835 --- Diff: metron-platform/metron-parsers/src/main/resources/META-INF/NOTICE --- @@ -37,4 +37,18 @@ Copyright 2006-2011 Google, Inc. Apache Software Foundation that were originally developed at iClick, Inc., software copyright (c) 1999. + (ASLv2) simple-syslog-5424 +The following NOTICE information applies: +simple-syslog-5424 +https://github.com/palindromicity/simple-syslog-5424 + +Copyright 2018 simple-syslog-5424 authors. + + (ASLv2) simple-syslog-5424 --- End diff -- `s/5424/3164/`? ---
[GitHub] metron issue #1277: METRON-1890 disable audio in vm so it does not grab the ...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1277 +1 by inspection ---
[GitHub] metron issue #1273: METRON-1886 Update to minimum Ansible version in documen...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1273 From what I can tell, `include_tasks` was [added in 2.4.0](https://github.com/ansible/ansible/blob/stable-2.4/CHANGELOG.md#major-changes) and the patch versions only include bugfixes. Can you point me to an error message that you're seeing, or a change in ansible that we need to make sure is included? ---
[GitHub] metron-bro-plugin-kafka pull request #19: METRON-1885: Remove version from b...
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/19 METRON-1885: Remove version from bro plugin btest ## Contributor Comments This changes the show-plugin btest so that it should be compatible with bro 2.5-2.7 (as far as we currently know). It is related to #8. ## Testing An easy way to test this is to use `bro-pkg` to install the plugin (which runs all of the tests by default), or you can go on a properly configured box (wrt `btest`) and run `cd metron-bro-plugin-kafka/tests/ && btest -d`. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron's Bro kafka writer plugin. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [X] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [X] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Apache Metron's [Vagrant full-dev environment](https://github.com/apache/metron/tree/master/metron-deployment/development/centos6) or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1885 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/19.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #19 commit 241d2261052715d939ba41728f898611a00146fc Author: Jon Zeolla Date: 2018-11-19T00:04:33Z METRON-1885: Remove version from bro plugin btest ---
[GitHub] metron-bro-plugin-kafka issue #18: Remove version from show-plugin btest out...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/18 Sounds good, I'm inclined to accept but there is a little bit of housekeeping to take care of first. Can you please take a look the "For all changes" section of our PR template and let me know if you have any questions? ---
[GitHub] metron-bro-plugin-kafka issue #18: Remove version from show-plugin btest out...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/18 So, I did put together a PR meant to handle this @ metron-bro-plugin-kafka#8 Do you happen to know if your approach is more generally accepted? It feels somewhat strange to say "version" without providing a version number at the very least, but I could see the benefits of having a show-plugin btest that works pre- and post- 2.7. ---
[GitHub] metron issue #1237: METRON-1825: Upgrade bro to 2.5.5
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1237 Yup, that was actually how I found METRON-1826 was an issue. Sorry, I should have noted that. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1261 If we'd want to replace full dev we would need to get skip tags passed in appropriately, I use that a lot. That said I'm not 100% that we need to do that all at once. ---
[GitHub] metron issue #1261: METRON-1860 [WIP] new developer option for ansible in do...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1261 I'm going to take a stab at a further look next week. For now I gave it a quick run-up and it was successful. ---
[GitHub] metron issue #1260: METRON-1868: metron-committer-common incorrectly checkin...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1260 If you ran `prepare_commit` and selected the metron-bro-plugin-kafka repo, when you get to `select_code` it would default to metron. ---
[GitHub] metron pull request #1260: METRON-1868: metron-committer-common incorrectly ...
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron/pull/1260 METRON-1868: metron-committer-common incorrectly checking CHOSEN_REPO ## Contributor Comments I found a minor bug when trying to use `prepare-commit` against the metron-bro-plugin-kafka repo. This should fix that. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [X] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [X] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [X] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [X] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [X] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron METRON-1868 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1260.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1260 commit f5a659707fe978af9ef365897db3fbcbe3b63f96 Author: Jon Zeolla Date: 2018-11-13T15:00:38Z METRON-1868: metron-committer-common incorrectly checking CHOSEN_REPO ---
[GitHub] metron-bro-plugin-kafka issue #13: METRON-1827: Update librdkafka in metron-...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/13 Ok, I've deconflicted and this one is ready for review. ---
[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1171 Yup, my testing was all successful. I didn't review it thoroughly enough to suggest it gets merged as is but I have no reason to hold it back. ---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/17#discussion_r232435113 --- Diff: README.md --- @@ -12,24 +12,72 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + +`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation. + +1. Install [librdkafka](https://github.com/edenhill/librdkafka), a native client library for Kafka. This plugin has been tested against the latest release of librdkafka, which at the time of this writing is v0.9.4. + +In order to use this plugin within a kerberized Kafka environment, you will also need `libsasl2` installed and will need to pass `--enable-sasl` to the `configure` script. + +``` +$ curl -L https://github.com/edenhill/librdkafka/archive/v0.9.4.tar.gz | tar xvz +$ cd librdkafka-0.9.4/ +$ ./configure --enable-sasl +$ make +$ sudo make install +``` + +1. Configure `bro-pkg` by following the quickstart guide [here](https://bro-package-manager.readthedocs.io/en/stable/quickstart.html). + +1. Install the plugin using `bro-pkg install`. + +``` +$ bro-pkg install apache/metron-bro-plugin-kafka --version master +The following packages will be INSTALLED: + bro/apache/metron-bro-plugin-kafka (master) + +Verify the following REQUIRED external dependencies: +(Ensure their installation on all relevant systems before proceeding): + from bro/apache/metron-bro-plugin-kafka (master): +librdkafka ~0.9.4 + +Proceed? [Y/n] +bro/apache/metron-bro-plugin-kafka asks for LIBRDKAFKA_ROOT (Path to librdkafka installation tree) ? [/usr/local/lib] +Saved answers to config file: /home/jonzeolla/.bro-pkg/config +Running unit tests for "bro/apache/metron-bro-plugin-kafka" +all 10 tests successful + + +Installing "bro/apache/metron-bro-plugin-kafka" +Installed "bro/apache/metron-bro-plugin-kafka" (master) +Loaded "bro/apache/metron-bro-plugin-kafka" +``` + --- End diff -- Sounds good, added a final step to validate manually. Initially, I left that out since `bro-pkg` does a very similar test, among others ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 Okay, pushed that fix and merged in master. ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 Yes that's why I said it depends on #16 and mentioned the order of review should be #16 -> #2 -> #17 -> #13 ---
[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1171 Yes, I outlined the specifics in my prior comments ---
[GitHub] metron issue #1171: METRON-1740 make parser support CONFIG and SYSTEM log ty...
Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/1171 +0 only performed functional testing. ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r232405844 --- Diff: README.md --- @@ -11,6 +11,32 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + --- End diff -- As a part of the review process, this improvement was split into #17 ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r232405772 --- Diff: README.md --- @@ -42,22 +68,47 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec The following examples highlight different ways that the plugin can be used. Simply add the Bro script language to your `local.bro` file (for example, `/usr/share/bro/site/local.bro`) as shown to demonstrate the example. -### Example 1 +### Example 1 - Send a list of logs to kafka The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`. * Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration table. - * By defining `topic_name` all records will be sent to the same Kafka topic. - * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. + * The `topic_name` will default to send all records to a single Kafka topic called 'bro'. + * Defining `logs_to_send` will send the HTTP and DNS records to the brokers specified in your `Kafka::kafka_conf`. ``` @load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); --- End diff -- As a part of the review process, this improvement was split into #17 ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r232405800 --- Diff: README.md --- @@ -144,23 +194,35 @@ event bro_init() &priority=-5 Notes * `logs_to_send` is mutually exclusive with `$pred`, thus for each log you want to set `$pred` on, you must individually setup a `Log::add_filter` and refrain from including that log in `logs_to_send`. + * In Bro 2.5.x the bro project introduced a [logger function](https://www.bro.org/sphinx/cluster/index.html#logger) which removes the logging functions from the manager thread, and taking advantage of that is highly recommended. If you are running this plugin on Bro 2.4.x, you may encounter issues where the manager thread is taking on too much responsibility and pinning a single CPU core without the ability to spread the load across additional cores. In this case, it may be in your best interest to prefer using a bro logging predicate over filtering in your Metron cluster [using Stellar](https://github.com/apache/metron/tree/master/metron-stellar/stellar-common) in order to lesson the load of that thread. --- End diff -- As a part of the review process, this improvement was split into #17 ---
[GitHub] metron-bro-plugin-kafka pull request #2: METRON-1304: Allow metron-bro-plugi...
Github user JonZeolla commented on a diff in the pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/2#discussion_r232405751 --- Diff: README.md --- @@ -11,6 +11,32 @@ This software is a part of the [Apache Metron](http://metron.apache.org/) projec ## Installation +### `bro-pkg` Installation + +`bro-pkg` is the preferred mechanism for installing this plugin, as it will dynamically retrieve, build, test, and load the plugin. Note, that you will still need to [activate](#activation) and configure the plugin after your installation. + --- End diff -- As a part of the review process, this improvement was split into #17 ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 Suggested order of review: #16 -> #2 -> #17 -> #13 ---
[GitHub] metron-bro-plugin-kafka pull request #17: METRON-1866: Improve metron-bro-pl...
GitHub user JonZeolla opened a pull request: https://github.com/apache/metron-bro-plugin-kafka/pull/17 METRON-1866: Improve metron-bro-plugin-kafka documentation ## Contributor Comments This depends on METRON-1304. Pure documentation change You can merge this pull request into a Git repository by running: $ git pull https://github.com/JonZeolla/metron-bro-plugin-kafka METRON-1866 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron-bro-plugin-kafka/pull/17.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #17 commit d0e59108238d7f36c707e7f44a0f00bbbdd73c74 Author: Jon Zeolla Date: 2018-11-09T21:49:01Z METRON-1866 ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 Note that this one now depends on apache/metron-bro-plugin-kafka#16 to work properly because I split out the btest bugfix. ---
[GitHub] metron-bro-plugin-kafka issue #2: METRON-1304: Allow metron-bro-plugin-kafka...
Github user JonZeolla commented on the issue: https://github.com/apache/metron-bro-plugin-kafka/pull/2 Good to go on this one @nickwallen ---
