[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection URL: https://github.com/apache/metron-bro-plugin-kafka/pull/36#discussion_r323012133 ## File path: tests/Baseline/kafka.resolved-topic-override-and-config/output ## @@ -0,0 +1,2 @@ +Kafka topic set to test-topic +Kafka topic set to xxx-topic Review comment: Maybe it's just me but I would prefer if we use a different placeholder throughout This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for
bro plugin topic_name selection
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/36#discussion_r323011857
##
File path: docker/run_end_to_end.sh
##
@@ -196,58 +206,58 @@ rc=$?; if [[ ${rc} != 0 ]]; then
exit ${rc}
fi
+if [[ "$NO_PCAP" = false ]]; then
+ # for each pcap in the data directory, we want to
+ # run bro then read the output from kafka
+ # and output both of them to the same directory named
+ # for the date/pcap
-# for each pcap in the data directory, we want to
-# run bro then read the output from kafka
-# and output both of them to the same directory named
-# for the date/pcap
+ for file in "${DATA_PATH}"/**/*.pcap*
+ do
+# get the file name
+BASE_FILE_NAME=$(basename "${file}")
+DOCKER_DIRECTORY_NAME=${BASE_FILE_NAME//\./_}
-for file in "${DATA_PATH}"/**/*.pcap*
-do
- # get the file name
- BASE_FILE_NAME=$(basename "${file}")
- DOCKER_DIRECTORY_NAME=${BASE_FILE_NAME//\./_}
+mkdir "${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}" || exit 1
+echo "MADE ${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}"
- mkdir "${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}" || exit 1
- echo "MADE ${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}"
+# get the current offset in kafka
+# this is where we are going to _start_
+OFFSET=$(bash "${SCRIPT_DIR}"/docker_run_get_offset_kafka.sh
--kafka-topic="${KAFKA_TOPIC}" | sed "s/^${KAFKA_TOPIC}:0:\(.*\)$/\1/")
+echo "OFFSET--> ${OFFSET}"
- # get the current offset in kafka
- # this is where we are going to _start_
- OFFSET=$(bash "${SCRIPT_DIR}"/docker_run_get_offset_kafka.sh
--kafka-topic="${KAFKA_TOPIC}" | sed "s/^${KAFKA_TOPIC}:0:\(.*\)$/\1/")
- echo "OFFSET--> ${OFFSET}"
+bash "${SCRIPT_DIR}"/docker_execute_process_data_file.sh
--pcap-file-name="${BASE_FILE_NAME}"
--output-directory-name="${DOCKER_DIRECTORY_NAME}"
+rc=$?; if [[ ${rc} != 0 ]]; then
+ echo "ERROR> FAILED TO PROCESS ${file} DATA. CHECK LOGS, please run the
finish_end_to_end.sh when you are done."
+ exit ${rc}
+fi
- bash "${SCRIPT_DIR}"/docker_execute_process_data_file.sh
--pcap-file-name="${BASE_FILE_NAME}"
--output-directory-name="${DOCKER_DIRECTORY_NAME}"
- rc=$?; if [[ ${rc} != 0 ]]; then
-echo "ERROR> FAILED TO PROCESS ${file} DATA. CHECK LOGS, please run the
finish_end_to_end.sh when you are done."
-exit ${rc}
- fi
+
KAFKA_OUTPUT_FILE="${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}/kafka-output.log"
+bash "${SCRIPT_DIR}"/docker_run_consume_kafka.sh --offset="${OFFSET}"
--kafka-topic="${KAFKA_TOPIC}" | "${ROOT_DIR}"/remove_timeout_message.sh | tee
"${KAFKA_OUTPUT_FILE}"
+
+rc=$?; if [[ ${rc} != 0 ]]; then
+ echo "ERROR> FAILED TO PROCESS ${DATA_PATH} DATA. CHECK LOGS"
+fi
-
KAFKA_OUTPUT_FILE="${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}/kafka-output.log"
- bash "${SCRIPT_DIR}"/docker_run_consume_kafka.sh --offset="${OFFSET}"
--kafka-topic="${KAFKA_TOPIC}" | "${ROOT_DIR}"/remove_timeout_message.sh | tee
"${KAFKA_OUTPUT_FILE}"
+"${SCRIPT_DIR}"/split_kakfa_output_by_log.sh
--log-directory="${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}"
Review comment:
The file also needs to be renamed to fix this typo
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for
bro plugin topic_name selection
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/36#discussion_r323011168
##
File path: tests/kafka/resolved-topic-override-and-config.bro
##
@@ -0,0 +1,37 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# @TEST-EXEC: bro -r ../../../tests/pcaps/exercise-traffic.pcap
../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+
+redef Kafka::logs_to_send = set(Conn::LOG);
+redef Kafka::topic_name = "test-topic";
+redef Kafka::mock = T;
+
+event bro_init() =-10
+{
+local xxx_filter: Log::Filter = [
+$name = "kafka-xxx",
+$writer = Log::WRITER_KAFKAWRITER,
+$path = "kakfa_xxx",
Review comment:
`kafka_`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for
bro plugin topic_name selection
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/36#discussion_r323011602
##
File path: docker/run_end_to_end.sh
##
@@ -196,58 +206,58 @@ rc=$?; if [[ ${rc} != 0 ]]; then
exit ${rc}
fi
+if [[ "$NO_PCAP" = false ]]; then
+ # for each pcap in the data directory, we want to
+ # run bro then read the output from kafka
+ # and output both of them to the same directory named
+ # for the date/pcap
-# for each pcap in the data directory, we want to
-# run bro then read the output from kafka
-# and output both of them to the same directory named
-# for the date/pcap
+ for file in "${DATA_PATH}"/**/*.pcap*
+ do
+# get the file name
+BASE_FILE_NAME=$(basename "${file}")
+DOCKER_DIRECTORY_NAME=${BASE_FILE_NAME//\./_}
-for file in "${DATA_PATH}"/**/*.pcap*
-do
- # get the file name
- BASE_FILE_NAME=$(basename "${file}")
- DOCKER_DIRECTORY_NAME=${BASE_FILE_NAME//\./_}
+mkdir "${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}" || exit 1
+echo "MADE ${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}"
- mkdir "${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}" || exit 1
- echo "MADE ${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}"
+# get the current offset in kafka
+# this is where we are going to _start_
+OFFSET=$(bash "${SCRIPT_DIR}"/docker_run_get_offset_kafka.sh
--kafka-topic="${KAFKA_TOPIC}" | sed "s/^${KAFKA_TOPIC}:0:\(.*\)$/\1/")
+echo "OFFSET--> ${OFFSET}"
- # get the current offset in kafka
- # this is where we are going to _start_
- OFFSET=$(bash "${SCRIPT_DIR}"/docker_run_get_offset_kafka.sh
--kafka-topic="${KAFKA_TOPIC}" | sed "s/^${KAFKA_TOPIC}:0:\(.*\)$/\1/")
- echo "OFFSET--> ${OFFSET}"
+bash "${SCRIPT_DIR}"/docker_execute_process_data_file.sh
--pcap-file-name="${BASE_FILE_NAME}"
--output-directory-name="${DOCKER_DIRECTORY_NAME}"
+rc=$?; if [[ ${rc} != 0 ]]; then
+ echo "ERROR> FAILED TO PROCESS ${file} DATA. CHECK LOGS, please run the
finish_end_to_end.sh when you are done."
+ exit ${rc}
+fi
- bash "${SCRIPT_DIR}"/docker_execute_process_data_file.sh
--pcap-file-name="${BASE_FILE_NAME}"
--output-directory-name="${DOCKER_DIRECTORY_NAME}"
- rc=$?; if [[ ${rc} != 0 ]]; then
-echo "ERROR> FAILED TO PROCESS ${file} DATA. CHECK LOGS, please run the
finish_end_to_end.sh when you are done."
-exit ${rc}
- fi
+
KAFKA_OUTPUT_FILE="${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}/kafka-output.log"
+bash "${SCRIPT_DIR}"/docker_run_consume_kafka.sh --offset="${OFFSET}"
--kafka-topic="${KAFKA_TOPIC}" | "${ROOT_DIR}"/remove_timeout_message.sh | tee
"${KAFKA_OUTPUT_FILE}"
+
+rc=$?; if [[ ${rc} != 0 ]]; then
+ echo "ERROR> FAILED TO PROCESS ${DATA_PATH} DATA. CHECK LOGS"
+fi
-
KAFKA_OUTPUT_FILE="${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}/kafka-output.log"
- bash "${SCRIPT_DIR}"/docker_run_consume_kafka.sh --offset="${OFFSET}"
--kafka-topic="${KAFKA_TOPIC}" | "${ROOT_DIR}"/remove_timeout_message.sh | tee
"${KAFKA_OUTPUT_FILE}"
+"${SCRIPT_DIR}"/split_kakfa_output_by_log.sh
--log-directory="${TEST_OUTPUT_PATH}/${DOCKER_DIRECTORY_NAME}"
Review comment:
`split_kafka_output`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for
bro plugin topic_name selection
URL:
https://github.com/apache/metron-bro-plugin-kafka/pull/36#discussion_r323011538
##
File path: tests/kafka/resolved-topic-override-only.bro
##
@@ -0,0 +1,34 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# @TEST-EXEC: bro -r ../../../tests/pcaps/exercise-traffic.pcap
../../../scripts/Apache/Kafka/ %INPUT > output
+# @TEST-EXEC: btest-diff output
+
+module Kafka;
+
+
+redef Kafka::mock = T;
+event bro_init() =-10
+{
+local xxx_filter: Log::Filter = [
+$name = "kafka-xxx",
+$writer = Log::WRITER_KAFKAWRITER,
+$path = "kakfa_xxx",
Review comment:
`kafka`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron-bro-plugin-kafka] JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
JonZeolla commented on a change in pull request #36: METRON-2069 Add btests for bro plugin topic_name selection URL: https://github.com/apache/metron-bro-plugin-kafka/pull/36#discussion_r323009365 ## File path: docker/run_end_to_end.sh ## @@ -68,6 +69,15 @@ for i in "$@"; do SKIP_REBUILD_BRO=true shift # past argument ;; + # + # NO_PCAP + # + # --skip-docker-build Review comment: `--no-pcap` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to
HBase 2.0.2
URL: https://github.com/apache/metron/pull/1506#discussion_r322977881
##
File path:
metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/adapters/cif/CIFHbaseAdapter.java
##
@@ -92,6 +95,13 @@ protected Map getCIFObject(String key) {
return output;
}
+ private static byte[] getQualifier(Cell cell) {
Review comment:
This is duplicated between this class and AbstractConverter. Might it be
worth an HBaseUtils class for common static methods like this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to
HBase 2.0.2
URL: https://github.com/apache/metron/pull/1506#discussion_r322979944
##
File path: metron-stellar/stellar-common/pom.xml
##
@@ -331,6 +331,10 @@
com.google.common
org.apache.metron.guava.${guava_version}
+
+com.google.thirdparty
Review comment:
Just curious - `com.google.thirdparty` - is that from us somewhere else, or
does Google also do some repackaging that we depend on? Wondering if we're
double-relocating our own deps.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to
HBase 2.0.2
URL: https://github.com/apache/metron/pull/1506#discussion_r322978557
##
File path:
metron-platform/metron-hbase-server/src/main/java/org/apache/metron/hbase/coprocessor/EnrichmentCoprocessor.java
##
@@ -66,7 +69,7 @@
* @see https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html;>https://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/coprocessor/RegionObserver.html
* @see EnrichmentConfigurations Available options.
*/
-public class EnrichmentCoprocessor extends BaseRegionObserver {
+public class EnrichmentCoprocessor implements RegionObserver,
RegionCoprocessor {
Review comment:
The changes for this are a lot smaller than I was anticipating. This is
really great. We have a link to the RegionObserver docs, should we add one to
RegionCoprocessor as well?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to
HBase 2.0.2
URL: https://github.com/apache/metron/pull/1506#discussion_r322980605
##
File path:
metron-platform/metron-hbase/metron-hbase-common/src/test/java/org/apache/metron/hbase/mock/MockHTable.java
##
@@ -44,651 +35,757 @@
import org.apache.hadoop.hbase.client.Durability;
import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.Increment;
+import org.apache.hadoop.hbase.client.Mutation;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.ResultScanner;
import org.apache.hadoop.hbase.client.Row;
import org.apache.hadoop.hbase.client.RowMutations;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.Table;
+import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.client.coprocessor.Batch;
+import org.apache.hadoop.hbase.client.metrics.ScanMetrics;
import org.apache.hadoop.hbase.filter.CompareFilter;
import org.apache.hadoop.hbase.filter.Filter;
import org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel;
import org.apache.hadoop.hbase.util.Bytes;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.NavigableMap;
+import java.util.NavigableSet;
+import java.util.NoSuchElementException;
+import java.util.Set;
+import java.util.concurrent.ConcurrentSkipListMap;
+
/**
* MockHTable.
*
- * This implementation is a selected excerpt from
https://gist.github.com/agaoglu/613217
+ * This implementation is a selected excerpt from
https://gist.github.com/agaoglu/613217 and
+ *
https://github.com/rayokota/hgraphdb/blob/07c551f39a92b7ee2c8b48edcc7c0b314f6c3e33/src/main/java/org/apache/hadoop/hbase/client/mock/MockHTable.java.
*/
public class MockHTable implements Table {
Review comment:
This looks like it was potentially some fun to get working. I'll submit a
separate review for this class as there's a lot to digest. But if our Storm
integration tests are passing with this, I'm pretty confident it will be good
to go.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
mmiklavc commented on a change in pull request #1506: METRON-2188 Upgrade to HBase 2.0.2 URL: https://github.com/apache/metron/pull/1506#discussion_r322981358 ## File path: metron-platform/metron-hbase/metron-hbase-common/pom.xml ## @@ -52,6 +52,10 @@ org.slf4j slf4j-log4j12 + Review comment: Any thoughts, comments, or lessons learned on how you came around to the various exclusions? With our recent maven/pom changes, the process changes slightly. We haven't done the best job sharing some of this dependency tribal knowledge with the broader dev community for Metron, so it's probably worth having some details in this PR that we can use in the future. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen opened a new pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
nickwallen opened a new pull request #1506: METRON-2188 Upgrade to HBase 2.0.2
URL: https://github.com/apache/metron/pull/1506
This upgrades the feature branch `feature/METRON-2088-support-hdp-3.1` to
HBase 2.0.2. The scope of the changes include:
* Replace `HBaseTestingUtility` API references that are no longer supported.
* Replace references of `Put.add` with `Put.addColumn`.
* Replaced references of `Cell.getQualifier` and `Cell.getValue`.
* Update the `MockHTable` to function with HBase 2.0.2.
* Update the Enrichment Coprocessor to reflect changes made in the
coprocessor APIs.
## Acceptance Testing
This PR should be tested using the centos7 development environment.
1. Start up the centos7 dev environment.
```
cd metron-deployment/development/centos7
vagrant destroy -f
vagrant up
```
### Basics
Ensure that we can continue to parse, enrich, and index telemetry. Verify
data is flowing through the system, from parsing to indexing
1. Open Ambari and navigate to the Metron service
http://node1:8080/#/main/services/METRON/summary
1. Open the Alerts UI. Verify alerts show up in the main UI - click the
search icon (you may need to wait a moment for them to appear)
1. Go to the Alerts UI and ensure that an ever increasing number of
telemetry from Bro, Snort, and YAF are visible by watching the total alert
count increase over time.
1. Ensure that geoip enrichment is occurring. The telemetry should contain
fields like `enrichments:geo:ip_src_addr:location_point`.
1. Head back to Ambari and select the Kibana service
http://node1:8080/#/main/services/KIBANA/summary
1. Open the Kibana dashboard via the "Metron UI" option in the quick links
1. Verify the dashboard is populating
### Streaming Enrichments
1. Create a Streaming Enrichment [by following these
instructions](https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment).
1. Launch the Stellar REPL.
```
source /etc/default/metron
cd $METRON_HOME
$METRON_HOME/bin/stellar -z $ZOOKEEPER
```
1. Define the streaming enrichment and save it as a new source of
telemetry.
```
[Stellar]>>> conf := SHELL_EDIT(conf)
{
"parserClassName": "org.apache.metron.parsers.csv.CSVParser",
"writerClassName":
"org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter",
"sensorTopic": "user",
"parserConfig": {
"shew.table": "enrichment",
"shew.cf": "t",
"shew.keyColumns": "ip",
"shew.enrichmentType": "user",
"columns": {
"user": 0,
"ip": 1
}
}
}
[Stellar]>>>
[Stellar]>>> CONFIG_PUT("PARSER", conf, "user")
```
1. Go to the Management UI and start the new parser called 'user'.
1. Create some test telemetry.
```
[Stellar]>>> msgs := ["user1,192.168.1.1", "user2,192.168.1.2",
"user3,192.168.1.3"]
[user1,192.168.1.1, user2,192.168.1.2, user3,192.168.1.3]
[Stellar]>>> KAFKA_PUT("user", msgs)
3
[Stellar]>>> KAFKA_PUT("user", msgs)
3
[Stellar]>>> KAFKA_PUT("user", msgs)
3
```
1. Ensure that the enrichments are persisted in HBase.
```
[Stellar]>>> ENRICHMENT_GET('user', '192.168.1.1', 'enrichment', 't')
{original_string=user1,192.168.1.1,
guid=a6caf3c1-2506-4eb7-b33e-7c05b77cd72c, user=user1, timestamp=1551813589399,
source.type=user}
[Stellar]>>> ENRICHMENT_GET('user', '192.168.1.2', 'enrichment', 't')
{original_string=user2,192.168.1.2,
guid=49e4b8fa-c797-44f0-b041-cfb47983d54a, user=user2, timestamp=1551813589399,
source.type=user}
[Stellar]>>> ENRICHMENT_GET('user', '192.168.1.3', 'enrichment', 't')
{original_string=user3,192.168.1.3,
guid=324149fd-6c4c-42a3-b579-e218c032ea7f, user=user3, timestamp=1551813589402,
source.type=user}
```
### Enrichment Coprocessor
1. Confirm that the 'user' enrichment added in the previous section was
'found' by the coprocessor.
* Go to Swagger.
* Click the `sensor-enrichment-config-controller` option.
* Click the `GET
/api/v1/sensor/enrichment/config/list/available/enrichments` option.
1. Click the "Try it out!" button. You should see a array returned with
the value of each enrichment type that you have loaded.
```
[
"user"
]
```
### Enrichment Stellar Functions in Storm
1. Follow [instructions similar to
[jira] [Updated] (METRON-2188) Upgrade to HBase 2.0.2
[ https://issues.apache.org/jira/browse/METRON-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-2188: --- Description: Upgrade Metron to function with HBase 2.0.2. (was: Upgrade the enrichment coprocessor to support HBase 2.0.2) > Upgrade to HBase 2.0.2 > -- > > Key: METRON-2188 > URL: https://issues.apache.org/jira/browse/METRON-2188 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > Upgrade Metron to function with HBase 2.0.2. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2188) Upgrade to HBase 2.0.2
[ https://issues.apache.org/jira/browse/METRON-2188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen updated METRON-2188: --- Summary: Upgrade to HBase 2.0.2 (was: Upgrade HBase Coprocessor for HBase 2.0.2) > Upgrade to HBase 2.0.2 > -- > > Key: METRON-2188 > URL: https://issues.apache.org/jira/browse/METRON-2188 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > Upgrade the enrichment coprocessor to support HBase 2.0.2 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [metron] ottobackwards commented on issue #1501: METRON-2246 rpm-docker - minimise use of bind mounts due to performance
ottobackwards commented on issue #1501: METRON-2246 rpm-docker - minimise use of bind mounts due to performance URL: https://github.com/apache/metron/pull/1501#issuecomment-530036173 @tigerquoll, for my issue, i was going to take it a step further and just do a ROOT level docker file, and copy/add . /code. Mainly because i want to do full END to END in the container: - run ansible -> builds code -> builds rpm -> deploys to vagrant or "other" host. Having gotten that far in my thinking I stopped however, because I haven't had time to think through all the restructuring it would take, given the way we have the docker stuff buried and purposed right now. Does that make sense? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron-bro-plugin-kafka] ottobackwards opened a new pull request #36: METRON-2069 Add btests for bro plugin topic_name selection
ottobackwards opened a new pull request #36: METRON-2069 Add btests for bro plugin topic_name selection URL: https://github.com/apache/metron-bro-plugin-kafka/pull/36 - add btests for different configuration options - add docker run_end_to_end option to surpess pcaps so you can just build the plugin and run btests as part of bro-pkg - add new plugin configuration for `mock`. When this is set to `T`, the plugin will not create or call anything todo with Kafka, this is so you can test without kafka running ## Testing ```bash >run_end_to_end.sh --no-pcap and observer tests pass ``` ``` bash >run_end_to_end.sh ``` In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed via: ``` bro-pkg test $GITHUB_USERNAME/metron-bro-plugin-kafka --version $BRANCH ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [-] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] run in docker with run_end_to_end.sh This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (METRON-2069) Add btests for bro plugin topic_name selection
[ https://issues.apache.org/jira/browse/METRON-2069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Otto Fowler reassigned METRON-2069: --- Assignee: Otto Fowler > Add btests for bro plugin topic_name selection > -- > > Key: METRON-2069 > URL: https://issues.apache.org/jira/browse/METRON-2069 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Otto Fowler >Priority: Major > Fix For: Next + 1 > > Time Spent: 10m > Remaining Estimate: 0h > > We should add btests for the bro plugin topic_name selection to avoid > regressions with the critical selection logic. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2251) Update with the latest master branch
Shane Ardell created METRON-2251: Summary: Update with the latest master branch Key: METRON-2251 URL: https://issues.apache.org/jira/browse/METRON-2251 Project: Metron Issue Type: Sub-task Reporter: Shane Ardell This branch is really out-of-date with the current master branch of Metron. In addition to resolving conflicts, both UIs are up to Angular 7, which means we have to update code to accommodate for breaking changes, stricter type checking, etc. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[GitHub] [metron] mmiklavc commented on issue #1505: METRON-2250 Missing services in HDP 3.1 metron mpack and installer stuck
mmiklavc commented on issue #1505: METRON-2250 Missing services in HDP 3.1 metron mpack and installer stuck URL: https://github.com/apache/metron/pull/1505#issuecomment-529947210 Closing as it looks like this will not be sufficient to fix the issue. Fixing this issue will require an Ambari code change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc closed pull request #1505: METRON-2250 Missing services in HDP 3.1 metron mpack and installer stuck
mmiklavc closed pull request #1505: METRON-2250 Missing services in HDP 3.1 metron mpack and installer stuck URL: https://github.com/apache/metron/pull/1505 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
