[metron] branch master updated: METRON-2332 Enable Tuning of the Profiler's Parallelism from Ambari (nickwallen) closes apache/metron#1574
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 19dd9b1 METRON-2332 Enable Tuning of the Profiler's Parallelism from
Ambari (nickwallen) closes apache/metron#1574
19dd9b1 is described below
commit 19dd9b173b5fbe3c4dc1e08a5d0498a1c441c6bc
Author: nickwallen
AuthorDate: Thu Dec 5 10:37:30 2019 -0500
METRON-2332 Enable Tuning of the Profiler's Parallelism from Ambari
(nickwallen) closes apache/metron#1574
---
metron-analytics/metron-profiler-storm/README.md | 36 ++--
.../src/main/config/profiler.properties| 7 ++-
.../src/main/flux/profiler/remote.yaml | 7 ++-
.../storm/integration/ProfilerIntegrationTest.java | 7 ++-
.../CURRENT/configuration/metron-profiler-env.xml | 36 ++--
.../CURRENT/package/scripts/params/params_linux.py | 5 +++
.../package/templates/profiler.properties.j2 | 7 ++-
.../METRON/CURRENT/themes/metron_theme.json| 50 ++
8 files changed, 145 insertions(+), 10 deletions(-)
diff --git a/metron-analytics/metron-profiler-storm/README.md
b/metron-analytics/metron-profiler-storm/README.md
index c952cb7..c31b6fa 100644
--- a/metron-analytics/metron-profiler-storm/README.md
+++ b/metron-analytics/metron-profiler-storm/README.md
@@ -325,11 +325,41 @@ The units used to specify the `profiler.window.lag`.
This value should be defin
The number of worker processes to create for the Profiler topology. This
property is useful for performance tuning the Profiler.
-### `profiler.executors`
+### `profiler.acker.executors`
-*Default*: 0
+*Default*: 1
+
+The number of threads dedicated to tuple acking. This should most often be
equal to the number of partitions in the inbound Kafka topic.
+
+### `profiler.spout.parallelism`
+
+*Default*: 1
+
+The initial number of executors (threads) for the Profiler's Kafka spout. This
is also known as the parallelism hint.
+
+### `profiler.splitter.parallelism`
+
+*Default*: 1
+
+The initial number of executors (threads) for the Profiler's splitter
component. This is also known as the parallelism hint.
+
+### `profiler.builder.parallelism`
+
+*Default*: 1
+
+The initial number of executors (threads) for the Profiler's builder
component. This is also known as the parallelism hint.
+
+### `profiler.hbase.writer.parallelism`
+
+*Default*: 1
+
+The initial number of executors (threads) for the Profiler's HBase writer
component. This is also known as the parallelism hint.
+
+### `profiler.kafka.writer.parallelism`
+
+*Default*: 1
-The number of executors to spawn per component for the Profiler topology.
This property is useful for performance tuning the Profiler.
+The initial number of executors (threads) for the Profiler's Kafka writer
component. This is also known as the parallelism hint.
### `profiler.ttl`
diff --git
a/metron-analytics/metron-profiler-storm/src/main/config/profiler.properties
b/metron-analytics/metron-profiler-storm/src/main/config/profiler.properties
index dc30838..69efa93 100644
--- a/metron-analytics/metron-profiler-storm/src/main/config/profiler.properties
+++ b/metron-analytics/metron-profiler-storm/src/main/config/profiler.properties
@@ -23,7 +23,12 @@
topology.worker.childopts=
topology.auto-credentials=
profiler.workers=1
-profiler.executors=0
+profiler.acker.executors=1
+profiler.spout.parallelism=1
+profiler.splitter.parallelism=1
+profiler.builder.parallelism=1
+profiler.hbase.writer.parallelism=1
+profiler.kafka.writer.parallelism=1
topology.message.timeout.secs=30
topology.max.spout.pending=10
topology.fall.back.on.java.serialization=true
diff --git
a/metron-analytics/metron-profiler-storm/src/main/flux/profiler/remote.yaml
b/metron-analytics/metron-profiler-storm/src/main/flux/profiler/remote.yaml
index e16a782..3921967 100644
--- a/metron-analytics/metron-profiler-storm/src/main/flux/profiler/remote.yaml
+++ b/metron-analytics/metron-profiler-storm/src/main/flux/profiler/remote.yaml
@@ -18,7 +18,7 @@ name: "profiler"
config:
topology.workers: ${profiler.workers}
-topology.acker.executors: ${profiler.executors}
+topology.acker.executors: ${profiler.acker.executors}
topology.worker.childopts: ${topology.worker.childopts}
topology.auto-credentials: ${topology.auto-credentials}
topology.message.timeout.secs: ${topology.message.timeout.secs}
@@ -143,6 +143,7 @@ spouts:
className: "org.apache.metron.storm.kafka.flux.StormKafkaSpout"
constructorArgs:
- ref: "kafkaConfig"
+parallelism: ${profiler.spout.parallelism}
bolts:
@@ -150,6 +151,7 @@ bolts:
className: "org.apache.metron.profiler.storm.ProfileSplitterBolt"
constructorArgs:
- "${kafka.zk}"
+
[metron] branch master updated (2068820 -> fc4bf49)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/metron.git. from 2068820 METRON-2317 [UI] Delete confirmation dialogue looks visually broken (tiborm via sardell) closes apache/metron#1562 add fc4bf49 METRON-2321 Remove Legacy AWS Deployment Path (nickwallen) closes apache/metron#1565 No new revisions were added by this update. Summary of changes: Upgrading.md | 3 + metron-deployment/README.md| 25 -- metron-deployment/amazon-ec2/.gitignore| 5 - metron-deployment/amazon-ec2/README.md | 322 - metron-deployment/amazon-ec2/ansible.cfg | 32 -- metron-deployment/amazon-ec2/conf/defaults.yml | 97 --- metron-deployment/amazon-ec2/conf/ec2.ini | 105 --- metron-deployment/amazon-ec2/playbook.yml | 70 - metron-deployment/amazon-ec2/run.sh| 67 - metron-deployment/amazon-ec2/tasks/check-hosts.yml | 20 -- .../amazon-ec2/tasks/check-volume.yml | 26 -- .../amazon-ec2/tasks/create-hosts.yml | 54 .../amazon-ec2/tasks/create-keypair.yml| 29 -- .../tasks/create-open-inbound-security-group.yml | 26 -- .../tasks/create-open-outbound-security-group.yml | 26 -- .../amazon-ec2/tasks/create-security-group.yml | 28 -- metron-deployment/amazon-ec2/tasks/create-vpc.yml | 50 .../amazon-ec2/tasks/expand-volume.yml | 30 -- .../amazon-ec2/tasks/mount-volume.yml | 32 -- .../ansible/playbooks/ambari_install.yml | 7 - .../ansible/playbooks/metron_install.yml | 7 - .../ansible/playbooks/sensor_install.yml | 7 - .../ansible/playbooks/solr_install.yml | 7 - .../ansible/roles/sensor-stubs/README.md | 2 +- .../ansible/roles/sensor-test-mode/README.md | 2 +- 25 files changed, 5 insertions(+), 1074 deletions(-) delete mode 100644 metron-deployment/amazon-ec2/.gitignore delete mode 100644 metron-deployment/amazon-ec2/README.md delete mode 100644 metron-deployment/amazon-ec2/ansible.cfg delete mode 100644 metron-deployment/amazon-ec2/conf/defaults.yml delete mode 100755 metron-deployment/amazon-ec2/conf/ec2.ini delete mode 100644 metron-deployment/amazon-ec2/playbook.yml delete mode 100755 metron-deployment/amazon-ec2/run.sh delete mode 100644 metron-deployment/amazon-ec2/tasks/check-hosts.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/check-volume.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/create-hosts.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/create-keypair.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/create-open-inbound-security-group.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/create-open-outbound-security-group.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/create-security-group.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/create-vpc.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/expand-volume.yml delete mode 100644 metron-deployment/amazon-ec2/tasks/mount-volume.yml
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2314 HDFSIndexingIntegrationTests Fails with SLF4J/Log4j Delegation Loop (nickwallen) closes apache/metron#1559
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 889f113 METRON-2314 HDFSIndexingIntegrationTests Fails with
SLF4J/Log4j Delegation Loop (nickwallen) closes apache/metron#1559
889f113 is described below
commit 889f113f51e9d57b6189ee32f6eec6461a19db9c
Author: nickwallen
AuthorDate: Mon Nov 18 09:54:58 2019 -0500
METRON-2314 HDFSIndexingIntegrationTests Fails with SLF4J/Log4j Delegation
Loop (nickwallen) closes apache/metron#1559
---
.../metron-indexing/metron-indexing-storm/pom.xml| 12
1 file changed, 12 insertions(+)
diff --git a/metron-platform/metron-indexing/metron-indexing-storm/pom.xml
b/metron-platform/metron-indexing/metron-indexing-storm/pom.xml
index 5327ee5..f626982 100644
--- a/metron-platform/metron-indexing/metron-indexing-storm/pom.xml
+++ b/metron-platform/metron-indexing/metron-indexing-storm/pom.xml
@@ -60,6 +60,18 @@
metron-integration-test
${project.parent.version}
test
+
+
+
+ org.slf4j
+ log4j-over-slf4j
+
+
org.apache.metron
[metron] 01/01: Merge remote-tracking branch 'upstream/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit 69163fab7cf5a14754c96e7115492f0ce9e1ebca Merge: 78a6f38 12e16df Author: nickallen AuthorDate: Wed Nov 13 12:22:35 2019 -0500 Merge remote-tracking branch 'upstream/master' into feature/METRON-2088-support-hdp-3.1 metron-analytics/metron-maas-service/README.md | 209 - metron-analytics/metron-profiler-common/README.md | 10 +- .../metron/profiler/spark/BatchProfiler.java | 4 +- .../spark/BatchProfilerIntegrationTest.java| 29 ++- .../alerts/alerts-list/alerts-list.component.ts| 1 + .../auto-polling/auto-polling.service.spec.ts | 12 +- .../auto-polling/auto-polling.service.ts | 9 +- .../metron-elasticsearch-storm/pom.xml | 11 ++ .../metron-indexing/metron-indexing-storm/pom.xml | 23 +-- .../metron-solr/metron-solr-storm/pom.xml | 6 + 10 files changed, 269 insertions(+), 45 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2223 Reconcile Versions in Root POM with HDP 3.1 (nickwallen) closes apache/metron#1551
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/feature/METRON-2088-support-hdp-3.1 by this push: new 78a6f38 METRON-2223 Reconcile Versions in Root POM with HDP 3.1 (nickwallen) closes apache/metron#1551 78a6f38 is described below commit 78a6f38daa1ff4602e32d1c11a2d5d726d9975e4 Author: nickwallen AuthorDate: Wed Nov 13 12:19:35 2019 -0500 METRON-2223 Reconcile Versions in Root POM with HDP 3.1 (nickwallen) closes apache/metron#1551 --- README.md | 2 +- .../ambari_config/tasks/dependencies-CentOS-6.yml | 26 .../ambari_slave/tasks/install-agent-CentOS-6.yml | 19 --- .../tasks/install-service-CentOS-6.yml | 22 .../manual-install/Manual_Install_CentOS6.md | 18 +-- metron-platform/metron-storm-kafka/pom.xml | 2 +- pom.xml| 137 + 7 files changed, 68 insertions(+), 158 deletions(-) diff --git a/README.md b/README.md index a3bc14e..f7977a5 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ $ mvn clean install -DskipTests Build with the HDP profile: ``` -$ mvn clean install -PHDP-2.5.0.0 +$ mvn clean install -PHDP-3.1 ``` You can swap "install" for "package" in the commands above if you don't want to deploy the artifacts to your local .m2 repo. diff --git a/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-6.yml b/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-6.yml deleted file mode 100644 index 72a96d4..000 --- a/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-6.yml +++ /dev/null @@ -1,26 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -- name: Install urllib3 with pip - pip: -name: urllib3 -version: 1.10.2 - -- name: Install requests with pip - pip: -name: requests -version: 2.6.1 diff --git a/metron-deployment/ansible/roles/ambari_slave/tasks/install-agent-CentOS-6.yml b/metron-deployment/ansible/roles/ambari_slave/tasks/install-agent-CentOS-6.yml deleted file mode 100644 index 8f16b92..000 --- a/metron-deployment/ansible/roles/ambari_slave/tasks/install-agent-CentOS-6.yml +++ /dev/null @@ -1,19 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# -- name: Install ambari-agent on CentOS 6 - yum: name=ambari-agent diff --git a/metron-deployment/ansible/roles/sensor-stubs/tasks/install-service-CentOS-6.yml b/metron-deployment/ansible/roles/sensor-stubs/tasks/install-service-CentOS-6.yml deleted file mode 100644 index 022e49f..000 --- a/metron-deployment/ansible/roles/sensor-stubs/tasks/install-service-CentOS-6.yml +++ /dev/null @@ -1,22 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -
[metron] branch master updated: METRON-2308 Fix 'Degrees' Example Profile (nickwallen) closes apache/metron#1555
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 12e16df METRON-2308 Fix 'Degrees' Example Profile (nickwallen) closes
apache/metron#1555
12e16df is described below
commit 12e16df4c340be169ff0d0190c732948fdcd2e44
Author: nickwallen
AuthorDate: Wed Nov 13 10:06:30 2019 -0500
METRON-2308 Fix 'Degrees' Example Profile (nickwallen) closes
apache/metron#1555
---
metron-analytics/metron-profiler-common/README.md | 10 ++
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/metron-analytics/metron-profiler-common/README.md
b/metron-analytics/metron-profiler-common/README.md
index fe4c2ed..5841efc 100644
--- a/metron-analytics/metron-profiler-common/README.md
+++ b/metron-analytics/metron-profiler-common/README.md
@@ -363,15 +363,17 @@ This profile captures the vertex degree of a host. If you
view network communica
"profile": "in-degrees",
"onlyif": "source.type == 'yaf'",
"foreach": "ip_dst_addr",
- "update": { "in": "HLLP_ADD(in, ip_src_addr)" },
- "result": "HLLP_CARDINALITY(in)"
+ "init": { "estimator": "HLLP_INIT()" },
+ "update": { "estimator": "HLLP_ADD(estimator, ip_src_addr)" },
+ "result": "HLLP_CARDINALITY(estimator)"
},
{
"profile": "out-degrees",
"onlyif": "source.type == 'yaf'",
"foreach": "ip_src_addr",
- "update": { "out": "HLLP_ADD(out, ip_dst_addr)" },
- "result": "HLLP_CARDINALITY(out)"
+ "init": { "estimator": "HLLP_INIT()" },
+ "update": { "estimator": "HLLP_ADD(estimator, ip_dst_addr)" },
+ "result": "HLLP_CARDINALITY(estimator)"
}
]
}
[metron] branch master updated: METRON-2284 Metron Profiler for Spark Doesn't Work as Expected (nickwallen) closes apache/metron#1556
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 8686f52 METRON-2284 Metron Profiler for Spark Doesn't Work as
Expected (nickwallen) closes apache/metron#1556
8686f52 is described below
commit 8686f5265e68a5fb861f944b4b3b086c807618f2
Author: nickwallen
AuthorDate: Tue Nov 12 14:56:35 2019 -0500
METRON-2284 Metron Profiler for Spark Doesn't Work as Expected (nickwallen)
closes apache/metron#1556
---
.../metron/profiler/spark/BatchProfiler.java | 4 +--
.../spark/BatchProfilerIntegrationTest.java| 29 +-
2 files changed, 30 insertions(+), 3 deletions(-)
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
index 571545e..43b42be 100644
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
+++
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
@@ -87,13 +87,13 @@ public class BatchProfiler implements Serializable {
// find all routes for each message
Dataset routes = telemetry
-.flatMap(messageRouterFunction(profilerProps, profiles, globals),
Encoders.bean(MessageRoute.class));
+.flatMap(messageRouterFunction(profilerProps, profiles, globals),
Encoders.kryo(MessageRoute.class));
LOG.debug("Generated {} message route(s)", routes.cache().count());
// build the profiles
Dataset measurements = routes
.groupByKey(new GroupByPeriodFunction(profilerProps),
Encoders.STRING())
-.mapGroups(new ProfileBuilderFunction(profilerProps, globals),
Encoders.bean(ProfileMeasurementAdapter.class));
+.mapGroups(new ProfileBuilderFunction(profilerProps, globals),
Encoders.kryo(ProfileMeasurementAdapter.class));
LOG.debug("Produced {} profile measurement(s)",
measurements.cache().count());
// write the profile measurements to HBase
diff --git
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
index b36cf8c..72fd283 100644
---
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
+++
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
@@ -19,19 +19,24 @@
*/
package org.apache.metron.profiler.spark;
+import com.google.common.collect.Maps;
import org.adrianwalker.multilinestring.Multiline;
import org.apache.metron.hbase.mock.MockHBaseTableProvider;
+import org.apache.metron.profiler.MessageRoute;
import org.apache.metron.profiler.client.stellar.FixedLookback;
import org.apache.metron.profiler.client.stellar.GetProfile;
import org.apache.metron.profiler.client.stellar.WindowLookback;
+import org.apache.metron.profiler.spark.function.MessageRouterFunction;
import org.apache.metron.stellar.common.DefaultStellarStatefulExecutor;
import org.apache.metron.stellar.common.StellarStatefulExecutor;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.functions.resolver.SimpleFunctionResolver;
import org.apache.spark.SparkConf;
import org.apache.spark.SparkException;
+import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Encoders;
import org.apache.spark.sql.SparkSession;
+import org.json.simple.JSONObject;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
@@ -41,7 +46,9 @@ import org.junit.rules.TemporaryFolder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.io.IOException;
import java.lang.invoke.MethodHandles;
+import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
@@ -62,6 +69,7 @@ import static
org.apache.metron.profiler.spark.BatchProfilerConfig.TELEMETRY_INP
import static org.apache.metron.profiler.spark.reader.TelemetryReaders.JSON;
import static org.apache.metron.profiler.spark.reader.TelemetryReaders.ORC;
import static org.apache.metron.profiler.spark.reader.TelemetryReaders.PARQUET;
+import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
/**
@@ -145,6 +153,14 @@ public class BatchProfilerIntegrationTest {
*"init": { "count": 0 },
*"update": { "count": "count + 1" },
*"result": "count"
+ * },
+ * {
+
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2305 Unable to Add Enrichment Coprocessor with Kerberos (MohanDV via nickwallen) closes apache/metron#1549
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 018e90f METRON-2305 Unable to Add Enrichment Coprocessor with
Kerberos (MohanDV via nickwallen) closes apache/metron#1549
018e90f is described below
commit 018e90fe664295afeed391f678849a76a92919fe
Author: MohanDV
AuthorDate: Wed Oct 30 13:00:42 2019 -0400
METRON-2305 Unable to Add Enrichment Coprocessor with Kerberos (MohanDV via
nickwallen) closes apache/metron#1549
---
.../METRON/CURRENT/package/scripts/enrichment_commands.py | 3 ++-
.../METRON/CURRENT/package/scripts/enrichment_master.py | 4 ++--
2 files changed, 4 insertions(+), 3 deletions(-)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
index 68769bb..0b99a70 100755
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_commands.py
@@ -268,7 +268,7 @@ class EnrichmentCommands:
self.__params.hbase_principal_name,
execute_user=self.__params.hbase_user)
-cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
+cmd = "echo \"grant '{0}', 'CRW', '{1}'\" | hbase shell -n"
add_enrichment_acl_cmd = cmd.format(self.__params.metron_user,
self.__params.enrichment_hbase_table)
Execute(add_enrichment_acl_cmd,
tries=3,
@@ -278,6 +278,7 @@ class EnrichmentCommands:
user=self.__params.hbase_user
)
+cmd = "echo \"grant '{0}', 'RW', '{1}'\" | hbase shell -n"
add_enrichment_list_acl_cmd = cmd.format(self.__params.metron_user,
self.__params.enrichment_list_hbase_table)
Execute(add_enrichment_list_acl_cmd,
tries=3,
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
index 15d08e1..66a48a5 100755
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/enrichment_master.py
@@ -69,10 +69,10 @@ class Enrichment(Script):
commands.init_kafka_acls()
if not commands.is_hbase_configured():
commands.create_hbase_tables()
-if not commands.is_hbase_coprocessor_configured():
-commands.load_enrichment_coprocessor()
if params.security_enabled and not commands.is_hbase_acl_configured():
commands.set_hbase_acls()
+if not commands.is_hbase_coprocessor_configured():
+commands.load_enrichment_coprocessor()
if not commands.is_maxmind_configured():
commands.init_maxmind()
[metron] 01/01: Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit b48cdc2684ee3f0d1e30f22a6a9cf80a248e9e82 Merge: 345f991 37a8e2e Author: Nick Allen AuthorDate: Wed Oct 30 12:45:18 2019 -0400 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 .../metron-profiler-spark/src/main/config/batch-profiler.properties | 2 +- site/community/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (345f991 -> b48cdc2)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 345f991 METRON-2233 Deprecate CentOS 6 Development Environment (nickwallen) closes apache/metron#1542 add e645ffe METRON-2300 Fix Brad Kolarov's Apache ID (billierinaldi via mmiklavc) closes apache/metron#1541 add 37a8e2e METRON-2303 Change Default HDFS Port for Batch Profiler (nickwallen) closes apache/metron#1548 new b48cdc2 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../metron-profiler-spark/src/main/config/batch-profiler.properties | 2 +- site/community/index.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
[metron] branch master updated: METRON-2303 Change Default HDFS Port for Batch Profiler (nickwallen) closes apache/metron#1548
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 37a8e2e METRON-2303 Change Default HDFS Port for Batch Profiler (nickwallen) closes apache/metron#1548 37a8e2e is described below commit 37a8e2e8211dd2c8a56c8e75b516e3029c421099 Author: nickwallen AuthorDate: Wed Oct 30 12:38:31 2019 -0400 METRON-2303 Change Default HDFS Port for Batch Profiler (nickwallen) closes apache/metron#1548 --- .../metron-profiler-spark/src/main/config/batch-profiler.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-analytics/metron-profiler-spark/src/main/config/batch-profiler.properties b/metron-analytics/metron-profiler-spark/src/main/config/batch-profiler.properties index 400c526..e9a8f59 100644 --- a/metron-analytics/metron-profiler-spark/src/main/config/batch-profiler.properties +++ b/metron-analytics/metron-profiler-spark/src/main/config/batch-profiler.properties @@ -19,7 +19,7 @@ spark.app.name=Batch Profiler spark.master=local -profiler.batch.input.path=hdfs://localhost:9000/apps/metron/indexing/indexed/*/* +profiler.batch.input.path=hdfs://localhost:8020/apps/metron/indexing/indexed/*/* profiler.batch.input.format=text profiler.period.duration=15
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2233 Deprecate CentOS 6 Development Environment (nickwallen) closes apache/metron#1542
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 345f991 METRON-2233 Deprecate CentOS 6 Development Environment
(nickwallen) closes apache/metron#1542
345f991 is described below
commit 345f99157e4ed3fa3730909c53eea8f09a3167cf
Author: nickwallen
AuthorDate: Wed Oct 30 12:35:00 2019 -0400
METRON-2233 Deprecate CentOS 6 Development Environment (nickwallen) closes
apache/metron#1542
---
README.md | 2 +-
dev-utilities/release-utils/metron-rc-check| 2 +-
metron-contrib/metron-docker/README.md | 2 +-
metron-deployment/Kerberos-manual-setup.md | 12 +--
metron-deployment/README.md| 2 +-
.../ansible/playbooks/ambari_install.yml | 2 +-
.../ansible/playbooks/metron_install.yml | 2 +-
.../ansible/playbooks/sensor_install.yml | 2 +-
.../ansible/playbooks/solr_install.yml | 2 +-
.../ansible/roles/ambari_common/defaults/main.yml | 1 -
.../ambari_common/tasks/ambari-repo-CentOS-6.yml | 22 -
.../ambari_common/tasks/firewall-CentOS-6.yml | 20
.../roles/ambari_common/tasks/nodejs-CentOS-6.yml | 22 -
.../ansible/roles/sensor-stubs/README.md | 2 +-
metron-deployment/development/README.md| 4 +-
metron-deployment/development/centos6/.gitignore | 1 -
metron-deployment/development/centos6/Vagrantfile | 107 -
metron-deployment/development/centos6/ansible.cfg | 28 --
.../centos6/ansible/inventory/group_vars/all | 86 -
.../development/centos6/ansible/inventory/hosts| 47 -
.../development/centos6/ansible/playbook.yml | 24 -
.../development/{centos6 => centos7}/README.md | 30 --
.../packaging/docker/rpm-docker/Dockerfile | 2 +-
metron-interface/metron-rest/README.md | 4 +-
.../metron-solr/metron-solr-common/README.md | 10 +-
25 files changed, 48 insertions(+), 390 deletions(-)
diff --git a/README.md b/README.md
index 1f4e030..a3bc14e 100644
--- a/README.md
+++ b/README.md
@@ -83,7 +83,7 @@ Option 3 is more likely to have the latest code.
# Getting Started
-To start exploring the capabilities of Apache Metron [follow these
instructions to launch Metron in a single-node
VM](metron-deployment/development/centos6) on your own hardware.
+To start exploring the capabilities of Apache Metron [follow these
instructions to launch Metron in a single-node
VM](metron-deployment/development/centos7) on your own hardware.
# Building Metron
diff --git a/dev-utilities/release-utils/metron-rc-check
b/dev-utilities/release-utils/metron-rc-check
index 5b3a5d9..66752e1 100755
--- a/dev-utilities/release-utils/metron-rc-check
+++ b/dev-utilities/release-utils/metron-rc-check
@@ -274,7 +274,7 @@ echo ""
read -p " run vagrant full_dev? [yN] " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
- cd
"$WORK/apache-metron_${METRON_VERSION}-$RC/metron-deployment/development/centos6"
|| exit 1
+ cd
"$WORK/apache-metron_${METRON_VERSION}-$RC/metron-deployment/development/centos7"
|| exit 1
if [[ ${DID_BUILD} -ne 1 ]]; then
vagrant up
else
diff --git a/metron-contrib/metron-docker/README.md
b/metron-contrib/metron-docker/README.md
index 761cc6e..0dcf739 100644
--- a/metron-contrib/metron-docker/README.md
+++ b/metron-contrib/metron-docker/README.md
@@ -19,7 +19,7 @@ limitations under the License.
Metron Docker is a [Docker Compose](https://docs.docker.com/compose/overview/)
application that is intended only for development and integration testing of
Metron. These images can quickly spin-up the underlying components on which
Apache Metron runs.
-None of the core Metron components are setup or launched automatically with
these Docker images. You will need to manually setup and start the Metron
components that you require. You should not expect to see telemetry being
parsed, enriched, or indexed. If you are looking to try-out, experiment or
demo Metron capabilities on a single node, then the [Vagrant-driven
VM](../../metron-deployment/development/centos6) is what you need. Use this
instead of Vagrant when:
+None of the core Metron components are setup or launched automatically with
these Docker images. You will need to manually setup and start the Metron
components that you require. You should not expect to see telemetry being
parsed, enriched, or indexed. If you are looking to try-out, experiment or
demo Metron capabilities on a single node, then the [Vagrant-driven
VM](../../metron-deployment/development/centos7) is what you need. Use thi
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2301 Building Against Wrong Storm Flux Version (nickwallen) closes apache/metron#1544
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 01f6107 METRON-2301 Building Against Wrong Storm Flux Version
(nickwallen) closes apache/metron#1544
01f6107 is described below
commit 01f6107438a615612c378b708677be778ed4dad8
Author: nickwallen
AuthorDate: Wed Oct 30 08:23:52 2019 -0400
METRON-2301 Building Against Wrong Storm Flux Version (nickwallen) closes
apache/metron#1544
---
.../metron-common-storm/pom.xml| 2 +-
.../metron-elasticsearch-storm/pom.xml | 2 +-
metron-platform/metron-integration-test/pom.xml| 2 +-
.../components/FluxTopologyComponent.java | 69 +++---
metron-platform/metron-pcap-backend/pom.xml| 2 +-
.../metron-solr/metron-solr-storm/pom.xml | 2 +-
pom.xml| 4 +-
7 files changed, 41 insertions(+), 42 deletions(-)
diff --git
a/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
b/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
index 2b9fbed..4426802 100644
--- a/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
+++ b/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
@@ -72,7 +72,7 @@
org.apache.storm
flux-core
-${global_flux_version}
+${global_storm_version}
org.apache.zookeeper
diff --git
a/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
b/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
index a84e45d..3e71aba 100644
--- a/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
+++ b/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
@@ -62,7 +62,7 @@
org.apache.storm
flux-core
-${global_flux_version}
+${global_storm_version}
org.apache.storm
diff --git a/metron-platform/metron-integration-test/pom.xml
b/metron-platform/metron-integration-test/pom.xml
index 6bb8291..3906a8f 100644
--- a/metron-platform/metron-integration-test/pom.xml
+++ b/metron-platform/metron-integration-test/pom.xml
@@ -45,7 +45,7 @@
org.apache.storm
flux-core
- ${global_flux_version}
+ ${global_storm_version}
org.apache.curator
diff --git
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
index 1a1ceb0..85a05b0 100644
---
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
+++
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
@@ -17,22 +17,6 @@
*/
package org.apache.metron.integration.components;
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.FileWriter;
-import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.io.Writer;
-import java.lang.invoke.MethodHandles;
-import java.lang.reflect.InvocationTargetException;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.nio.file.Paths;
-import java.util.Comparator;
-import java.util.Map;
-import java.util.Properties;
-import java.util.regex.Pattern;
import org.apache.commons.io.FileUtils;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
@@ -55,6 +39,19 @@ import org.junit.Assert;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.io.File;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.invoke.MethodHandles;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Pattern;
+
public class FluxTopologyComponent implements InMemoryComponent {
protected static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
@@ -249,7 +246,7 @@ public class FluxTopologyComponent implements
InMemoryComponent {
}
private void startTopology(String topologyName, File topologyLoc, File
templateFile, Properties properties) throws IOException,
ClassNotFoundException, NoSuchMethodException, InvocationTargetException,
InstantiationException, IllegalAccessException, TException,
NoSuchFieldException{
-TopologyDef topologyDef = loadYaml(topologyName
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2222 Remove Overrides for Storm 1.0.x (nickwallen) closes apache/metron#1545
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new e4b10cc METRON- Remove Overrides for Storm 1.0.x (nickwallen)
closes apache/metron#1545
e4b10cc is described below
commit e4b10cce4d170122f6b2825114ecace1381fcf90
Author: nickwallen
AuthorDate: Tue Oct 29 17:26:17 2019 -0400
METRON- Remove Overrides for Storm 1.0.x (nickwallen) closes
apache/metron#1545
---
.../metron-storm-kafka-override/pom.xml| 165 --
.../spout/KafkaSpoutRetryExponentialBackoff.java | 336 -
.../apache/storm/kafka/spout/internal/Timer.java | 65
.../storm/kafka/spout/internal/TimerTest.java | 36 ---
metron-platform/metron-storm-kafka/pom.xml | 90 +-
metron-platform/pom.xml| 1 -
6 files changed, 85 insertions(+), 608 deletions(-)
diff --git a/metron-platform/metron-storm-kafka-override/pom.xml
b/metron-platform/metron-storm-kafka-override/pom.xml
deleted file mode 100644
index fe086f2..000
--- a/metron-platform/metron-storm-kafka-override/pom.xml
+++ /dev/null
@@ -1,165 +0,0 @@
-
-
-
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-4.0.0
-
-org.apache.metron
-metron-platform
-0.7.2
-
-metron-storm-kafka-override
-metron-storm-kafka-override
-Components that extend the Storm/Kafka spout
-https://metron.apache.org/
-
-UTF-8
-
UTF-8
-1.10
-16.0.1
-
-
-
-com.google.guava
-guava
-${guava_version}
-
-
-org.apache.storm
-storm-kafka-client
-${global_storm_kafka_version}
-
-
-org.apache.kafka
-kafka-clients
-${global_kafka_version}
-
-
-org.apache.storm
-storm-core
-${global_storm_version}
-provided
-
-
-org.apache.logging.log4j
-log4j-core
-
-
-org.apache.logging.log4j
-log4j-api
-
-
-servlet-api
-javax.servlet
-
-
-log4j-over-slf4j
-org.slf4j
-
-
-log4j-slf4j-impl
-org.apache.logging.log4j
-
-
-
-
-
-
-
-org.apache.maven.plugins
-maven-shade-plugin
-${global_shade_version}
-
-true
-uber
-
true
-
-
-
-package
-
-shade
-
-
-
-
- *:*
-
-META-INF/*.SF
-META-INF/*.DSA
-META-INF/*.RSA
-
-
-
-
-
-com.google.common
-
org.apache.metron.guava.${guava_version}
-
-
-com.google.thirdparty
-
org.apache.metron.storm.kafka.override.guava.thirdparty
-
-
-org.apache.commons.lang
-
org.apache.metron.storm.kafka.override.commons.lang
-
-
-com.fasterxml
-
org.apache.metron.storm.kafka.override.com.fasterxml
-
-
-
-
-storm:storm-core:*
-storm:storm-lib:*
-org.slf4j.impl*
-org.slf4
[metron] branch METRON-2223 deleted (was e9613e4)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch METRON-2223 in repository https://gitbox.apache.org/repos/asf/metron.git. was e9613e4 Start on changes to root pom This change permanently discards the following revisions: discard e9613e4 Start on changes to root pom discard 522de84 METRON- Remove Overrides for Storm 1.0.x (nickwallen) closes apache/metron#1545 discard c033bc7 METRON-2301 Building Against Wrong Storm Flux Version (nickwallen) closes apache/metron#1544 discard 3188696 METRON-2232 Upgrade to Hadoop 3.1.1 (mmiklavc via nickwallen) closes apache/metron#1523
[metron] 03/04: METRON-2222 Remove Overrides for Storm 1.0.x (nickwallen) closes apache/metron#1545
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch METRON-2223
in repository https://gitbox.apache.org/repos/asf/metron.git
commit 522de8410df14c314ea3b6597a9d6e9c2f0e8353
Author: nickwallen
AuthorDate: Tue Oct 29 11:07:30 2019 -0400
METRON- Remove Overrides for Storm 1.0.x (nickwallen) closes
apache/metron#1545
---
.../metron-storm-kafka-override/pom.xml| 165 --
.../spout/KafkaSpoutRetryExponentialBackoff.java | 336 -
.../apache/storm/kafka/spout/internal/Timer.java | 65
.../storm/kafka/spout/internal/TimerTest.java | 36 ---
metron-platform/metron-storm-kafka/pom.xml | 87 +-
metron-platform/pom.xml| 1 -
6 files changed, 82 insertions(+), 608 deletions(-)
diff --git a/metron-platform/metron-storm-kafka-override/pom.xml
b/metron-platform/metron-storm-kafka-override/pom.xml
deleted file mode 100644
index fe086f2..000
--- a/metron-platform/metron-storm-kafka-override/pom.xml
+++ /dev/null
@@ -1,165 +0,0 @@
-
-
-
-http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
-4.0.0
-
-org.apache.metron
-metron-platform
-0.7.2
-
-metron-storm-kafka-override
-metron-storm-kafka-override
-Components that extend the Storm/Kafka spout
-https://metron.apache.org/
-
-UTF-8
-
UTF-8
-1.10
-16.0.1
-
-
-
-com.google.guava
-guava
-${guava_version}
-
-
-org.apache.storm
-storm-kafka-client
-${global_storm_kafka_version}
-
-
-org.apache.kafka
-kafka-clients
-${global_kafka_version}
-
-
-org.apache.storm
-storm-core
-${global_storm_version}
-provided
-
-
-org.apache.logging.log4j
-log4j-core
-
-
-org.apache.logging.log4j
-log4j-api
-
-
-servlet-api
-javax.servlet
-
-
-log4j-over-slf4j
-org.slf4j
-
-
-log4j-slf4j-impl
-org.apache.logging.log4j
-
-
-
-
-
-
-
-org.apache.maven.plugins
-maven-shade-plugin
-${global_shade_version}
-
-true
-uber
-
true
-
-
-
-package
-
-shade
-
-
-
-
- *:*
-
-META-INF/*.SF
-META-INF/*.DSA
-META-INF/*.RSA
-
-
-
-
-
-com.google.common
-
org.apache.metron.guava.${guava_version}
-
-
-com.google.thirdparty
-
org.apache.metron.storm.kafka.override.guava.thirdparty
-
-
-org.apache.commons.lang
-
org.apache.metron.storm.kafka.override.commons.lang
-
-
-com.fasterxml
-
org.apache.metron.storm.kafka.override.com.fasterxml
-
-
-
-
-storm:storm-core:*
-storm:storm-lib:*
-org.slf4j.impl*
-org.slf4j:slf4j-log4j*
-
-
-
-
-
-.yaml
-
[metron] 02/04: METRON-2301 Building Against Wrong Storm Flux Version (nickwallen) closes apache/metron#1544
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch METRON-2223
in repository https://gitbox.apache.org/repos/asf/metron.git
commit c033bc731a3767d4a61dd72d1d3709d8cf3138f8
Author: nickwallen
AuthorDate: Tue Oct 29 10:56:25 2019 -0400
METRON-2301 Building Against Wrong Storm Flux Version (nickwallen) closes
apache/metron#1544
---
.../metron-common-storm/pom.xml| 2 +-
.../metron-elasticsearch-storm/pom.xml | 2 +-
metron-platform/metron-integration-test/pom.xml| 2 +-
.../components/FluxTopologyComponent.java | 69 +++---
metron-platform/metron-pcap-backend/pom.xml| 2 +-
.../metron-solr/metron-solr-storm/pom.xml | 2 +-
pom.xml| 4 +-
7 files changed, 41 insertions(+), 42 deletions(-)
diff --git
a/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
b/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
index 2b9fbed..4426802 100644
--- a/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
+++ b/metron-platform/metron-common-streaming/metron-common-storm/pom.xml
@@ -72,7 +72,7 @@
org.apache.storm
flux-core
-${global_flux_version}
+${global_storm_version}
org.apache.zookeeper
diff --git
a/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
b/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
index a84e45d..3e71aba 100644
--- a/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
+++ b/metron-platform/metron-elasticsearch/metron-elasticsearch-storm/pom.xml
@@ -62,7 +62,7 @@
org.apache.storm
flux-core
-${global_flux_version}
+${global_storm_version}
org.apache.storm
diff --git a/metron-platform/metron-integration-test/pom.xml
b/metron-platform/metron-integration-test/pom.xml
index 6bb8291..3906a8f 100644
--- a/metron-platform/metron-integration-test/pom.xml
+++ b/metron-platform/metron-integration-test/pom.xml
@@ -45,7 +45,7 @@
org.apache.storm
flux-core
- ${global_flux_version}
+ ${global_storm_version}
org.apache.curator
diff --git
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
index 1a1ceb0..85a05b0 100644
---
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
+++
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
@@ -17,22 +17,6 @@
*/
package org.apache.metron.integration.components;
-import java.io.File;
-import java.io.FileOutputStream;
-import java.io.FileWriter;
-import java.io.IOException;
-import java.io.OutputStreamWriter;
-import java.io.Writer;
-import java.lang.invoke.MethodHandles;
-import java.lang.reflect.InvocationTargetException;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.nio.file.Paths;
-import java.util.Comparator;
-import java.util.Map;
-import java.util.Properties;
-import java.util.regex.Pattern;
import org.apache.commons.io.FileUtils;
import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
@@ -55,6 +39,19 @@ import org.junit.Assert;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import java.io.File;
+import java.io.IOException;
+import java.io.StringReader;
+import java.lang.invoke.MethodHandles;
+import java.lang.reflect.InvocationTargetException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Comparator;
+import java.util.Map;
+import java.util.Properties;
+import java.util.regex.Pattern;
+
public class FluxTopologyComponent implements InMemoryComponent {
protected static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
@@ -249,7 +246,7 @@ public class FluxTopologyComponent implements
InMemoryComponent {
}
private void startTopology(String topologyName, File topologyLoc, File
templateFile, Properties properties) throws IOException,
ClassNotFoundException, NoSuchMethodException, InvocationTargetException,
InstantiationException, IllegalAccessException, TException,
NoSuchFieldException{
-TopologyDef topologyDef = loadYaml(topologyName, topologyLoc,
templateFile, properties);
+TopologyDef topologyDef = loadYaml(topologyLoc, templateFile, properties);
Config conf = FluxBuilder.buildConfig(topologyDef);
ExecutionContext context = new ExecutionContext(topologyDef, conf
[metron] 04/04: Start on changes to root pom
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch METRON-2223
in repository https://gitbox.apache.org/repos/asf/metron.git
commit e9613e4efa24be2c7401f6d68893b09f3e471c3f
Author: nickallen
AuthorDate: Tue Oct 29 11:53:28 2019 -0400
Start on changes to root pom
---
metron-interface/pom.xml | 2 +-
metron-platform/metron-common/pom.xml | 2 +-
pom.xml | 50 +--
3 files changed, 9 insertions(+), 45 deletions(-)
diff --git a/metron-interface/pom.xml b/metron-interface/pom.xml
index 2934368..8968966 100644
--- a/metron-interface/pom.xml
+++ b/metron-interface/pom.xml
@@ -51,7 +51,7 @@
junit
junit
-4.12
+${global_junit_version}
test
diff --git a/metron-platform/metron-common/pom.xml
b/metron-platform/metron-common/pom.xml
index b4114cc..04b12fa 100644
--- a/metron-platform/metron-common/pom.xml
+++ b/metron-platform/metron-common/pom.xml
@@ -175,7 +175,7 @@
org.krakenapps
kraken-pcap
-${global_pcap_version}
+${global_kraken_pcap_version}
slf4j-api
diff --git a/pom.xml b/pom.xml
index 90de889..25df5ed 100644
--- a/pom.xml
+++ b/pom.xml
@@ -86,30 +86,16 @@
@ApacheMetron
UTF-8
UTF-8
-
-0.10.0
-2.7.1
-1.5.2
-
1.8.0
2.6.2
4.5
3.7
3.3
-1.1.1
-12.0
-1.0.3
-1.2.2
-1.7.1
-${base_hadoop_version}
-${base_flume_version}
-5.6.14
+1.7.1
1.1.1
3.0.2
4.12
17.0
-
2.2.5
-3.7
1.8
1.10.19
1.7.0
@@ -125,10 +111,10 @@
8.0
2.1
0.0.3
-2.3.1
4.3.2
0.66.19
0.8.3
+1.7.25
@@ -138,43 +124,21 @@
true
-3.1.0.0
2.0.0
-1.7.25
1.2.1
1.2.3
0.8.0
-7.4.0
-3.1.1
+3.1.1
2.0.2
17.0
+2.3.1
-3.4.6
-2.7.1
-
2.7.1
-
-
-
-HDP-2.5.0.0
-
-2.5.0.0
-1245
-1.1.1
-12.0
-1.2.2
-1.0.1
-
${base_storm_version}.${hdp_version}-${build_number}
-
${base_kafka_version}.${hdp_version}-${build_number}
-0.7.3
-6.6.2
-2.0.2
-17.0
-
+http://curator.apache.org/zk-compatibility.html -->
3.4.6
2.7.1
2.7.1
+7.4.0
+
5.6.14
[metron] branch METRON-2223 created (now e9613e4)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch METRON-2223 in repository https://gitbox.apache.org/repos/asf/metron.git. at e9613e4 Start on changes to root pom This branch includes the following new commits: new 3188696 METRON-2232 Upgrade to Hadoop 3.1.1 (mmiklavc via nickwallen) closes apache/metron#1523 new c033bc7 METRON-2301 Building Against Wrong Storm Flux Version (nickwallen) closes apache/metron#1544 new 522de84 METRON- Remove Overrides for Storm 1.0.x (nickwallen) closes apache/metron#1545 new e9613e4 Start on changes to root pom The 4 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2297 Enrichment Topology Unable to Load Geo IP Data from HDFS (nickwallen) closes apache/metron#1539
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 0a619e0 METRON-2297 Enrichment Topology Unable to Load Geo IP Data
from HDFS (nickwallen) closes apache/metron#1539
0a619e0 is described below
commit 0a619e082f34d65269f03ed4c6e16ba8bb5a0dbb
Author: nickwallen
AuthorDate: Wed Oct 23 15:39:19 2019 -0400
METRON-2297 Enrichment Topology Unable to Load Geo IP Data from HDFS
(nickwallen) closes apache/metron#1539
---
metron-analytics/metron-profiler-storm/pom.xml | 5 +
.../CURRENT/configuration/metron-security-env.xml | 10 +-
.../CURRENT/package/scripts/params/params_linux.py | 7 +-
.../storm/security/auth/kerberos/AutoTGT.java | 254 +
.../metron-elasticsearch-storm/pom.xml | 5 +
.../metron-enrichment-storm/pom.xml| 5 +
.../metron-indexing/metron-indexing-storm/pom.xml | 5 +
metron-platform/metron-pcap-backend/pom.xml| 5 +
.../metron-solr/metron-solr-storm/pom.xml | 5 +
9 files changed, 299 insertions(+), 2 deletions(-)
diff --git a/metron-analytics/metron-profiler-storm/pom.xml
b/metron-analytics/metron-profiler-storm/pom.xml
index 84c0631..da049ac 100644
--- a/metron-analytics/metron-profiler-storm/pom.xml
+++ b/metron-analytics/metron-profiler-storm/pom.xml
@@ -148,6 +148,11 @@
org.apache.metron
+metron-common-storm
+${project.parent.version}
+
+
+org.apache.metron
metron-writer-storm
${project.parent.version}
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
index beb0451..fc91a4a 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
@@ -159,7 +159,6 @@
-
metron.ldap.ssl.truststore
LDAP Truststore
@@ -240,4 +239,13 @@
Name of the role at the authentication provider that provides
administrative access to Metron.
+
+topology_auto_credentials
+Topology Auto Credentials
+The value of Storm's `topology.auto-credentials`. A list of
plugins used to unpack credentials on the Storm worker. This value is only used
when Kerberos has been enabled.
+['org.apache.metron.storm.security.auth.kerberos.AutoTGT']
+
+ true
+
+
\ No newline at end of file
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index b7fbcf4..d0026dd 100755
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -258,7 +258,12 @@ profiler_topology_worker_childopts = client_jaas_arg if
security_enabled else ''
indexing_topology_worker_childopts = client_jaas_arg if security_enabled else
''
pcap_topology_worker_childopts = client_jaas_arg if security_enabled else ''
metron_jvm_flags += (' ' + client_jaas_arg) if security_enabled else ''
-topology_auto_credentials =
config['configurations']['storm-site'].get('nimbus.credential.renewers.classes',
[])
+
+# the user-defined `topology.auto-credentials` are only used if security is
enabled
+topology_auto_credentials = []
+if security_enabled:
+topology_auto_credentials =
config['configurations']['metron-security-env'].get('topology_auto_credentials',
[])
+
# Needed for storm.config, because it needs Java String
topology_auto_credentials_double_quotes =
str(topology_auto_credentials).replace("'", '"')
diff --git
a/metron-platform/metron-common-streaming/metron-common-storm/src/main/java/org/apache/metron/storm/security/auth/kerberos/AutoTGT.java
b/metron-platform/metron-common-streaming/metron-common-storm/src/main/java/org/apache/metron/storm/security/auth/kerberos/AutoTGT.java
new file mode 100644
index 000..adf0327
--- /dev/null
+++
b/metron-platform/metron-common-streaming/metron-common-storm/src/main/java/org/apache/metron/storm/security/auth/kerber
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2298 'User Base DN' Missing from Security Configuration (nickwallen) closes apache/metron#1538
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 0b6cd7d METRON-2298 'User Base DN' Missing from Security
Configuration (nickwallen) closes apache/metron#1538
0b6cd7d is described below
commit 0b6cd7da29795dee0ae90b09eb2afac2b460de9f
Author: nickwallen
AuthorDate: Wed Oct 23 15:23:07 2019 -0400
METRON-2298 'User Base DN' Missing from Security Configuration (nickwallen)
closes apache/metron#1538
---
.../common-services/METRON/CURRENT/themes/metron_theme.json| 10 ++
1 file changed, 10 insertions(+)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
index f6285c7..06c9013 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
@@ -897,6 +897,10 @@
"subsection-name": "subsection-security-ldap"
},
{
+ "config": "metron-security-env/metron.ldap.user.basedn",
+ "subsection-name": "subsection-security-ldap"
+},
+{
"config": "metron-security-env/metron.ldap.user.dnpattern",
"subsection-name": "subsection-security-ldap"
},
@@ -1688,6 +1692,12 @@
}
},
{
+"config": "metron-security-env/metron.ldap.user.basedn",
+"widget": {
+ "type": "text-field"
+}
+ },
+ {
"config": "metron-security-env/metron.ldap.user.dnpattern",
"widget": {
"type": "text-field"
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2279 Unable to Index to Solr with Kerberos (nickwallen) closes apache/metron#1528
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 8a23972 METRON-2279 Unable to Index to Solr with Kerberos
(nickwallen) closes apache/metron#1528
8a23972 is described below
commit 8a239727b759aba7a90ac4d494e6c81cae275068
Author: nickwallen
AuthorDate: Thu Oct 10 12:53:49 2019 -0400
METRON-2279 Unable to Index to Solr with Kerberos (nickwallen) closes
apache/metron#1528
---
metron-platform/metron-solr/metron-solr-storm/pom.xml | 17 +
1 file changed, 17 insertions(+)
diff --git a/metron-platform/metron-solr/metron-solr-storm/pom.xml
b/metron-platform/metron-solr/metron-solr-storm/pom.xml
index f160447..da12b0e 100644
--- a/metron-platform/metron-solr/metron-solr-storm/pom.xml
+++ b/metron-platform/metron-solr/metron-solr-storm/pom.xml
@@ -29,6 +29,8 @@
UTF-8
1.10
${global_hbase_guava_version}
+
+4.5.3
@@ -170,6 +172,21 @@
org.apache.metron.guava.${guava_version}
+
+org.apache.http
+
org.apache.metron.http.${http_client_version}
+
+
com.fasterxml.jackson
org.apache.metron.jackson
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2275 Solr Indexing Topology Fails to Start on Secure Cluster with HDP 3.1 (nickwallen) closes apache/metron#1526
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 5738ea3 METRON-2275 Solr Indexing Topology Fails to Start on Secure
Cluster with HDP 3.1 (nickwallen) closes apache/metron#1526
5738ea3 is described below
commit 5738ea3a5129e93cb8904ad379a984a0a5825902
Author: nickwallen
AuthorDate: Tue Oct 8 11:46:59 2019 -0400
METRON-2275 Solr Indexing Topology Fails to Start on Secure Cluster with
HDP 3.1 (nickwallen) closes apache/metron#1526
---
metron-platform/metron-solr/metron-solr-storm/pom.xml | 11 +++
1 file changed, 11 insertions(+)
diff --git a/metron-platform/metron-solr/metron-solr-storm/pom.xml
b/metron-platform/metron-solr/metron-solr-storm/pom.xml
index 38688fb..f160447 100644
--- a/metron-platform/metron-solr/metron-solr-storm/pom.xml
+++ b/metron-platform/metron-solr/metron-solr-storm/pom.xml
@@ -28,6 +28,7 @@
UTF-8
UTF-8
1.10
+${global_hbase_guava_version}
@@ -159,6 +160,16 @@
+
+com.google.common
+
org.apache.metron.guava.${guava_version}
+
+
com.fasterxml.jackson
org.apache.metron.jackson
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2265 Update Kerberos settings (merrimanr via nickwallen) closes apache/metron#1519
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 939c4ea METRON-2265 Update Kerberos settings (merrimanr via
nickwallen) closes apache/metron#1519
939c4ea is described below
commit 939c4ea6b43d73c086dddbb86078d8d6878fc83f
Author: merrimanr
AuthorDate: Thu Oct 3 13:26:07 2019 -0400
METRON-2265 Update Kerberos settings (merrimanr via nickwallen) closes
apache/metron#1519
---
metron-deployment/Kerberos-manual-setup.md | 65 +++---
.../roles/sensor-stubs/templates/start-bro-stub| 1 +
.../roles/sensor-stubs/templates/start-snort-stub | 1 +
.../roles/sensor-stubs/templates/start-yaf-stub| 1 +
.../CURRENT/package/scripts/params/params_linux.py | 2 +-
.../METRON/CURRENT/package/templates/metron.j2 | 12 ++--
.../org/apache/metron/rest/config/KafkaConfig.java | 2 +-
.../apache/metron/rest/config/KafkaConfigTest.java | 2 +-
metron-platform/elasticsearch-shaded/pom.xml | 22 +---
.../metron-common/src/main/scripts/stellar | 9 +++
metron-platform/metron-data-management/pom.xml | 7 +++
.../metron-elasticsearch-storm/pom.xml | 4 ++
.../AbstractLuceneMetaAlertUpdateDaoTest.java | 5 +-
.../apache/metron/management/KafkaFunctions.java | 11 +++-
.../metron-parsing/metron-parsing-storm/pom.xml| 12
15 files changed, 114 insertions(+), 42 deletions(-)
diff --git a/metron-deployment/Kerberos-manual-setup.md
b/metron-deployment/Kerberos-manual-setup.md
index d50da34..aeedd95 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -70,7 +70,6 @@ Setup
1. Create the `metron` user's home directory in HDFS.
```
-sudo -u hdfs hdfs dfs -mkdir /user/metron
sudo -u hdfs hdfs dfs -chown metron:hdfs /user/metron
sudo -u hdfs hdfs dfs -chmod 770 /user/metron
```
@@ -84,6 +83,38 @@ Setup a KDC
yum -y install krb5-server krb5-libs krb5-workstation
```
+1. Uncomment all commented lines in `/etc/krb5.conf`. The contents of
`/etc/krb5.conf` should look like:
+
+ ```
+ # Configuration snippets may be placed in this directory as well
+ includedir /etc/krb5.conf.d/
+
+ [logging]
+default = FILE:/var/log/krb5libs.log
+kdc = FILE:/var/log/krb5kdc.log
+admin_server = FILE:/var/log/kadmind.log
+
+ [libdefaults]
+dns_lookup_realm = false
+ticket_lifetime = 24h
+renew_lifetime = 7d
+forwardable = true
+rdns = false
+pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
+default_realm = EXAMPLE.COM
+default_ccache_name = KEYRING:persistent:%{uid}
+
+ [realms]
+EXAMPLE.COM = {
+ kdc = node1
+ admin_server = node1
+}
+
+ [domain_realm]
+.example.com = EXAMPLE.COM
+example.com = EXAMPLE.COM
+ ```
+
1. Define the current host as the KDC.
```
@@ -92,7 +123,7 @@ Setup a KDC
cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts
```
-1. Ensure that the KDC can issue renewable tickets. This may be necessary on a
real cluster, but should not be on a [single VM](development/centos6/README.md).
+1. Ensure that the KDC can issue renewable tickets.
Edit `/var/kerberos/krb5kdc/kdc.conf` and ensure the following is added to
the `realm` section
@@ -109,14 +140,14 @@ Setup a KDC
1. Start the KDC and ensure that it starts on boot.
```
-/etc/rc.d/init.d/krb5kdc start
+krb5kdc
chkconfig krb5kdc on
```
1. Start the Kerberos Admin service and ensure that it starts on boot.
```
-/etc/rc.d/init.d/kadmin start
+kadmind
chkconfig kadmin on
```
@@ -203,6 +234,7 @@ Enable Kerberos
```
kadmin.local -q "ktadd -k metron.headless.keytab met...@example.com"
+mkdir /etc/security/keytabs
cp metron.headless.keytab /etc/security/keytabs
chown metron:hadoop /etc/security/keytabs/metron.headless.keytab
chmod 440 /etc/security/keytabs/metron.headless.keytab
@@ -417,9 +449,12 @@ Push Data
1. Push some sample data to one of the parser topics. E.g for Bro we took raw
data from
[metron/metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput](../metron-platform/metron-integration-test/src/main/sample/data/bro/raw/BroExampleOutput)
```
-cat sample-bro.txt |
${KAFKA_HOME}/kafka-broker/bin/kafka-console-producer.sh \
+source /etc/default/metron
+kinit -kt $METRON_SERVICE_KEYTAB $METRON_PRINCIPAL_NAME
+export KAFKA_OPTS=$CLIENT_JAAS_ARG
+cat sample-bro.txt | ${KAFKA_HOME}/bin/kafka-console-producer.sh \
--broker-list ${BROKERLIST} \
---security-protocol SASL_PLAINTEXT \
+--producer-property securit
[metron] 01/01: Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit 34bc08a0f5a6b3f982164294e1df1682cc6c689b Merge: 9c0d395 080e074 Author: Nick Allen AuthorDate: Thu Oct 3 10:04:57 2019 -0400 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 .travis.yml| 41 +++-- .../ansible/roles/ambari_master/tasks/ambari.yml | 6 +++ metron-interface/metron-rest/README.md | 19 metron-interface/metron-rest/pom.xml | 49 + .../readme-images/debug-configuration.png | Bin 0 -> 119283 bytes .../readme-images/debug-maven-profile.png | Bin 0 -> 240634 bytes 6 files changed, 102 insertions(+), 13 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (9c0d395 -> 34bc08a)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 9c0d395 METRON-2264 Upgrade metron-hbase-client to HBase 2.0.2 (nickwallen) closes apache/metron#1518 add 3a666d5 METRON-2235 Increase server startup timeout (tigerquoll via mmiklavc) closes apache/metron#1496 add eaee9f8 METRON-2266 REST debug instructions (merrimanr via nickwallen) closes apache/metron#1520 add 080e074 METRON-2271 Reorganize Travis Build (nickwallen) closes apache/metron#1522 new 34bc08a Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .travis.yml| 41 +++-- .../ansible/roles/ambari_master/tasks/ambari.yml | 6 +++ metron-interface/metron-rest/README.md | 19 metron-interface/metron-rest/pom.xml | 49 + .../readme-images/debug-configuration.png | Bin 0 -> 119283 bytes .../readme-images/debug-maven-profile.png | Bin 0 -> 240634 bytes 6 files changed, 102 insertions(+), 13 deletions(-) create mode 100644 metron-interface/metron-rest/readme-images/debug-configuration.png create mode 100644 metron-interface/metron-rest/readme-images/debug-maven-profile.png
[metron] branch master updated: METRON-2271 Reorganize Travis Build (nickwallen) closes apache/metron#1522
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 080e074 METRON-2271 Reorganize Travis Build (nickwallen) closes apache/metron#1522 080e074 is described below commit 080e074e9d3861f102e74a46a30f6b90ac2da1c9 Author: nickwallen AuthorDate: Thu Oct 3 08:55:16 2019 -0400 METRON-2271 Reorganize Travis Build (nickwallen) closes apache/metron#1522 --- .travis.yml | 41 - 1 file changed, 28 insertions(+), 13 deletions(-) diff --git a/.travis.yml b/.travis.yml index 13ecf46..02fdd28 100644 --- a/.travis.yml +++ b/.travis.yml @@ -19,13 +19,6 @@ sudo: required addons: chrome: stable -env: - - SCRIPT="mvn surefire:test@unit-tests -T 2C" - - SCRIPT="mvn surefire:test@integration-tests" - - SCRIPT="mvn clean site --projects site-book" - - SCRIPT="mvn test --projects metron-interface/metron-config,metron-interface/metron-alerts" - - SCRIPT="./dev-utilities/build-utils/verify_licenses.sh" - install: true language: java jdk: @@ -38,14 +31,36 @@ before_install: - npm config set cache $HOME/.npm-cache --global - npm config set prefix $HOME/.npm-prefix --global -install: - - time mvn install -T 2C -q -DskipTests=true -Dmaven.javadoc.skip=true -B -V - -script: - - time $SCRIPT - matrix: fast_finish: true + include: +- name: Unit Tests + script: +- time mvn install -T 2C -q -DskipTests=true -Dmaven.javadoc.skip=true -B -V -pl '!:metron-config,!:metron-alerts' +- time mvn surefire:test@unit-tests -T 2C + +- name: Integration Tests + script: +- time mvn install -T 2C -q -DskipTests=true -Dmaven.javadoc.skip=true -B -V -pl '!:metron-config,!:metron-alerts' +- time mvn surefire:test@integration-tests + +- name: Alerts UI Tests + script: +- time mvn install -T 2C -q -DskipTests=true -Dmaven.javadoc.skip=true -B -V -pl ':metron-alerts' +- time mvn test -pl ':metron-alerts' + +- name: Management UI Tests + script: +- time mvn install -T 2C -q -DskipTests=true -Dmaven.javadoc.skip=true -B -V -pl ':metron-config' +- time mvn test -pl ':metron-config' + +- name: Site Book + script: +- time mvn clean site -pl site-book + +- name: Verify Licenses + script: +- time ./dev-utilities/build-utils/verify_licenses.sh before_cache: - rm -rf $HOME/.m2/repository/org/apache/metron
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2264 Upgrade metron-hbase-client to HBase 2.0.2 (nickwallen) closes apache/metron#1518
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 9c0d395 METRON-2264 Upgrade metron-hbase-client to HBase 2.0.2
(nickwallen) closes apache/metron#1518
9c0d395 is described below
commit 9c0d3952c92fd57a8bea8f78f4759f917f8be5f0
Author: nickwallen
AuthorDate: Wed Sep 25 10:40:26 2019 -0400
METRON-2264 Upgrade metron-hbase-client to HBase 2.0.2 (nickwallen) closes
apache/metron#1518
---
metron-platform/metron-hbase-client/pom.xml | 5 +
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/metron-platform/metron-hbase-client/pom.xml
b/metron-platform/metron-hbase-client/pom.xml
index fe497df..17647bf 100644
--- a/metron-platform/metron-hbase-client/pom.xml
+++ b/metron-platform/metron-hbase-client/pom.xml
@@ -36,15 +36,12 @@
UTF-8
UTF-8
-
-
-1.1.2
org.apache.hbase
hbase-shaded-client
-${shaded.client.version}
+${global_hbase_version}
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (f300ce3 -> 09bd629)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from f300ce3 METRON-2261 Isolate Curator Dependencies (nickwallen) closes apache/metron#1515 add 0e2eadf METRON-2247 rpm-docker: Provide an option to bypass running rpmlint (tigerquoll via nickwallen) closes apache/metron#1503 add 92220ce METRON-2257 Metron-Alerts GUI testing failing on MacOS builds (sardell) closes apache/metron#1513 new 09bd629 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../roles/metron-builder/tasks/build-rpms.yml | 4 +- metron-deployment/development/centos6/README.md| 22 +++ metron-deployment/development/centos6/Vagrantfile | 1 + metron-deployment/development/centos6/ansible.cfg | 1 + metron-deployment/development/centos7/Vagrantfile | 4 +- metron-deployment/development/centos7/ansible.cfg | 1 + metron-deployment/development/ubuntu14/Vagrantfile | 1 + .../packaging/docker/rpm-docker/build.sh | 6 +- metron-interface/metron-alerts/package-lock.json | 167 +++-- metron-interface/metron-alerts/package.json| 2 +- 10 files changed, 91 insertions(+), 118 deletions(-)
[metron] 01/01: Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit 09bd6295ee56a1c593900561e6b7fa8f501020f5 Merge: f300ce3 92220ce Author: Nick Allen AuthorDate: Mon Sep 23 15:43:20 2019 -0400 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 .../roles/metron-builder/tasks/build-rpms.yml | 4 +- metron-deployment/development/centos6/README.md| 22 +++ metron-deployment/development/centos6/Vagrantfile | 1 + metron-deployment/development/centos6/ansible.cfg | 1 + metron-deployment/development/centos7/Vagrantfile | 4 +- metron-deployment/development/centos7/ansible.cfg | 1 + metron-deployment/development/ubuntu14/Vagrantfile | 1 + .../packaging/docker/rpm-docker/build.sh | 6 +- metron-interface/metron-alerts/package-lock.json | 167 +++-- metron-interface/metron-alerts/package.json| 2 +- 10 files changed, 91 insertions(+), 118 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2261 Isolate Curator Dependencies (nickwallen) closes apache/metron#1515
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new f300ce3 METRON-2261 Isolate Curator Dependencies (nickwallen) closes
apache/metron#1515
f300ce3 is described below
commit f300ce36dd705bbb27e90d19502a1f05d2f902f5
Author: nickwallen
AuthorDate: Fri Sep 20 11:05:43 2019 -0400
METRON-2261 Isolate Curator Dependencies (nickwallen) closes
apache/metron#1515
---
metron-analytics/metron-maas-common/pom.xml| 45 --
metron-analytics/metron-maas-service/pom.xml | 76 +++---
metron-analytics/metron-profiler-spark/pom.xml | 7 +
metron-analytics/metron-profiler-storm/pom.xml | 35 +
metron-interface/metron-rest/pom.xml | 23 ++-
.../metron-common-storm/pom.xml| 21 ++-
metron-platform/metron-common/pom.xml | 90 +++-
metron-platform/metron-data-management/pom.xml | 40 +
.../metron-elasticsearch-common/pom.xml| 10 ++
.../metron-enrichment-storm/pom.xml| 25
metron-platform/metron-hbase-server/pom.xml| 30
.../metron-hbase/metron-hbase-common/pom.xml | 62 ++--
.../metron-indexing/metron-indexing-common/pom.xml | 16 +-
metron-platform/metron-integration-test/pom.xml| 161 +++--
metron-platform/metron-management/pom.xml | 40 -
.../metron-parsing/metron-parsers-common/pom.xml | 32 ++--
metron-platform/metron-pcap-backend/pom.xml| 26 +++-
metron-platform/metron-pcap/pom.xml| 15 ++
.../metron-solr/metron-solr-common/pom.xml | 10 ++
.../metron-solr/metron-solr-storm/pom.xml | 15 ++
metron-platform/metron-test-utilities/pom.xml | 64 ++--
.../metron-writer/metron-writer-common/pom.xml | 45 ++
.../metron-writer/metron-writer-storm/pom.xml | 25
metron-platform/metron-zookeeper/pom.xml | 53 +--
metron-stellar/stellar-common/pom.xml | 99 -
pom.xml| 11 +-
26 files changed, 959 insertions(+), 117 deletions(-)
diff --git a/metron-analytics/metron-maas-common/pom.xml
b/metron-analytics/metron-maas-common/pom.xml
index b404111..4fa3163 100644
--- a/metron-analytics/metron-maas-common/pom.xml
+++ b/metron-analytics/metron-maas-common/pom.xml
@@ -1,15 +1,15 @@
-
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
@@ -54,11 +54,25 @@
org.apache.curator
curator-recipes
${global_curator_version}
+
+
+
+ org.apache.zookeeper
+ zookeeper
+
+
org.apache.curator
curator-x-discovery
${global_curator_version}
+
+
+
+ org.apache.zookeeper
+ zookeeper
+
+
org.apache.httpcomponents
@@ -83,8 +97,15 @@
org.apache.curator
curator-test
- ${curator.version}
+ ${global_curator_test_version}
test
+
+
+
+ org.apache.zookeeper
+ zookeeper
+
+
diff --git a/metron-analytics/metron-maas-service/pom.xml
b/metron-analytics/metron-maas-service/pom.xml
index ebb58bd..4e81be6 100644
--- a/metron-analytics/metron-maas-service/pom.xml
+++ b/metron-analytics/metron-maas-service/pom.xml
@@ -1,15 +1,15 @@
-
http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
@@ -59,6 +59,23 @@
hadoop-yarn-server-common
${hadoop.version}
provided
+
+
+
+org.apache.curator
+curator-framework
+
+
+
+org.apache.curator
+curator-client
+
+
+
+org.apache.curator
+curator-recipes
+
+
org.apache.hadoop
@@ -94,6 +111,13 @@
hadoop-yarn-common
${hadoop.version}
provided
+
+
+
+org.apache.curator
+curator-framework
+
+
org.apache.hadoop
@@ -119,16 +143,19 @@
provided
- org.apache.curator
- curator-recipes
+
+org.apache.curator
+curator-framework
- org.apache.curator
- curator-client
+
+org.apache.curator
+curator-client
- org.apache.curator
- curator-framework
+
+org.apache.curator
+curator-recipes
org.slf4j
@@ -211
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2250 Missing services in HDP 3.1 metron mpack and installer stuck (MohanDV via nickwallen) closes apache/metron#1512
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new f6f0234 METRON-2250 Missing services in HDP 3.1 metron mpack and
installer stuck (MohanDV via nickwallen) closes apache/metron#1512
f6f0234 is described below
commit f6f023402f325b34ba192711dc14978b1ab12e60
Author: MohanDV
AuthorDate: Tue Sep 17 11:18:23 2019 -0400
METRON-2250 Missing services in HDP 3.1 metron mpack and installer stuck
(MohanDV via nickwallen) closes apache/metron#1512
---
.../METRON/CURRENT/service_advisor.py | 55 +-
1 file changed, 54 insertions(+), 1 deletion(-)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
index 6584a37..5bf4e33 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/service_advisor.py
@@ -18,7 +18,7 @@ limitations under the License.
"""
import os
import traceback
-
+import sys
import imp
SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
@@ -195,3 +195,56 @@ class
METRON${metron.short.version}ServiceAdvisor(service_advisor.ServiceAdvisor
}
return storm_site_desired_values
+
+# need to override this method from ServiceAdvisor to work around
https://issues.apache.org/jira/browse/AMBARI-25375
+def colocateService(self, hostsComponentsMap, serviceComponents):
+self.validateMetronComponentLayout(hostsComponentsMap,
serviceComponents)
+
+
+# This method is added as work around for
https://issues.apache.org/jira/browse/AMBARI-25375
+# Note: This only affects the recommendations in Ambari -
+# users can still modify the hosts for the components as they see fit
+def validateMetronComponentLayout(self, hostsComponentsMap,
serviceComponents):
+metronComponents = [
+{'name': 'METRON_ALERTS_UI'},
+{'name': 'METRON_CLIENT'},
+{'name': 'METRON_ENRICHMENT_MASTER'},
+{'name': 'METRON_INDEXING'},
+{'name': 'METRON_MANAGEMENT_UI'},
+{'name': 'METRON_PARSERS'},
+{'name': 'METRON_PCAP'},
+{'name': 'METRON_PROFILER'},
+{'name': 'METRON_REST'}
+]
+
+# find any metron components that have not been assigned to a host
+unassignedMetronComponents = []
+for component in metronComponents:
+ if (component not in hostsComponentsMap.values()):
+unassignedMetronComponents.append(component)
+
+if len(unassignedMetronComponents) > 0:
+ # assign each unassigned metron component to a host
+ default_host = self.getDefaultHost(metronComponents,
hostsComponentsMap)
+ for unassigned in unassignedMetronComponents:
+ self.logger.info("Anassigned component {0} to host
{1}".format(unassigned, default_host))
+ hostsComponentsMap[default_host].append(unassigned)
+
+
+def getDefaultHost(self, metronComponents, hostsComponentsMap):
+# fist, attempt to colocate metron; suggest a host where metron is
already assigned
+default_host = None
+for component in metronComponents:
+if default_host is None:
+for host, hostComponents in hostsComponentsMap.items():
+if component in hostComponents:
+ default_host = host
+ break
+
+# if there are no assigned metron components, just choose the first
known host
+if default_host is None:
+default_host = hostsComponentsMap.keys()[0]
+self.logger.info("No hosts found with Metron components. Using
first known host: host={0}".format(default_host))
+
+return default_host
+
[metron] branch master updated: METRON-2247 rpm-docker: Provide an option to bypass running rpmlint (tigerquoll via nickwallen) closes apache/metron#1503
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 0e2eadf METRON-2247 rpm-docker: Provide an option to bypass running
rpmlint (tigerquoll via nickwallen) closes apache/metron#1503
0e2eadf is described below
commit 0e2eadf7dc7aa9db3e77f97b43c5846625162059
Author: tigerquoll
AuthorDate: Tue Sep 17 08:33:43 2019 -0400
METRON-2247 rpm-docker: Provide an option to bypass running rpmlint
(tigerquoll via nickwallen) closes apache/metron#1503
---
.../roles/metron-builder/tasks/build-rpms.yml | 4 +++-
metron-deployment/development/centos6/README.md| 22 ++
metron-deployment/development/centos6/Vagrantfile | 1 +
metron-deployment/development/centos6/ansible.cfg | 1 +
metron-deployment/development/centos7/Vagrantfile | 4 +---
metron-deployment/development/centos7/ansible.cfg | 1 +
metron-deployment/development/ubuntu14/Vagrantfile | 1 +
.../packaging/docker/rpm-docker/build.sh | 6 +-
8 files changed, 35 insertions(+), 5 deletions(-)
diff --git
a/metron-deployment/ansible/roles/metron-builder/tasks/build-rpms.yml
b/metron-deployment/ansible/roles/metron-builder/tasks/build-rpms.yml
index 7a5b6bd..f3883b8 100644
--- a/metron-deployment/ansible/roles/metron-builder/tasks/build-rpms.yml
+++ b/metron-deployment/ansible/roles/metron-builder/tasks/build-rpms.yml
@@ -15,8 +15,10 @@
# limitations under the License.
#
---
-- name: Build Metron RPM Packages
+- name: Build Metron RPM Packages {{ '(skipping rpmlint)' if SKIP_RPMLINT is
defined else '' }}
shell: "{{ item }}"
+ environment:
+SKIP_RPMLINT: "{{ SKIP_RPMLINT if SKIP_RPMLINT is defined else '' }}"
args:
chdir: "{{ metron_build_dir }}/metron-deployment"
with_items:
diff --git a/metron-deployment/development/centos6/README.md
b/metron-deployment/development/centos6/README.md
index 5ca3ed5..a00c5cb 100644
--- a/metron-deployment/development/centos6/README.md
+++ b/metron-deployment/development/centos6/README.md
@@ -78,6 +78,28 @@ Any platform that supports these tools is suitable, but the
following instructio
vagrant provision
```
+### Deployment optimizations
+
+1. Set environment variable
+```
+export ANSIBLE_ARGS='--extra-vars "SKIP_RPMLINT=1"'
+```
+To disable running rpmlint as part of the dev deployment task - this can
save a couple of minutes of time on the deployment.
+Either add this variable to your profile, or use it on the command line
like
+
+```
+env ANSIBLE_ARGS='--extra-vars "SKIP_RPMLINT=1"' vagrant up
+```
+
+### Deployment debugging
+
+1. To enable more verbose logging of ansible actions during the deployment,
use
+```
+env ANSIBLE_ARGS=' -' vagrant up
+```
+As this can produce large amounts of logging, it is best to redirect
output to a file for later analysis.
+
+
### Explore Metron
Navigate to the following resources to explore your newly minted Apache Metron
environment.
diff --git a/metron-deployment/development/centos6/Vagrantfile
b/metron-deployment/development/centos6/Vagrantfile
index 50571b4..21ca2bd 100644
--- a/metron-deployment/development/centos6/Vagrantfile
+++ b/metron-deployment/development/centos6/Vagrantfile
@@ -102,5 +102,6 @@ Vagrant.configure(2) do |config|
ansible.skip_tags = ansibleSkipTags.split(",") if ansibleSkipTags != ''
ansible.inventory_path = "ansible/inventory"
ansible.compatibility_mode = "auto"
+ansible.raw_arguments = Shellwords.shellsplit(ENV['ANSIBLE_ARGS']) if
ENV['ANSIBLE_ARGS']
end
end
diff --git a/metron-deployment/development/centos6/ansible.cfg
b/metron-deployment/development/centos6/ansible.cfg
index 66f0afd..70ac5de 100644
--- a/metron-deployment/development/centos6/ansible.cfg
+++ b/metron-deployment/development/centos6/ansible.cfg
@@ -21,6 +21,7 @@ roles_path = ../../ansible/roles
pipelining = True
log_path = ./ansible.log
callback_plugins = ../../ansible/callback_plugins
+callback_whitelist = profile_tasks
# fix for "ssh throws 'unix domain socket too long' " problem
[ssh_connection]
diff --git a/metron-deployment/development/centos7/Vagrantfile
b/metron-deployment/development/centos7/Vagrantfile
index 4706e42..28db76a 100644
--- a/metron-deployment/development/centos7/Vagrantfile
+++ b/metron-deployment/development/centos7/Vagrantfile
@@ -64,7 +64,6 @@ Vagrant.configure(2) do |config|
if Vagrant.has_plugin?("vagrant-cachier")
config.cache.enable :yum
config.cache.scope = :box
-
config.cache.synced_folder_opts = {
type: :nfs,
mount_options: ['rw', 'vers=3', 'tcp', 'nolock']
@@ -102,7 +101,6 @@ Vagrant.configure(2) do |config|
ansible.skip_tags =
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (802cbdd -> eeccdc4)
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git.
from 802cbdd METRON-2188 Upgrade to HBase 2.0.2 (nickwallen) closes
apache/metron#1506
add 722eb66 METRON-2211 [UI] Alerts UI should optionally render timestamp
in local time (sardell) closes apache/metron#1495
add b1536d7 METRON-2254 Intermittent Test Failure in
RestFunctionsIntegrationTest (nickwallen) closes apache/metron#1510
new eeccdc4 Merge remote-tracking branch 'apache/master' into
feature/METRON-2088-support-hdp-3.1
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../alerts-list/table-view/table-view.component.ts | 13 +++-
.../alerts-list/tree-view/tree-view.component.ts | 6 +-
.../configure-rows/configure-rows.component.html | 3 +
.../configure-rows.component.spec.ts | 2 +
.../alerts/configure-rows/configure-rows.module.ts | 8 ++-
.../timezone-config.component.spec.ts | 76 ++
.../timezone-config/timezone-config.component.ts} | 27
.../timezone-config.service.spec.ts| 55
.../timezone-config/timezone-config.service.ts}| 30 +
.../functions/RestFunctionsIntegrationTest.java| 18 +++--
10 files changed, 200 insertions(+), 38 deletions(-)
create mode 100644
metron-interface/metron-alerts/src/app/alerts/configure-rows/timezone-config/timezone-config.component.spec.ts
copy metron-interface/metron-alerts/src/app/{shared/switch/switch.component.ts
=> alerts/configure-rows/timezone-config/timezone-config.component.ts} (58%)
create mode 100644
metron-interface/metron-alerts/src/app/alerts/configure-rows/timezone-config/timezone-config.service.spec.ts
copy metron-interface/metron-alerts/src/app/{shared/login-guard.ts =>
alerts/configure-rows/timezone-config/timezone-config.service.ts} (61%)
[metron] 01/01: Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit eeccdc4f6d96ec3c3d19880e88b79278578d5963 Merge: 802cbdd b1536d7 Author: Nick Allen AuthorDate: Fri Sep 13 11:32:56 2019 -0400 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 .../alerts-list/table-view/table-view.component.ts | 13 +++- .../alerts-list/tree-view/tree-view.component.ts | 6 +- .../configure-rows/configure-rows.component.html | 3 + .../configure-rows.component.spec.ts | 2 + .../alerts/configure-rows/configure-rows.module.ts | 8 ++- .../timezone-config.component.spec.ts | 76 ++ .../timezone-config.component.ts} | 32 - .../timezone-config.service.spec.ts| 55 .../timezone-config.service.ts}| 34 ++ .../functions/RestFunctionsIntegrationTest.java| 18 +++-- 10 files changed, 206 insertions(+), 41 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2188 Upgrade to HBase 2.0.2 (nickwallen) closes apache/metron#1506
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/feature/METRON-2088-support-hdp-3.1 by this push: new 802cbdd METRON-2188 Upgrade to HBase 2.0.2 (nickwallen) closes apache/metron#1506 802cbdd is described below commit 802cbdd71a972f281a445e8025040f863987964f Author: nickwallen AuthorDate: Thu Sep 12 13:06:39 2019 -0400 METRON-2188 Upgrade to HBase 2.0.2 (nickwallen) closes apache/metron#1506 --- dependencies_with_url.csv | 33 + metron-analytics/metron-profiler-storm/pom.xml |1 + .../CURRENT/package/scripts/enrichment_commands.py |2 +- .../UpdateControllerIntegrationTest.java |4 + metron-platform/metron-common/pom.xml |7 + metron-platform/metron-data-management/pom.xml | 32 +- .../mr/LeastRecentlyUsedPrunerIntegrationTest.java |5 +- ...pleEnrichmentFlatFileLoaderIntegrationTest.java |3 +- .../metron-elasticsearch-common/pom.xml| 12 + .../enrichment/adapters/cif/CIFHbaseAdapter.java |8 +- .../enrichment/converter/AbstractConverter.java|5 +- .../lookup/accesstracker/AccessTrackerUtil.java|2 +- .../simplehbase/SimpleHBaseAdapterTest.java|2 +- .../metron-enrichment-storm/pom.xml| 36 +- metron-platform/metron-hbase-server/pom.xml|6 + .../hbase/coprocessor/EnrichmentCoprocessor.java | 17 +- .../EnrichmentCoprocessorIntegrationTest.java |4 +- .../metron-hbase/metron-hbase-common/pom.xml | 17 + .../java/org/apache/metron/hbase/HBaseUtils.java | 49 + .../apache/metron/hbase/client/HBaseClient.java|4 +- .../metron/hbase/client/HBaseClientTest.java |5 +- .../metron/hbase/mock/MockHBaseTableProvider.java |3 +- .../org/apache/metron/hbase/mock/MockHTable.java | 1345 +++- .../metron-parsing/metron-parsers/pom.xml |9 + metron-stellar/stellar-common/pom.xml |7 + pom.xml|4 + 26 files changed, 942 insertions(+), 680 deletions(-) diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv index 70bfb85..623a021 100644 --- a/dependencies_with_url.csv +++ b/dependencies_with_url.csv @@ -81,7 +81,9 @@ org.json4s:json4s-ast_2.11:jar:3.2.11:compile,ASLv2,https://github.com/json4s/js org.json4s:json4s-core_2.11:jar:3.2.11:compile,ASLv2,https://github.com/json4s/json4s org.json4s:json4s-jackson_2.11:jar:3.2.11:compile,ASLv2,https://github.com/json4s/json4s org.jruby.jcodings:jcodings:jar:1.0.8:compile,MIT License,https://github.com/jruby/jcodings +org.jruby.jcodings:jcodings:jar:1.0.18:compile,MIT License,https://github.com/jruby/jcodings org.jruby.joni:joni:jar:2.1.2:compile,MIT License,https://github.com/jruby/joni +org.jruby.joni:joni:jar:2.1.11:compile,MIT License,https://github.com/jruby/joni org.lz4:lz4-java:jar:1.4.0:compile,ASLv2,https://github.com/lz4/lz4-java org.mitre.taxii:taxii:jar:1.1.0.1:compile,The BSD 3-Clause License,https://github.com/TAXIIProject/java-taxii org.mitre:stix:jar:1.2.0.2:compile,The BSD 3-Clause License,https://github.com/STIXProject/java-stix @@ -107,9 +109,11 @@ com.sun.jersey:jersey-server:jar:1.9:compile,CDDL 1.1,https://jersey.java.net/ com.thoughtworks.paranamer:paranamer:jar:2.3:compile,BSD,https://github.com/paul-hammant/paranamer javax.servlet.jsp:jsp-api:jar:2.1:runtime,CDDL,http://oracle.com javax.servlet.jsp:jsp-api:jar:2.1:compile,CDDL,http://oracle.com +javax.servlet.jsp:javax.servlet.jsp-api:jar:2.3.1:compile,CDDL,http://oracle.com javax.servlet:servlet-api:jar:2.5:compile,CDDL,http://oracle.com net.jcip:jcip-annotations:jar:1.0:compile,Public,http://jcip.net/ org.codehaus.jettison:jettison:jar:1.1:compile,ASLv2,https://github.com/codehaus/jettison +org.codehaus.jettison:jettison:jar:1.3.8:compile,ASLv2,https://github.com/codehaus/jettison org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile,BSD 3-clause,https://github.com/fusesource/leveldbjni org.hamcrest:hamcrest-core:jar:1.1:runtime,BSD 2-clause,http://hamcrest.org/JavaHamcrest/ org.hamcrest:hamcrest-core:jar:1.3:compile,BSD 2-clause,http://hamcrest.org/JavaHamcrest/ @@ -146,6 +150,7 @@ com.101tec:zkclient:jar:0.10:compile,The Apache Software License, Version 2.0,ht com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile,Apache License, Version 2.0,http://stephenc.github.com/findbugs-annotations com.github.tony19:named-regexp:jar:0.2.3:compile,Apache License, Version 2.0, com.google.code.findbugs:jsr305:jar:1.3.9:compile,The Apache Software License, Version 2.0,http://findbugs.sourceforge.net/ +com.google.code.findbugs:jsr305:jar:2.0.1:compile,The Apache Software License, Version 2.0,http://findbugs.sourceforge.net
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2252 PcapTopologyIntegrationTest Intermittent Failures (nickwallen) closes apache/metron#1508
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new b25f9a5 METRON-2252 PcapTopologyIntegrationTest Intermittent Failures
(nickwallen) closes apache/metron#1508
b25f9a5 is described below
commit b25f9a52d3186d72d1e3000f1e608e57b294c10a
Author: nickwallen
AuthorDate: Thu Sep 12 09:57:22 2019 -0400
METRON-2252 PcapTopologyIntegrationTest Intermittent Failures (nickwallen)
closes apache/metron#1508
---
.../src/main/java/org/apache/metron/integration/utils/KafkaUtil.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/utils/KafkaUtil.java
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/utils/KafkaUtil.java
index 0292808..f9271aa 100644
---
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/utils/KafkaUtil.java
+++
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/utils/KafkaUtil.java
@@ -36,6 +36,7 @@ public class KafkaUtil {
Thread.sleep(sleepBetween);
}
}
+// wait for all messages to be sent
+producer.flush();
}
-
}
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (f77946d -> 9e53a83)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from f77946d METRON-2231 Revert METRON-2175, METRON-2176, METRON-2177 in HDP 3.1 upgrade feature branch (nickwallen) closes apache/metron#1497 add 891ebd5 METRON-2150 [UI] User not able to filter by multiple values of the same field on Alerts UI (tiborm via sardell) closes apache/metron#1443 add 42ce5ad METRON-2148 Stellar REST POST function (merrimanr) closes apache/metron#1440 add f36bba7 Metron-2180 IS_DATE stellar function is lenient while parsing date/time (MohanDV) closes apache/metron#1461 add 5425f8f METRON-2183 Update to Angular v7 (sardell) closes apache/metron#1463 add b516a75 METRON-2161 [UI] CSS positioning bugs in Alerts and MGMT UI (sardell) closes apache/metron#1446 add 753e8ea METRON-2174 [UI] Grouped alerts total can differ from search alerts total (sardell) closes apache/metron#1455 add 7e39143 METRON-2079 Fix documentation for installing Ansible for fulldev Centos 6 (mmiklavc) closes apache/metron#1450 add 43f74fc METRON-2185 Update Simple-Syslog dependency to fix error in Structured Data (ottobackwards) closes apache/metron#1466 add ed63eac METRON-2172 Solr Updates Not Tested in Integration Test (nickwallen) closes apache/metron#1465 add 5e1e3bd METRON-2140: [UI] Implement logic behind show/hide RESOLVE and DISMISS items in Alerts UI (tiborm via mmiklavc) closes apache/metron#1459 add 65681a0 METRON-2129 [UI] Clearing the search bar resets alert filter range to 'All Time' (ruffle1986 via mmiklavc) closes apache/metron#1420 add 6306c93 METRON-2130 [UI] Numeric steppers on the Management UI seems broken (ruffle1986 via mmiklavc) closes apache/metron#1421 add 6256535 METRON-2191 [UI] Checkbox selector on Alerts UI is broken (tiborm via sardell) closes apache/metron#1468 add 8a5e7e7 METRON-2192 [UI] All time time range is broken on Alerts UI (tiborm via sardell) closes apache/metron#1469 add 855de4a METRON-2194 Update Ambari tooltip to specify single quotes for parser names with hyphens (mmiklavc) closes apache/metron#1471 add 340827f METRON-2189 Optimize imports in mpack python scripts (mmiklavc) closes apache/metron#1467 add 8c4b53d METRON-2197 Add debugging info output for Solr queries (mmiklavc) closes apache/metron#1475 add 0239432 METRON-2202 Add parameter validation for the stellar field validation functions (MohanDV via mmiklavc) closes apache/metron#1476 add 771d66b METRON-2195 Add defensive log level checks when constructing logs is expensive (tigerquoll via mmiklavc) closes apache/metron#1473 add 3bea6ad METRON-2205 Cease querying on filter or time-range change (tiborm via sardell) closes apache/metron#1478 add f6054a4 METRON-614: Eliminate use of the default Charset (justinleet) closes apache/metron#1341 add 88f4d2c METRON-614: Eliminate use of the default Charset (mmiklavc via justinleet) closes apache/metron#1341 add 5e93677 METRON-2076 Fixed up flakey stellar timezone test (tigerquoll via mmiklavc) closes apache/metron#1487 add 7219606 METRON-2212 Add debugging developer docs to hbase-server README (mmiklavc) closes apache/metron#1481 add 4ab1fdb METRON-2179 [UI] Make navigation in both UIs consistent (sardell) closes apache/metron#1464 add e6ca3c7 METRON-2149 Shaded jar classifier is not consistent (merrimanr) closes apache/metron#1436 add 1a6d77f METRON-2238 Streaming enrichments regression (merrimanr via mmiklavc) closes apache/metron#1498 add 888c4bc METRON-2199 [UI] Add ability to turn off query building in Alerts UI search input (sardell) closes apache/metron#1477 add edc449f METRON-2243 [UI] Update npm dependencies to resolve audit warnings (sardell) closes apache/metron#1500 add 5eed666 METRON-2221 Notebook import fails through Zeppelin REST API (anandsubbu) closes apache/metron#1489 add 67fa5a4 METRON-2227 Increase Kafka test harness timeout (tigerquoll via mmiklavc) closes apache/metron#1493 add 3b96474 METRON-2201 The description for the IS_IP method default behavior needs to corrected as per implementation (MohanDV via mmiklavc) closes apache/metron#1474 add c402e64 METRON-2217 Migrate current HBase client from HTableInterface to Table (mmiklavc) closes apache/metron#1483 new e635965 Merge remote-tracking branch 'apache/master' into MERGE-MASTER-SEPT new 9e53a83 METRON-2248 Merge Master into Feature Branch The 2 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: Upgrading.md
[metron] 01/02: Merge remote-tracking branch 'apache/master' into MERGE-MASTER-SEPT
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit e6359655733395c4c2a45b126611a823909075f7 Merge: f77946d c402e64 Author: Nick Allen AuthorDate: Wed Sep 4 17:15:12 2019 -0400 Merge remote-tracking branch 'apache/master' into MERGE-MASTER-SEPT Upgrading.md | 9 + dependencies_with_url.csv | 2 +- metron-analytics/metron-maas-common/pom.xml| 1 + .../java/org/apache/metron/maas/util/RESTUtil.java | 4 +- metron-analytics/metron-maas-service/pom.xml | 6 + .../apache/metron/maas/service/runner/Runner.java | 3 +- .../metron/maas/service/MaasIntegrationTest.java | 7 +- metron-analytics/metron-profiler-client/pom.xml| 8 + .../src/main/assembly/assembly.xml | 2 +- .../profiler/client/HBaseProfilerClient.java |34 +- .../metron/profiler/client/stellar/GetProfile.java |61 +- .../profiler/client/stellar/VerboseProfile.java|57 +- .../profiler/client/HBaseProfilerClientTest.java |30 +- .../metron/profiler/client/ProfileWriter.java |25 +- .../profiler/client/stellar/GetProfileTest.java|35 +- .../client/stellar/VerboseProfileTest.java |50 +- metron-analytics/metron-profiler-common/pom.xml| 6 + .../profiler/DefaultMessageDistributorTest.java| 6 +- .../profiler/hbase/SaltyRowKeyBuilderTest.java |33 +- metron-analytics/metron-profiler-repl/pom.xml | 8 + .../src/main/assembly/assembly.xml | 2 +- .../profiler/repl/StandAloneProfilerTest.java | 3 +- metron-analytics/metron-profiler-spark/pom.xml | 3 + .../src/main/assembly/assembly.xml | 2 +- .../metron/profiler/spark/BatchProfilerConfig.java |11 +- .../spark/function/MessageRouterFunction.java | 9 +- .../src/main/scripts/start_batch_profiler.sh | 4 +- metron-analytics/metron-profiler-storm/pom.xml | 8 + .../metron/profiler/storm/ProfileSplitterBolt.java |11 +- .../src/main/scripts/start_profiler_topology.sh| 3 +- .../profiler/storm/ProfileSplitterBoltTest.java| 6 +- .../storm/integration/ConfigUploadComponent.java | 3 +- .../storm/integration/ProfilerIntegrationTest.java | 4 +- metron-analytics/metron-statistics/pom.xml | 3 + metron-contrib/metron-performance/pom.xml | 8 + .../src/main/assembly/assembly.xml | 2 +- .../metron/performance/load/LoadOptions.java | 8 +- .../load/monitor/writers/CSVWriter.java| 5 +- .../src/main/scripts/load_tool.sh | 2 +- .../metron/performance/load/LoadOptionsTest.java | 5 +- metron-deployment/development/centos6/README.md| 5 +- .../5.6.14/package/scripts/elastic_commands.py |14 +- .../5.6.14/package/scripts/elastic_master.py | 7 +- .../5.6.14/package/scripts/elastic_slave.py| 3 +- .../5.6.14/package/scripts/service_check.py| 4 +- .../KIBANA/5.6.14/package/scripts/common.py| 8 +- .../KIBANA/5.6.14/package/scripts/kibana_master.py | 2 +- .../CURRENT/configuration/metron-parsers-env.xml | 2 +- .../common-services/METRON/CURRENT/metainfo.xml| 5 + .../CURRENT/package/scripts/alerts_ui_commands.py | 4 +- .../CURRENT/package/scripts/alerts_ui_master.py|10 +- .../package/scripts/dashboard/dashboardindex.py|10 +- .../CURRENT/package/scripts/enrichment_commands.py | 6 +- .../CURRENT/package/scripts/enrichment_master.py |10 +- .../CURRENT/package/scripts/indexing_commands.py |79 +- .../CURRENT/package/scripts/indexing_master.py |32 +- .../package/scripts/management_ui_commands.py | 6 +- .../package/scripts/management_ui_master.py| 7 +- .../CURRENT/package/scripts/metron_client.py |11 +- .../CURRENT/package/scripts/metron_security.py | 7 +- .../CURRENT/package/scripts/metron_service.py |12 +- .../CURRENT/package/scripts/params/params.py | 2 +- .../CURRENT/package/scripts/params/params_linux.py | 9 +- .../package/scripts/params/status_params.py| 4 +- .../CURRENT/package/scripts/parser_commands.py | 9 +- .../CURRENT/package/scripts/parser_master.py | 7 +- .../CURRENT/package/scripts/pcap_commands.py |11 +- .../METRON/CURRENT/package/scripts/pcap_master.py |13 +- .../CURRENT/package/scripts/profiler_commands.py | 6 +- .../CURRENT/package/scripts/profiler_master.py |13 +- .../CURRENT/package/scripts/rest_commands.py | 6 +- .../METRON/CURRENT/package/scripts/rest_master.py | 8 +- .../CURRENT/package/scripts/service_check.py |14
[metron] 02/02: METRON-2248 Merge Master into Feature Branch
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
commit 9e53a83e68d5dfa0486ec885f3659a7302573dd4
Author: Nick Allen
AuthorDate: Thu Sep 5 11:17:44 2019 -0400
METRON-2248 Merge Master into Feature Branch
---
metron-platform/metron-enrichment/metron-enrichment-storm/pom.xml | 6 ++
metron-platform/metron-parsing/metron-parsing-storm/pom.xml | 7 +++
metron-platform/metron-solr/metron-solr-storm/pom.xml | 6 ++
3 files changed, 19 insertions(+)
diff --git a/metron-platform/metron-enrichment/metron-enrichment-storm/pom.xml
b/metron-platform/metron-enrichment/metron-enrichment-storm/pom.xml
index 832d070..6aee1e6 100644
--- a/metron-platform/metron-enrichment/metron-enrichment-storm/pom.xml
+++ b/metron-platform/metron-enrichment/metron-enrichment-storm/pom.xml
@@ -33,6 +33,12 @@
${global_hbase_guava_version}
+
+
+org.apache.commons
+commons-lang3
+3.2
+
diff --git a/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
b/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
index 9bd898e..c983be5 100644
--- a/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
+++ b/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
@@ -30,6 +30,13 @@
UTF-8
+
+
+ org.apache.commons
+ commons-lang3
+ 3.2
+
+
org.apache.metron
diff --git a/metron-platform/metron-solr/metron-solr-storm/pom.xml
b/metron-platform/metron-solr/metron-solr-storm/pom.xml
index b515ceb..3d55616 100644
--- a/metron-platform/metron-solr/metron-solr-storm/pom.xml
+++ b/metron-platform/metron-solr/metron-solr-storm/pom.xml
@@ -83,6 +83,12 @@
test-jar
+org.apache.metron
+stellar-common
+${project.parent.version}
+provided
+
+
org.apache.storm
flux-core
${global_flux_version}
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2241 Profiler Integration Test Fails After Storm 1.2.1 Upgrade (nickwallen) closes apache/metron#1499
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/feature/METRON-2088-support-hdp-3.1 by this push: new 1004b69 METRON-2241 Profiler Integration Test Fails After Storm 1.2.1 Upgrade (nickwallen) closes apache/metron#1499 1004b69 is described below commit 1004b694b2157acb9c8dda51fa49dd088d47fbbe Author: nickwallen AuthorDate: Wed Sep 4 14:35:54 2019 -0400 METRON-2241 Profiler Integration Test Fails After Storm 1.2.1 Upgrade (nickwallen) closes apache/metron#1499 --- pom.xml | 1 + 1 file changed, 1 insertion(+) diff --git a/pom.xml b/pom.xml index 3cbd2bc..b632115 100644 --- a/pom.xml +++ b/pom.xml @@ -146,6 +146,7 @@ 2.0.0 1.7.25 1.2.1 +1.2.3 0.8.0 7.4.0
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2224 Upgrade to Zeppelin 0.8.0 (merrimanr via nickwallen) closes apache/metron#1491
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 1a41a2c METRON-2224 Upgrade to Zeppelin 0.8.0 (merrimanr via
nickwallen) closes apache/metron#1491
1a41a2c is described below
commit 1a41a2cf1bd394cdf15eb83e3a182b4e1addaff5
Author: merrimanr
AuthorDate: Fri Aug 23 13:37:21 2019 -0400
METRON-2224 Upgrade to Zeppelin 0.8.0 (merrimanr via nickwallen) closes
apache/metron#1491
---
dependencies_with_url.csv | 5 +
metron-stellar/stellar-zeppelin/README.md | 7 ---
metron-stellar/stellar-zeppelin/pom.xml | 2 +-
.../apache/metron/stellar/zeppelin/StellarInterpreter.java| 11 +++
.../metron/stellar/zeppelin/StellarInterpreterTest.java | 9 +
pom.xml | 3 +++
6 files changed, 25 insertions(+), 12 deletions(-)
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index 04f7935..0065f9b 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -50,6 +50,7 @@
javax.xml.bind:jaxb-api:jar:2.2.2:compile,CDDL,https://jaxb.dev.java.net/
javax.xml.bind:jaxb-api:jar:2.3.0:compile,CDDL,https://jaxb.dev.java.net/
javax.xml.stream:stax-api:jar:1.0-2:compile,COMMON DEVELOPMENT AND
DISTRIBUTION LICENSE (CDDL) Version
1.0,https://docs.oracle.com/javase/7/docs/api/javax/xml/stream/package-summary.html
jline:jline:jar:0.9.94:compile,BSD,http://jline.sourceforge.net
+jline:jline:jar:2.12.1:compile,BSD,http://jline.sourceforge.net
junit:junit:jar:4.12:compile,Eclipse Public License 1.0,http://junit.org
junit:junit:jar:4.4:compile,Common Public License Version 1.0,http://junit.org
net.razorvine:pyrolite:jar:4.13:compile,MIT,https://github.com/irmen/Pyrolite
@@ -61,6 +62,7 @@
org.abego.treelayout:org.abego.treelayout.core:jar:1.0.1:compile,BSD 3-Clause "N
org.adrianwalker:multiline-string:jar:0.1.2:compile,Common Public License
Version 1.0,https://github.com/benelog/multiline
org.antlr:antlr4-runtime:jar:4.5:compile,BSD 3-Clause
License,http://www.antlr.org
org.bouncycastle:bcprov-jdk15on:jar:1.52:compile,MIT,https://www.bouncycastle.org/license.html
+org.bouncycastle:bcpkix-jdk15on:jar:1.52:compile,MIT,https://www.bouncycastle.org/license.html
org.clojure:clojure:jar:1.6.0:compile,Eclipse Public License
1.0,http://clojure.org/
org.clojure:clojure:jar:1.7.0:compile,Eclipse Public License
1.0,http://clojure.org/
org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile,Apache
v2,http://jackson.codehaus.org
@@ -185,6 +187,7 @@
com.fasterxml:classmate:jar:1.3.4:compile,ASLv2,http://github.com/cowtowncoder/j
com.google.code.gson:gson:jar:2.2.2:compile,The Apache Software License,
Version 2.0,http://code.google.com/p/google-gson/
com.google.code.gson:gson:jar:2.2.4:compile,The Apache Software License,
Version 2.0,http://code.google.com/p/google-gson/
com.google.code.gson:gson:jar:2.7:compile,The Apache Software License, Version
2.0,http://code.google.com/p/google-gson/
+com.google.code.gson:gson:jar:2.8.5:compile,The Apache Software License,
Version 2.0,http://code.google.com/p/google-gson/
com.google.guava:guava:jar:11.0.2:compile,ASLv2,
com.google.guava:guava:jar:12.0.1:compile,ASLv2,
com.google.guava:guava:jar:12.0:compile,ASLv2,
@@ -224,6 +227,7 @@
commons-collections:commons-collections:jar:3.2.1:compile,ASLv2,http://commons.a
commons-collections:commons-collections:jar:3.2.2:compile,ASLv2,http://commons.apache.org/collections/
commons-configuration:commons-configuration:jar:1.10:compile,ASLv2,http://commons.apache.org/configuration/
commons-configuration:commons-configuration:jar:1.6:compile,The Apache
Software License, Version
2.0,http://commons.apache.org/${pom.artifactId.substring(8)}/
+commons-configuration:commons-configuration:jar:1.9:compile,The Apache
Software License, Version
2.0,http://commons.apache.org/${pom.artifactId.substring(8)}/
commons-daemon:commons-daemon:jar:1.0.13:compile,ASLv2,http://commons.apache.org/daemon/
commons-digester:commons-digester:jar:1.8.1:compile,ASLv2,http://commons.apache.org/digester/
commons-digester:commons-digester:jar:1.8:compile,The Apache Software License,
Version 2.0,http://jakarta.apache.org/commons/digester/
@@ -504,3 +508,4 @@ org.elasticsearch:securesm:jar:1.2:compile
com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile,ASLv2,http://stephenc.github.io/jcip-annotations/
com.nimbusds:nimbus-jose-jwt:jar:4.41.2:compile,ASLv2,https://bitbucket.org/connect2id/nimbus-jose-jwt/wiki/Home
tomcat:jasper-compiler:jar:5.5.23:compile,ASLv2,https://tomcat.apache.org/
+org.danilopianini:gson-extras:jar:0.2.1:compile,ASLv2,https://github.com/DanySK/gson-extras
diff
[metron] branch master updated: METRON-2172 Solr Updates Not Tested in Integration Test (nickwallen) closes apache/metron#1465
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new ed63eac METRON-2172 Solr Updates Not Tested in Integration Test
(nickwallen) closes apache/metron#1465
ed63eac is described below
commit ed63eac260e442f8c6f17dd93f6d87fc946cae0d
Author: nickwallen
AuthorDate: Fri Jul 19 13:00:38 2019 -0400
METRON-2172 Solr Updates Not Tested in Integration Test (nickwallen) closes
apache/metron#1465
---
.../ElasticsearchUpdateIntegrationTest.java| 5 --
.../metron/indexing/dao/UpdateIntegrationTest.java | 84 +-
.../integration/HBaseDaoIntegrationTest.java | 5 --
.../org/apache/metron/solr/dao/SolrUtilities.java | 5 +-
.../apache/metron/solr/dao/SolrSearchDaoTest.java | 22 --
.../apache/metron/solr/dao/SolrUtilitiesTest.java | 5 +-
.../SolrRetrieveLatestIntegrationTest.java | 22 +++---
.../integration/SolrUpdateIntegrationTest.java | 54 +-
8 files changed, 122 insertions(+), 80 deletions(-)
diff --git
a/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
b/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
index 16760d2..97afd41 100644
---
a/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
+++
b/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
@@ -160,11 +160,6 @@ public class ElasticsearchUpdateIntegrationTest extends
UpdateIntegrationTest {
return es.getAllIndexedDocs(index, SENSOR_NAME + "_doc");
}
- @Override
- protected MockHTable getMockHTable() {
-return table;
- }
-
/**
* Install the index template to ensure that "guid" is of type "keyword". The
* {@link
org.apache.metron.elasticsearch.dao.ElasticsearchRetrieveLatestDao} cannot find
diff --git
a/metron-platform/metron-indexing/metron-indexing-common/src/test/java/org/apache/metron/indexing/dao/UpdateIntegrationTest.java
b/metron-platform/metron-indexing/metron-indexing-common/src/test/java/org/apache/metron/indexing/dao/UpdateIntegrationTest.java
index 6333d32..ab41c08 100644
---
a/metron-platform/metron-indexing/metron-indexing-common/src/test/java/org/apache/metron/indexing/dao/UpdateIntegrationTest.java
+++
b/metron-platform/metron-indexing/metron-indexing-common/src/test/java/org/apache/metron/indexing/dao/UpdateIntegrationTest.java
@@ -17,12 +17,12 @@ package org.apache.metron.indexing.dao;
import org.adrianwalker.multilinestring.Multiline;
import org.apache.commons.collections.MapUtils;
import org.apache.metron.common.Constants;
-import org.apache.metron.hbase.mock.MockHTable;
import org.apache.metron.indexing.dao.search.AlertComment;
import org.apache.metron.indexing.dao.update.CommentAddRemoveRequest;
import org.apache.metron.indexing.dao.update.Document;
import org.apache.metron.indexing.dao.update.OriginalNotFoundException;
import org.apache.metron.indexing.dao.update.PatchRequest;
+import org.json.simple.parser.ParseException;
import org.junit.Assert;
import org.junit.Test;
@@ -35,9 +35,10 @@ import java.util.Optional;
import java.util.UUID;
import java.util.stream.Collectors;
-import static org.apache.metron.indexing.dao.IndexDao.COMMENTS_FIELD;
import static org.hamcrest.CoreMatchers.hasItem;
+import static org.apache.metron.indexing.dao.IndexDao.COMMENTS_FIELD;
+
public abstract class UpdateIntegrationTest {
/**
@@ -272,29 +273,81 @@ public abstract class UpdateIntegrationTest {
return new Document(message1, guid, SENSOR_NAME, timestamp);
}
- private List getComments(Document withComment) {
-List> commentsField =
List.class.cast(withComment.getDocument().get(COMMENTS_FIELD));
+ private static List getComments(Document withComment) throws
ParseException {
+return getComments(withComment.getDocument());
+ }
+
+ private static List getComments(Map fields)
throws ParseException {
List comments = new ArrayList<>();
-if(commentsField != null) {
- comments = commentsField
- .stream()
- .map(map -> new AlertComment(map))
- .collect(Collectors.toList());
+boolean hasComments = fields.containsKey(COMMENTS_FIELD);
+if(hasComments) {
+ List commentsField = List.class.cast(fields.get(COMMENTS_FIELD));
+ for (Object commentObject: commentsField) {
+if (commentObject instanceof Map) {
+ // comments are stored as maps in
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2175 Introduce HBase Connection Abstractions for HBase 2.0.2 (nickwallen) closes apache/metron#1456
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new ef3a76c METRON-2175 Introduce HBase Connection Abstractions for HBase
2.0.2 (nickwallen) closes apache/metron#1456
ef3a76c is described below
commit ef3a76c799d48856d8e4999226ec9392f298a2f7
Author: nickwallen
AuthorDate: Thu Jul 18 09:56:31 2019 -0400
METRON-2175 Introduce HBase Connection Abstractions for HBase 2.0.2
(nickwallen) closes apache/metron#1456
---
.../metron/profiler/client/ProfileWriter.java | 6 +-
.../spark/function/HBaseWriterFunction.java| 4 +-
.../org/apache/metron/hbase/bolt/HBaseBolt.java| 8 +-
.../apache/metron/hbase/bolt/HBaseBoltTest.java| 6 +-
.../org/apache/metron/rest/config/HBaseConfig.java | 6 +-
.../impl/SensorEnrichmentConfigServiceImpl.java| 6 +-
.../org/apache/metron/rest/config/TestConfig.java | 6 +-
.../SensorEnrichmentConfigServiceImplTest.java | 6 +-
.../metron-elasticsearch-common/pom.xml| 18 ++
.../metron/hbase/coprocessor/HBaseCacheWriter.java | 4 +-
.../java/org/apache/metron/hbase/ColumnList.java | 97 ++-
.../metron/hbase/HBaseProjectionCriteria.java | 10 +-
.../apache/metron/hbase/client/HBaseClient.java| 317 -
.../metron/hbase/client/HBaseClientFactory.java| 70 +
.../hbase/client/HBaseConnectionFactory.java | 59
.../metron/hbase/client/HBaseTableClient.java | 285 ++
.../hbase/client/HBaseTableClientFactory.java | 54
.../metron/hbase/client/HBaseWriterParams.java | 51
.../{HBaseClient.java => LegacyHBaseClient.java} | 4 +-
.../metron/hbase/client/HBaseClientTest.java | 10 +-
.../HBaseTableClientIntegrationTest.java | 286 +++
metron-platform/metron-pcap/pom.xml| 5 +
pom.xml| 26 +-
23 files changed, 1039 insertions(+), 305 deletions(-)
diff --git
a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java
b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java
index 4e00164..d4e5e66 100644
---
a/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java
+++
b/metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/ProfileWriter.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hbase.client.Durability;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.metron.hbase.HTableProvider;
import org.apache.metron.hbase.ColumnList;
-import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.LegacyHBaseClient;
import org.apache.metron.profiler.ProfileMeasurement;
import org.apache.metron.profiler.ProfilePeriod;
import org.apache.metron.profiler.hbase.ColumnBuilder;
@@ -47,13 +47,13 @@ public class ProfileWriter {
private RowKeyBuilder rowKeyBuilder;
private ColumnBuilder columnBuilder;
- private HBaseClient hbaseClient;
+ private LegacyHBaseClient hbaseClient;
private HBaseProfilerClient client;
public ProfileWriter(RowKeyBuilder rowKeyBuilder, ColumnBuilder
columnBuilder, HTableInterface table, long periodDurationMillis) {
this.rowKeyBuilder = rowKeyBuilder;
this.columnBuilder = columnBuilder;
-this.hbaseClient = new HBaseClient((c, t) -> table,
table.getConfiguration(), table.getName().getNameAsString());
+this.hbaseClient = new LegacyHBaseClient((c, t) -> table,
table.getConfiguration(), table.getName().getNameAsString());
this.client = new HBaseProfilerClient(table, rowKeyBuilder, columnBuilder,
periodDurationMillis);
}
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
index cfabd94..6a090cf 100644
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
+++
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/HBaseWriterFunction.java
@@ -26,7 +26,7 @@ import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Durability;
import org.apache.metron.hbase.HTableProvider;
import org.apache.metron.hbase.TableProvider;
-import org.apache.metron.hbase.client.HBaseClient;
+import org.apache.metron.hbase.client.LegacyHBaseClient;
import org.apache.metron.profiler.ProfileMeasurement;
import org.apache.metron.profiler.hbase.ColumnBuilder
[metron] 01/01: Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
commit 51c0ad0648fa017aa1b2cfd51a83b488b7b831d9
Merge: 1430d62 bf673a2
Author: Nick Allen
AuthorDate: Wed Jul 3 09:54:52 2019 -0400
Merge remote-tracking branch 'apache/master' into
feature/METRON-2088-support-hdp-3.1
.travis.yml| 5 +-
Upgrading.md | 5 +
metron-deployment/development/README.md| 9 +
metron-deployment/packaging/ambari/.gitignore | 2 -
.../packaging/ambari/metron-mpack/pom.xml | 6 +-
.../configuration/metron-enrichment-env.xml| 82 ---
.../CURRENT/package/scripts/enrichment_commands.py | 15 +-
.../CURRENT/package/scripts/enrichment_master.py | 9 +-
.../CURRENT/package/scripts/params/params_linux.py | 12 -
.../METRON/CURRENT/themes/metron_theme.json| 63 +--
.../packaging/docker/rpm-docker/SPECS/metron.spec | 6 +-
metron-platform/Performance-tuning-guide.md| 10 +-
.../dao/ElasticsearchRetrieveLatestDao.java| 8 +
.../ElasticsearchUpdateIntegrationTest.java| 57 +-
.../metron-enrichment-storm/README.md | 15 +-
.../metron-enrichment-storm/enrichment_arch.png| Bin 113606 -> 0 bytes
.../main/config/enrichment-splitjoin.properties| 63 ---
.../main/config/enrichment-splitjoin.properties.j2 | 63 ---
...nt-unified.properties => enrichment.properties} | 0
...fied.properties.j2 => enrichment.properties.j2} | 0
.../src/main/flux/enrichment/remote-splitjoin.yaml | 593 -
.../{remote-unified.yaml => remote.yaml} | 0
.../metron/enrichment/bolt/EnrichmentJoinBolt.java | 122 -
.../enrichment/bolt/EnrichmentSplitterBolt.java| 184 ---
.../apache/metron/enrichment/bolt/JoinBolt.java| 189 ---
.../apache/metron/enrichment/bolt/SplitBolt.java | 113
.../enrichment/bolt/ThreatIntelJoinBolt.java | 123 -
.../enrichment/bolt/ThreatIntelSplitterBolt.java | 67 ---
.../src/main/scripts/start_enrichment_topology.sh | 7 +-
.../enrichment/bolt/EnrichmentJoinBoltTest.java| 96
.../bolt/EnrichmentSplitterBoltTest.java | 106
.../metron/enrichment/bolt/JoinBoltTest.java | 193 ---
.../metron/enrichment/bolt/SplitBoltTest.java | 123 -
.../enrichment/bolt/ThreatIntelJoinBoltTest.java | 222
.../bolt/ThreatIntelSplitterBoltTest.java | 46 --
.../integration/EnrichmentIntegrationTest.java | 87 +--
.../UnifiedEnrichmentIntegrationTest.java | 96
.../apache/metron/indexing/dao/MultiIndexDao.java | 3 +-
.../metron/indexing/dao/update/PatchException.java | 27 +
.../metron/indexing/dao/update/PatchOperation.java | 28 +
.../metron/indexing/dao/update/PatchUtils.java | 105
.../metron/indexing/dao/update/UpdateDao.java | 3 +-
.../metron/indexing/dao/MultiIndexDaoTest.java | 173 +-
.../metron/indexing/dao/UpdateIntegrationTest.java | 56 +-
.../metron/indexing/dao/update/PatchUtilsTest.java | 263 +
.../apache/metron/pcap/mr/FileFilterUtilTest.java | 37 +-
46 files changed, 790 insertions(+), 2702 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (1430d62 -> 51c0ad0)
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git.
from 1430d62 METRON-2161 Ambari client exception occurred: No JSON object
could be decoded (nickwallen) closes apache/metron#1447
add f2f3bb4 METRON-2155 Cache Maven in Travis CI Builds (nickwallen)
closes apache/metron#1445
add c9604c2 METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116
expected:<1> but was:<2> (mmiklavc) closes apache/metron#1452
add 02f88a3 METRON-2164 Remove the Split-Join Enrichment Topology
(nickwallen) closes apache/metron#1448
add 8792882 METRON-2061 Solr documents with date fields cannot be updated
with Dao classes (merrimanr) closes apache/metron#1374
add c8f2c9a METRON-2084 Add documentation notice for MacOS Mojave users
for new security permissions (mmiklavc) closes apache/metron#1449
add bf673a2 METRON-2168 Elasticsearch Updates Not Tested in Integration
Test (nickwallen) closes apache/metron#1451
new 51c0ad0 Merge remote-tracking branch 'apache/master' into
feature/METRON-2088-support-hdp-3.1
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.travis.yml| 5 +-
Upgrading.md | 5 +
metron-deployment/development/README.md| 9 +
metron-deployment/packaging/ambari/.gitignore | 2 -
.../packaging/ambari/metron-mpack/pom.xml | 6 +-
.../configuration/metron-enrichment-env.xml| 82 ---
.../CURRENT/package/scripts/enrichment_commands.py | 15 +-
.../CURRENT/package/scripts/enrichment_master.py | 9 +-
.../CURRENT/package/scripts/params/params_linux.py | 12 -
.../METRON/CURRENT/themes/metron_theme.json| 63 +--
.../packaging/docker/rpm-docker/SPECS/metron.spec | 6 +-
metron-platform/Performance-tuning-guide.md| 10 +-
.../dao/ElasticsearchRetrieveLatestDao.java| 8 +
.../ElasticsearchUpdateIntegrationTest.java| 57 +-
.../metron-enrichment-storm/README.md | 15 +-
.../metron-enrichment-storm/enrichment_arch.png| Bin 113606 -> 0 bytes
.../main/config/enrichment-splitjoin.properties| 63 ---
.../main/config/enrichment-splitjoin.properties.j2 | 63 ---
...nt-unified.properties => enrichment.properties} | 0
...fied.properties.j2 => enrichment.properties.j2} | 0
.../src/main/flux/enrichment/remote-splitjoin.yaml | 593 -
.../{remote-unified.yaml => remote.yaml} | 0
.../metron/enrichment/bolt/EnrichmentJoinBolt.java | 122 -
.../enrichment/bolt/EnrichmentSplitterBolt.java| 184 ---
.../apache/metron/enrichment/bolt/JoinBolt.java| 189 ---
.../apache/metron/enrichment/bolt/SplitBolt.java | 113
.../enrichment/bolt/ThreatIntelJoinBolt.java | 123 -
.../enrichment/bolt/ThreatIntelSplitterBolt.java | 67 ---
.../src/main/scripts/start_enrichment_topology.sh | 7 +-
.../enrichment/bolt/EnrichmentJoinBoltTest.java| 96
.../bolt/EnrichmentSplitterBoltTest.java | 106
.../metron/enrichment/bolt/JoinBoltTest.java | 193 ---
.../metron/enrichment/bolt/SplitBoltTest.java | 123 -
.../enrichment/bolt/ThreatIntelJoinBoltTest.java | 222
.../bolt/ThreatIntelSplitterBoltTest.java | 46 --
.../integration/EnrichmentIntegrationTest.java | 87 +--
.../UnifiedEnrichmentIntegrationTest.java | 96
.../apache/metron/indexing/dao/MultiIndexDao.java | 3 +-
.../metron/indexing/dao/update/PatchException.java | 16 +-
.../metron/indexing/dao/update/PatchOperation.java | 18 +-
.../metron/indexing/dao/update/PatchUtils.java | 105
.../metron/indexing/dao/update/UpdateDao.java | 3 +-
.../metron/indexing/dao/MultiIndexDaoTest.java | 173 +-
.../metron/indexing/dao/UpdateIntegrationTest.java | 56 +-
.../metron/indexing/dao/update/PatchUtilsTest.java | 263 +
.../apache/metron/pcap/mr/FileFilterUtilTest.java | 37 +-
46 files changed, 750 insertions(+), 2721 deletions(-)
delete mode 100644
metron-platform/metron-enrichment/metron-enrichment-storm/enrichment_arch.png
delete mode 100644
metron-platform/metron-enrichment/metron-enrichment-storm/src/main/config/enrichment-splitjoin.properties
delete mode 100755
metron-platform/metron-enrichment/metron-enrichment-storm/src/main/config/enrichment-splitjoin.properties.j2
rename
metron-platform/metron-enrichment/metron-enrichment-storm/src/main/config/{enrichment-unified.properties
=> enrichment.properties} (100%)
rename
metron-platform/metron-enrichment/metr
[metron] branch master updated: METRON-2168 Elasticsearch Updates Not Tested in Integration Test (nickwallen) closes apache/metron#1451
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new bf673a2 METRON-2168 Elasticsearch Updates Not Tested in Integration
Test (nickwallen) closes apache/metron#1451
bf673a2 is described below
commit bf673a2e9c00a59716e70feea55d581af91145f1
Author: nickwallen
AuthorDate: Wed Jul 3 09:48:53 2019 -0400
METRON-2168 Elasticsearch Updates Not Tested in Integration Test
(nickwallen) closes apache/metron#1451
---
.../dao/ElasticsearchRetrieveLatestDao.java| 8 +
.../ElasticsearchUpdateIntegrationTest.java| 57 ++-
.../apache/metron/indexing/dao/MultiIndexDao.java | 3 +-
.../metron/indexing/dao/MultiIndexDaoTest.java | 173 +++--
.../metron/indexing/dao/UpdateIntegrationTest.java | 56 +--
5 files changed, 262 insertions(+), 35 deletions(-)
diff --git
a/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchRetrieveLatestDao.java
b/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchRetrieveLatestDao.java
index 95d27db..8044d62 100644
---
a/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchRetrieveLatestDao.java
+++
b/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchRetrieveLatestDao.java
@@ -43,6 +43,14 @@ import static
org.elasticsearch.index.query.QueryBuilders.boolQuery;
import static org.elasticsearch.index.query.QueryBuilders.termsQuery;
import static org.elasticsearch.index.query.QueryBuilders.typeQuery;
+/**
+ * A {@link RetrieveLatestDao} that retrieves documents from Elasticsearch.
+ *
+ * The index being searched must have a field "guid" of type "keyword",
+ * otherwise no documents can be found by "guid".
+ *
+ * See
https://www.elastic.co/guide/en/elasticsearch/reference/5.6/query-dsl-term-query.html
+ */
public class ElasticsearchRetrieveLatestDao implements RetrieveLatestDao {
private ElasticsearchClient client;
diff --git
a/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
b/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
index 6489206..16760d2 100644
---
a/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
+++
b/metron-platform/metron-elasticsearch/metron-elasticsearch-common/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
@@ -19,9 +19,15 @@ package org.apache.metron.elasticsearch.integration;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.google.common.collect.Iterables;
+import org.adrianwalker.multilinestring.Multiline;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
+import org.apache.http.HttpEntity;
+import org.apache.http.entity.ContentType;
+import org.apache.http.nio.entity.NStringEntity;
import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.elasticsearch.client.ElasticsearchClient;
+import org.apache.metron.elasticsearch.client.ElasticsearchClientFactory;
import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
import
org.apache.metron.elasticsearch.integration.components.ElasticSearchComponent;
import org.apache.metron.hbase.mock.MockHBaseTableProvider;
@@ -29,17 +35,21 @@ import org.apache.metron.hbase.mock.MockHTable;
import org.apache.metron.indexing.dao.AccessConfig;
import org.apache.metron.indexing.dao.HBaseDao;
import org.apache.metron.indexing.dao.IndexDao;
-import org.apache.metron.indexing.dao.MultiIndexDao;
import org.apache.metron.indexing.dao.UpdateIntegrationTest;
import org.apache.metron.integration.UnableToStartException;
+import org.elasticsearch.action.support.WriteRequest;
+import org.elasticsearch.client.Response;
+import org.hamcrest.CoreMatchers;
import org.junit.After;
import org.junit.AfterClass;
+import org.junit.Assert;
import org.junit.Before;
import org.junit.BeforeClass;
import java.io.File;
import java.io.IOException;
import java.text.SimpleDateFormat;
+import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.List;
@@ -55,11 +65,27 @@ public class ElasticsearchUpdateIntegrationTest extends
UpdateIntegrationTest {
private static final String TABLE_NAME = "modifications";
private static
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2161 Ambari client exception occurred: No JSON object could be decoded (nickwallen) closes apache/metron#1447
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 1430d62 METRON-2161 Ambari client exception occurred: No JSON object
could be decoded (nickwallen) closes apache/metron#1447
1430d62 is described below
commit 1430d628ef2671983fd656b0b9fd1454adcf82e2
Author: nickwallen
AuthorDate: Fri Jun 28 10:00:53 2019 -0400
METRON-2161 Ambari client exception occurred: No JSON object could be
decoded (nickwallen) closes apache/metron#1447
---
.../roles/ambari_config/tasks/dependencies-CentOS-7.yml | 15 ++-
1 file changed, 6 insertions(+), 9 deletions(-)
diff --git
a/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
b/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
index 72a96d4..e5105a5 100644
---
a/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
+++
b/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
@@ -15,12 +15,9 @@
# limitations under the License.
#
---
-- name: Install urllib3 with pip
- pip:
-name: urllib3
-version: 1.10.2
-
-- name: Install requests with pip
- pip:
-name: requests
-version: 2.6.1
+- name: Install Python dependencies
+ yum: name={{item}}
+ with_items:
+ - python-urllib3
+ - python-requests
+ when: ansible_distribution == "CentOS"
[metron] branch master updated: METRON-2155 Cache Maven in Travis CI Builds (nickwallen) closes apache/metron#1445
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new f2f3bb4 METRON-2155 Cache Maven in Travis CI Builds (nickwallen) closes apache/metron#1445 f2f3bb4 is described below commit f2f3bb4877b02ce40e9712a9bd0c91f6e4d40d75 Author: nickwallen AuthorDate: Tue Jun 25 18:41:18 2019 -0400 METRON-2155 Cache Maven in Travis CI Builds (nickwallen) closes apache/metron#1445 --- .travis.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.travis.yml b/.travis.yml index 0cb408f..13ecf46 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,10 +31,10 @@ language: java jdk: - oraclejdk8 before_install: - - curl --retry 10 -O https://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip - - unzip -qq apache-maven-3.3.9-bin.zip - export M2_HOME=$PWD/apache-maven-3.3.9 - export PATH=$M2_HOME/bin:$PATH + - '[[ -d $M2_HOME ]] || curl --retry 10 -O https://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip' + - '[[ -d $M2_HOME ]] || unzip -qq apache-maven-3.3.9-bin.zip' - npm config set cache $HOME/.npm-cache --global - npm config set prefix $HOME/.npm-prefix --global @@ -58,3 +58,4 @@ cache: - metron-interface/metron-config/node_modules - $HOME/.m2 - $HOME/.npm + - $M2_HOME
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (58f4519 -> 11f9663)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 58f4519 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 add adf477d METRON-2089 [UI] Adding loading state to Alerts details view (tiborm via sardell) closes apache/metron#1390 add 5a3fd55 METRON-2111 Update public web site to point at 0.7.1 new release (justinleet) closes apache/metron#1406 add fcd681d METRON-1997 Replace Threat Triage Score Field Slider with Text Box (ruffle1986 via sardell) closes apache/metron#1334 add b8df052 METRON-2107 Add architecture diagram item to PR checklist (mmiklavc) closes apache/metron#1401 add 2653157 METRON-2058 UI: Actions - Add to Alert can still be selected from dropdown when no alerts are selected. (ruffle1986 via sardell) closes apache/metron#1373 add 506c18c METRON-2085 [UI] Alerts UI Details Pane: naming meta alerts is broken (ruffle1986 via sardell) closes apache/metron#1388 add 22cc622 METRON-2118 Added a LEEF parser (simonellistonball via mmiklavc) closes apache/metron#1408 add 3b04460 METRON-1788 Batch profiler pull profile information from zookeeper (tigerquoll via mmiklavc) closes apache/metron#1383 add a6281d3 METRON-2113 Update version to 0.7.2 (justinleet) closes apache/metron#1407 add 373ac51 METRON-2109 Add option to use Metron GUID as the id in Elasticsearch (merrimanr) closes apache/metron#1403 add 9b70adf METRON-2123 Expand Stellar JOIN to work on all Iterables (mmiklavc) closes apache/metron#1416 add ccb5bec METRON-2143 Travis Build Fails to Download Maven (nickwallen) closes apache/metron#1433 add bfe662d METRON-2128 LEEF config file is missing in RPM spec file (simonellistonball via mmiklavc) closes apache/metron#1419 add b9a130c METRON-2087 Remove Storm dependency from metron-indexing (merrimanr) closes apache/metron#1389 add 3754ff3 METRON-2112 Normalize parser original_string handling (mmiklavc) closes apache/metron#1409 add 2dd753e METRON-2152 Add debug logging for when sensor batchTimeout exceeds the calculated maximum (mmiklavc) closes apache/metron#1437 add a8f2d3a METRON-2083 Fix broken links in root metron README (mmiklavc) closes apache/metron#1435 add dbf8bf9 METRON-2145 Clarify RPM build documentation (mmiklavc) closes apache/metron#1434 add e16ff31 METRON-2127 Update Maven repositories to https (justinleet via mmiklavc) closes apache/metron#1417 add a1487ec METRON-2153 ParserIntegrationTest should print failed messages (merrimanr) closes apache/metron#1438 add 5bd7e01 METRON-2102 [UI] Adding click-through navigation to Alerts table (tiborm via mmiklavc) closes apache/metron#1431 add 3c13173 METRON-2141 Cache REST API status update calls to the Storm UI (mmiklavc) closes apache/metron#1439 add 9621ec7 METRON-2092 [UI] Config UI does not require you to set a grok timestamp field by default (ruffle1986 via sardell) closes apache/metron#1393 add 38b8a78 METRON-2073 Create in-memory use case for enrichment with map type and flatfile summarizer (merrimanr) closes apache/metron#1399 add ee1b2c3 METRON-1253 Manual pasting of timestamps into the timestamp picker (ruffle1986 via sardell) closes apache/metron#1380 add 09bbdf3 METRON-2142 Install solar schema as metron user (MohanDV via nickwallen) closes apache/metron#1432 add a2a46e6 METRON-2156 Remove Storm dependency from metron-hbase (merrimanr) closes apache/metron#1441 new 11f9663 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/PULL_REQUEST_TEMPLATE.md | 2 + .travis.yml| 2 +- README.md | 4 +- metron-analytics/metron-maas-common/pom.xml| 2 +- metron-analytics/metron-maas-service/pom.xml | 2 +- metron-analytics/metron-profiler-client/pom.xml| 6 +- .../metron/profiler/client/ProfileWriter.java | 2 +- metron-analytics/metron-profiler-common/pom.xml| 4 +- .../metron/profiler/hbase/ColumnBuilder.java | 2 +- .../profiler/hbase/ValueOnlyColumnBuilder.java | 2 +- metron-analytics/metron-profiler-repl/pom.xml | 2 +- metron-analytics/metron-profiler-spark/README.md | 61 +- metron-analytics/metron-profiler-spark/pom.xml | 12 +- .../profiler/spark/cli/BatchProfilerCLI.java | 126 ++- .../spark/cli/BatchProfilerCLIOptions.java | 15 +-
[metron] 01/01: Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit 11f966395fc18d832306f3b5f6353e9a76947e6c Merge: 58f4519 a2a46e6 Author: Nick Allen AuthorDate: Tue Jun 25 17:54:32 2019 -0400 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 .github/PULL_REQUEST_TEMPLATE.md | 2 + .travis.yml| 2 +- README.md | 4 +- metron-analytics/metron-maas-common/pom.xml| 2 +- metron-analytics/metron-maas-service/pom.xml | 2 +- metron-analytics/metron-profiler-client/pom.xml| 6 +- .../metron/profiler/client/ProfileWriter.java | 2 +- metron-analytics/metron-profiler-common/pom.xml| 4 +- .../metron/profiler/hbase/ColumnBuilder.java | 2 +- .../profiler/hbase/ValueOnlyColumnBuilder.java | 2 +- metron-analytics/metron-profiler-repl/pom.xml | 2 +- metron-analytics/metron-profiler-spark/README.md | 61 +- metron-analytics/metron-profiler-spark/pom.xml | 12 +- .../profiler/spark/cli/BatchProfilerCLI.java | 126 ++- .../spark/cli/BatchProfilerCLIOptions.java | 15 +- .../src/main/scripts/start_batch_profiler.sh | 18 +- .../profiler/spark/cli/BatchProfilerCLITest.java | 28 +- .../spark/cli/BatchProfilerZKIntegrationTest.java | 82 ++ metron-analytics/metron-profiler-storm/pom.xml | 15 +- .../org/apache/metron/hbase/bolt/BatchHelper.java | 0 .../org/apache/metron/hbase/bolt/HBaseBolt.java| 2 +- .../metron/hbase/bolt/mapper/HBaseMapper.java | 1 + .../metron/profiler/storm/ProfileHBaseMapper.java | 2 +- .../apache/metron/hbase/bolt/HBaseBoltTest.java| 4 +- .../apache/metron/hbase/bolt/mapper}/Widget.java | 2 +- .../metron/hbase/bolt/mapper}/WidgetMapper.java| 5 +- metron-analytics/metron-statistics/pom.xml | 2 +- metron-analytics/pom.xml | 2 +- metron-contrib/metron-docker/pom.xml | 2 +- metron-contrib/metron-performance/pom.xml | 2 +- metron-contrib/pom.xml | 2 +- metron-deployment/amazon-ec2/conf/defaults.yml | 2 +- .../ansible/playbooks/docker_probe_install.yml | 2 +- .../ansible/roles/ambari_master/defaults/main.yml | 2 +- .../centos6/ansible/inventory/group_vars/all | 2 +- .../ubuntu14/ansible/inventory/group_vars/all | 2 +- .../packaging/ambari/elasticsearch-mpack/pom.xml | 4 +- .../packaging/ambari/metron-mpack/pom.xml | 10 +- .../CURRENT/configuration/metron-rest-env.xml | 13 +- .../common-services/METRON/CURRENT/metainfo.xml| 12 +- .../CURRENT/package/scripts/indexing_commands.py | 16 +- .../CURRENT/package/scripts/params/params_linux.py | 3 + .../package/templates/alerts-ui-app-config.json.j2 | 3 +- .../METRON/CURRENT/package/templates/metron.j2 | 3 + .../METRON/CURRENT/themes/metron_theme.json| 20 + .../metron-mpack/src/main/resources/mpack.json | 2 +- .../packaging/docker/deb-docker/README.md | 2 +- .../packaging/docker/deb-docker/pom.xml| 4 +- .../packaging/docker/rpm-docker/README.md | 4 +- .../packaging/docker/rpm-docker/SPECS/metron.spec | 124 ++- .../packaging/docker/rpm-docker/pom.xml| 22 +- metron-deployment/pom.xml | 2 +- metron-interface/metron-alerts/README.md | 4 + .../cypress/fixtures/context-menu.conf.json| 49 ++ .../integration/alert-list/context-menu.spec.js| 89 ++ .../cypress/integration/pcap/pcap.spec.js | 2 +- metron-interface/metron-alerts/package-lock.json | 2 +- metron-interface/metron-alerts/package.json| 2 +- metron-interface/metron-alerts/pom.xml | 2 +- .../metron-alerts/scripts/package.json | 2 +- .../alert-details/alert-details.component.html | 56 +- .../alert-details/alert-details.component.scss | 17 +- .../alert-details/alert-details.component.spec.ts | 63 ++ .../alert-details/alert-details.component.ts | 41 +- .../alerts/alerts-list/alerts-list.component.html | 12 +- .../alerts/alerts-list/alerts-list.component.ts| 64 +- .../table-view/table-view.component.html | 89 +- .../table-view/table-view.component.spec.ts| 2 + .../alerts-list/table-view/table-view.component.ts | 9 +- .../src/app/app.module.spec.ts}| 7 +- .../metron-alerts/src/app/model/alert-source.ts| 1 + .../src/app/service/app-config.service.spec.ts | 154 .../src/app/service/app-config.service.ts | 21 +- .../src/app/shared/context-menu/README.md | 203 + .../context-menu.component.html} | 16
[metron] branch master updated: METRON-2143 Travis Build Fails to Download Maven (nickwallen) closes apache/metron#1433
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new ccb5bec METRON-2143 Travis Build Fails to Download Maven (nickwallen) closes apache/metron#1433 ccb5bec is described below commit ccb5bec1dd101850c18faee128bbca77af815880 Author: nickwallen AuthorDate: Sun May 26 08:41:13 2019 -0400 METRON-2143 Travis Build Fails to Download Maven (nickwallen) closes apache/metron#1433 --- .travis.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 58f8861..0cb408f 100644 --- a/.travis.yml +++ b/.travis.yml @@ -31,7 +31,7 @@ language: java jdk: - oraclejdk8 before_install: - - wget https://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip + - curl --retry 10 -O https://archive.apache.org/dist/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.zip - unzip -qq apache-maven-3.3.9-bin.zip - export M2_HOME=$PWD/apache-maven-3.3.9 - export PATH=$M2_HOME/bin:$PATH
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (1245fd1 -> 58f4519)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 1245fd1 METRON-2097 Install Metron MPack in Ambari 2.7.3.0 (nickwallen) closes apache/metron#1397 add 024aa20 METRON-2094 Create CentOS 7 Development Environment (nickwallen) closes apache/metron#1395 add 4390e38 METRON-2093 Metron RC check script is outdated (justinleet) closes apache/metron#1394 add f3f5c97 METRON-2018 Update prepare-commit to add Bro plugin tests (ottobackwards) closes apache/metron#1348 add e1d1901 METRON-2100 Update developer documentation for full dev management UI parser aggregation feature gap (mmiklavc) closes apache/metron#1398 add 57bbab2 METRON-2106 Escalation topic setting in Ambari has no effect (merrimanr) closes apache/metron#1400 add d92e0a2 METRON-2075 Site book build support for MacOS that has GNU sed installed (tigerquoll via mmiklavc) closes apache/metron#1384 add fb80e32 METRON-1989 Tooltip for ES mpack path_data is incorrect (JonZeolla via mmiklavc) closes apache/metron#1329 new 58f4519 Merge remote-tracking branch 'apache/master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: Upgrading.md | 7 ++-- .../committer-utils/metron-committer-common| 43 +- dev-utilities/release-utils/metron-rc-check| 21 ++- .../release-utils/prepare-release-candidate| 3 +- metron-deployment/README.md| 35 ++ .../ambari_config/tasks/dependencies-CentOS-7.yml | 14 --- .../5.6.14/configuration/elastic-site.xml | 2 +- .../metron-rest/src/main/scripts/metron-rest.sh| 2 +- site-book/bin/generate-md.sh | 3 +- 9 files changed, 105 insertions(+), 25 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2097 Install Metron MPack in Ambari 2.7.3.0 (nickwallen) closes apache/metron#1397
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 1245fd1 METRON-2097 Install Metron MPack in Ambari 2.7.3.0
(nickwallen) closes apache/metron#1397
1245fd1 is described below
commit 1245fd160a5ae826a8e2900528ab12c64350fdc6
Author: nickwallen
AuthorDate: Thu May 9 13:54:59 2019 -0400
METRON-2097 Install Metron MPack in Ambari 2.7.3.0 (nickwallen) closes
apache/metron#1397
---
.../ambari_config/tasks/dependencies-CentOS-7.yml | 14 +++--
.../roles/ambari_gather_facts/tasks/main.yml | 36 ++
.../roles/sensor-stubs/templates/start-bro-stub| 7 +++--
.../roles/sensor-stubs/templates/start-snort-stub | 7 +++--
.../roles/sensor-stubs/templates/start-yaf-stub| 7 +++--
.../centos7/ansible/inventory/group_vars/all | 6 +++-
.../5.6.14/configuration/elastic-env.xml | 10 +++---
.../ELASTICSEARCH/5.6.14/package/scripts/params.py | 22 +++--
.../KIBANA/5.6.14/configuration/kibana-env.xml | 12 +---
.../KIBANA/5.6.14/package/scripts/params.py| 21 +++--
.../src/main/resources/mpack.json | 11 +--
.../METRON/CURRENT/configuration/metron-env.xml| 16 +-
.../CURRENT/package/scripts/params/params.py | 22 +++--
.../CURRENT/package/scripts/params/params_linux.py | 20 +---
.../package/scripts/params/status_params.py| 14 ++---
.../METRON/CURRENT/themes/metron_theme.json| 1 +
.../metron-mpack/src/main/resources/mpack.json | 16 +++---
17 files changed, 155 insertions(+), 87 deletions(-)
diff --git
a/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
b/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
index 72a96d4..d7fc829 100644
---
a/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
+++
b/metron-deployment/ansible/roles/ambari_config/tasks/dependencies-CentOS-7.yml
@@ -15,12 +15,8 @@
# limitations under the License.
#
---
-- name: Install urllib3 with pip
- pip:
-name: urllib3
-version: 1.10.2
-
-- name: Install requests with pip
- pip:
-name: requests
-version: 2.6.1
+- name: Install Ambari dependencies
+ yum: name={{ item }}
+ with_items:
+ - python-urllib3
+ - python-requests
diff --git a/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml
b/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml
index 25f0982..c76c7bf 100644
--- a/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml
+++ b/metron-deployment/ansible/roles/ambari_gather_facts/tasks/main.yml
@@ -56,8 +56,8 @@
#
- name: "Ask Ambari: core_site_tag"
shell: >
-{{ curl }} '{{ base_url }}/hosts/{{ namenode_host
}}/host_components/NAMENODE' \
- | python -c '{{ parse_json
}}["HostRoles"]["actual_configs"]["core-site"]["default"]'
+{{ curl }} '{{ base_url }}/configurations?type=core-site' \
+ | python -c '{{ parse_json }}["items"][-1]["tag"]'
args:
warn: false
register: core_site_tag_response
@@ -104,17 +104,16 @@
# kafka_broker_tag
#
- name: "Ask Ambari: kafka_broker_tag"
- uri:
-url: "{{ base_url }}/hosts/{{ kafka_broker_hosts[0]
}}/host_components/KAFKA_BROKER"
-user: "{{ ambari_user }}"
-password: "{{ ambari_password }}"
-force_basic_auth: yes
-return_content: yes
+ shell: >
+{{ curl }} '{{ base_url }}/configurations?type=kafka-broker' \
+ | python -c '{{ parse_json }}["items"][-1]["tag"]'
+ args:
+warn: false
register: kafka_broker_tag_response
when: kafka_broker_tag is undefined
- set_fact:
-kafka_broker_tag: "{{ (kafka_broker_tag_response.content |
from_json).HostRoles.actual_configs['kafka-broker'].default }}"
+kafka_broker_tag: "{{ kafka_broker_tag_response.stdout_lines[0] }}"
when: kafka_broker_tag is undefined
#
@@ -158,17 +157,16 @@
# zookeeper_tag
#
- name: "Ask Ambari: zookeeper_tag"
- uri:
-url: "http://{{ groups.ambari_master[0] }}:{{ ambari_port
}}/api/v1/clusters/{{ cluster_name }}/hosts/{{ zookeeper_hosts[0]
}}/host_components/ZOOKEEPER_SERVER"
-user: "{{ ambari_user }}"
-password: "{{ ambari_password }}"
-force_basic_auth: yes
-return_content: yes
+ shell: >
+{{ curl }} '{{ base_url }}/configurations?type=zookeeper-env' \
+ | python -c '{{ parse_json }}["items"][-1]["tag"]'
+ args:
+warn: false
register: zookeeper_tag_response
when: zookeeper_tag is undefined
- set_fact:
-
[metron] branch master updated: METRON-2094 Create CentOS 7 Development Environment (nickwallen) closes apache/metron#1395
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 024aa20 METRON-2094 Create CentOS 7 Development Environment
(nickwallen) closes apache/metron#1395
024aa20 is described below
commit 024aa201bc4cbe00af71c57b9e0fe27b51305030
Author: nickwallen
AuthorDate: Wed May 1 10:48:45 2019 -0400
METRON-2094 Create CentOS 7 Development Environment (nickwallen) closes
apache/metron#1395
---
.../ansible/playbooks/metron_full_install.yml | 3 -
.../ansible/roles/ambari_common/defaults/main.yml | 3 +-
...ri-repo-centos.yml => ambari-repo-CentOS-6.yml} | 2 +-
...ri-repo-centos.yml => ambari-repo-CentOS-7.yml} | 2 +-
...i-repo-ubuntu.yml => ambari-repo-Ubuntu-14.yml} | 1 -
.../{iptables-centos.yml => firewall-CentOS-6.yml} | 2 +-
.../{iptables-centos.yml => firewall-CentOS-7.yml} | 4 +-
...{iptables-ubuntu.yml => firewall-Ubuntu-14.yml} | 0
.../ansible/roles/ambari_common/tasks/main.yml | 19 +---
.../{iptables-centos.yml => nodejs-CentOS-6.yml} | 8 +-
.../{iptables-centos.yml => nodejs-CentOS-7.yml} | 8 +-
.../{iptables-centos.yml => nodejs-Ubuntu-14.yml} | 8 +-
.../ansible/roles/ambari_common/tasks/nodejs.yml | 36 ---
.../ansible/roles/ambari_config/defaults/main.yml | 5 +
...{dependencies.yml => dependencies-CentOS-6.yml} | 7 +-
...{dependencies.yml => dependencies-CentOS-7.yml} | 7 +-
...dependencies.yml => dependencies-Ubuntu-14.yml} | 5 +-
.../ansible/roles/ambari_config/tasks/main.yml | 2 +-
.../roles/ambari_gather_facts/defaults/main.yml| 4 +
.../ambari_slave/tasks/{main.yml => hostname.yml} | 11 ---
.../tasks/install-agent-CentOS-6.yml} | 4 +-
...nstall-agent.yml => install-agent-CentOS-7.yml} | 12 ++-
.../tasks/install-agent-Ubuntu-14.yml} | 4 +-
.../ansible/roles/ambari_slave/tasks/main.yml | 21 +---
.../ansible/roles/enable-swap/defaults/main.yml| 4 +-
.../enable-swap/tasks/{main.yml => check-swap.yml} | 15 ++-
.../ansible/roles/enable-swap/tasks/main.yml | 23 ++---
.../tasks/{enable-swap.yml => setup-swap.yml} | 9 +-
.../tasks/install-service-CentOS-6.yml}| 8 +-
.../tasks/install-service-CentOS-7.yml}| 20 ++--
.../tasks/install-service-Ubuntu-14.yml} | 8 +-
.../ansible/roles/sensor-stubs/tasks/main.yml | 8 +-
.../templates/sensor-stubs-bro.service | 23 +
.../templates/sensor-stubs-snort.service | 23 +
.../templates/sensor-stubs-yaf.service | 23 +
metron-deployment/development/centos7/Vagrantfile | 108 +
.../centos7/ansible.cfg} | 17 ++--
.../centos7/ansible/inventory/group_vars/all} | 12 +--
.../centos7/ansible/inventory/hosts} | 35 ++-
.../centos7/ansible/playbook.yml} | 12 +--
40 files changed, 346 insertions(+), 180 deletions(-)
diff --git a/metron-deployment/ansible/playbooks/metron_full_install.yml
b/metron-deployment/ansible/playbooks/metron_full_install.yml
index ab72b85..c6e8d18 100644
--- a/metron-deployment/ansible/playbooks/metron_full_install.yml
+++ b/metron-deployment/ansible/playbooks/metron_full_install.yml
@@ -33,9 +33,6 @@
- import_playbook: sensor_install.yml
-#
-# deployment report
-#
- hosts: metron
become: false
roles:
diff --git a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
index e7968f5..344638e 100644
--- a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
+++ b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
@@ -17,5 +17,6 @@
---
hadoop_logrotate_frequency: daily
hadoop_logrotate_retention: 30
-centos_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.0/ambari.repo
+centos6_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.0/ambari.repo
+centos7_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari.repo
ubuntu_ambari_repo:
http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.6.1.0
diff --git
a/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-centos.yml
b/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
similarity index 91%
copy from
metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-centos.yml
copy to
metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
index 8c1bc33..7e6d9d2 100644
--- a/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-centos.yml
+++
b/metron-deployment/ansible/roles/ambari_common/tasks/ambari-rep
[metron] branch feature/METRON-2088-support-hdp-3.1 updated: METRON-2094 Create CentOS 7 Development Environment (nickwallen) closes apache/metron#1395
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to
refs/heads/feature/METRON-2088-support-hdp-3.1 by this push:
new 2f5ba1f METRON-2094 Create CentOS 7 Development Environment
(nickwallen) closes apache/metron#1395
2f5ba1f is described below
commit 2f5ba1f7b011c19571227a75b2ea387def94b74a
Author: nickwallen
AuthorDate: Tue Apr 30 16:31:00 2019 -0400
METRON-2094 Create CentOS 7 Development Environment (nickwallen) closes
apache/metron#1395
---
.../ansible/playbooks/metron_full_install.yml | 3 -
.../ansible/roles/ambari_common/defaults/main.yml | 3 +-
...ri-repo-centos.yml => ambari-repo-CentOS-6.yml} | 2 +-
...ri-repo-centos.yml => ambari-repo-CentOS-7.yml} | 2 +-
...i-repo-ubuntu.yml => ambari-repo-Ubuntu-14.yml} | 1 -
.../{iptables-centos.yml => firewall-CentOS-6.yml} | 2 +-
.../{iptables-centos.yml => firewall-CentOS-7.yml} | 4 +-
...{iptables-ubuntu.yml => firewall-Ubuntu-14.yml} | 0
.../ansible/roles/ambari_common/tasks/main.yml | 19 +---
.../{iptables-centos.yml => nodejs-CentOS-6.yml} | 8 +-
.../{iptables-centos.yml => nodejs-CentOS-7.yml} | 8 +-
.../{iptables-centos.yml => nodejs-Ubuntu-14.yml} | 8 +-
.../ansible/roles/ambari_common/tasks/nodejs.yml | 36 ---
.../ansible/roles/ambari_config/defaults/main.yml | 5 +
...{dependencies.yml => dependencies-CentOS-6.yml} | 7 +-
...{dependencies.yml => dependencies-CentOS-7.yml} | 7 +-
...dependencies.yml => dependencies-Ubuntu-14.yml} | 5 +-
.../ansible/roles/ambari_config/tasks/main.yml | 2 +-
.../roles/ambari_gather_facts/defaults/main.yml| 4 +
.../ambari_slave/tasks/{main.yml => hostname.yml} | 11 ---
.../tasks/install-agent-CentOS-6.yml} | 4 +-
...nstall-agent.yml => install-agent-CentOS-7.yml} | 12 ++-
.../tasks/install-agent-Ubuntu-14.yml} | 4 +-
.../ansible/roles/ambari_slave/tasks/main.yml | 21 +---
.../ansible/roles/enable-swap/defaults/main.yml| 4 +-
.../enable-swap/tasks/{main.yml => check-swap.yml} | 15 ++-
.../ansible/roles/enable-swap/tasks/main.yml | 23 ++---
.../tasks/{enable-swap.yml => setup-swap.yml} | 9 +-
.../tasks/install-service-CentOS-6.yml}| 8 +-
.../tasks/install-service-CentOS-7.yml}| 20 ++--
.../tasks/install-service-Ubuntu-14.yml} | 8 +-
.../ansible/roles/sensor-stubs/tasks/main.yml | 8 +-
.../templates/sensor-stubs-bro.service | 23 +
.../templates/sensor-stubs-snort.service | 23 +
.../templates/sensor-stubs-yaf.service | 23 +
metron-deployment/development/centos7/Vagrantfile | 108 +
.../centos7/ansible.cfg} | 17 ++--
.../centos7/ansible/inventory/group_vars/all} | 12 +--
.../centos7/ansible/inventory/hosts} | 35 ++-
.../centos7/ansible/playbook.yml} | 12 +--
40 files changed, 346 insertions(+), 180 deletions(-)
diff --git a/metron-deployment/ansible/playbooks/metron_full_install.yml
b/metron-deployment/ansible/playbooks/metron_full_install.yml
index ab72b85..c6e8d18 100644
--- a/metron-deployment/ansible/playbooks/metron_full_install.yml
+++ b/metron-deployment/ansible/playbooks/metron_full_install.yml
@@ -33,9 +33,6 @@
- import_playbook: sensor_install.yml
-#
-# deployment report
-#
- hosts: metron
become: false
roles:
diff --git a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
index e7968f5..344638e 100644
--- a/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
+++ b/metron-deployment/ansible/roles/ambari_common/defaults/main.yml
@@ -17,5 +17,6 @@
---
hadoop_logrotate_frequency: daily
hadoop_logrotate_retention: 30
-centos_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.0/ambari.repo
+centos6_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos6/2.x/updates/2.6.1.0/ambari.repo
+centos7_ambari_install_url:
http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.1.0/ambari.repo
ubuntu_ambari_repo:
http://public-repo-1.hortonworks.com/ambari/ubuntu14/2.x/updates/2.6.1.0
diff --git
a/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-centos.yml
b/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
similarity index 91%
copy from
metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-centos.yml
copy to
metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-CentOS-6.yml
index 8c1bc33..7e6d9d2 100644
--- a/metron-deployment/ansible/roles/ambari_common/tasks/ambari-repo-centos.yml
+++
b/met
[metron] branch feature/METRON-2088-support-hdp-3.1 created (now f99cc54)
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. at f99cc54 METRON-2090 Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency (merrimanr) closes apache/metron#1391 No new revisions were added by this update.
[metron] branch master updated: METRON-2082 Update the README document steps to run Batch Profiler (MohanDV via nickwallen) closes apache/metron#1387
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new c58062a METRON-2082 Update the README document steps to run Batch
Profiler (MohanDV via nickwallen) closes apache/metron#1387
c58062a is described below
commit c58062a263927a4c41edd0315d58077e0a91446a
Author: MohanDV
AuthorDate: Wed Apr 17 11:32:27 2019 -0400
METRON-2082 Update the README document steps to run Batch Profiler (MohanDV
via nickwallen) closes apache/metron#1387
---
metron-analytics/metron-profiler-spark/README.md | 6 ++
1 file changed, 6 insertions(+)
diff --git a/metron-analytics/metron-profiler-spark/README.md
b/metron-analytics/metron-profiler-spark/README.md
index 5ed5f4b..5ee8510 100644
--- a/metron-analytics/metron-profiler-spark/README.md
+++ b/metron-analytics/metron-profiler-spark/README.md
@@ -67,6 +67,12 @@ For an introduction to the Profiler, see the [Profiler
README](../metron-profile
hdfs dfs -cat /apps/metron/indexing/indexed/*/* | wc -l
```
+1. Copy the `hbase-site.xml` file from `/etc/hbase/conf` to
`/etc/spark2/conf`. It is advised to create a symlink to avoid the duplication
of file, also to keep consistency between files while config updates.
+
+ ```
+ln -s /etc/hbase/conf/hbase-site.xml /etc/spark2/conf/hbase-site.xml
+ ```
+
1. Review the Batch Profiler's properties located at
`$METRON_HOME/config/batch-profiler.properties`. See [Configuring the
Profiler](#configuring-the-profiler) for more information on these properties.
1. You may want to edit the log4j properties that sits in your config
directory in `${SPARK_HOME}` or create one. It may be helpful to turn on
`DEBUG` logging for the Profiler by adding the following line.
[metron] branch master updated: METRON-2006 Reenable JacoCo code coverage (justinleet via nickwallen) closes apache/metron#1339
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new d6763a8 METRON-2006 Reenable JacoCo code coverage (justinleet via
nickwallen) closes apache/metron#1339
d6763a8 is described below
commit d6763a8fad94e7b25bdf03e7b896eb435852c8e5
Author: justinleet
AuthorDate: Wed Apr 17 10:34:38 2019 -0400
METRON-2006 Reenable JacoCo code coverage (justinleet via nickwallen)
closes apache/metron#1339
---
pom.xml | 8 +---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/pom.xml b/pom.xml
index 7e886aa..1202aff 100644
--- a/pom.xml
+++ b/pom.xml
@@ -136,6 +136,8 @@
2.3.1
4.3.2
0.66.19
+0.8.3
+
@@ -175,7 +177,7 @@
${global_surefire_version}
--Xmx2048m
+@{argLine} -Xmx2048m
true
false
@@ -287,7 +289,7 @@
org.jacoco
jacoco-maven-plugin
-0.7.9
+${global_jacoco_version}
@@ -507,7 +509,7 @@
org.jacoco
jacoco-maven-plugin
-0.7.9
+${global_jacoco_version}
[metron] branch master updated: METRON-2056 Support LDAP Bind Authentication (nickwallen) closes apache/metron#1371
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new cf7fcae METRON-2056 Support LDAP Bind Authentication (nickwallen)
closes apache/metron#1371
cf7fcae is described below
commit cf7fcaeb3cb22755cf732505a2d6fc62783d7af9
Author: nickwallen
AuthorDate: Fri Apr 5 13:23:04 2019 -0400
METRON-2056 Support LDAP Bind Authentication (nickwallen) closes
apache/metron#1371
---
.../CURRENT/configuration/metron-security-env.xml | 21 +-
.../metron/rest/config/WebSecurityConfig.java | 48 ++
2 files changed, 42 insertions(+), 27 deletions(-)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
index 06c14ca..beb0451 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
@@ -44,7 +44,7 @@
metron.ldap.url
LDAP URL
ldap://localhost:33389
-LDAP Server URL
+The URL to the LDAP server in the form ldap://[host]:[port]
or ldaps://[host]:[port].
false
@@ -54,7 +54,7 @@
metron.ldap.user.dnpattern
uid={0},ou=people,dc=hadoop,dc=apache,dc=org
User dn pattern
-LDAP user DN
+Pattern used to create a distinguished name (DN) from a
username. This pattern is used for creating a DN string for direct user
authentication. The pattern argument {0} will be replaced with the username at
runtime.
false
@@ -64,7 +64,7 @@
metron.ldap.user.searchbase
User Search Base
ou=people,dc=hadoop,dc=apache,dc=org
-LDAP user searchbase
+The location from which the search starts for user
entries.
false
true
@@ -75,7 +75,7 @@
metron.ldap.group.searchbase
Group Search Base
ou=groups,dc=hadoop,dc=apache,dc=org
-LDAP group searchbase
+The location from which the search starts for group
entries.
false
true
@@ -86,7 +86,7 @@
metron.ldap.group.searchfilter
Group Search Filter
member={0}
-LDAP group search filter
+The search filter used to locate a group. The pattern
argument {0} will be replaced with the username at runtime.
false
true
@@ -108,7 +108,7 @@
metron.ldap.user.searchfilter
User Search Filter
-Search filter used for Bind Authentication
+The search filter used to locate a user. The pattern argument
{0} will be replaced with the username at runtime.
false
true
@@ -119,9 +119,10 @@
metron.ldap.user.password
userPassword
User password attribute
-LDAP attribute for the user password
+The name of an attribute containing the SHA-encoded user
password. This attribute is used to perform a remote compare operation to
authenticate the user. To use bind authentication, leave this field
blank.
false
+ true
@@ -129,7 +130,7 @@
metron.ldap.group.roleattribute
LDAP group role attribute
cn
-The LDAP group attribute to be used for determining
roles
+The attribute of a group that defines the group
name.
false
@@ -139,7 +140,7 @@
metron.ldap.bind.dn
Bind User
uid=admin,ou=people,dc=hadoop,dc=apache,dc=org
-Full distinguished name (DN), of an LDAP user account that
has privileges to search for users.
+The fully distinguished name (DN) of an LDAP user account
that has privileges to search for users. For example,
`uid=admin,ou=people,dc=hadoop,dc=apache,dc=org` or `admin@mycompany.local`.
false
@@ -151,7 +152,7 @@
Bind User Password
PASSWORD
-Password for the account that can search for
users
+Password for the bind user account.
false
true
diff --git
a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
index 3244e67..2513df0 100644
---
a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
+++
b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/WebSecurityConfig.java
@@ -17,6 +17,7 @@
*/
package org.apache.metron.rest.config;
+import org.apache.commons.lang3.StringUtils;
import org.apache.metron.rest.MetronRestConstants;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -27,6 +28,7
[metron] branch master updated: Revert "METRON-2023 [UI] Covering Grok Parser Creation with Cypress tests (tiborm via sardell) closes apache/metron#1364"
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new cad679a Revert "METRON-2023 [UI] Covering Grok Parser Creation with
Cypress tests (tiborm via sardell) closes apache/metron#1364"
cad679a is described below
commit cad679a5948ce0ee9866e61bbf75b1f5f255682c
Author: Nick Allen
AuthorDate: Thu Apr 4 13:10:25 2019 -0400
Revert "METRON-2023 [UI] Covering Grok Parser Creation with Cypress tests
(tiborm via sardell) closes apache/metron#1364"
This reverts commit 2f1af5cc50ba0e5e940885a1eaf5096c40a49072.
---
.../metron-alerts/alerts_ui_cypress.yml| 23 +
metron-interface/metron-alerts/package-lock.json | 709
metron-interface/metron-alerts/package.json| 16 +-
metron-interface/metron-alerts/pom.xml | 14 +-
metron-interface/metron-config/cypress.json| 9 -
.../metron-config/cypress/fixtures/config.json | 351
.../config-list-available.json | 19 -
.../cypress/integration/grok-parser.spec.js| 85 --
.../metron-config/cypress/support/commands.js | 30 -
.../metron-config/cypress/support/index.js | 18 -
metron-interface/metron-config/package-lock.json | 919 +
metron-interface/metron-config/package.json| 7 +-
metron-interface/metron-config/pom.xml | 14 +-
.../sensors/sensor-grok/sensor-grok.component.html | 6 +-
.../sensor-parser-config.component.html| 10 +-
.../sensor-parser-list.component.html | 2 +-
.../shared/ace-editor/ace-editor.component.html| 2 +-
.../shared/sample-data/sample-data.component.html | 2 +-
18 files changed, 439 insertions(+), 1797 deletions(-)
diff --git a/metron-interface/metron-alerts/alerts_ui_cypress.yml
b/metron-interface/metron-alerts/alerts_ui_cypress.yml
new file mode 100644
index 000..ddf663a
--- /dev/null
+++ b/metron-interface/metron-alerts/alerts_ui_cypress.yml
@@ -0,0 +1,23 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+port: 4200
+
+rest:
+ host: localhost
+ port: 4200
+
+dirPath: dist
\ No newline at end of file
diff --git a/metron-interface/metron-alerts/package-lock.json
b/metron-interface/metron-alerts/package-lock.json
index b167d07..8ce6443 100644
--- a/metron-interface/metron-alerts/package-lock.json
+++ b/metron-interface/metron-alerts/package-lock.json
@@ -3554,9 +3554,9 @@
}
},
"@cypress/xvfb": {
- "version": "1.2.4",
- "resolved": "https://registry.npmjs.org/@cypress/xvfb/-/xvfb-1.2.4.tgz;,
- "integrity":
"sha512-skbBzPggOVYCbnGgV+0dmBdW/s77ZkAOXIC1knS8NagwDjBrNC1LuXtQJeiN6l+m7lzmHtaoUw/ctJKdqkG57Q==",
+ "version": "1.2.3",
+ "resolved": "https://registry.npmjs.org/@cypress/xvfb/-/xvfb-1.2.3.tgz;,
+ "integrity":
"sha512-yYrK+/bgL3hwoRHMZG4r5fyLniCy1pXex5fimtewAY6vE/jsVs8Q37UsEO03tFlcmiLnQ3rBNMaZBYTi/+C1cw==",
"dev": true,
"requires": {
"debug": "^3.1.0",
@@ -3564,9 +3564,9 @@
},
"dependencies": {
"debug": {
- "version": "3.2.6",
- "resolved": "https://registry.npmjs.org/debug/-/debug-3.2.6.tgz;,
- "integrity":
"sha512-mel+jf7nrtEl5Pn1Qx46zARXKDpBbvzezse7p7LqINmdoIk8PYP5SySaxEmYv6TZ0JyEKA1hsCId6DIhgITtWQ==",
+ "version": "3.2.5",
+ "resolved": "https://registry.npmjs.org/debug/-/debug-3.2.5.tgz;,
+ "integrity":
"sha512-D61LaDQPQkxJ5AUM2mbSJRbPkNs/TmdmOeLAi1hgDkpDfIfetSrjmWhccwtuResSwMbACjx/xXQofvM9CE/aeg==",
"dev": true,
"requires": {
"ms": "^2.1.1"
@@ -5470,6 +5470,18 @@
"integrity": "sha1-1YSBOWBgdUDdsg
[metron] branch master updated: METRON-2038 Enrichment Loader Fails When Run as MR Job (nickwallen) closes apache/metron#1358
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new e63faf8 METRON-2038 Enrichment Loader Fails When Run as MR Job
(nickwallen) closes apache/metron#1358
e63faf8 is described below
commit e63faf84f1b227c44b2d6b224b89c74449bcf0cc
Author: nickwallen
AuthorDate: Wed Mar 20 09:59:43 2019 -0400
METRON-2038 Enrichment Loader Fails When Run as MR Job (nickwallen) closes
apache/metron#1358
---
dependencies_with_url.csv | 7 ++-
metron-platform/metron-data-management/pom.xml | 9 +
.../src/main/scripts/flatfile_loader.sh| 23 --
3 files changed, 19 insertions(+), 20 deletions(-)
diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv
index 0ae7ab4..04f7935 100644
--- a/dependencies_with_url.csv
+++ b/dependencies_with_url.csv
@@ -194,6 +194,7 @@ com.google.guava:guava:jar:18.0:compile,ASLv2,
com.google.inject.extensions:guice-servlet:jar:3.0:compile,ASLv2,
com.google.inject:guice:jar:3.0:compile,ASLv2,
com.googlecode.disruptor:disruptor:jar:2.10.4:compile,The Apache Software
License, Version 2.0,http://code.google.com/p/disruptor/
+com.lmax:disruptor:jar:3.3.0:compile,The Apache Software License, Version
2.0,https://github.com/LMAX-Exchange/disruptor/
com.lmax:disruptor:jar:3.3.2:compile,The Apache Software License, Version
2.0,https://github.com/LMAX-Exchange/disruptor/
com.googlecode.json-simple:json-simple:jar:1.1:compile,The Apache Software
License, Version 2.0,http://code.google.com/p/json-simple/
com.googlecode.json-simple:json-simple:jar:1.1.1:compile,The Apache Software
License, Version 2.0,http://code.google.com/p/json-simple/
@@ -301,6 +302,10 @@
org.jboss.resteasy:resteasy-jaxrs:jar:3.0.1.Final:compile,ASLv2,
org.joda:joda-convert:jar:1.2:compile,Apache
2,http://joda-convert.sourceforge.net
org.mortbay.jetty:jetty-util:jar:6.1.26:compile,ASLv2,
org.mortbay.jetty:jetty:jar:6.1.26:compile,ASLv2,
+org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile,ASLv2,
+org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile,ASLv2,
+org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile,ASLv2,
+org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile,ASLv2,
org.noggit:noggit:jar:0.6:compile,Apache License, Version
2.0,http://github.com/yonik/noggit
org.scannotation:scannotation:jar:1.0.3:compile,Apache License
V2.0,http://scannotation.sf.net
org.slf4j:log4j-over-slf4j:jar:1.6.6:compile,Apache Software
Licenses,http://www.slf4j.org
@@ -498,4 +503,4 @@ joda-time:joda-time:jar:2.10:compile
org.elasticsearch:securesm:jar:1.2:compile
com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile,ASLv2,http://stephenc.github.io/jcip-annotations/
com.nimbusds:nimbus-jose-jwt:jar:4.41.2:compile,ASLv2,https://bitbucket.org/connect2id/nimbus-jose-jwt/wiki/Home
-
+tomcat:jasper-compiler:jar:5.5.23:compile,ASLv2,https://tomcat.apache.org/
diff --git a/metron-platform/metron-data-management/pom.xml
b/metron-platform/metron-data-management/pom.xml
index 72412ed..22e3947 100644
--- a/metron-platform/metron-data-management/pom.xml
+++ b/metron-platform/metron-data-management/pom.xml
@@ -186,7 +186,6 @@
hadoop-common
-provided
org.apache.hbase
@@ -210,7 +209,6 @@
hadoop-common
-provided
com.opencsv
@@ -344,7 +342,6 @@
test
-
@@ -380,7 +377,6 @@
-
@@ -395,6 +391,11 @@
false
${project.name}
+
+
diff --git
a/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
b/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
index f92d418..feccc33 100755
--- a/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
+++ b/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
@@ -1,5 +1,5 @@
#!/bin/bash
-#
+#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -7,15 +7,15 @@
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obt
[metron] branch master updated: METRON-2035 Allow User to Configure Role Names for Access Control (nickwallen) closes apache/metron#1355
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new ef1ce3c METRON-2035 Allow User to Configure Role Names for Access
Control (nickwallen) closes apache/metron#1355
ef1ce3c is described below
commit ef1ce3c3d82758452fb63ca9d5cee31d266031ee
Author: nickwallen
AuthorDate: Wed Mar 20 09:52:16 2019 -0400
METRON-2035 Allow User to Configure Role Names for Access Control
(nickwallen) closes apache/metron#1355
---
.../CURRENT/configuration/metron-security-env.xml | 15 +-
.../CURRENT/package/scripts/params/params_linux.py | 4 +
.../METRON/CURRENT/package/templates/metron.j2 | 4 +
.../METRON/CURRENT/themes/metron_theme.json| 47 +++-
metron-interface/metron-rest/README.md | 272 -
.../src/main/config/rest_application.yml | 4 +
.../rest/config/MetronAuthoritiesMapper.java | 125 ++
.../metron/rest/config/WebSecurityConfig.java | 41 +---
.../rest/config/MetronAuthoritiesMapperTest.java | 101
9 files changed, 461 insertions(+), 152 deletions(-)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
index ab3e532..06c14ca 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-security-env.xml
@@ -225,5 +225,18 @@
false
-
+
+metron_user_role
+User Role Name
+USER
+Name of the role at the authentication provider that provides
user access to Metron.
+
+
+
+metron_admin_role
+Admin Role Name
+ADMIN
+Name of the role at the authentication provider that provides
administrative access to Metron.
+
+
\ No newline at end of file
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index e97c0e8..095acda 100755
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -294,6 +294,10 @@ metron_ldap_group_role =
config['configurations']['metron-security-env']['metron
metron_ldap_ssl_truststore =
config['configurations']['metron-security-env']['metron.ldap.ssl.truststore']
metron_ldap_ssl_truststore_password =
config['configurations']['metron-security-env']['metron.ldap.ssl.truststore.password']
+# Roles
+metron_user_role =
config['configurations']['metron-security-env']['metron_user_role']
+metron_admin_role =
config['configurations']['metron-security-env']['metron_admin_role']
+
# REST
metron_rest_pid_dir =
config['configurations']['metron-rest-env']['metron_rest_pid_dir']
metron_rest_pid = 'metron-rest.pid'
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
index 08d4281..936118c 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
@@ -44,6 +44,10 @@
METRON_LDAP_GROUP_SEARCHFILTER="{{metron_ldap_group_searchfilter}}"
METRON_LDAP_GROUP_ROLE="{{metron_ldap_group_role}}"
METRON_LDAP_SSL_TRUSTSTORE="{{metron_ldap_ssl_truststore}}"
+#Roles
+METRON_USER_ROLE="{{metron_user_role}}"
+METRON_ADMIN_ROLE="{{metron_admin_role}}"
+
ZOOKEEPER="{{zookeeper_quorum}}"
BROKERLIST="{{kafka_brokers}}"
HADOOP_CONF_DIR="/etc/hadoop/conf/"
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
index 65b2767..69084e3 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/the
[metron] branch master updated: METRON-2012 Unable to Execute Stellar Functions Against HBase in the REPL (nickwallen) closes apache/metron#1345
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 1ae45a9 METRON-2012 Unable to Execute Stellar Functions Against HBase
in the REPL (nickwallen) closes apache/metron#1345
1ae45a9 is described below
commit 1ae45a97dc73ee468f6d45052fbfa63c3bca19b8
Author: nickwallen
AuthorDate: Wed Mar 6 13:56:16 2019 -0500
METRON-2012 Unable to Execute Stellar Functions Against HBase in the REPL
(nickwallen) closes apache/metron#1345
---
metron-analytics/metron-profiler-client/pom.xml| 314 ++---
.../profiler/client/stellar/WindowLookback.java| 25 +-
.../metron/profiler/client/window/Window.java | 15 +-
.../client/stellar/WindowLookbackTest.java | 11 +-
metron-analytics/metron-profiler-common/pom.xml| 64 +
.../metron/profiler/DefaultMessageDistributor.java | 2 -
metron-analytics/metron-profiler-repl/pom.xml | 6 +
metron-analytics/metron-profiler-storm/pom.xml | 8 +-
metron-contrib/metron-performance/pom.xml | 10 +-
metron-platform/elasticsearch-shaded/pom.xml | 9 +-
metron-platform/metron-common/pom.xml | 6 +-
metron-platform/metron-data-management/pom.xml | 30 +-
metron-platform/metron-elasticsearch/pom.xml | 95 +--
metron-platform/metron-enrichment/pom.xml | 28 +-
metron-platform/metron-indexing/pom.xml| 32 +--
metron-platform/metron-management/pom.xml | 6 +-
metron-platform/metron-pcap-backend/pom.xml| 27 +-
metron-platform/metron-solr/pom.xml| 5 +-
.../metron-storm-kafka-override/pom.xml| 9 +-
metron-platform/metron-writer/pom.xml | 12 +-
metron-stellar/stellar-common/pom.xml | 5 +-
21 files changed, 299 insertions(+), 420 deletions(-)
diff --git a/metron-analytics/metron-profiler-client/pom.xml
b/metron-analytics/metron-profiler-client/pom.xml
index 0238214..b667991 100644
--- a/metron-analytics/metron-profiler-client/pom.xml
+++ b/metron-analytics/metron-profiler-client/pom.xml
@@ -25,68 +25,33 @@
UTF-8
UTF-8
+${global_hbase_guava_version}
-org.apache.hadoop
-hadoop-auth
-${global_hadoop_version}
-provided
-
-
-org.slf4j
-slf4j-log4j12
-
-
-
-
-org.apache.hadoop
-hadoop-common
-${global_hadoop_version}
-
-
-servlet-api
-javax.servlet
-
-
-commons-httpclient
-commons-httpclient
-
-
-org.slf4j
-slf4j-log4j12
-
-
-provided
-
-
de.jollyday
jollyday
0.5.2
+com.github.ben-manes.caffeine
+caffeine
+${global_caffeine_version}
+
+
org.apache.metron
metron-profiler-common
${project.parent.version}
org.apache.metron
-metron-hbase
+metron-common
${project.parent.version}
-org.slf4j
-slf4j-log4j12
+com.google.guava
+guava
-
-
-
-org.apache.metron
-metron-hbase
-${project.parent.version}
-test
-test-jar
-
org.slf4j
slf4j-log4j12
@@ -94,87 +59,45 @@
-org.apache.hadoop
-hadoop-mapreduce-client-core
-${global_hadoop_version}
-
-
-servlet-api
-javax.servlet
-
-
-commons-httpclient
-commons-httpclient
-
-
-org.slf4j
-slf4j-log4j12
-
-
-provided
+com.google.guava
+guava
+${guava_version}
-org.apache.hbase
-hbase-client
-${global_hbase_version}
+org.apache.metron
+metron-hbase
+${project.parent.version}
org.apache.hadoop
-hadoop-auth
+hadoop-hdfs
[metron] branch master updated: METRON-1778 Out-of-order timestamps may delay flush in Storm Profiler (nickwallen) closes apache/metron#1197
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new fec36c7 METRON-1778 Out-of-order timestamps may delay flush in Storm
Profiler (nickwallen) closes apache/metron#1197
fec36c7 is described below
commit fec36c7ed0b2d29f4dc48574adf02c7fcb13e5fb
Author: nickwallen
AuthorDate: Tue Feb 26 14:34:39 2019 -0500
METRON-1778 Out-of-order timestamps may delay flush in Storm Profiler
(nickwallen) closes apache/metron#1197
---
.../profiler/storm/FixedFrequencyFlushSignal.java | 82 ++-
.../storm/FixedFrequencyFlushSignalTest.java | 151 +++--
2 files changed, 179 insertions(+), 54 deletions(-)
diff --git
a/metron-analytics/metron-profiler-storm/src/main/java/org/apache/metron/profiler/storm/FixedFrequencyFlushSignal.java
b/metron-analytics/metron-profiler-storm/src/main/java/org/apache/metron/profiler/storm/FixedFrequencyFlushSignal.java
index 02503c2..db4f99b 100644
---
a/metron-analytics/metron-profiler-storm/src/main/java/org/apache/metron/profiler/storm/FixedFrequencyFlushSignal.java
+++
b/metron-analytics/metron-profiler-storm/src/main/java/org/apache/metron/profiler/storm/FixedFrequencyFlushSignal.java
@@ -32,14 +32,14 @@ public class FixedFrequencyFlushSignal implements
FlushSignal {
protected static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
- * The latest known timestamp.
+ * Tracks the min timestamp.
*/
- private long currentTime;
+ private long minTime;
/**
- * The time when the next flush should occur.
+ * Tracks the max timestamp.
*/
- private long flushTime;
+ private long maxTime;
/**
* The amount of time between flushes in milliseconds.
@@ -47,7 +47,6 @@ public class FixedFrequencyFlushSignal implements FlushSignal
{
private long flushFrequency;
public FixedFrequencyFlushSignal(long flushFrequencyMillis) {
-
if(flushFrequencyMillis < 0) {
throw new IllegalArgumentException("flush frequency must be >= 0");
}
@@ -61,9 +60,8 @@ public class FixedFrequencyFlushSignal implements FlushSignal
{
*/
@Override
public void reset() {
-flushTime = 0;
-currentTime = 0;
-
+minTime = Long.MAX_VALUE;
+maxTime = Long.MIN_VALUE;
LOG.debug("Flush counters reset");
}
@@ -74,31 +72,34 @@ public class FixedFrequencyFlushSignal implements
FlushSignal {
*/
@Override
public void update(long timestamp) {
+if(LOG.isWarnEnabled()) {
+ checkIfOutOfOrder(timestamp);
+}
-if(timestamp > currentTime) {
-
- // need to update current time
- LOG.debug("Updating current time; last={}, new={}", currentTime,
timestamp);
- currentTime = timestamp;
-
-} else if ((currentTime - timestamp) > flushFrequency) {
-
- // significantly out-of-order timestamps
- LOG.warn("Timestamps out-of-order by '{}' ms. This may indicate a
problem in the data. last={}, current={}",
- (currentTime - timestamp),
- timestamp,
- currentTime);
+if(timestamp < minTime) {
+ minTime = timestamp;
}
-if(flushTime == 0) {
+if(timestamp > maxTime) {
+ maxTime = timestamp;
+}
+ }
- // set the next time to flush
- flushTime = currentTime + flushFrequency;
- LOG.debug("Setting flush time; '{}' ms until flush; flushTime={},
currentTime={}, flushFreq={}",
- timeToNextFlush(),
- flushTime,
- currentTime,
- flushFrequency);
+ /**
+ * Checks if the timestamp is significantly out-of-order.
+ *
+ * @param timestamp The last timestamp.
+ */
+ private void checkIfOutOfOrder(long timestamp) {
+// do not warn if this is the first timestamp we've seen, which will
always be 'out-of-order'
+if (maxTime > Long.MIN_VALUE) {
+
+ long outOfOrderBy = maxTime - timestamp;
+ if (Math.abs(outOfOrderBy) > flushFrequency) {
+LOG.warn("Timestamp out-of-order by {} ms. This may indicate a problem
in the data. " +
+"timestamp={}, maxKnown={}, flushFreq={} ms",
+outOfOrderBy, timestamp, maxTime, flushFrequency);
+ }
}
}
@@ -109,27 +110,20 @@ public class FixedFrequencyFlushSignal implements
FlushSignal {
*/
@Override
public boolean isTimeToFlush() {
+boolean flush = false;
-boolean flush = currentTime > flushTime;
-LOG.debug("Flush={}, '{}' ms until flush; currentTime={}, flushTime={}",
-flush,
-timeToNextFlush(),
-currentTime,
-flushTime);
+long flushTime = minTime + flushFrequency;
+if(maxTime >= flushTime) {
+
[metron] branch master updated: METRON-1996 Solr search throws NPE for group search if the group parameter is null or empty (MohanDV via nickwallen) closes apache/metron#1333
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new cc30c0e METRON-1996 Solr search throws NPE for group search if the
group parameter is null or empty (MohanDV via nickwallen) closes
apache/metron#1333
cc30c0e is described below
commit cc30c0e21656fa95b488adb47a3c9fcec94af67a
Author: MohanDV
AuthorDate: Tue Feb 26 11:20:14 2019 -0500
METRON-1996 Solr search throws NPE for group search if the group parameter
is null or empty (MohanDV via nickwallen) closes apache/metron#1333
---
.../org/apache/metron/solr/dao/SolrSearchDao.java | 3 ++
.../apache/metron/solr/dao/SolrSearchDaoTest.java | 36 ++
2 files changed, 39 insertions(+)
diff --git
a/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
b/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
index 4a8d482..134d28c 100644
---
a/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
+++
b/metron-platform/metron-solr/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
@@ -105,6 +105,9 @@ public class SolrSearchDao implements SearchDao {
@Override
public GroupResponse group(GroupRequest groupRequest) throws
InvalidSearchException {
try {
+ if (groupRequest.getGroups() == null || groupRequest.getGroups().size()
== 0) {
+throw new InvalidSearchException("At least 1 group must be provided.");
+ }
String groupNames =
groupRequest.getGroups().stream().map(Group::getField).collect(
Collectors.joining(","));
SolrQuery query = new SolrQuery()
diff --git
a/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/dao/SolrSearchDaoTest.java
b/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/dao/SolrSearchDaoTest.java
index fe27a55..bf9458f 100644
---
a/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/dao/SolrSearchDaoTest.java
+++
b/metron-platform/metron-solr/src/test/java/org/apache/metron/solr/dao/SolrSearchDaoTest.java
@@ -136,6 +136,42 @@ public class SolrSearchDaoTest {
}
@Test
+ public void searchShouldThrowInvalidSearchExceptionOnNullGroup() throws
Exception {
+exception.expect(InvalidSearchException.class);
+exception.expectMessage("At least 1 group must be provided.");
+
+GroupRequest groupRequest = mock(GroupRequest.class);
+GroupResponse groupResponse = mock(GroupResponse.class);
+
+solrSearchDao = spy(new SolrSearchDao(client, accessConfig));
+when(groupRequest.getQuery()).thenReturn("query");
+when(groupRequest.getGroups()).thenReturn(null);
+when(groupRequest.getScoreField()).thenReturn(Optional.of("scoreField"));
+when(groupRequest.getIndices()).thenReturn(Arrays.asList("bro", "snort"));
+
+assertEquals(groupResponse, solrSearchDao.group(groupRequest));
+verifyNoMoreInteractions(client);
+ }
+
+ @Test
+ public void searchShouldThrowInvalidSearchExceptionOnEmptyGroup() throws
Exception {
+exception.expect(InvalidSearchException.class);
+exception.expectMessage("At least 1 group must be provided.");
+
+GroupRequest groupRequest = mock(GroupRequest.class);
+GroupResponse groupResponse = mock(GroupResponse.class);
+
+solrSearchDao = spy(new SolrSearchDao(client, accessConfig));
+when(groupRequest.getQuery()).thenReturn("query");
+when(groupRequest.getGroups()).thenReturn(Collections.EMPTY_LIST);
+when(groupRequest.getScoreField()).thenReturn(Optional.of("scoreField"));
+when(groupRequest.getIndices()).thenReturn(Arrays.asList("bro", "snort"));
+
+assertEquals(groupResponse, solrSearchDao.group(groupRequest));
+verifyNoMoreInteractions(client);
+ }
+
+ @Test
public void searchShouldThrowSearchResultSizeException() throws Exception {
exception.expect(InvalidSearchException.class);
exception.expectMessage("Search result size must be less than 100");
[metron] branch master updated: METRON-2010 Unable to Build Metron Due to Inaccessible Repository (nickwallen) closes apache/metron#1343
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 2fb8524 METRON-2010 Unable to Build Metron Due to Inaccessible Repository (nickwallen) closes apache/metron#1343 2fb8524 is described below commit 2fb8524ab46047aaf408691ac1d3143836371ab4 Author: nickwallen AuthorDate: Thu Feb 21 16:45:05 2019 -0500 METRON-2010 Unable to Build Metron Due to Inaccessible Repository (nickwallen) closes apache/metron#1343 --- metron-analytics/pom.xml | 7 --- metron-interface/pom.xml | 9 - pom.xml | 7 --- 3 files changed, 23 deletions(-) diff --git a/metron-analytics/pom.xml b/metron-analytics/pom.xml index 04d00aa..704bbc7 100644 --- a/metron-analytics/pom.xml +++ b/metron-analytics/pom.xml @@ -83,13 +83,6 @@ http://clojars.org/repo - multiline-release-repo - https://raw.github.com/benelog/multiline/master/maven-repository - - false - - - sonatype.org https://oss.sonatype.org/content/groups/public/ diff --git a/metron-interface/pom.xml b/metron-interface/pom.xml index 8858127..65f2c95 100644 --- a/metron-interface/pom.xml +++ b/metron-interface/pom.xml @@ -65,13 +65,4 @@ - - -multiline-release-repo - https://raw.github.com/benelog/multiline/master/maven-repository - -false - - - diff --git a/pom.xml b/pom.xml index 6ad3181..8b660c6 100644 --- a/pom.xml +++ b/pom.xml @@ -63,13 +63,6 @@ http://repo.hortonworks.com/content/repositories/releases/ default - -multiline-release-repo - https://raw.github.com/benelog/multiline/master/maven-repository - -false - - @ApacheMetron
[metron] branch master updated: METRON-1986 Batch Profiler Fails to Resolve Stats Stellar Functions (nickwallen) closes apache/metron#1328
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 2283487 METRON-1986 Batch Profiler Fails to Resolve Stats Stellar
Functions (nickwallen) closes apache/metron#1328
2283487 is described below
commit 22834879017355fa456f42a6e71a3ed44d0b43fe
Author: nickwallen
AuthorDate: Thu Feb 14 17:33:22 2019 -0500
METRON-1986 Batch Profiler Fails to Resolve Stats Stellar Functions
(nickwallen) closes apache/metron#1328
---
metron-analytics/metron-profiler-spark/pom.xml | 37 --
.../spark/BatchProfilerIntegrationTest.java| 83 --
2 files changed, 90 insertions(+), 30 deletions(-)
diff --git a/metron-analytics/metron-profiler-spark/pom.xml
b/metron-analytics/metron-profiler-spark/pom.xml
index 000d884..40bd551 100644
--- a/metron-analytics/metron-profiler-spark/pom.xml
+++ b/metron-analytics/metron-profiler-spark/pom.xml
@@ -52,7 +52,6 @@
org.apache.metron
metron-profiler-client
${project.parent.version}
-test
org.antlr
@@ -62,6 +61,11 @@
org.apache.metron
+stellar-common
+${project.parent.version}
+
+
+org.apache.metron
metron-common
${project.parent.version}
@@ -182,8 +186,7 @@
-
+
.yaml
LICENSE.txt
@@ -191,16 +194,36 @@
NOTICE.txt
-
-
+
+
+
+
+
+
+org.atteo.classindex
+classindex-transformer
+${global_classindex_version}
+
+
maven-assembly-plugin
diff --git
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
index 9ea151a..b36cf8c 100644
---
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
+++
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
@@ -70,29 +70,6 @@ import static org.junit.Assert.assertTrue;
public class BatchProfilerIntegrationTest {
private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- /**
- * {
- * "timestampField": "timestamp",
- * "profiles": [
- * {
- *"profile": "count-by-ip",
- *"foreach": "ip_src_addr",
- *"init": { "count": 0 },
- *"update": { "count" : "count + 1" },
- *"result": "count"
- * },
- * {
- *"profile": "total-count",
- *"foreach": "'total'",
- *"init": { "count": 0 },
- *"update": { "count": "count + 1" },
- *"result": "count"
- * }
- * ]
- * }
- */
- @Multiline
- private static String profileJson;
private static SparkSession spark;
private Properties profilerProperties;
private Properties readerProperties;
@@ -152,6 +129,30 @@ public class BatchProfilerIntegrationTest {
}
/**
+ * {
+ * "timestampField": "timestamp",
+ * "profiles": [
+ * {
+ *"profile": "count-by-ip",
+ *"foreach": "ip_src_addr",
+ *"init": { "count": 0 },
+ *"update": { "count" : "count + 1" },
+ *"result": "count"
+ * },
+ * {
+ *"profile": "total-count",
+ *"
[metron] branch master updated: METRON-1985 Improve Error Handling When Cannot Connect to HBase (nickwallen) closes apache/metron#1327
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 9641aec METRON-1985 Improve Error Handling When Cannot Connect to
HBase (nickwallen) closes apache/metron#1327
9641aec is described below
commit 9641aec6284102d16db2fbfd1dc5960d6767008a
Author: nickwallen
AuthorDate: Tue Feb 12 12:30:36 2019 -0500
METRON-1985 Improve Error Handling When Cannot Connect to HBase
(nickwallen) closes apache/metron#1327
---
.../apache/metron/hbase/client/HBaseClient.java| 40 ---
.../metron/hbase/client/HBaseClientTest.java | 57 --
2 files changed, 87 insertions(+), 10 deletions(-)
diff --git
a/metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/client/HBaseClient.java
b/metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/client/HBaseClient.java
index f0a0a10..c416f47 100644
---
a/metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/client/HBaseClient.java
+++
b/metron-platform/metron-hbase/src/main/java/org/apache/metron/hbase/client/HBaseClient.java
@@ -25,6 +25,8 @@ import java.io.IOException;
import java.lang.invoke.MethodHandles;
import java.util.ArrayList;
import java.util.List;
+
+import org.apache.commons.collections4.CollectionUtils;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.client.Durability;
import org.apache.hadoop.hbase.client.Get;
@@ -39,6 +41,8 @@ import
org.apache.metron.hbase.bolt.mapper.HBaseProjectionCriteria;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
+import static org.apache.commons.collections4.CollectionUtils.size;
+
/**
* A client that interacts with HBase.
*/
@@ -67,7 +71,9 @@ public class HBaseClient implements Closeable {
try {
this.table = provider.getTable(configuration, tableName);
} catch (Exception e) {
- throw new RuntimeException(e);
+ String msg = String.format("Unable to open connection to HBase for table
'%s'", tableName);
+ LOG.error(msg, e);
+ throw new RuntimeException(msg, e);
}
}
@@ -142,9 +148,10 @@ public class HBaseClient implements Closeable {
table.batch(mutations, result);
mutations.clear();
-} catch (InterruptedException | IOException e) {
- LOG.warn("Error performing a mutation to HBase.", e);
- throw new RuntimeException(e);
+} catch (Exception e) {
+ String msg = String.format("'%d' HBase write(s) failed on table '%s'",
size(mutations), tableName(table));
+ LOG.error(msg, e);
+ throw new RuntimeException(msg, e);
}
return mutationCount;
@@ -187,8 +194,9 @@ public class HBaseClient implements Closeable {
return results;
} catch (Exception e) {
- LOG.warn("Could not perform HBase lookup.", e);
- throw new RuntimeException(e);
+ String msg = String.format("'%d' HBase read(s) failed on table '%s'",
size(gets), tableName(table));
+ LOG.error(msg, e);
+ throw new RuntimeException(msg, e);
}
}
@@ -197,7 +205,9 @@ public class HBaseClient implements Closeable {
*/
@Override
public void close() throws IOException {
-table.close();
+if(table != null) {
+ table.close();
+}
}
/**
@@ -276,4 +286,20 @@ public class HBaseClient implements Closeable {
cols.getCounters().forEach(cnt -> inc.addColumn(cnt.getFamily(),
cnt.getQualifier(), cnt.getIncrement()));
return inc;
}
+
+ /**
+ * Returns the name of the HBase table.
+ * Attempts to avoid any null pointers that might be encountered along
the way.
+ * @param table The table to retrieve the name of.
+ * @return The name of the table
+ */
+ private static String tableName(HTableInterface table) {
+String tableName = "null";
+if(table != null) {
+ if(table.getName() != null) {
+tableName = table.getName().getNameAsString();
+ }
+}
+return tableName;
+ }
}
diff --git
a/metron-platform/metron-hbase/src/test/java/org/apache/metron/hbase/client/HBaseClientTest.java
b/metron-platform/metron-hbase/src/test/java/org/apache/metron/hbase/client/HBaseClientTest.java
index ca1f5a6..e6247d0 100644
---
a/metron-platform/metron-hbase/src/test/java/org/apache/metron/hbase/client/HBaseClientTest.java
+++
b/metron-platform/metron-hbase/src/test/java/org/apache/metron/hbase/client/HBaseClientTest.java
@@ -20,18 +20,24 @@
package org.apache.metron.hbase.client;
-import org.apache.hadoop.hbase.TableName;
-import org.apache.hadoop.hbase.client.*;
-import org.apache.storm.tuple.Tuple;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.HBaseTestingUtility;
+import org.apache.hadoop.hbase.clien
[metron] branch master updated: METRON-1974 Batch Profiler Should Handle Errant Profiles Better (nickwallen) closes apache/metron#1326
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 0cf78d2 METRON-1974 Batch Profiler Should Handle Errant Profiles
Better (nickwallen) closes apache/metron#1326
0cf78d2 is described below
commit 0cf78d223f24eaf7112bfb7fb52d8778f8a1a2fa
Author: nickwallen
AuthorDate: Tue Feb 12 12:25:29 2019 -0500
METRON-1974 Batch Profiler Should Handle Errant Profiles Better
(nickwallen) closes apache/metron#1326
---
.../spark/function/GroupByPeriodFunction.java | 50 -
.../spark/function/ProfileBuilderFunction.java | 31 ---
.../spark/BatchProfilerIntegrationTest.java| 59 +++-
.../spark/function/GroupByPeriodFunctionTest.java | 63 ++
.../spark/function/ProfileBuilderFunctionTest.java | 59
5 files changed, 229 insertions(+), 33 deletions(-)
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/GroupByPeriodFunction.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/GroupByPeriodFunction.java
index 1b602f4..d06f7e9 100644
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/GroupByPeriodFunction.java
+++
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/GroupByPeriodFunction.java
@@ -47,6 +47,8 @@ public class GroupByPeriodFunction implements
MapFunction
*/
private TimeUnit periodDurationUnits;
+ private static final String SEPARATOR = "__";
+
public GroupByPeriodFunction(Properties profilerProperties) {
periodDurationUnits =
TimeUnit.valueOf(PERIOD_DURATION_UNITS.get(profilerProperties, String.class));
periodDuration = PERIOD_DURATION.get(profilerProperties, Integer.class);
@@ -55,6 +57,52 @@ public class GroupByPeriodFunction implements
MapFunction
@Override
public String call(MessageRoute route) {
ProfilePeriod period = ProfilePeriod.fromTimestamp(route.getTimestamp(),
periodDuration, periodDurationUnits);
-return route.getProfileDefinition().getProfile() + "-" + route.getEntity()
+ "-" + period.getPeriod();
+return new StringBuilder()
+.append(route.getProfileDefinition().getProfile())
+.append(SEPARATOR)
+.append(route.getEntity())
+.append(SEPARATOR)
+.append(period.getPeriod())
+.toString();
+ }
+
+ /**
+ * @param groupKey The group key used to group {@link MessageRoute}s.
+ * @return The name of the profile.
+ */
+ public static String profileFromKey(String groupKey) {
+String[] pieces = groupKey.split(SEPARATOR);
+if(pieces.length == 3) {
+ return pieces[0];
+} else {
+ return "unknown";
+}
+ }
+
+ /**
+ * @param groupKey The group key used to group {@link MessageRoute}s.
+ * @return The name of the entity.
+ */
+ public static String entityFromKey(String groupKey) {
+String[] pieces = groupKey.split(SEPARATOR);
+if(pieces.length == 3) {
+ return pieces[1];
+} else {
+ return "unknown";
+}
}
+
+ /**
+ * @param groupKey The group key used to group {@link MessageRoute}s.
+ * @return The period identifier.
+ */
+ public static String periodFromKey(String groupKey) {
+String[] pieces = groupKey.split(SEPARATOR);
+if(pieces.length == 3) {
+ return pieces[2];
+} else {
+ return "unknown";
+}
+ }
+
}
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/ProfileBuilderFunction.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/ProfileBuilderFunction.java
index 273695b..7283b48 100644
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/ProfileBuilderFunction.java
+++
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/ProfileBuilderFunction.java
@@ -39,9 +39,13 @@ import java.util.stream.Collectors;
import java.util.stream.Stream;
import java.util.stream.StreamSupport;
+import static java.lang.String.format;
import static java.util.Comparator.comparing;
import static
org.apache.metron.profiler.spark.BatchProfilerConfig.PERIOD_DURATION;
import static
org.apache.metron.profiler.spark.BatchProfilerConfig.PERIOD_DURATION_UNITS;
+import static
org.apache.metron.profiler.spark.function.GroupByPeriodFunction.entityFromKey;
+import static
org.apache.metron.profiler.spark.function.GroupByPeriodFunction.periodFromKey;
+import static
org.apache.metron.profiler.spark.function.GroupByPeriodFunction.profileFromKey;
/**
[metron] branch master updated: METRON-1970 Add Metadata to Error Messages Generated During Parsing (nickwallen) closes apache/metron#1325
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 1b6ef88 METRON-1970 Add Metadata to Error Messages Generated During
Parsing (nickwallen) closes apache/metron#1325
1b6ef88 is described below
commit 1b6ef88c79d60022542cda7e9abbea7e720773cc
Author: nickwallen
AuthorDate: Tue Feb 12 12:18:09 2019 -0500
METRON-1970 Add Metadata to Error Messages Generated During Parsing
(nickwallen) closes apache/metron#1325
---
.../java/org/apache/metron/common/Constants.java | 1 +
.../apache/metron/common/error/MetronError.java| 79 +++---
.../metron/common/error/MetronErrorTest.java | 47 +
metron-platform/metron-parsing/README.md | 51 ++
.../apache/metron/parsers/ParserRunnerImpl.java| 3 +
.../metron/parsers/ParserRunnerImplTest.java | 16 -
6 files changed, 141 insertions(+), 56 deletions(-)
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
index 5054508..a0b5bce 100644
---
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
+++
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/Constants.java
@@ -93,6 +93,7 @@ public class Constants {
,RAW_MESSAGE_BYTES("raw_message_bytes")
,ERROR_FIELDS("error_fields")
,ERROR_HASH("error_hash")
+,METADATA("metadata")
;
private String name;
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
index 0493be6..89044de 100644
---
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
+++
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/error/MetronError.java
@@ -25,10 +25,14 @@ import java.net.InetAddress;
import java.net.UnknownHostException;
import java.util.ArrayList;
import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
+import java.util.Objects;
import java.util.Optional;
import java.util.Set;
import java.util.UUID;
+
import org.apache.commons.lang.exception.ExceptionUtils;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.metron.common.Constants;
@@ -45,6 +49,7 @@ public class MetronError {
private ErrorType errorType = ErrorType.DEFAULT_ERROR;
private Set errorFields;
private List rawMessages;
+ private Map metadata = new HashMap<>();
public MetronError withMessage(String message) {
this.message = message;
@@ -71,6 +76,10 @@ public class MetronError {
return this;
}
+ public MetronError withMetadata(Map metadata) {
+this.metadata.putAll(metadata);
+return this;
+ }
public MetronError addRawMessage(Object rawMessage) {
if (rawMessage != null) {
@@ -95,25 +104,28 @@ public class MetronError {
public JSONObject getJSONObject() {
JSONObject errorMessage = new JSONObject();
errorMessage.put(Constants.GUID, UUID.randomUUID().toString());
-errorMessage.put(Constants.SENSOR_TYPE, "error");
-if (sensorTypes.size() == 1) {
- errorMessage.put(ErrorFields.FAILED_SENSOR_TYPE.getName(),
sensorTypes.iterator().next());
-} else {
- errorMessage
- .put(ErrorFields.FAILED_SENSOR_TYPE.getName(), new
JSONArray().addAll(sensorTypes));
-}
+errorMessage.put(Constants.SENSOR_TYPE, Constants.ERROR_TYPE);
errorMessage.put(ErrorFields.ERROR_TYPE.getName(), errorType.getType());
-
+addFailedSensorType(errorMessage);
addMessageString(errorMessage);
addStacktrace(errorMessage);
addTimestamp(errorMessage);
addHostname(errorMessage);
addRawMessages(errorMessage);
addErrorHash(errorMessage);
+addMetadata(errorMessage);
return errorMessage;
}
+ private void addFailedSensorType(JSONObject errorMessage) {
+if (sensorTypes.size() == 1) {
+ errorMessage.put(ErrorFields.FAILED_SENSOR_TYPE.getName(),
sensorTypes.iterator().next());
+} else {
+ errorMessage.put(ErrorFields.FAILED_SENSOR_TYPE.getName(), new
JSONArray().addAll(sensorTypes));
+}
+ }
+
@SuppressWarnings({"unchecked"})
private void addMessageString(JSONObject errorMessage) {
if (message != null) {
@@ -192,44 +204,31 @@ public class MetronError {
}
}
- @Override
- public boolean equals(Object o) {
-if (this == o) {
- return true;
-}
-if (o == null || getClass() != o.getClass()) {
- return false;
+ private void addMetadata(JSONObject errorMessage) {
+if(metadata != n
[metron] branch master updated: METRON-1939 Update version to 0.7.1 (justinleet via nickwallen) closes apache/metron#1303
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new aa85ffa METRON-1939 Update version to 0.7.1 (justinleet via nickwallen) closes apache/metron#1303 aa85ffa is described below commit aa85ffab39b50eeff46b47c68bc670c5e7c6a1d0 Author: justinleet AuthorDate: Tue Jan 8 11:40:20 2019 -0500 METRON-1939 Update version to 0.7.1 (justinleet via nickwallen) closes apache/metron#1303 --- metron-analytics/metron-maas-common/pom.xml | 2 +- metron-analytics/metron-maas-service/pom.xml| 2 +- metron-analytics/metron-profiler-client/pom.xml | 2 +- metron-analytics/metron-profiler-common/pom.xml | 2 +- metron-analytics/metron-profiler-repl/pom.xml | 2 +- metron-analytics/metron-profiler-spark/pom.xml | 2 +- metron-analytics/metron-profiler-storm/pom.xml | 2 +- metron-analytics/metron-statistics/pom.xml | 2 +- metron-analytics/pom.xml| 2 +- metron-contrib/metron-docker/pom.xml| 2 +- metron-contrib/metron-performance/pom.xml | 2 +- metron-contrib/pom.xml | 2 +- metron-deployment/amazon-ec2/conf/defaults.yml | 2 +- metron-deployment/ansible/playbooks/docker_probe_install.yml| 2 +- metron-deployment/ansible/roles/ambari_master/defaults/main.yml | 2 +- .../development/centos6/ansible/inventory/group_vars/all| 2 +- .../development/ubuntu14/ansible/inventory/group_vars/all | 2 +- metron-deployment/packaging/ambari/elasticsearch-mpack/pom.xml | 4 ++-- metron-deployment/packaging/ambari/metron-mpack/pom.xml | 4 ++-- .../packaging/ambari/metron-mpack/src/main/resources/mpack.json | 2 +- metron-deployment/packaging/docker/deb-docker/pom.xml | 4 ++-- metron-deployment/packaging/docker/rpm-docker/pom.xml | 4 ++-- metron-deployment/pom.xml | 2 +- metron-interface/metron-alerts/package-lock.json| 2 +- metron-interface/metron-alerts/package.json | 2 +- metron-interface/metron-alerts/pom.xml | 2 +- metron-interface/metron-alerts/scripts/package.json | 2 +- metron-interface/metron-config/package-lock.json| 2 +- metron-interface/metron-config/package.json | 2 +- metron-interface/metron-config/pom.xml | 2 +- metron-interface/metron-config/scripts/package.json | 2 +- metron-interface/metron-rest-client/pom.xml | 2 +- metron-interface/metron-rest/pom.xml| 2 +- metron-interface/metron-rest/src/main/resources/application.yml | 2 +- metron-interface/pom.xml| 2 +- metron-platform/Performance-tuning-guide.md | 4 ++-- metron-platform/README.md | 2 +- metron-platform/elasticsearch-shaded/pom.xml| 2 +- metron-platform/metron-common/pom.xml | 2 +- metron-platform/metron-common/src/main/scripts/cluster_info.py | 2 +- metron-platform/metron-data-management/pom.xml | 2 +- metron-platform/metron-elasticsearch/pom.xml| 2 +- metron-platform/metron-enrichment/pom.xml | 2 +- metron-platform/metron-hbase-client/pom.xml | 2 +- metron-platform/metron-hbase/pom.xml| 2 +- metron-platform/metron-indexing/pom.xml | 2 +- metron-platform/metron-integration-test/pom.xml | 2 +- metron-platform/metron-job/pom.xml | 2 +- metron-platform/metron-management/pom.xml | 2 +- metron-platform/metron-parsing/metron-parsers-common/pom.xml| 2 +- metron-platform/metron-parsing/metron-parsers/pom.xml | 2 +- metron-platform/metron-parsing/metron-parsing-storm/pom.xml | 2 +- metron-platform/metron-parsing/pom.xml | 2 +- metron-platform/metron-pcap-backend/pom.xml | 2 +- .../metron-pcap-backend/src/main/scripts/pcap_zeppelin_run.sh | 2 +- metron-platform/metron-pcap/pom.xml | 2 +- metron-platform/metron-solr/pom.xml | 2 +- metron-platform/metron-storm-kafka-override/pom.xml
[metron] branch master updated: METRON-685 Scores in Threat Triage should be a Stellar Statement (nickwallen) closes apache/metron#1311
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 39fa0f1 METRON-685 Scores in Threat Triage should be a Stellar
Statement (nickwallen) closes apache/metron#1311
39fa0f1 is described below
commit 39fa0f134fbf4d16da2001792971135d9f24cf4c
Author: nickwallen
AuthorDate: Mon Jan 7 13:53:35 2019 -0500
METRON-685 Scores in Threat Triage should be a Stellar Statement
(nickwallen) closes apache/metron#1311
---
.../enrichment/threatintel/RiskLevelRule.java | 63 ++-
.../enrichment/threatintel/RuleScore.java | 37 +--
.../enrichment/threatintel/ThreatTriageConfig.java | 2 +-
.../threatintel/ThreatTriageConfigTest.java| 115 +
metron-platform/metron-enrichment/README.md| 58 +--
.../metron/enrichment/utils/ThreatIntelUtils.java | 2 +-
.../threatintel/triage/ThreatTriageProcessor.java | 24 +++--
.../threatintel/triage/ThreatTriageTest.java | 55 ++
.../metron/management/ThreatTriageFunctions.java | 10 +-
.../management/ThreatTriageFunctionsTest.java | 55 --
10 files changed, 326 insertions(+), 95 deletions(-)
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/RiskLevelRule.java
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/RiskLevelRule.java
index 94ab0c8..0443c27 100644
---
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/RiskLevelRule.java
+++
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/RiskLevelRule.java
@@ -17,6 +17,10 @@
*/
package org.apache.metron.common.configuration.enrichment.threatintel;
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+import java.util.Objects;
+
/**
* This class represents a rule that is used to triage threats.
*
@@ -34,24 +38,24 @@ public class RiskLevelRule {
/**
* The name of the rule. This field is optional.
*/
- String name;
+ private String name;
/**
* A description of the rule. This field is optional.
*/
- String comment;
+ private String comment;
/**
* A predicate, in the form of a Stellar expression, that determines whether
* the rule is applied to an alert or not. This field is required.
*/
- String rule;
+ private String rule;
/**
- * A numeric value that represents the score that is applied to the alert.
This
- * field is required.
+ * A Stellar expression that when evaluated results in a numeric score. The
expression
+ * can refer to fields within the message undergoing triage.
*/
- Number score;
+ private String scoreExpression;
/**
* Allows a rule author to provide contextual information when a rule is
applied
@@ -60,7 +64,7 @@ public class RiskLevelRule {
* This is expected to be a valid Stellar expression and can refer to any of
the
* fields within the message itself.
*/
- String reason;
+ private String reason;
public String getName() {
return name;
@@ -86,12 +90,26 @@ public class RiskLevelRule {
this.rule = rule;
}
- public Number getScore() {
-return score;
+ @JsonProperty("score")
+ public String getScoreExpression() {
+return scoreExpression;
}
- public void setScore(Number score) {
-this.score = score;
+ @JsonProperty("score")
+ public void setScoreExpression(Object scoreExpression) {
+if(scoreExpression instanceof Number) {
+ // a numeric value was provided
+ scoreExpression = Number.class.cast(scoreExpression).toString();
+
+} else if (scoreExpression instanceof String) {
+ // a stellar expression was provided
+ scoreExpression = String.class.cast(scoreExpression);
+
+} else {
+ throw new IllegalArgumentException(String.format("Expected 'score' to be
number or string, but got '%s'", scoreExpression));
+}
+
+this.scoreExpression = scoreExpression.toString();
}
public String getReason() {
@@ -105,25 +123,18 @@ public class RiskLevelRule {
@Override
public boolean equals(Object o) {
if (this == o) return true;
-if (o == null || getClass() != o.getClass()) return false;
-
+if (!(o instanceof RiskLevelRule)) return false;
RiskLevelRule that = (RiskLevelRule) o;
-
-if (name != null ? !name.equals(that.name) : that.name != null) return
false;
-if (comment != null ? !comment.equals(that.comment) : that.comment !=
null) return false;
-if (rule != null ? !rule.equals(that.rule) : that.rule != null) return
false;
-if (score != null ? !score.equals(that.score) : that.score != null) return
false;
-return
[metron] branch master updated: METRON-1954 Commit Script Fails to Extract Usernames with Numerals (nickwallen) closes apache/metron#1313
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 8c1171f METRON-1954 Commit Script Fails to Extract Usernames with
Numerals (nickwallen) closes apache/metron#1313
8c1171f is described below
commit 8c1171fa5bcc1c4a015d9160b626f452c150
Author: nickwallen
AuthorDate: Sun Dec 23 15:21:36 2018 -0500
METRON-1954 Commit Script Fails to Extract Usernames with Numerals
(nickwallen) closes apache/metron#1313
---
dev-utilities/committer-utils/metron-committer-common | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/dev-utilities/committer-utils/metron-committer-common
b/dev-utilities/committer-utils/metron-committer-common
index 23ec4e9..30a9bfd 100644
--- a/dev-utilities/committer-utils/metron-committer-common
+++ b/dev-utilities/committer-utils/metron-committer-common
@@ -239,7 +239,7 @@ function setup_code {
#
function get_contributor_info {
# use github api to retrieve the contributor's login
-USER=`curl -s
https://api.github.com/repos/apache/${CHOSEN_REPO}/pulls/${PR} | grep login |
head -1 | awk -F":" '{print $2}' | sed 's/[^a-zA-Z.@_-]//g'`
+USER=`curl -s
https://api.github.com/repos/apache/${CHOSEN_REPO}/pulls/${PR} | grep login |
head -1 | awk -F":" '{print $2}' | sed 's/[^a-zA-Z0-9.@_-]//g'`
read -p " github contributor's username [$USER]: " INPUT
[ -n "$INPUT" ] && USER=${INPUT}
[metron] branch master updated: METRON-1953 Stellar REPL Fails on Start (nickwallen) closes apache/metron#1312
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new d1b534f METRON-1953 Stellar REPL Fails on Start (nickwallen) closes
apache/metron#1312
d1b534f is described below
commit d1b534f31afc2f241d5ea656eeab5268cc39b84b
Author: nickwallen
AuthorDate: Sun Dec 23 15:05:42 2018 -0500
METRON-1953 Stellar REPL Fails on Start (nickwallen) closes
apache/metron#1312
---
metron-platform/metron-common/src/main/scripts/stellar | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/metron-platform/metron-common/src/main/scripts/stellar
b/metron-platform/metron-common/src/main/scripts/stellar
index a538e13..142cc29 100644
--- a/metron-platform/metron-common/src/main/scripts/stellar
+++ b/metron-platform/metron-common/src/main/scripts/stellar
@@ -38,7 +38,7 @@ set -u
export METRON_PARSERS_PLATFORM="${METRON_PARSERS_PLATFORM:-storm}"
export HBASE_CONFIGS=$(hbase classpath)
-export STELLAR_LIB=$(find $METRON_HOME/lib/ -name
metron-parsers-${METRON_PARSERS_PLATFORM}*.jar)
+export STELLAR_LIB=$(find $METRON_HOME/lib/ -name
metron-parsing-${METRON_PARSERS_PLATFORM}*.jar)
export MANAGEMENT_LIB=$(find $METRON_HOME/lib/ -name metron-management*.jar)
export PROFILER_LIB=$(find $METRON_HOME/lib/ -name metron-profiler-repl*.jar)
java $METRON_JVMFLAGS -cp
"${CONTRIB:-$METRON_HOME/contrib/*}:$STELLAR_LIB:$MANAGEMENT_LIB:$PROFILER_LIB:$HBASE_CONFIGS"
org.apache.metron.stellar.common.shell.cli.StellarShell "$@"
[metron] branch master updated: METRON-1938 Add Parser Debugger to READMEs (nickwallen) closes apache/metron#1304
This is an automated email from the ASF dual-hosted git repository. nickallen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new ec11855 METRON-1938 Add Parser Debugger to READMEs (nickwallen) closes apache/metron#1304 ec11855 is described below commit ec118557cb6a9bfe91528702c7bf093977fa80d7 Author: nickwallen AuthorDate: Tue Dec 18 14:21:42 2018 -0500 METRON-1938 Add Parser Debugger to READMEs (nickwallen) closes apache/metron#1304 --- metron-platform/metron-management/README.md| 527 + .../apache/metron/management/ParserFunctions.java | 14 +- 2 files changed, 341 insertions(+), 200 deletions(-) diff --git a/metron-platform/metron-management/README.md b/metron-platform/metron-management/README.md index bf939c2..f364627 100644 --- a/metron-platform/metron-management/README.md +++ b/metron-platform/metron-management/README.md @@ -25,7 +25,7 @@ have been added: * Stellar enrichments in the Enrichment topology * Stellar threat triage rules -Additionally, some shell functions have been added to +Additionally, some shell functions have been added to * provide the ability to refer to the Stellar expression used to create a variable * print structured data in a way that is easier to view (i.e. tabular) @@ -45,7 +45,6 @@ project. * [Manage Stellar Field Transformations](#manage-stellar-field-transformations) * [Manage Stellar Enrichments](#manage-stellar-enrichments) * [Manage Threat Triage Rules](#manage-threat-triage-rules) -* [Simulate Threat Triage Rules](#simulate-threat-triage-rules) ## Functions @@ -60,13 +59,13 @@ The functions are split roughly into a few sections: ### Grok Functions -* `GROK_EVAL` + `GROK_EVAL` * Description: Evaluate a grok expression for a statement. * Input: * grokExpression - The grok expression to evaluate * data - Either a data message or a list of data messages to evaluate using the grokExpression * Returns: The Map associated with the grok expression being evaluated on the list of messages. -* `GROK_PREDICT` + `GROK_PREDICT` * Description: Discover a grok statement for an input doc * Input: * data - The data to discover a grok expression from @@ -74,73 +73,73 @@ The functions are split roughly into a few sections: ### File Functions -* Local Files - * `LOCAL_LS` -* Description: Lists the contents of a directory. -* Input: - * path - The path of the file -* Returns: The contents of the directory in tabular form sorted by last modification date. - * `LOCAL_RM` -* Description: Removes the path -* Input: - * path - The path of the file or directory. - * recursive - Recursively remove or not (optional and defaulted to false) -* Returns: boolean - true if successful, false otherwise - * `LOCAL_READ` -* Description: Retrieves the contents as a string of a file. -* Input: - * path - The path of the file -* Returns: The contents of the file and null otherwise. - * `LOCAL_READ_LINES` -* Description: Retrieves the contents of a file as a list of strings. -* Input: - * path - The path of the file -* Returns: A list of lines - * `LOCAL_WRITE` -* Description: Writes the contents of a string to a local file -* Input: - * content - The content to write out - * path - The path of the file -* Returns: true if the file was written and false otherwise. -* HDFS Files - * `HDFS_LS` -* Description: Lists the contents of a directory in HDFS. -* Input: - * path - The path of the file -* Returns: The contents of the directory in tabular form sorted by last modification date. - * `HDFS_RM` -* Description: Removes the path in HDFS. -* Input: - * path - The path of the file or directory. - * recursive - Recursively remove or not (optional and defaulted to false) -* Returns: boolean - true if successful, false otherwise - * `HDFS_READ` -* Description: Retrieves the contents as a string of a file in HDFS. -* Input: - * path - The path of the file -* Returns: The contents of the file and null otherwise. - * `HDFS_READ_LINES` -* Description: Retrieves the contents of a HDFS file as a list of strings. -* Input: - * path - The path of the file -* Returns: A list of lines - * `HDFS_WRITE` -* Description: Writes the contents of a string to a HDFS file -* Input: - * content - The content to write out - * path - The path of the file -* Returns: true if the file was written and false otherwise. + Local Files +# `LOCAL_LS` + * Description: Lists the contents of a directory. + * Input: +* path - The path of the file + * Returns: The contents of the directory in tabular form sorted by last modification date. +# `LOCAL_RM` + * Description: Removes
[metron] branch master updated: METRON-1925 Provide Verbose View of Profile Results in REPL (nickwallen) closes apache/metron#1292
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 2e6caa9 METRON-1925 Provide Verbose View of Profile Results in REPL
(nickwallen) closes apache/metron#1292
2e6caa9 is described below
commit 2e6caa908c0b7708281d76b6744d63706fe610b2
Author: nickwallen
AuthorDate: Tue Dec 18 08:59:08 2018 -0500
METRON-1925 Provide Verbose View of Profile Results in REPL (nickwallen)
closes apache/metron#1292
---
.../profiler/client/HBaseProfilerClient.java | 133 ++--
.../metron/profiler/client/ProfilerClient.java | 36 ++--
.../metron/profiler/client/stellar/GetProfile.java | 66 +++---
.../client/stellar/ProfilerClientConfig.java | 1 +
.../metron/profiler/client/stellar/Util.java | 16 +-
.../profiler/client/stellar/VerboseProfile.java| 222 +
.../profiler/client/HBaseProfilerClientTest.java | 167 +---
.../metron/profiler/client/ProfileWriter.java | 20 +-
.../profiler/client/stellar/GetProfileTest.java| 38 ++--
.../client/stellar/VerboseProfileTest.java | 220
.../org/apache/metron/profiler/ProfilePeriod.java | 3 +
metron-stellar/stellar-common/README.md| 12 +-
12 files changed, 673 insertions(+), 261 deletions(-)
diff --git
a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
index de2d42c..2e537da 100644
---
a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
+++
b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
@@ -24,18 +24,17 @@ import org.apache.hadoop.hbase.client.Get;
import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
+import org.apache.metron.common.utils.SerDeUtils;
+import org.apache.metron.profiler.ProfileMeasurement;
import org.apache.metron.profiler.ProfilePeriod;
import org.apache.metron.profiler.hbase.ColumnBuilder;
import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import org.apache.metron.common.utils.SerDeUtils;
import java.io.IOException;
import java.util.ArrayList;
-import java.util.Arrays;
import java.util.List;
import java.util.Optional;
import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
/**
* The default implementation of a ProfilerClient that fetches profile data
persisted in HBase.
@@ -57,30 +56,16 @@ public class HBaseProfilerClient implements ProfilerClient {
*/
private ColumnBuilder columnBuilder;
- public HBaseProfilerClient(HTableInterface table, RowKeyBuilder
rowKeyBuilder, ColumnBuilder columnBuilder) {
+ private long periodDurationMillis;
+
+ public HBaseProfilerClient(HTableInterface table,
+ RowKeyBuilder rowKeyBuilder,
+ ColumnBuilder columnBuilder,
+ long periodDurationMillis) {
setTable(table);
setRowKeyBuilder(rowKeyBuilder);
setColumnBuilder(columnBuilder);
- }
-
- /**
- * Fetches all of the data values associated with a Profile.
- *
- * @param clazz The type of values stored by the profile.
- * @param profile The name of the profile.
- * @param entity The name of the entity.
- * @param groups The groups used to sort the profile data.
- * @param durationAgo How far in the past to fetch values from.
- * @param unitThe time unit of 'durationAgo'.
- * @param defaultValue The default value to specify. If empty, the result
will be sparse.
- * @param The type of values stored by the Profile.
- * @return A list of values.
- */
- @Override
- public List fetch(Class clazz, String profile, String entity,
List groups, long durationAgo, TimeUnit unit, Optional defaultValue)
{
-long end = System.currentTimeMillis();
-long start = end - unit.toMillis(durationAgo);
-return fetch(clazz, profile, entity, groups, start, end, defaultValue);
+this.periodDurationMillis = periodDurationMillis;
}
/**
@@ -97,21 +82,15 @@ public class HBaseProfilerClient implements ProfilerClient {
* @return A list of values.
*/
@Override
- public List fetch(Class clazz, String profile, String entity,
List groups, long start, long end, Optional defaultValue) {
-byte[] columnFamily = Bytes.toBytes(columnBuilder.getColumnFamily());
-byte[] columnQualifier = columnBuilder.getColumnQualifier("value");
-
-// find all the row keys that satisfy this fetch
-List keysToFetch = rowKeyBuilder.rowKeys(profi
[metron] branch master updated: METRON-1892 Parser Debugger Should Load Config From Zookeeper (nickwallen) closes apache/metron#1278
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 90582b3 METRON-1892 Parser Debugger Should Load Config From Zookeeper
(nickwallen) closes apache/metron#1278
90582b3 is described below
commit 90582b30902b9506d16547c0cdac16e8a7e3a66e
Author: nickwallen
AuthorDate: Mon Dec 17 09:57:06 2018 -0500
METRON-1892 Parser Debugger Should Load Config From Zookeeper (nickwallen)
closes apache/metron#1278
---
.../org/apache/metron/management/Functions.java| 23 +-
.../apache/metron/management/ParserFunctions.java | 52 ++--
.../metron/management/StellarParserRunner.java | 7 +-
.../metron/management/ParserFunctionsTest.java | 95 +-
4 files changed, 166 insertions(+), 11 deletions(-)
diff --git
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
index 5e591aa..e67f0ed 100644
---
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
+++
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
@@ -19,11 +19,16 @@
package org.apache.metron.management;
+import org.apache.curator.framework.CuratorFramework;
import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.ParseException;
import java.util.List;
+import java.util.Optional;
import static java.lang.String.format;
+import static
org.apache.metron.stellar.dsl.Context.Capabilities.ZOOKEEPER_CLIENT;
/**
* Contains utility functionality that is useful across all of the Stellar
management functions.
@@ -39,10 +44,10 @@ public class Functions {
* @param args All of the arguments.
* @param The type of the argument expected.
*/
- public static T getArg(String argName, int index, Class clazz,
List args) {
+ public static T getArg(String argName, int index, Class clazz,
List args) throws ParseException {
if(index >= args.size()) {
String msg = format("missing '%s'; expected at least %d argument(s),
found %d", argName, index+1, args.size());
- throw new IllegalArgumentException(msg);
+ throw new ParseException(msg);
}
return ConversionUtils.convert(args.get(index), clazz);
@@ -69,4 +74,18 @@ public class Functions {
return result;
}
+
+ /**
+ * Retrieves the Zookeeper client from the execution context.
+ *
+ * @param context The execution context.
+ * @return A Zookeeper client, if one exists. Otherwise, an exception is
thrown.
+ */
+ public static CuratorFramework getZookeeperClient(Context context) throws
ParseException {
+return context
+.getCapability(ZOOKEEPER_CLIENT, false)
+.filter(CuratorFramework.class::isInstance)
+.map(CuratorFramework.class::cast)
+.orElseThrow(() -> new ParseException("Missing ZOOKEEPER_CLIENT;
zookeeper connection required"));
+ }
}
diff --git
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/ParserFunctions.java
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/ParserFunctions.java
index 6a02812..fcb91f9 100644
---
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/ParserFunctions.java
+++
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/ParserFunctions.java
@@ -19,7 +19,9 @@
package org.apache.metron.management;
-import com.fasterxml.jackson.core.JsonProcessingException;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.metron.common.configuration.SensorParserConfig;
import org.apache.metron.stellar.dsl.BaseStellarFunction;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.ParseException;
@@ -31,7 +33,9 @@ import java.util.List;
import java.util.Map;
import static java.lang.String.format;
+import static
org.apache.metron.common.configuration.ConfigurationsUtils.readSensorParserConfigFromZookeeper;
import static org.apache.metron.management.Functions.getArg;
+import static org.apache.metron.management.Functions.getZookeeperClient;
import static org.apache.metron.management.Functions.hasArg;
/**
@@ -49,25 +53,32 @@ public class ParserFunctions {
},
returns = "A parser that can be used to parse messages."
)
- public static class InitializeFunction extends BaseStellarFunction {
+ public static class InitializeFunction implements StellarFunction {
@Override
-public Object apply(List args) throws ParseException {
[metron] branch master updated: METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID (nickwallen) closes apache/metron#1269
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new eb2aee1 METRON-1879 Allow Elasticsearch to Auto-Generate the Document
ID (nickwallen) closes apache/metron#1269
eb2aee1 is described below
commit eb2aee1820cae811c6491d68aed32ccc055922e4
Author: nickwallen
AuthorDate: Thu Dec 13 14:29:16 2018 -0500
METRON-1879 Allow Elasticsearch to Auto-Generate the Document ID
(nickwallen) closes apache/metron#1269
---
.../e2e/alerts-list/alerts-list.e2e-spec.ts| 19 +-
.../configure-table/configure-table.e2e-spec.ts| 17 +-
.../table-view/table-view.component.html | 2 +-
.../alerts-list/table-view/table-view.component.ts | 9 +-
.../alerts-list/tree-view/tree-view.component.ts | 2 +-
.../app/service/elasticsearch-localstorage-impl.ts | 2 +-
.../bulk/ElasticsearchBulkDocumentWriter.java | 17 +-
.../dao/ElasticsearchMetaAlertUpdateDao.java | 16 +-
.../dao/ElasticsearchRequestSubmitter.java | 3 +-
.../dao/ElasticsearchRetrieveLatestDao.java| 127 -
.../bulk/ElasticsearchBulkDocumentWriterTest.java | 35 ++-
.../dao/ElasticsearchRequestSubmitterTest.java | 12 +-
...ticsearchBulkDocumentWriterIntegrationTest.java | 159
.../apache/metron/indexing/dao/MultiIndexDao.java | 115 ++---
.../indexing/dao/metaalert/MetaAlertConstants.java | 14 +
.../metron/indexing/dao/update/Document.java | 124 ++---
.../metron/indexing/dao/update/UpdateDao.java | 34 ++-
.../dao/metaalert/MetaAlertIntegrationTest.java| 284 +++--
18 files changed, 670 insertions(+), 321 deletions(-)
diff --git
a/metron-interface/metron-alerts/e2e/alerts-list/alerts-list.e2e-spec.ts
b/metron-interface/metron-alerts/e2e/alerts-list/alerts-list.e2e-spec.ts
index e3709ab..70f52de 100644
--- a/metron-interface/metron-alerts/e2e/alerts-list/alerts-list.e2e-spec.ts
+++ b/metron-interface/metron-alerts/e2e/alerts-list/alerts-list.e2e-spec.ts
@@ -24,9 +24,9 @@ import { loadTestData, deleteTestData } from
'../utils/e2e_util';
describe('Test spec for all ui elements & list view', function() {
let page: MetronAlertsPage;
let loginPage: LoginPage;
- let columnNames = [ '', 'Score', 'id', 'timestamp', 'source:type',
'ip_src_addr', 'enrichm...:country',
+ let columnNames = [ '', 'Score', 'guid', 'timestamp', 'source:type',
'ip_src_addr', 'enrichm...:country',
'ip_dst_addr', 'host', 'alert_status', '', '', ''];
- let colNamesColumnConfig = [ 'score', 'id', 'timestamp', 'source:type',
'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
+ let colNamesColumnConfig = [ 'score', 'guid', 'timestamp', 'source:type',
'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
'ip_dst_addr', 'host', 'alert_status' ];
beforeAll(async function() : Promise {
@@ -136,16 +136,17 @@ describe('Test spec for all ui elements & list view',
function() {
});
it('should select columns from table configuration', async function() :
Promise {
-let newColNamesColumnConfig = [ 'score', 'timestamp', 'source:type',
'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
- 'ip_dst_addr', 'host', 'alert_status', 'guid' ];
-
await page.clickConfigureTable();
-expect(await page.getSelectedColumnNames()).toEqual(colNamesColumnConfig,
'for default selected column names');
-await page.toggleSelectCol('id');
+expect(await page.getSelectedColumnNames()).toEqual(colNamesColumnConfig,
'expect default selected column names');
+
+// remove the 'guid' column and add the 'id' column
await page.toggleSelectCol('guid');
-expect(await
page.getSelectedColumnNames()).toEqual(newColNamesColumnConfig, 'for guid added
to selected column names');
-await page.saveConfigureColumns();
+await page.toggleSelectCol('id');
+let expectedColumns = [ 'score', 'timestamp', 'source:type',
'ip_src_addr', 'enrichments:geo:ip_dst_addr:country',
+ 'ip_dst_addr', 'host', 'alert_status', 'id' ];
+expect(await page.getSelectedColumnNames()).toEqual(expectedColumns,
'expect "id" field added and "guid" field removed from visible columns');
+await page.saveConfigureColumns();
});
it('should have all time-range controls', async function() : Promise {
diff --git
a/metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts
b/metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts
index c3636f5..39504a9 100644
---
a/metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts
+++
b/metron-interface/metron-alerts/e2e/alerts-list/configure-table/configure-table.e2e-spec.ts
@@ -24,7 +24,7 @@ import {loadTe
[metron] branch master updated: METRON-1849 Elasticsearch Index Write Functionality Should be Shared (nickwallen) closes apache/metron#1254
This is an automated email from the ASF dual-hosted git repository.
nickallen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new ec3b98f METRON-1849 Elasticsearch Index Write Functionality Should be
Shared (nickwallen) closes apache/metron#1254
ec3b98f is described below
commit ec3b98f762ce3726ed9a33abdb446957d1865dca
Author: nickwallen
AuthorDate: Tue Dec 11 12:59:08 2018 -0500
METRON-1849 Elasticsearch Index Write Functionality Should be Shared
(nickwallen) closes apache/metron#1254
---
.../elasticsearch/bulk/BulkDocumentWriter.java | 45 +++
.../bulk/BulkDocumentWriterResults.java| 68
.../bulk/ElasticsearchBulkDocumentWriter.java | 166 ++
.../metron/elasticsearch/bulk/WriteFailure.java| 48 +++
.../metron/elasticsearch/bulk/WriteSuccess.java| 36 +++
.../metron/elasticsearch/dao/ElasticsearchDao.java | 2 +-
.../elasticsearch/dao/ElasticsearchUpdateDao.java | 144 -
.../elasticsearch/writer/ElasticsearchWriter.java | 157 +
.../elasticsearch/writer/TupleBasedDocument.java | 44 +++
.../bulk/ElasticsearchBulkDocumentWriterTest.java | 178 ++
.../components/ElasticSearchComponent.java | 46 +--
.../writer/ElasticsearchWriterTest.java| 360 ++---
.../src/test/resources/log4j.properties| 0
13 files changed, 994 insertions(+), 300 deletions(-)
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
new file mode 100644
index 000..34f543e
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriter.java
@@ -0,0 +1,45 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.bulk;
+
+import org.apache.metron.indexing.dao.update.Document;
+
+/**
+ * Writes documents to an index in bulk.
+ *
+ * @param The type of document to write.
+ */
+public interface BulkDocumentWriter {
+
+/**
+ * Add a document to the batch.
+ * @param document The document to write.
+ * @param index The name of the index to write to.
+ */
+void addDocument(D document, String index);
+
+/**
+ * @return The number of documents waiting to be written.
+ */
+int size();
+
+/**
+ * Write all documents in the batch.
+ */
+BulkDocumentWriterResults write();
+}
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
new file mode 100644
index 000..90e5ce3
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/bulk/BulkDocumentWriterResults.java
@@ -0,0 +1,68 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.elasticsearch.bulk;
+
+import org.apache.metron.indexing.dao.update.Document;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * The result of writing documents in bulk using a {@link B
metron git commit: METRON-1867 Remove `/api/v1/update/replace` endpoint (nickwallen) closes apache/metron#1284
Repository: metron
Updated Branches:
refs/heads/master d749961da -> 285ba5eb6
METRON-1867 Remove `/api/v1/update/replace` endpoint (nickwallen) closes
apache/metron#1284
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/285ba5eb
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/285ba5eb
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/285ba5eb
Branch: refs/heads/master
Commit: 285ba5eb604f2e286f72641308a1424234739402
Parents: d749961
Author: nickwallen
Authored: Fri Dec 7 16:17:48 2018 -0500
Committer: nickallen
Committed: Fri Dec 7 16:17:48 2018 -0500
--
.../src/app/model/replace-request.ts| 24 --
metron-interface/metron-rest/README.md | 26 --
.../rest/controller/UpdateController.java | 12 -
.../metron/rest/service/UpdateService.java | 2 -
.../rest/service/impl/UpdateServiceImpl.java| 10 -
.../UpdateControllerIntegrationTest.java| 116 +++---
.../elasticsearch/dao/ElasticsearchDao.java | 17 +-
.../indexing/dao/update/ReplaceRequest.java | 77
.../metron/indexing/dao/update/UpdateDao.java | 22 +-
.../indexing/dao/InMemoryMetaAlertDao.java | 23 +-
.../indexing/dao/UpdateIntegrationTest.java | 369 +--
.../integration/HBaseDaoIntegrationTest.java| 7 -
12 files changed, 229 insertions(+), 476 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/285ba5eb/metron-interface/metron-alerts/src/app/model/replace-request.ts
--
diff --git a/metron-interface/metron-alerts/src/app/model/replace-request.ts
b/metron-interface/metron-alerts/src/app/model/replace-request.ts
deleted file mode 100644
index d7c18b2..000
--- a/metron-interface/metron-alerts/src/app/model/replace-request.ts
+++ /dev/null
@@ -1,24 +0,0 @@
-/**
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-export class ReplaceRequest {
- replacement: {[key: string]: any};
- guid: string;
- sensorType: string;
- index: string;
-}
http://git-wip-us.apache.org/repos/asf/metron/blob/285ba5eb/metron-interface/metron-rest/README.md
--
diff --git a/metron-interface/metron-rest/README.md
b/metron-interface/metron-rest/README.md
index f85707f..a3cbad7 100644
--- a/metron-interface/metron-rest/README.md
+++ b/metron-interface/metron-rest/README.md
@@ -373,7 +373,6 @@ Request and Response objects are JSON formatted. The JSON
schemas are available
| [ `GET /api/v1/storm/{name}`](#get-apiv1stormname)|
| [ `GET /api/v1/storm/supervisors`](#get-apiv1stormsupervisors)|
| [ `PATCH /api/v1/update/patch`](#patch-apiv1updatepatch)|
-| [ `PUT /api/v1/update/replace`](#put-apiv1updatereplace)|
| [ `POST /api/v1/update/add/comment`](#put-apiv1updateaddcomment)|
| [ `POST /api/v1/update/remove/comment`](#put-apiv1updateremovecomment)|
| [ `GET /api/v1/user`](#get-apiv1user)|
@@ -978,31 +977,6 @@ Request and Response objects are JSON formatted. The JSON
schemas are available
* Returns:
* 200 - Nothing
* 404 - Document not found
-
-### `PUT /api/v1/update/replace`
- * Description: Replace a document
- * Input:
-* request - Replacement request
- * guid - The Patch UUID
- * sensorType - The sensor type
- * replacement - A Map representing the replaced document
-* Example replacing a `bro` message with guid of `000-000-` :
-```
-{
- "guid" : "000-000-",
- "sensorType" : "bro",
- "replacement" : {
-"source:type": "bro",
-"guid" : "bro_index_2017.01.01.01:1",
-"ip_src_addr":"192.168.1.2",
-"ip_src_port": 8009,
-"timestamp":200,
-"rejected":false
- }
-}
-```
- * Returns:
-* 200 -
metron git commit: METRON-1810 Storm Profiler Intermittent Test Failure (nickwallen) closes apache/metron#1289
Repository: metron
Updated Branches:
refs/heads/master e81a5c102 -> d749961da
METRON-1810 Storm Profiler Intermittent Test Failure (nickwallen) closes
apache/metron#1289
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/d749961d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/d749961d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/d749961d
Branch: refs/heads/master
Commit: d749961daaa77624e1cb6878548479d86c8fe668
Parents: e81a5c1
Author: nickwallen
Authored: Fri Dec 7 14:24:48 2018 -0500
Committer: nickallen
Committed: Fri Dec 7 14:24:48 2018 -0500
--
.../profiler/DefaultMessageDistributor.java | 75 +++---
.../profiler/DefaultMessageDistributorTest.java | 12 +--
metron-analytics/metron-profiler-storm/pom.xml | 42 ++--
.../integration/ProfilerIntegrationTest.java| 103 ---
.../metron/integration/utils/TestUtils.java | 6 +-
5 files changed, 121 insertions(+), 117 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/d749961d/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageDistributor.java
--
diff --git
a/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageDistributor.java
b/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageDistributor.java
index 0e50467..ef2ff2c 100644
---
a/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageDistributor.java
+++
b/metron-analytics/metron-profiler-common/src/main/java/org/apache/metron/profiler/DefaultMessageDistributor.java
@@ -21,11 +21,11 @@
package org.apache.metron.profiler;
import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.CacheWriter;
import com.github.benmanes.caffeine.cache.Caffeine;
import com.github.benmanes.caffeine.cache.RemovalCause;
import com.github.benmanes.caffeine.cache.RemovalListener;
import com.github.benmanes.caffeine.cache.Ticker;
-import com.google.common.util.concurrent.MoreExecutors;
import org.apache.commons.lang.builder.HashCodeBuilder;
import org.apache.metron.common.configuration.profiler.ProfileConfig;
import org.apache.metron.stellar.dsl.Context;
@@ -41,7 +41,6 @@ import java.util.List;
import java.util.Optional;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Executor;
-import java.util.concurrent.ExecutorService;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;
@@ -105,7 +104,7 @@ public class DefaultMessageDistributor implements
MessageDistributor, Serializab
long periodDurationMillis,
long profileTimeToLiveMillis,
long maxNumberOfRoutes) {
-this(periodDurationMillis, profileTimeToLiveMillis, maxNumberOfRoutes,
Ticker.systemTicker(), Optional.empty());
+this(periodDurationMillis, profileTimeToLiveMillis, maxNumberOfRoutes,
Ticker.systemTicker());
}
/**
@@ -116,14 +115,12 @@ public class DefaultMessageDistributor implements
MessageDistributor, Serializab
* @param maxNumberOfRoutes The max number of unique routes to maintain.
After this is exceeded, lesser
* used routes will be evicted from the internal
cache.
* @param ticker The ticker used to drive time for the caches. Only needs
set for testing.
- * @param cacheMaintenanceExecutor The executor responsible for running
cache maintenance tasks. Only needed for testing.
*/
public DefaultMessageDistributor(
long periodDurationMillis,
long profileTimeToLiveMillis,
long maxNumberOfRoutes,
- Ticker ticker,
- Optional cacheMaintenanceExecutor) {
+ Ticker ticker) {
if(profileTimeToLiveMillis < periodDurationMillis) {
throw new IllegalStateException(format(
@@ -138,11 +135,8 @@ public class DefaultMessageDistributor implements
MessageDistributor, Serializab
.newBuilder()
.maximumSize(maxNumberOfRoutes)
.expireAfterAccess(profileTimeToLiveMillis, TimeUnit.MILLISECONDS)
-.removalListener(new ActiveCacheRemovalListener())
-.ticker(ticker);
-if (cacheMaintenanceExecutor.isPresent()) {
- activeCacheBuilder.executor(cacheMaintenanceExecutor.get());
-}
+.ticker(ticker)
+.writer(new ActiveCacheWriter());
if (LOG.isDebugEnabled()) {
activeCacheBuilder.recordStats();
}
@@ -153,11 +147,8 @@ public class DefaultMessageDistributor implements
MessageDistributor, Serializab
.newBuilder()
.maximumSize(maxNumberOf
metron git commit: METRON-1869 Unable to Sort an Escalated Meta Alert (nickwallen) closes apache/metron#1280
Repository: metron
Updated Branches:
refs/heads/master 4ef65e09e -> 877b51014
METRON-1869 Unable to Sort an Escalated Meta Alert (nickwallen) closes
apache/metron#1280
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/877b5101
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/877b5101
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/877b5101
Branch: refs/heads/master
Commit: 877b510146456f2eed6eb12c35b1124c35b00aa5
Parents: 4ef65e0
Author: nickwallen
Authored: Tue Dec 4 09:51:19 2018 -0500
Committer: nickallen
Committed: Tue Dec 4 09:51:19 2018 -0500
--
.../package/files/metaalert_index.template | 3 +
.../ElasticsearchMetaAlertIntegrationTest.java | 3 +-
.../dao/metaalert/MetaAlertIntegrationTest.java | 75 +++-
3 files changed, 79 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/877b5101/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
--
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
index 040c411..0c9978d 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
@@ -29,6 +29,9 @@
"score": {
"type": "keyword"
},
+"alert_status": {
+ "type": "keyword"
+},
"status": {
"type": "keyword"
},
http://git-wip-us.apache.org/repos/asf/metron/blob/877b5101/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
index cba0f65..eb821a8 100644
---
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
+++
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchMetaAlertIntegrationTest.java
@@ -115,7 +115,8 @@ public class ElasticsearchMetaAlertIntegrationTest extends
MetaAlertIntegrationT
"ip_src_addr" : { "type" : "keyword" },
"score" : { "type" : "integer" },
"metron_alert" : { "type" : "nested" },
- "source:type" : { "type" : "keyword"}
+ "source:type" : { "type" : "keyword"},
+ "alert_status": { "type": "keyword" }
}
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/877b5101/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/metaalert/MetaAlertIntegrationTest.java
--
diff --git
a/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/metaalert/MetaAlertIntegrationTest.java
b/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/metaalert/MetaAlertIntegrationTest.java
index 90bee80..f1355a6 100644
---
a/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/metaalert/MetaAlertIntegrationTest.java
+++
b/metron-platform/metron-indexing/src/test/java/org/apache/metron/indexing/dao/metaalert/MetaAlertIntegrationTest.java
@@ -59,6 +59,8 @@ import org.json.simple.parser.ParseException;
import org.junit.Assert;
import org.junit.Test;
+import static org.apache.metron.integration.utils.TestUtils.assertEventually;
+
public abstract class MetaAlertIntegrationTest {
private static final String META_INDEX_FLAG = "%META_INDEX%";
@@ -145,7 +147,6 @@ public abstract class MetaAlertIntegrationTest {
@Multiline
public static String statusPatchRequest;
-
@Test
public void shouldGetAllMetaAl
[metron] Git Push Summary
Repository: metron Updated Branches: refs/heads/feature/METRON-1699-create-batch-profiler [deleted] fa3be8d32
metron git commit: METRON-1845 Correct Test Data Load in Elasticsearch Integration Tests (nickwallen) closes apache/metron#1247
Repository: metron
Updated Branches:
refs/heads/master 42fc699ce -> 89a2beda4
METRON-1845 Correct Test Data Load in Elasticsearch Integration Tests
(nickwallen) closes apache/metron#1247
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/89a2beda
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/89a2beda
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/89a2beda
Branch: refs/heads/master
Commit: 89a2beda4f07911c8b3cd7dee8a2c3426838d161
Parents: 42fc699
Author: nickwallen
Authored: Wed Nov 28 20:10:50 2018 -0500
Committer: nickallen
Committed: Wed Nov 28 20:10:50 2018 -0500
--
.../elasticsearch/dao/ElasticsearchDao.java | 16 ++-
.../dao/ElasticsearchUpdateDao.java | 9 ++
.../ElasticsearchIndexingIntegrationTest.java | 15 ++-
.../ElasticsearchMetaAlertIntegrationTest.java | 78 +++---
.../ElasticsearchSearchIntegrationTest.java | 106 ---
.../ElasticsearchUpdateIntegrationTest.java | 68 ++--
.../components/ElasticSearchComponent.java | 96 -
.../dao/metaalert/MetaAlertIntegrationTest.java | 3 +-
8 files changed, 227 insertions(+), 164 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/89a2beda/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
index 210e1ce..dcd6fdb 100644
---
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchDao.java
@@ -39,6 +39,7 @@ import org.apache.metron.indexing.dao.update.Document;
import org.apache.metron.indexing.dao.update.OriginalNotFoundException;
import org.apache.metron.indexing.dao.update.PatchRequest;
import org.apache.metron.indexing.dao.update.ReplaceRequest;
+import org.elasticsearch.action.support.WriteRequest;
import org.elasticsearch.index.query.QueryBuilder;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -63,8 +64,9 @@ public class ElasticsearchDao implements IndexDao {
private ElasticsearchRequestSubmitter requestSubmitter;
private AccessConfig accessConfig;
+ private WriteRequest.RefreshPolicy refreshPolicy;
- protected ElasticsearchDao(ElasticsearchClient client,
+ public ElasticsearchDao(ElasticsearchClient client,
AccessConfig config,
ElasticsearchSearchDao searchDao,
ElasticsearchUpdateDao updateDao,
@@ -83,6 +85,7 @@ public class ElasticsearchDao implements IndexDao {
public ElasticsearchDao() {
//uninitialized.
+refreshPolicy = WriteRequest.RefreshPolicy.NONE;
}
public AccessConfig getAccessConfig() {
@@ -100,10 +103,10 @@ public class ElasticsearchDao implements IndexDao {
this.accessConfig = config;
this.columnMetadataDao = new ElasticsearchColumnMetadataDao(this.client);
this.requestSubmitter = new ElasticsearchRequestSubmitter(this.client);
- this.searchDao = new ElasticsearchSearchDao(client, accessConfig,
columnMetadataDao,
- requestSubmitter);
+ this.searchDao = new ElasticsearchSearchDao(client, accessConfig,
columnMetadataDao, requestSubmitter);
this.retrieveLatestDao = new ElasticsearchRetrieveLatestDao(client);
- this.updateDao = new ElasticsearchUpdateDao(client, accessConfig,
retrieveLatestDao);
+ this.updateDao = new ElasticsearchUpdateDao(client, accessConfig,
retrieveLatestDao)
+ .withRefreshPolicy(refreshPolicy);
}
if (columnMetadataDao == null) {
@@ -187,6 +190,11 @@ public class ElasticsearchDao implements IndexDao {
return this.updateDao.removeCommentFromAlert(request, latest);
}
+ public ElasticsearchDao withRefreshPolicy(WriteRequest.RefreshPolicy
refreshPolicy) {
+this.refreshPolicy = refreshPolicy;
+return this;
+ }
+
protected Optional getIndexName(String guid, String sensorType)
throws IOException {
return updateDao.getIndexName(guid, sensorType);
}
http://git-wip-us.apache.org/repos/asf/metron/blob/89a2beda/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDao.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/dao/ElasticsearchUpdateDao.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apa
metron git commit: METRON-1888 Default Topology Settings in MPack Cause Profiler to Stall (nickwallen) closes apache/metron#1276
Repository: metron Updated Branches: refs/heads/master e1a957be3 -> 42fc699ce METRON-1888 Default Topology Settings in MPack Cause Profiler to Stall (nickwallen) closes apache/metron#1276 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/42fc699c Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/42fc699c Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/42fc699c Branch: refs/heads/master Commit: 42fc699ce1df04fd065d131293a8322a095c1e9c Parents: e1a957b Author: nickwallen Authored: Tue Nov 27 12:44:58 2018 -0500 Committer: nickallen Committed: Tue Nov 27 12:44:58 2018 -0500 -- .../METRON/CURRENT/configuration/metron-profiler-env.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/42fc699c/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml -- diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml index c45d799..07b8d11 100644 --- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml +++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-profiler-env.xml @@ -165,7 +165,7 @@ profiler_topology_max_spout_pending Profiler Topology Spout Max Pending Tuples Spout Max Pending Tuples -300 + true
metron git commit: METRON-1825 Upgrade bro to 2.5.5 (JonZeolla via nickwallen) closes apache/metron#1237
Repository: metron
Updated Branches:
refs/heads/master 12e3ae548 -> 952b707df
METRON-1825 Upgrade bro to 2.5.5 (JonZeolla via nickwallen) closes
apache/metron#1237
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/952b707d
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/952b707d
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/952b707d
Branch: refs/heads/master
Commit: 952b707dfba7d584dfe10cb650c3e5ddf6c4789d
Parents: 12e3ae5
Author: JonZeolla
Authored: Wed Nov 21 14:08:01 2018 -0500
Committer: nickallen
Committed: Wed Nov 21 14:08:01 2018 -0500
--
metron-deployment/ansible/playbooks/docker_probe_install.yml | 2 +-
metron-deployment/ansible/roles/bro/vars/main.yml | 2 +-
.../development/centos6/ansible/inventory/group_vars/all | 2 +-
.../development/ubuntu14/ansible/inventory/group_vars/all | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/952b707d/metron-deployment/ansible/playbooks/docker_probe_install.yml
--
diff --git a/metron-deployment/ansible/playbooks/docker_probe_install.yml
b/metron-deployment/ansible/playbooks/docker_probe_install.yml
index 824aa40..d42c57a 100644
--- a/metron-deployment/ansible/playbooks/docker_probe_install.yml
+++ b/metron-deployment/ansible/playbooks/docker_probe_install.yml
@@ -32,7 +32,7 @@
vars:
metron_version: 0.6.1
metron_directory: /usr/metron/{{ metron_version }}
-bro_version: "2.5.4"
+bro_version: "2.5.5"
fixbuf_version: "1.7.1"
yaf_version: "2.8.0"
daq_version: "2.0.6-1"
http://git-wip-us.apache.org/repos/asf/metron/blob/952b707d/metron-deployment/ansible/roles/bro/vars/main.yml
--
diff --git a/metron-deployment/ansible/roles/bro/vars/main.yml
b/metron-deployment/ansible/roles/bro/vars/main.yml
index 9e6f221..e37d10a 100644
--- a/metron-deployment/ansible/roles/bro/vars/main.yml
+++ b/metron-deployment/ansible/roles/bro/vars/main.yml
@@ -17,7 +17,7 @@
---
bro_home: /usr/local/bro
bro_bin: "{{ bro_home }}/bin"
-bro_version: 2.5.4
+bro_version: 2.5.5
bro_daemon_log: /var/log/bro.log
bro_topic: bro
http://git-wip-us.apache.org/repos/asf/metron/blob/952b707d/metron-deployment/development/centos6/ansible/inventory/group_vars/all
--
diff --git
a/metron-deployment/development/centos6/ansible/inventory/group_vars/all
b/metron-deployment/development/centos6/ansible/inventory/group_vars/all
index 694ecb7..f2a3dc9 100644
--- a/metron-deployment/development/centos6/ansible/inventory/group_vars/all
+++ b/metron-deployment/development/centos6/ansible/inventory/group_vars/all
@@ -37,7 +37,7 @@ enrichment_hbase_table: enrichment
# metron
metron_version: 0.6.1
metron_directory: /usr/metron/{{ metron_version }}
-bro_version: "2.5.4"
+bro_version: "2.5.5"
fixbuf_version: "1.7.1"
yaf_version: "2.8.0"
daq_version: "2.0.6-1"
http://git-wip-us.apache.org/repos/asf/metron/blob/952b707d/metron-deployment/development/ubuntu14/ansible/inventory/group_vars/all
--
diff --git
a/metron-deployment/development/ubuntu14/ansible/inventory/group_vars/all
b/metron-deployment/development/ubuntu14/ansible/inventory/group_vars/all
index f26e7f8..0896694 100644
--- a/metron-deployment/development/ubuntu14/ansible/inventory/group_vars/all
+++ b/metron-deployment/development/ubuntu14/ansible/inventory/group_vars/all
@@ -37,7 +37,7 @@ enrichment_hbase_table: enrichment
# metron
metron_version: 0.6.1
metron_directory: /usr/metron/{{ metron_version }}
-bro_version: "2.5.4"
+bro_version: "2.5.5"
fixbuf_version: "1.7.1"
yaf_version: "2.8.0"
daq_version: "2.0.6-1"
metron git commit: METRON-1874 Create a Parser Debugger (nickwallen) closes apache/metron#1265
Repository: metron
Updated Branches:
refs/heads/master 575ba03b9 -> d599efb08
METRON-1874 Create a Parser Debugger (nickwallen) closes apache/metron#1265
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/d599efb0
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/d599efb0
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/d599efb0
Branch: refs/heads/master
Commit: d599efb08021a85317680ef52ad2e50405621647
Parents: 575ba03
Author: nickwallen
Authored: Tue Nov 20 13:54:45 2018 -0500
Committer: nickallen
Committed: Tue Nov 20 13:54:45 2018 -0500
--
.../org/apache/metron/management/Functions.java | 72 +
.../metron/management/KafkaFunctions.java | 19 +-
.../metron/management/ParserFunctions.java | 161 +++
.../metron/management/StellarParserRunner.java | 155 +++
.../metron/management/ParserFunctionsTest.java | 264 +++
.../management/StellarParserRunnerTest.java | 148 +++
6 files changed, 801 insertions(+), 18 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/d599efb0/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
--
diff --git
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
new file mode 100644
index 000..5e591aa
--- /dev/null
+++
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/Functions.java
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ *
+ */
+
+package org.apache.metron.management;
+
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+
+import java.util.List;
+
+import static java.lang.String.format;
+
+/**
+ * Contains utility functionality that is useful across all of the Stellar
management functions.
+ */
+public class Functions {
+
+ /**
+ * Get an argument from the Stellar function arguments
+ *
+ * @param argName The name of the argument.
+ * @param index The index within the list of arguments.
+ * @param clazz The type expected.
+ * @param args All of the arguments.
+ * @param The type of the argument expected.
+ */
+ public static T getArg(String argName, int index, Class clazz,
List args) {
+if(index >= args.size()) {
+ String msg = format("missing '%s'; expected at least %d argument(s),
found %d", argName, index+1, args.size());
+ throw new IllegalArgumentException(msg);
+}
+
+return ConversionUtils.convert(args.get(index), clazz);
+ }
+
+ /**
+ * Returns true if an argument of a specific type at a given index exists.
Otherwise returns
+ * false if an argument does not exist at the index or is not the expected
type.
+ *
+ * @param argName The name of the argument.
+ * @param index The index within the list of arguments.
+ * @param clazz The type expected.
+ * @param args All of the arguments.
+ * @param The type of argument expected.
+ */
+ public static boolean hasArg(String argName, int index, Class clazz,
List args) {
+boolean result = false;
+
+if(args.size() > index) {
+ if(clazz.isAssignableFrom(args.get(index).getClass())) {
+return true;
+ }
+}
+
+return result;
+ }
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/d599efb0/metron-platform/metron-management/src/main/java/org/apache/metron/management/KafkaFunctions.java
--
diff --git
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/KafkaFunctions.java
b/metron-platform/metron-management/src/main/java/org/apache/metron/management/KafkaFunctions.java
index 76418b6..78026fb 100644
---
a/metron-platform/metron-management/src/main/java/org/apache/metron/management/KafkaFunctions.java
+++
b/metron-platform/met
metron git commit: METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes apache/metron#1270
Repository: metron
Updated Branches:
refs/heads/master 05d309f91 -> 575ba03b9
METRON-1880 Use Caffeine for Profiler Caching (nickwallen) closes
apache/metron#1270
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/575ba03b
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/575ba03b
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/575ba03b
Branch: refs/heads/master
Commit: 575ba03b9d91d96bf5c2bd1dbee95681ae3131d0
Parents: 05d309f
Author: nickwallen
Authored: Tue Nov 20 13:31:46 2018 -0500
Committer: nickallen
Committed: Tue Nov 20 13:31:46 2018 -0500
--
.../metron-profiler-common/README.md| 14 ++--
metron-analytics/metron-profiler-common/pom.xml | 5 ++
.../profiler/DefaultMessageDistributor.java | 84
.../profiler/DefaultMessageDistributorTest.java | 27 ---
4 files changed, 79 insertions(+), 51 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/575ba03b/metron-analytics/metron-profiler-common/README.md
--
diff --git a/metron-analytics/metron-profiler-common/README.md
b/metron-analytics/metron-profiler-common/README.md
index 8f26aaf..fe4c2ed 100644
--- a/metron-analytics/metron-profiler-common/README.md
+++ b/metron-analytics/metron-profiler-common/README.md
@@ -28,7 +28,7 @@ The Profiler is a feature extraction mechanism that can
generate a profile descr
This is achieved by summarizing the telemetry data consumed by Metron over
tumbling windows. A summary statistic is applied to the data received within a
given window. Collecting these values across many windows result in a time
series that is useful for analysis.
-Any field contained within a message can be used to generate a profile. A
profile can even be produced by combining fields that originate in different
data sources. A user has considerable power to transform the data used in a
profile by leveraging the Stellar language.
+Any field contained within a message can be used to generate a profile. A
profile can even be produced by combining fields that originate in different
data sources. A user has considerable power to transform the data used in a
profile by leveraging the Stellar language.
There are three separate ports of the Profiler that share this common code
base.
* The [Storm Profiler](../metron-profiler-storm/README.md) builds low-latency
profiles over streaming data sets.
@@ -58,12 +58,12 @@ Let's start with a simple example. The following profile
maintains a count of th
"profile": "hello-world",
"foreach": "ip_src_addr",
"init": {
-"count": 0
+"count": "0"
},
"update": {
"count": "count + 1"
},
- "result": "count",
+ "result": "count"
}
]
}
@@ -321,7 +321,7 @@ It is important to note that the Profiler can persist any
serializable Object, n
```
$ source /etc/default/metron
$ bin/stellar -z $ZOOKEEPER
-
+
[Stellar]>>> stats := PROFILE_GET( "example4", "10.0.0.1",
PROFILE_FIXED(30, "MINUTES"))
[org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9,
...]
```
@@ -330,10 +330,10 @@ It is important to note that the Profiler can persist any
serializable Object, n
```
[Stellar]>>> aStat := GET_FIRST(stats)
org.apache.metron.common.math.stats.OnlineStatisticsProvider@79fe4ab9
-
+
[Stellar]>>> STATS_MEAN(aStat)
15979.0625
-
+
[Stellar]>>> STATS_PERCENTILE(aStat, 90)
30310.958
```
@@ -341,7 +341,7 @@ It is important to note that the Profiler can persist any
serializable Object, n
1. Merge all of the profile measurements over the past 30 minutes into a
single sketch and calculate the 90th percentile.
```
[Stellar]>>> merged := STATS_MERGE( stats)
-
+
[Stellar]>>> STATS_PERCENTILE(merged, 90)
29810.992
```
http://git-wip-us.apache.org/repos/asf/metron/blob/575ba03b/metron-analytics/metron-profiler-common/pom.xml
--
diff --git a/metron-analytics/metron-profiler-common/pom.xml
b/metron-analytics/metron-profiler-common/pom.xml
index 630709e..eff70c4 100644
--- a/metron-analytics/metron-profiler-common/pom.xml
+++ b/metron-analytics/metron-profiler-common/pom.xml
@@ -28,6 +28,11 @@
+com.github.ben-manes.caffeine
+caffeine
+${global_caffeine_version}
+
+
org.apache.hadoo
[6/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts -- diff --git a/metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts b/metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts index d2066ea..2cb44e2 100644 --- a/metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts +++ b/metron-interface/metron-config/src/app/sensors/sensor-field-schema/sensor-field-schema.component.spec.ts @@ -19,505 +19,641 @@ /* tslint:disable:max-line-length */ import { TestBed, async, ComponentFixture } from '@angular/core/testing'; -import {Http} from '@angular/http'; -import {SimpleChanges, SimpleChange} from '@angular/core'; -import {SensorParserConfigService} from '../../service/sensor-parser-config.service'; -import {StellarService} from '../../service/stellar.service'; -import {MetronAlerts} from '../../shared/metron-alerts'; -import {SensorFieldSchemaModule} from './sensor-field-schema.module'; -import {SensorFieldSchemaComponent, FieldSchemaRow} from './sensor-field-schema.component'; -import {KafkaService} from '../../service/kafka.service'; -import {Observable} from 'rxjs/Observable'; -import {StellarFunctionDescription} from '../../model/stellar-function-description'; -import {SensorParserConfig} from '../../model/sensor-parser-config'; -import {SensorEnrichmentConfig, EnrichmentConfig, ThreatIntelConfig} from '../../model/sensor-enrichment-config'; -import {ParseMessageRequest} from '../../model/parse-message-request'; -import {AutocompleteOption} from '../../model/autocomplete-option'; -import {FieldTransformer} from '../../model/field-transformer'; -import {SensorEnrichmentConfigService} from '../../service/sensor-enrichment-config.service'; - +import { HttpClient } from '@angular/common/http'; +import { SimpleChanges, SimpleChange } from '@angular/core'; +import { SensorParserConfigService } from '../../service/sensor-parser-config.service'; +import { StellarService } from '../../service/stellar.service'; +import { MetronAlerts } from '../../shared/metron-alerts'; +import { SensorFieldSchemaModule } from './sensor-field-schema.module'; +import { + SensorFieldSchemaComponent, + FieldSchemaRow +} from './sensor-field-schema.component'; +import { KafkaService } from '../../service/kafka.service'; +import { Observable, throwError } from 'rxjs'; +import { StellarFunctionDescription } from '../../model/stellar-function-description'; +import { SensorParserConfig } from '../../model/sensor-parser-config'; +import { + SensorEnrichmentConfig, + EnrichmentConfig, + ThreatIntelConfig +} from '../../model/sensor-enrichment-config'; +import { ParseMessageRequest } from '../../model/parse-message-request'; +import { AutocompleteOption } from '../../model/autocomplete-option'; +import { FieldTransformer } from '../../model/field-transformer'; +import { SensorEnrichmentConfigService } from '../../service/sensor-enrichment-config.service'; class MockSensorParserConfigService { - -parseMessage(parseMessageRequest: ParseMessageRequest): Observable<{}> { -let parsedJson = { -'elapsed': 415, -'code': 200, -'ip_dst_addr': '207.109.73.154', -'original_string': '1467011157.401 415 127.0.0.1 TCP_MISS/200 337891 GET http://www.aliexpress.com/', -'method': 'GET', -'bytes': 337891, -'action': 'TCP_MISS', -'ip_src_addr': '127.0.0.1', -'url': 'http://www.aliexpress.com/af/shoes.html?', -'timestamp': '1467011157.401' -}; -return Observable.create((observable) => { -observable.next(parsedJson); -observable.complete(); -}); -} + parseMessage(parseMessageRequest: ParseMessageRequest): Observable<{}> { +let parsedJson = { + elapsed: 415, + code: 200, + ip_dst_addr: '207.109.73.154', + original_string: +'1467011157.401 415 127.0.0.1 TCP_MISS/200 337891 GET http://www.aliexpress.com/', + method: 'GET', + bytes: 337891, + action: 'TCP_MISS', + ip_src_addr: '127.0.0.1', + url: 'http://www.aliexpress.com/af/shoes.html?', + timestamp: '1467011157.401' +}; +return Observable.create(observable => { +
[1/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
Repository: metron
Updated Branches:
refs/heads/master 0e2a43484 -> 0c4c622b9
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
--
diff --git
a/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
b/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
index 5488209..b8cdf1f 100644
---
a/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
+++
b/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
@@ -15,17 +15,15 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import {async, TestBed, ComponentFixture} from '@angular/core/testing';
-import {KafkaService} from '../../service/kafka.service';
-import {Observable} from 'rxjs/Observable';
-import {SampleDataComponent} from './sample-data.component';
-import {SharedModule} from '../shared.module';
-import '../../rxjs-operators';
+import { async, TestBed, ComponentFixture } from '@angular/core/testing';
+import { KafkaService } from '../../service/kafka.service';
+import { Observable, throwError } from 'rxjs';
+import { SampleDataComponent } from './sample-data.component';
+import { SharedModule } from '../shared.module';
class MockKafkaService {
_sample: string[];
- _sampleCounter: number = 0;
-
+ _sampleCounter = 0;
public setSample(sampleMessages: string[]): void {
this._sample = sampleMessages;
@@ -33,7 +31,6 @@ class MockKafkaService {
}
public sample(name: string): Observable {
-
if (this._sampleCounter < this._sample.length) {
return Observable.create(observer => {
observer.next(this._sample[this._sampleCounter++]);
@@ -41,7 +38,7 @@ class MockKafkaService {
});
}
-return Observable.throw('Error');
+return throwError('Error');
}
}
@@ -58,24 +55,21 @@ describe('SampleDataComponent', () => {
beforeEach(async(() => {
TestBed.configureTestingModule({
imports: [SharedModule],
- declarations: [ SampleDataComponent],
+ declarations: [SampleDataComponent],
providers: [
SampleDataComponent,
-{provide: KafkaService, useClass: MockKafkaService}
+{ provide: KafkaService, useClass: MockKafkaService }
]
});
-
fixture = TestBed.createComponent(SampleDataComponent);
sampleDataComponent = fixture.componentInstance;
-kafkaService = fixture.debugElement.injector.get(KafkaService);
-
+kafkaService = TestBed.get(KafkaService);
}));
it('can instantiate SampleDataComponent', async(() => {
expect(sampleDataComponent instanceof SampleDataComponent).toBe(true);
}));
-
it('should emmit messages', async(() => {
let expectedMessage;
let successCount = 0;
@@ -140,11 +134,9 @@ describe('SampleDataComponent', () => {
sampleDataComponent.getPreviousSample();
expect(successCount).toEqual(7);
expect(failureCount).toEqual(1);
-
}));
it('should emmit messages on blur', async(() => {
-
let expectedMessage;
let successCount = 0;
@@ -155,9 +147,10 @@ describe('SampleDataComponent', () => {
expect(message).toEqual(expectedMessage);
});
-
expectedMessage = 'This is a simple message';
-fixture.debugElement.nativeElement.querySelector('textarea').value =
expectedMessage;
+fixture.debugElement.nativeElement.querySelector(
+ 'textarea'
+).value = expectedMessage;
sampleDataComponent.onBlur();
expect(successCount).toEqual(1);
@@ -165,16 +158,16 @@ describe('SampleDataComponent', () => {
expect(sampleDataComponent.sampleData.length).toEqual(1);
expect(sampleDataComponent.sampleData[0]).toEqual(expectedMessage);
-
expectedMessage = '';
-fixture.debugElement.nativeElement.querySelector('textarea').value =
expectedMessage;
+fixture.debugElement.nativeElement.querySelector(
+ 'textarea'
+).value = expectedMessage;
sampleDataComponent.onBlur();
expect(successCount).toEqual(2);
expect(sampleDataComponent.sampleDataIndex).toEqual(0);
expect(sampleDataComponent.sampleData.length).toEqual(1);
-
expectedMessage = sampleMessages[0];
sampleDataComponent.getNextSample();
@@ -182,7 +175,5 @@ describe('SampleDataComponent', () => {
expect(sampleDataComponent.sampleDataIndex).toEqual(1);
expect(sampleDataComponent.sampleData.length).toEqual(2);
expect(sampleDataComponent.sampleData[1]).toEqual(sampleMessages[0]);
-
}));
-
});
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/src/app/util/httpUtil.ts
--
diff --git a/metron-interface/metron-config/src/app/util/httpUtil.ts
[8/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/package-lock.json
--
diff --git a/metron-interface/metron-config/package-lock.json
b/metron-interface/metron-config/package-lock.json
index fff8cb7..f2a264d 100644
--- a/metron-interface/metron-config/package-lock.json
+++ b/metron-interface/metron-config/package-lock.json
@@ -4,180 +4,530 @@
"lockfileVersion": 1,
"requires": true,
"dependencies": {
-"@angular-cli/ast-tools": {
- "version": "1.0.16",
- "resolved":
"https://registry.npmjs.org/@angular-cli/ast-tools/-/ast-tools-1.0.16.tgz;,
- "integrity": "sha1-YxmULBol+4TjKUID6fejJmMvzlA=",
+"@angular-devkit/architect": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/architect/-/architect-0.8.3.tgz;,
+ "integrity":
"sha512-cFku50grgEJPg1CZZ0DXt4CkA6WnV6zN3hCXzpWbOfc/Id923Mml/jsEaoByeXHsRqb5rIZKZAhz7R509ya8OQ==",
"dev": true,
"requires": {
-"@angular/tsc-wrapped": "0.5.2",
-"denodeify": "1.2.1",
-"rxjs": "5.1.0",
-"typescript": "2.0.10"
+"@angular-devkit/core": "0.8.3",
+"rxjs": "6.2.2"
},
"dependencies": {
-"@angular/tsc-wrapped": {
- "version": "0.5.2",
- "resolved":
"https://registry.npmjs.org/@angular/tsc-wrapped/-/tsc-wrapped-0.5.2.tgz;,
- "integrity": "sha1-Lt30csRn/LM06pTe3aqnGZDFpII=",
+"@angular-devkit/core": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/core/-/core-0.8.3.tgz;,
+ "integrity":
"sha512-2KHt5osMs3zACYXev20ZU5SXdWoinoKwZkj2caj2LCj9W7QNHmsz34QvaygNq7YdJzF3jkXkdy0GSUgUgDke0w==",
"dev": true,
"requires": {
-"tsickle": "0.2.5"
+"ajv": "6.4.0",
+"chokidar": "2.0.4",
+"rxjs": "6.2.2",
+"source-map": "0.5.7"
}
-},
-"denodeify": {
- "version": "1.2.1",
- "resolved":
"https://registry.npmjs.org/denodeify/-/denodeify-1.2.1.tgz;,
- "integrity": "sha1-OjYof1A05pnnV3kBBSwubJQlFjE=",
+}
+ }
+},
+"@angular-devkit/build-angular": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/build-angular/-/build-angular-0.8.3.tgz;,
+ "integrity":
"sha512-NWwWV+6apvCGmllWjwwy9Pmj5uK5tVGL/xIVQgSGC5waLmW/vFWNRXCI50ji5UPP+vAeRi/pWdXWMxuoVA08FA==",
+ "dev": true,
+ "requires": {
+"@angular-devkit/architect": "0.8.3",
+"@angular-devkit/build-optimizer": "0.8.3",
+"@angular-devkit/build-webpack": "0.8.3",
+"@angular-devkit/core": "0.8.3",
+"@ngtools/webpack": "6.2.3",
+"ajv": "6.4.0",
+"autoprefixer": "8.6.5",
+"circular-dependency-plugin": "5.0.2",
+"clean-css": "4.2.1",
+"copy-webpack-plugin": "4.5.2",
+"file-loader": "1.1.11",
+"glob": "7.1.3",
+"html-webpack-plugin": "3.2.0",
+"istanbul": "0.4.5",
+"istanbul-instrumenter-loader": "3.0.1",
+"karma-source-map-support": "1.3.0",
+"less": "3.8.1",
+"less-loader": "4.1.0",
+"license-webpack-plugin": "1.5.0",
+"loader-utils": "1.1.0",
+"mini-css-extract-plugin": "0.4.3",
+"minimatch": "3.0.4",
+"node-sass": "4.9.3",
+"opn": "5.3.0",
+"parse5": "4.0.0",
+"portfinder": "1.0.17",
+"postcss": "6.0.23",
+"postcss-import": "11.1.0",
+"postcss-loader": "2.1.6",
+"postcss-url": "7.3.2",
+"raw-loader": "0.5.1",
+"rxjs": "6.2.2",
+"sass-loader": "7.1.0",
+"semver": "5.5.1",
+"source-map-loader": "0.2.4",
+"source-map-support": "0.5.9",
+"stats-webpack-plugin": "0.6.2",
+"style-loader": "0.21.0",
+"stylus": "0.54.5",
+"stylus-loader": "3.0.2",
+"tree-kill": "1.2.0",
+"uglifyjs-webpack-plugin": "1.3.0",
+"url-loader": "1.1.1",
+"webpack": "4.19.1",
+"webpack-dev-middleware": "3.3.0",
+"webpack-dev-server": "3.1.8",
+"webpack-merge": "4.1.4",
+"webpack-sources": "1.3.0",
+"webpack-subresource-integrity": "1.1.0-rc.6"
+ },
+ "dependencies": {
+"@angular-devkit/core": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/core/-/core-0.8.3.tgz;,
+ "integrity":
"sha512-2KHt5osMs3zACYXev20ZU5SXdWoinoKwZkj2caj2LCj9W7QNHmsz34QvaygNq7YdJzF3jkXkdy0GSUgUgDke0w==",
+ "dev": true,
+ "requires": {
+"ajv": "6.4.0",
+"chokidar": "2.0.4",
+"rxjs": "6.2.2",
+"source-map": "0.5.7"
+ }
+}
+ }
+},
+"@angular-devkit/build-optimizer": {
+ "version": "0.8.3",
+
[4/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
b/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
index 647e02f..1ba297c 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
@@ -15,30 +15,36 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import {Component, OnInit, ViewChild} from '@angular/core';
-import {FormGroup, Validators, FormControl} from '@angular/forms';
-import {SensorParserConfig} from '../../model/sensor-parser-config';
-import {SensorParserConfigService} from
'../../service/sensor-parser-config.service';
-import {Router, ActivatedRoute} from '@angular/router';
-import {MetronAlerts} from '../../shared/metron-alerts';
-import {SensorParserContext} from '../../model/sensor-parser-context';
-import {SensorEnrichmentConfigService} from
'../../service/sensor-enrichment-config.service';
-import {SensorEnrichmentConfig} from '../../model/sensor-enrichment-config';
-import {SensorFieldSchemaComponent} from
'../sensor-field-schema/sensor-field-schema.component';
-import {SensorRawJsonComponent} from
'../sensor-raw-json/sensor-raw-json.component';
-import {KafkaService} from '../../service/kafka.service';
-import {SensorIndexingConfigService} from
'../../service/sensor-indexing-config.service';
-import {IndexingConfigurations} from '../../model/sensor-indexing-config';
-import {RestError} from '../../model/rest-error';
-import {HdfsService} from '../../service/hdfs.service';
-import {GrokValidationService} from '../../service/grok-validation.service';
+import { Component, OnInit, ViewChild } from '@angular/core';
+import { FormGroup, Validators, FormControl } from '@angular/forms';
+import { SensorParserConfig } from '../../model/sensor-parser-config';
+import { SensorParserConfigService } from
'../../service/sensor-parser-config.service';
+import { Router, ActivatedRoute } from '@angular/router';
+import { MetronAlerts } from '../../shared/metron-alerts';
+import { SensorParserContext } from '../../model/sensor-parser-context';
+import { SensorEnrichmentConfigService } from
'../../service/sensor-enrichment-config.service';
+import { SensorEnrichmentConfig } from '../../model/sensor-enrichment-config';
+import { SensorFieldSchemaComponent } from
'../sensor-field-schema/sensor-field-schema.component';
+import { SensorRawJsonComponent } from
'../sensor-raw-json/sensor-raw-json.component';
+import { KafkaService } from '../../service/kafka.service';
+import { SensorIndexingConfigService } from
'../../service/sensor-indexing-config.service';
+import { IndexingConfigurations } from '../../model/sensor-indexing-config';
+import { RestError } from '../../model/rest-error';
+import { HdfsService } from '../../service/hdfs.service';
+import { GrokValidationService } from '../../service/grok-validation.service';
export enum Pane {
- GROK, RAWJSON, FIELDSCHEMA, THREATTRIAGE, STORMSETTINGS
+ GROK,
+ RAWJSON,
+ FIELDSCHEMA,
+ THREATTRIAGE,
+ STORMSETTINGS
}
export enum KafkaStatus {
- NO_TOPIC, NOT_EMITTING, EMITTING
+ NO_TOPIC,
+ NOT_EMITTING,
+ EMITTING
}
@Component({
@@ -46,9 +52,7 @@ export enum KafkaStatus {
templateUrl: 'sensor-parser-config.component.html',
styleUrls: ['sensor-parser-config.component.scss']
})
-
export class SensorParserConfigComponent implements OnInit {
-
sensorConfigForm: FormGroup;
transformsValidationForm: FormGroup;
@@ -73,7 +77,7 @@ export class SensorParserConfigComponent implements OnInit {
grokStatementValid = false;
availableParsers = {};
availableParserNames = [];
- grokStatement = '';
+ grokStatement = {};
patternLabel = '';
currentSensors = [];
@@ -81,7 +85,10 @@ export class SensorParserConfigComponent implements OnInit {
topicExists: boolean = false;
- transformsValidationResult: {map: any, keys: string[]} = {map: {}, keys: []};
+ transformsValidationResult: { map: any; keys: string[] } = {
+map: {},
+keys: []
+ };
transformsValidation: SensorParserContext = new SensorParserContext();
pane = Pane;
@@ -90,65 +97,94 @@ export class SensorParserConfigComponent implements OnInit {
kafkaStatus = KafkaStatus;
currentKafkaStatus = null;
- @ViewChild(SensorFieldSchemaComponent) sensorFieldSchema:
SensorFieldSchemaComponent;
- @ViewChild(SensorRawJsonComponent) sensorRawJson: SensorRawJsonComponent;
-
- constructor(private sensorParserConfigService: SensorParserConfigService,
private
[3/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
b/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
index 02f1fd9..6549b4b 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
@@ -16,153 +16,161 @@
* limitations under the License.
*/
-import {async, TestBed, ComponentFixture} from '@angular/core/testing';
-import {SensorStormSettingsComponent} from './sensor-storm-settings.component';
-import {SharedModule} from '../../shared/shared.module';
-import {SimpleChanges, SimpleChange} from '@angular/core';
-import {SensorParserConfig} from '../../model/sensor-parser-config';
-import {SensorStormSettingsModule} from './sensor-storm-settings.module';
-import '../../rxjs-operators';
+import { async, TestBed, ComponentFixture } from '@angular/core/testing';
+import { SensorStormSettingsComponent } from
'./sensor-storm-settings.component';
+import { SharedModule } from '../../shared/shared.module';
+import { SimpleChanges, SimpleChange } from '@angular/core';
+import { SensorParserConfig } from '../../model/sensor-parser-config';
+import { SensorStormSettingsModule } from './sensor-storm-settings.module';
describe('Component: SensorStormSettingsComponent', () => {
-
-let fixture: ComponentFixture;
-let component: SensorStormSettingsComponent;
-let sensorParserConfig: SensorParserConfig = new SensorParserConfig();
-sensorParserConfig.sensorTopic = 'bro';
-sensorParserConfig.parserClassName =
'org.apache.metron.parsers.bro.BasicBroParser';
-sensorParserConfig.parserConfig = {};
-sensorParserConfig.numWorkers = 2;
-sensorParserConfig.numAckers = 2;
-sensorParserConfig.spoutParallelism = 2;
-sensorParserConfig.spoutNumTasks = 2;
-sensorParserConfig.parserParallelism = 2;
-sensorParserConfig.parserNumTasks = 2;
-sensorParserConfig.errorWriterParallelism = 2;
-sensorParserConfig.errorWriterNumTasks = 2;
-sensorParserConfig.spoutConfig = {'spoutConfigProp': 'spoutConfigValue1'};
-sensorParserConfig.stormConfig = {'stormConfigProp': 'stormConfigValue1'};
-
-beforeEach(async(() => {
-TestBed.configureTestingModule({
-imports: [SharedModule, SensorStormSettingsModule],
-});
-
-fixture = TestBed.createComponent(SensorStormSettingsComponent);
-component = fixture.componentInstance;
-}));
-
-it('should create an instance', () => {
-expect(component).toBeDefined();
-});
-
-it('should create an instance', () => {
-spyOn(component, 'init');
-let changes: SimpleChanges = {'showStormSettings': new
SimpleChange(false, true)};
-
-component.ngOnChanges(changes);
-expect(component.init).toHaveBeenCalled();
-
-changes = {'showStormSettings': new SimpleChange(true, false)};
-component.ngOnChanges(changes);
-expect(component.init['calls'].count()).toEqual(1);
-
-fixture.destroy();
+ let fixture: ComponentFixture;
+ let component: SensorStormSettingsComponent;
+ let sensorParserConfig: SensorParserConfig = new SensorParserConfig();
+ sensorParserConfig.sensorTopic = 'bro';
+ sensorParserConfig.parserClassName =
+'org.apache.metron.parsers.bro.BasicBroParser';
+ sensorParserConfig.parserConfig = {};
+ sensorParserConfig.numWorkers = 2;
+ sensorParserConfig.numAckers = 2;
+ sensorParserConfig.spoutParallelism = 2;
+ sensorParserConfig.spoutNumTasks = 2;
+ sensorParserConfig.parserParallelism = 2;
+ sensorParserConfig.parserNumTasks = 2;
+ sensorParserConfig.errorWriterParallelism = 2;
+ sensorParserConfig.errorWriterNumTasks = 2;
+ sensorParserConfig.spoutConfig = { spoutConfigProp: 'spoutConfigValue1' };
+ sensorParserConfig.stormConfig = { stormConfigProp: 'stormConfigValue1' };
+
+ beforeEach(async(() => {
+TestBed.configureTestingModule({
+ imports: [SharedModule, SensorStormSettingsModule]
});
-it('should initialise the fields', () => {
-
-component.init();
-expect(component.newSensorParserConfig).toEqual(new
SensorParserConfig());
-
-component.sensorParserConfig = sensorParserConfig;
-component.init();
-expect(component.newSensorParserConfig).toEqual(sensorParserConfig);
-expect(component.newSpoutConfig).toEqual('{\n\t"spoutConfigProp":
"spoutConfigValue1"\n}');
-expect(component.newStormConfig).toEqual('{\n\t"stormConfigProp":
"stormConfigValue1"\n}');
-
-
[7/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/package.json
--
diff --git a/metron-interface/metron-config/package.json
b/metron-interface/metron-config/package.json
index 6e196d5..ffc92ff 100644
--- a/metron-interface/metron-config/package.json
+++ b/metron-interface/metron-config/package.json
@@ -7,10 +7,11 @@
},
"angular-cli": {},
"scripts": {
-"build": "./node_modules/angular-cli/bin/ng build -prod",
+"build": "ng build --prod",
"start": "ng serve",
"lint": "tslint \"src/**/*.ts\"",
-"test": "./node_modules/angular-cli/bin/ng test --watch=false",
+"test": "ng test --browsers=Chrome",
+"testCI": "ng test --watch=false --browsers=ChromeHeadless",
"pree2e": "webdriver-manager update",
"e2e": "./node_modules/.bin/protractor",
"e2e-all": "./node_modules/.bin/protractor --suite=all",
@@ -20,47 +21,54 @@
},
"private": true,
"dependencies": {
+"@angular-devkit/schematics": "^0.7.5",
+"@angular/cli": "^6.2.3",
+"@angular/common": "^6.1.8",
+"@angular/compiler": "^6.1.8",
+"@angular/core": "^6.1.8",
+"@angular/forms": "^6.1.8",
+"@angular/http": "^6.1.8",
+"@angular/platform-browser": "^6.1.8",
+"@angular/platform-browser-dynamic": "^6.1.8",
+"@angular/platform-server": "^6.1.8",
+"@angular/router": "^6.1.8",
"@types/ace": "0.0.32",
-"@types/bootstrap": "^3.3.32",
-"@types/jasmine": "2.2.30",
-"@types/jquery": "^2.0.32",
+"@types/bootstrap": "^4.1.2",
+"@types/jasmine": "~2.8.6",
+"@types/jasminewd2": "~2.0.3",
+"@types/jquery": "^3.3.6",
+"@types/node": "^10.9.4",
"@types/tether": "^1.1.27",
-"@angular/common": "2.0.0",
-"@angular/compiler": "2.0.0",
-"@angular/core": "2.0.0",
-"@angular/forms": "2.0.0",
-"@angular/http": "2.0.0",
-"@angular/platform-browser": "2.0.0",
-"@angular/platform-browser-dynamic": "2.0.0",
-"@angular/router": "3.0.0",
"ace-builds": "^1.2.5",
"bootstrap": "4.0.0-alpha.5",
-"core-js": "^2.4.1",
+"core-js": "^2.5.7",
"font-awesome": "^4.6.3",
"jquery": "^3.3.1",
-"rxjs": "5.0.0-beta.12",
+"karma-phantomjs-launcher": "^1.0.4",
+"puppeteer": "^1.8.0",
+"rxjs": "6.2.2",
"tether": "^1.3.4",
"ts-helpers": "^1.1.1",
-"zone.js": "^0.6.23"
+"zone.js": "^0.8.26"
},
"devDependencies": {
-"angular-cli": "1.0.0-beta.15",
+"@angular-devkit/build-angular": "^0.8.3",
+"@angular/compiler-cli": "^6.1.8",
+"@types/request": "2.0.3",
"buffer-shims": "^1.0.0",
-"codelyzer": "~0.0.26",
+"codelyzer": "~4.2.1",
"copy": "^0.3.0",
-"jasmine-core": "2.4.1",
-"jasmine-spec-reporter": "2.5.0",
-"karma": "1.2.0",
-"karma-chrome-launcher": "^2.0.0",
-"karma-cli": "^1.0.1",
-"karma-jasmine": "^1.0.2",
-"karma-phantomjs-launcher": "^1.0.4",
-"karma-remap-istanbul": "0.6.0",
+"jasmine-core": "~2.99.1",
+"jasmine-spec-reporter": "~4.2.1",
+"karma": "^3.0.0",
+"karma-chrome-launcher": "~2.2.0",
+"karma-coverage-istanbul-reporter": "~2.0.0",
+"karma-jasmine": "^1.1.2",
+"karma-jasmine-html-reporter": "^0.2.2",
"phantomjs-prebuilt": "^2.1.14",
-"protractor": "4.0.5",
-"ts-node": "1.2.1",
-"tslint": "3.13.0",
-"typescript": "~2.0.3",
-"@types/request": "2.0.3"
+"protractor": "^5.4.1",
+"ts-node": "~5.0.1",
+"tslint": "^5.11.0",
+"typescript": "~2.7.2"
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/pom.xml
--
diff --git a/metron-interface/metron-config/pom.xml
b/metron-interface/metron-config/pom.xml
index fd76447..c6ef5b5 100644
--- a/metron-interface/metron-config/pom.xml
+++ b/metron-interface/metron-config/pom.xml
@@ -63,7 +63,7 @@
generate-resources
-ng build
+npm run build
npm
@@ -72,13 +72,13 @@
-npm test
+test
+npm testCI
- npm
+npm
-test
- test
+run testCI
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/scripts/prepend_license_header.sh
--
diff --git a/metron-interface/metron-config/scripts/prepend_license_header.sh
b/metron-interface/metron-config/scripts/prepend_license_header.sh
index 1957cd6..6bf004f 100755
--- a/metron-interface/metron-config/scripts/prepend_license_header.sh
[5/9] metron git commit: METRON-1749 Update Angular to latest release in Management UI (sardell via nickwallen) closes apache/metron#1217
http://git-wip-us.apache.org/repos/asf/metron/blob/0c4c622b/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts -- diff --git a/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts b/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts index 7d41003..2904ef5 100644 --- a/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts +++ b/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts @@ -15,23 +15,23 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -import {Component, OnInit} from '@angular/core'; -import {KafkaService} from '../../service/kafka.service'; -import {Router, ActivatedRoute} from '@angular/router'; -import {KafkaTopic} from '../../model/kafka-topic'; -import {MetronAlerts} from '../../shared/metron-alerts'; -import {SensorParserConfigService} from '../../service/sensor-parser-config.service'; -import {StormService} from '../../service/storm.service'; -import {TopologyStatus} from '../../model/topology-status'; -import {SensorParserConfigHistoryService} from '../../service/sensor-parser-config-history.service'; -import {SensorParserConfigHistory} from '../../model/sensor-parser-config-history'; -import {SensorEnrichmentConfigService} from '../../service/sensor-enrichment-config.service'; -import {SensorEnrichmentConfig} from '../../model/sensor-enrichment-config'; -import {RiskLevelRule} from '../../model/risk-level-rule'; -import {HdfsService} from '../../service/hdfs.service'; -import {RestError} from '../../model/rest-error'; -import {GrokValidationService} from '../../service/grok-validation.service'; -import {SensorParserConfig} from '../../model/sensor-parser-config'; +import { Component, OnInit } from '@angular/core'; +import { KafkaService } from '../../service/kafka.service'; +import { Router, ActivatedRoute } from '@angular/router'; +import { KafkaTopic } from '../../model/kafka-topic'; +import { MetronAlerts } from '../../shared/metron-alerts'; +import { SensorParserConfigService } from '../../service/sensor-parser-config.service'; +import { StormService } from '../../service/storm.service'; +import { TopologyStatus } from '../../model/topology-status'; +import { SensorParserConfigHistoryService } from '../../service/sensor-parser-config-history.service'; +import { SensorParserConfigHistory } from '../../model/sensor-parser-config-history'; +import { SensorEnrichmentConfigService } from '../../service/sensor-enrichment-config.service'; +import { SensorEnrichmentConfig } from '../../model/sensor-enrichment-config'; +import { RiskLevelRule } from '../../model/risk-level-rule'; +import { HdfsService } from '../../service/hdfs.service'; +import { RestError } from '../../model/rest-error'; +import { GrokValidationService } from '../../service/grok-validation.service'; +import { SensorParserConfig } from '../../model/sensor-parser-config'; @Component({ selector: 'metron-config-sensor-parser-readonly', @@ -39,7 +39,6 @@ import {SensorParserConfig} from '../../model/sensor-parser-config'; styleUrls: ['sensor-parser-config-readonly.component.scss'] }) export class SensorParserConfigReadonlyComponent implements OnInit { - selectedSensorName: string; startStopInProgress: boolean = false; kafkaTopic: KafkaTopic = new KafkaTopic(); @@ -47,78 +46,149 @@ export class SensorParserConfigReadonlyComponent implements OnInit { sensorParserConfig: SensorParserConfig = new SensorParserConfig(); topologyStatus: TopologyStatus = new TopologyStatus(); sensorEnrichmentConfig: SensorEnrichmentConfig = new SensorEnrichmentConfig(); - grokStatement: string = ''; + grokStatement = {}; transformsConfigKeys: string[] = []; transformsConfigMap: {} = {}; rules: RiskLevelRule[] = []; transformLinkText = 'show more'; threatTriageLinkText = 'show more'; - editViewMetaData: {label?: string, value?: string, type?: string, model?: string, boldTitle?: boolean}[] = [ -{type: 'SEPARATOR', model: '', value: ''}, -{label: 'PARSER', model: 'sensorParserConfigHistory', value: 'parserClassName'}, -{label: 'LAST UPDATED', model: 'sensorParserConfigHistory', value: 'modifiedByDate'}, -{label: 'LAST EDITOR', model: 'sensorParserConfigHistory', value: 'modifiedBy'}, -{label: 'STATE', model: 'topologyStatus', value: 'sensorStatus'}, -{label: 'ORIGINATOR', model:
metron git commit: METRON-1870 Intermittent Stellar REST test failures (merrimanr via nickwallen) closes apache/metron#1263
Repository: metron
Updated Branches:
refs/heads/master 0303d2405 -> 0e2a43484
METRON-1870 Intermittent Stellar REST test failures (merrimanr via nickwallen)
closes apache/metron#1263
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/0e2a4348
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/0e2a4348
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/0e2a4348
Branch: refs/heads/master
Commit: 0e2a4348465406ad3aca7614bc38a71571b63b83
Parents: 0303d24
Author: merrimanr
Authored: Wed Nov 14 13:04:11 2018 -0500
Committer: nickallen
Committed: Wed Nov 14 13:04:11 2018 -0500
--
.../stellar/dsl/functions/RestFunctions.java | 4 ++--
.../stellar/dsl/functions/RestFunctionsTest.java | 17 +++--
2 files changed, 9 insertions(+), 12 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/0e2a4348/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
index 7134bfc..354322a 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
@@ -223,14 +223,14 @@ public class RestFunctions {
* @return
* @throws IOException
*/
-private Object doGet(RestConfig restConfig, HttpGet httpGet,
HttpClientContext httpClientContext) throws IOException {
+protected Object doGet(RestConfig restConfig, HttpGet httpGet,
HttpClientContext httpClientContext) throws IOException {
// Schedule a command to abort the httpGet request if the timeout is
exceeded
ScheduledFuture scheduledFuture =
scheduledExecutorService.schedule(httpGet::abort, restConfig.getTimeout(),
TimeUnit.MILLISECONDS);
CloseableHttpResponse response;
try {
response = httpClient.execute(httpGet, httpClientContext);
- } catch(IOException e) {
+ } catch(Exception e) {
// Report a timeout if the httpGet request was aborted. Otherwise
rethrow exception.
if (httpGet.isAborted()) {
throw new IOException(String.format("Total Stellar REST request time
to %s exceeded the configured timeout of %d ms.", httpGet.getURI().toString(),
restConfig.getTimeout()));
http://git-wip-us.apache.org/repos/asf/metron/blob/0e2a4348/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RestFunctionsTest.java
--
diff --git
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RestFunctionsTest.java
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RestFunctionsTest.java
index ba80f02..4b912ef 100644
---
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RestFunctionsTest.java
+++
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/RestFunctionsTest.java
@@ -23,10 +23,9 @@ import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
-import org.apache.http.client.HttpClient;
import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.HttpGet;
import org.apache.http.client.protocol.HttpClientContext;
-import org.apache.http.conn.routing.HttpRoute;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
@@ -66,7 +65,10 @@ import static
org.apache.metron.stellar.dsl.functions.RestConfig.STELLAR_REST_SE
import static org.apache.metron.stellar.dsl.functions.RestConfig.TIMEOUT;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNull;
+import static org.mockito.Matchers.any;
+import static org.mockito.Mockito.doThrow;
import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.spy;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.mockito.Mockito.verifyNoMoreInteractions;
@@ -552,15 +554,10 @@ public class RestFunctionsTest {
*/
@Test
public void restGetShouldHandleIOException() throws
IllegalArgumentException, IOException {
-Map globalConfig = new HashMap() {{
- put(STELLAR_REST_
metron git commit: METRON-1740 Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages (liuy-tnz via nickwallen) closes apache/metron#1171
Repository: metron
Updated Branches:
refs/heads/master 42068d951 -> 61b1ede0b
METRON-1740 Improve Palo Alto parser to handle CONFIG and SYSTEM syslog
messages (liuy-tnz via nickwallen) closes apache/metron#1171
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/61b1ede0
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/61b1ede0
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/61b1ede0
Branch: refs/heads/master
Commit: 61b1ede0bdfd28b04a3783371deb935d293df998
Parents: 42068d9
Author: liuy-tnz
Authored: Tue Nov 13 07:00:51 2018 -0500
Committer: nickallen
Committed: Tue Nov 13 07:00:51 2018 -0500
--
.../paloalto/BasicPaloAltoFirewallParser.java | 413 ---
.../apache/metron/parsers/paloalto/README.md| 32 ++
.../BasicPaloAltoFirewallParserTest.java| 244 +++
3 files changed, 541 insertions(+), 148 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/61b1ede0/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParser.java
--
diff --git
a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParser.java
b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParser.java
index 9051f09..c8e8b62 100644
---
a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParser.java
+++
b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParser.java
@@ -34,21 +34,26 @@ import java.util.regex.Pattern;
public class BasicPaloAltoFirewallParser extends BasicParser {
- private static boolean empty_attribute( final String s ) {
+ private static boolean empty_attribute(final String s) {
return s == null || s.trim().isEmpty() || s.equals("\"\"");
}
- private static String unquoted_attribute( String s ) {
+ private static String unquoted_attribute(String s) {
s = s.trim();
-if ( s.startsWith( "\"" ) && s.endsWith( "\"" ) )
- return s.substring( 1, s.length( ) - 1 );
+if (s.startsWith("\"") && s.endsWith("\""))
+ return s.substring(1, s.length() - 1);
return s;
}
- private static final Logger _LOG = LoggerFactory.getLogger
- (BasicPaloAltoFirewallParser.class);
+ private static final Logger _LOG =
LoggerFactory.getLogger(BasicPaloAltoFirewallParser.class);
private static final long serialVersionUID = 3147090149725343999L;
+
+ private static final String LogTypeConfig = "CONFIG";
+ private static final String LogTypeSystem = "SYSTEM";
+ private static final String LogTypeThreat = "THREAT";
+ private static final String LogTypeTraffic = "TRAFFIC";
+
public static final String PaloAltoDomain = "palo_alto_domain";
public static final String ReceiveTime = "receive_time";
public static final String SerialNum = "serial";
@@ -101,6 +106,21 @@ public class BasicPaloAltoFirewallParser extends
BasicParser {
public static final String ParentSessionStartTime =
"parent_session_start_time";
public static final String TunnelType = "tunnel_type";
+ //System
+ public static final String EventId = "event_id";
+ public static final String Object = "object";
+ public static final String Module = "module";
+ public static final String Description = "description";
+
+ //Config
+ public static final String Command = "command";
+ public static final String Admin = "admin";
+ public static final String Client = "client";
+ public static final String Result = "result";
+ public static final String ConfigurationPath = "configuration_path";
+ public static final String BeforeChangeDetail = "before_change_detail";
+ public static final String AfterChangeDetail = "after_change_detail";
+
//Threat
public static final String URL = "url";
public static final String HOST = "host";
@@ -113,7 +133,7 @@ public class BasicPaloAltoFirewallParser extends
BasicParser {
public static final String PCAPID = "pcap_id";
public static final String WFFileDigest = "filedigest";
public static final String WFCloud = "cloud";
- public static final String UserAgent= "user_agent";
+ public static final String UserAgent = "user_agent";
public static final String WFFileType = "filetype";
metron-bro-plugin-kafka git commit: METRON-1304 Allow metron-bro-plugin-kafka to include or exclude logs (JonZeolla via nickwallen) closes apache/metron-bro-plugin-kafka#2
Repository: metron-bro-plugin-kafka Updated Branches: refs/heads/master 279a2970b -> 772788d1a METRON-1304 Allow metron-bro-plugin-kafka to include or exclude logs (JonZeolla via nickwallen) closes apache/metron-bro-plugin-kafka#2 Project: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/repo Commit: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/commit/772788d1 Tree: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/tree/772788d1 Diff: http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/diff/772788d1 Branch: refs/heads/master Commit: 772788d1ab97a37abe8f8356c03b01c57aab4a8a Parents: 279a297 Author: JonZeolla Authored: Sat Nov 10 12:05:43 2018 -0500 Committer: nickallen Committed: Sat Nov 10 12:05:43 2018 -0500 -- README.md | 81 scripts/Apache/Kafka/__load__.bro | 2 + scripts/Apache/Kafka/logs-to-kafka.bro | 45 +++ scripts/__load__.bro| 2 + scripts/init.bro| 47 +--- tests/Baseline/kafka.l2s-l2e-no-overlap/output | 4 + tests/Baseline/kafka.l2s-set-l2e-set/output | 3 + tests/Baseline/kafka.l2s-set-l2e-unset/output | 3 + tests/Baseline/kafka.l2s-unset-l2e-set/output | 3 + tests/Baseline/kafka.l2s-unset-l2e-unset/output | 3 + .../kafka.send-all-active-logs-l2e-set/output | 7 ++ .../kafka.send-all-active-logs-l2e-unset/output | 7 ++ .../output | 7 ++ .../output | 7 ++ tests/kafka/l2s-l2e-no-overlap.bro | 29 +++ tests/kafka/l2s-set-l2e-set.bro | 28 +++ tests/kafka/l2s-set-l2e-unset.bro | 27 +++ tests/kafka/l2s-unset-l2e-set.bro | 27 +++ tests/kafka/l2s-unset-l2e-unset.bro | 25 ++ tests/kafka/send-all-active-logs-l2e-set.bro| 32 tests/kafka/send-all-active-logs-l2e-unset.bro | 31 .../send-all-active-logs-l2s-set-l2e-set.bro| 33 .../send-all-active-logs-l2s-set-l2e-unset.bro | 32 23 files changed, 447 insertions(+), 38 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron-bro-plugin-kafka/blob/772788d1/README.md -- diff --git a/README.md b/README.md index 90d8444..ae6b260 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ The following examples highlight different ways that the plugin can be used. Si The goal in this example is to send all HTTP and DNS records to a Kafka topic named `bro`. * Any configuration value accepted by librdkafka can be added to the `kafka_conf` configuration table. * By defining `topic_name` all records will be sent to the same Kafka topic. - * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. + * Defining `logs_to_send` will ensure that only HTTP and DNS records are sent. ``` @load packages/metron-bro-plugin-kafka/Apache/Kafka redef Kafka::logs_to_send = set(HTTP::LOG, DNS::LOG); @@ -60,6 +60,32 @@ redef Kafka::kafka_conf = table( ### Example 2 +This plugin has the ability send all active logs to kafka with the following configuration. + +``` +@load packages/metron-bro-plugin-kafka/Apache/Kafka +redef Kafka::send_all_active_logs = T; +redef Kafka::kafka_conf = table( +["metadata.broker.list"] = "localhost:9092" +); +``` + +### Example 3 + +You can also specify a blacklist of bro logs to ensure they aren't being sent to kafka regardless of the `Kafka::send_all_active_logs` and `Kafka::logs_to_send` configurations. In this example, we will send all of the enabled logs except for the Conn log. + +``` +@load packages/metron-bro-plugin-kafka/Apache/Kafka +redef Kafka::send_all_active_logs = T; +redef Kafka::logs_to_exclude = set(Conn::LOG); +redef Kafka::topic_name = "bro"; +redef Kafka::kafka_conf = table( +["metadata.broker.list"] = "localhost:9092" +); +``` + +### Example 4 + It is also possible to send each log stream to a uniquely named topic. The goal in this example is to send all HTTP records to a Kafka topic named `http` and all DNS records to a separate Kafka topic named `dns`. * The `topic_name` value must be set to an empty string. * The `$path` value of Bro's Log Writer mechanism is used to define the topic name. @@ -97,7 +123,7 @@ event bro_init() } ``` -### Example 3 +### Example 5 You may want to configure bro to filter log messages with certain characteristics from being sent to your kafka topics. For instance, Metron currently doesn't support IPv6 source or destination IPs in the default enrichments, so it may be helpful to filter those log messages fr
metron git commit: METRON-1857 Fix Metaalert Nested Alert Field Name in Index Template (nickwallen) closes apache/metron#1253
Repository: metron
Updated Branches:
refs/heads/master bf6b07f7c -> b9461e765
METRON-1857 Fix Metaalert Nested Alert Field Name in Index Template
(nickwallen) closes apache/metron#1253
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/b9461e76
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/b9461e76
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/b9461e76
Branch: refs/heads/master
Commit: b9461e76506988ad51d6fc0f3a442909945048f2
Parents: bf6b07f
Author: nickwallen
Authored: Tue Nov 6 10:04:48 2018 -0500
Committer: nickallen
Committed: Tue Nov 6 10:04:48 2018 -0500
--
.../METRON/CURRENT/package/files/metaalert_index.template| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/b9461e76/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
--
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
index 05d5e32..040c411 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/metaalert_index.template
@@ -5,8 +5,8 @@
"dynamic_templates": [
{
"alert_template": {
- "path_match": "alert.*",
- "match_mapping_type": "string",
+ "path_match": "metron_alert.*",
+ "match_mapping_type": "*",
"mapping": {
"type": "keyword"
}
