[metron] branch master updated: METRON-2341 [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest (mmiklavc) closes apache/metron#1552
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new dbacbc3 METRON-2341 [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest (mmiklavc) closes apache/metron#1552 dbacbc3 is described below commit dbacbc3cae9173466de14bbc49f7e49bf45b11e0 Author: mmiklavc AuthorDate: Thu Jan 16 22:32:15 2020 -0500 METRON-2341 [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest (mmiklavc) closes apache/metron#1552 --- metron-interface/metron-rest/pom.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-interface/metron-rest/pom.xml b/metron-interface/metron-rest/pom.xml index ca59cb4..4420b74 100644 --- a/metron-interface/metron-rest/pom.xml +++ b/metron-interface/metron-rest/pom.xml @@ -38,7 +38,7 @@ 5.0.5.RELEASE 2.6.4 2.4.0 -4.41.2 +7.9
[metron] branch master updated: METRON-2322 Add Ambari connection check to upgrade_helper script (mmiklavc) closes apache/metron#1566
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new b8ea1e2 METRON-2322 Add Ambari connection check to upgrade_helper
script (mmiklavc) closes apache/metron#1566
b8ea1e2 is described below
commit b8ea1e22aebdd4e228f4c278c3171c18af664de4
Author: mmiklavc
AuthorDate: Fri Dec 6 12:04:50 2019 -0700
METRON-2322 Add Ambari connection check to upgrade_helper script (mmiklavc)
closes apache/metron#1566
---
Upgrading.md | 3 +++
metron-platform/metron-common/README.md | 4
.../metron-common/src/main/scripts/upgrade_helper.sh | 20 +---
3 files changed, 20 insertions(+), 7 deletions(-)
diff --git a/Upgrading.md b/Upgrading.md
index d5a848b..e6bf612 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -22,6 +22,9 @@ configuration which are non-backwards compatible.
## 0.7.2 to 0.7.3
+### [METRON-2239: Metron Automated backup and
restore](https://issues.apache.org/jira/browse/METRON-2239)
+An upgrade helper script has been added to
`$METRON_HOME/bin/upgrade_helper.sh`. This script will assist in backing up and
restoring Ambari configuration and Metron configuration stored in Zookeeper.
You can see more details at [Metron Upgrade
Helper](metron-platform/metron-common#metron-upgrade-helper) and [Upgrade
Steps](Upgrade_steps.md).
+
### [METRON-2321: Remove Legacy AWS Deployment
Path](https://issues.apache.org/jira/browse/METRON-2321)
The automated Amazon AWS deployment mechanism (previously located at
`metron-deployment/amazon-ec2`) has been removed. It is not the preferred
installation path for deploying to AWS. Using Ambari and the Metron MPack is
the preferred installation path. To deploy Metron to AWS, provision EC2 nodes,
install Ambari, install the Metron MPack, then use Ambari to deploy Metron.
diff --git a/metron-platform/metron-common/README.md
b/metron-platform/metron-common/README.md
index 2cee5dd..f0874eb 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -501,3 +501,7 @@ $METRON_HOME/bin/upgrade_helper.sh backup node1:8080 admin
admin metron_cluster
$METRON_HOME/bin/upgrade_helper.sh restore node1:8080 admin admin
metron_cluster
```
+Note: Before issuing a restore, you should verify that the backup completed
successfully. If there is an issue connecting to the Ambari server, the
following message will appear in the script output.
+```
+**ERROR:** Unable to get cluster detail from Ambari. Check your username,
password, and cluster name. Skipping.
+```
diff --git a/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
b/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
index e272756..0f079d5 100755
--- a/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
+++ b/metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
@@ -57,12 +57,18 @@ if [ "$mode" == "backup" ]; then
fi
if [ -f "/var/lib/ambari-server/resources/scripts/configs.py" ]; then
echo Backing up Ambari config...
-for config_type in $(curl -u $username:$password -H "X-Requested-By:
ambari" -X GET
http://$ambari_address/api/v1/clusters/$cluster_name?fields=Clusters/desired_configs
| grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2 |
grep metron);
-do
-echo Saving $config_type
-/var/lib/ambari-server/resources/scripts/configs.py -u $username
-p $password -a get -l ${ambari_address%:*} -n $cluster_name -c $config_type -f
$AMBARI_CONFIG_DIR/${config_type}.json
-done
-echo Done backing up Ambari config...
+echo Checking connection...
+ret_status=$(curl -s -i -u "$username":"$password" -H "X-Requested-By:
ambari" -X GET http://$ambari_address/api/v1/clusters/$cluster_name | head -n
1 | sed -e 's/[[:space:]]*$//')
+if [ "HTTP/1.1 200 OK" == "$ret_status" ]; then
+for config_type in $(curl -s -u "$username":"$password" -H
"X-Requested-By: ambari" -X GET
http://$ambari_address/api/v1/clusters/$cluster_name?fields=Clusters/desired_configs
| grep '" : {' | grep -v Clusters | grep -v desired_configs | cut -d'"' -f2 |
grep metron);
+do
+echo Saving $config_type
+/var/lib/ambari-server/resources/scripts/configs.py -u
"$username" -p "$password" -a get -l ${ambari_address%:*} -n $cluster_name -c
$config_type -f $AMBARI_CONFIG_DIR/${config_type}.json
+done
+echo Done backing up Ambari config...
+else
+echo
[metron] branch master updated: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field (nickwallen via mmiklavc) closes apache/metron#1570
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 657028b METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage
Rule Reason Field (nickwallen via mmiklavc) closes apache/metron#1570
657028b is described below
commit 657028bc7f69fb7b4d54fae22b3c3fcb5d88b61b
Author: nickwallen
AuthorDate: Mon Nov 25 09:37:27 2019 -0700
METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason
Field (nickwallen via mmiklavc) closes apache/metron#1570
---
.../enrichment/threatintel/ThreatTriageConfig.java | 14 +---
.../metron-enrichment-common/README.md | 98 ++
2 files changed, 82 insertions(+), 30 deletions(-)
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
index f363f76..0582a70 100644
---
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
+++
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/enrichment/threatintel/ThreatTriageConfig.java
@@ -20,8 +20,6 @@ package
org.apache.metron.common.configuration.enrichment.threatintel;
import com.google.common.base.Joiner;
import org.apache.metron.common.aggregator.Aggregators;
-import org.apache.metron.stellar.common.StellarPredicateProcessor;
-import org.apache.metron.stellar.common.StellarProcessor;
import java.util.ArrayList;
import java.util.HashMap;
@@ -49,8 +47,6 @@ public class ThreatTriageConfig {
public void setRiskLevelRules(List riskLevelRules) {
List rules = new ArrayList<>();
Set ruleIndex = new HashSet<>();
-StellarPredicateProcessor predicateProcessor = new
StellarPredicateProcessor();
-StellarProcessor processor = new StellarProcessor();
for(RiskLevelRule rule : riskLevelRules) {
if(rule.getRule() == null || rule.getScoreExpression() == null) {
@@ -58,17 +54,9 @@ public class ThreatTriageConfig {
}
if(ruleIndex.contains(rule.getRule())) {
continue;
- }
- else {
+ } else {
ruleIndex.add(rule.getRule());
}
-
- // validate the fields which are expected to be valid Stellar expressions
- predicateProcessor.validate(rule.getRule());
- if(rule.getReason() != null) {
-processor.validate(rule.getReason());
- }
-
rules.add(rule);
}
this.riskLevelRules = rules;
diff --git
a/metron-platform/metron-enrichment/metron-enrichment-common/README.md
b/metron-platform/metron-enrichment/metron-enrichment-common/README.md
index c6e6efc..530be4a 100644
--- a/metron-platform/metron-enrichment/metron-enrichment-common/README.md
+++ b/metron-platform/metron-enrichment/metron-enrichment-common/README.md
@@ -217,26 +217,89 @@ The `triageConfig` field is also a complex field and it
bears some description:
| `riskLevelRules` | This is a list of rules (represented as Stellar
expressions) associated with scores with optional names and comments
| see below|
| `aggregator` | An aggregation function that takes all non-zero scores
representing the matching queries from `riskLevelRules` and aggregates them
into a single score. | `"MAX"`
|
-A risk level rule is of the following format:
-* `name` : The name of the threat triage rule
-* `comment` : A comment describing the rule
-* `rule` : The rule, represented as a Stellar statement
-* `score` : The score attributed to the rule. Can be either numeric or a
Stellar expression. The expression has access to all fields with the message
being triaged.
-* `reason` : Reason the rule tripped. Can be represented as a Stellar statement
-
-An example of a rule is as follows:
+
+ Risk Level Rules
+
+A message is triaged by applying a set of risk scoring rules. These rules are
used to calculate an overall threat score that can be used to prioritize
threats. For each message a rule may either apply and attribute to the overall
risk score or the rule may be ignored. A set of rules might look like the
following.
+
```
-"riskLevelRules" : [
-{
- "name" : "is internal",
- "comment" : "determines if the destination is internal.",
- "rule" : "IN_SUBNET(ip_dst_addr, '192.168.0.0/24')",
- "score" : 10,
- "reason" : "FORMAT('%s is internal', ip_dst_addr)"
-}
-]
+"riskLevelRules" :
[metron] branch master updated: METRON-2285 Batch Profiler Cannot Persist Data Sketches (nickwallen via mmiklavc) closes apache/metron#1564
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 101d427 METRON-2285 Batch Profiler Cannot Persist Data Sketches
(nickwallen via mmiklavc) closes apache/metron#1564
101d427 is described below
commit 101d4275ce09b819b2312407e427469370f69ef8
Author: nickwallen
AuthorDate: Mon Nov 25 09:30:59 2019 -0700
METRON-2285 Batch Profiler Cannot Persist Data Sketches (nickwallen via
mmiklavc) closes apache/metron#1564
---
.../metron/profiler/spark/BatchProfiler.java | 5 +-
.../profiler/spark/ProfileMeasurementAdapter.java | 132 -
.../spark/function/HBaseWriterFunction.java| 10 +-
.../spark/function/ProfileBuilderFunction.java | 12 +-
.../spark/BatchProfilerIntegrationTest.java| 57 ++---
.../spark/function/HBaseWriterFunctionTest.java| 21 ++--
.../spark/function/ProfileBuilderFunctionTest.java | 10 +-
7 files changed, 69 insertions(+), 178 deletions(-)
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
index 43b42be..96e1880 100644
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
+++
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/BatchProfiler.java
@@ -22,6 +22,7 @@ package org.apache.metron.profiler.spark;
import com.google.common.collect.Maps;
import org.apache.metron.common.configuration.profiler.ProfilerConfig;
import org.apache.metron.profiler.MessageRoute;
+import org.apache.metron.profiler.ProfileMeasurement;
import org.apache.metron.profiler.spark.function.GroupByPeriodFunction;
import org.apache.metron.profiler.spark.function.HBaseWriterFunction;
import org.apache.metron.profiler.spark.function.MessageRouterFunction;
@@ -91,9 +92,9 @@ public class BatchProfiler implements Serializable {
LOG.debug("Generated {} message route(s)", routes.cache().count());
// build the profiles
-Dataset measurements = routes
+Dataset measurements = routes
.groupByKey(new GroupByPeriodFunction(profilerProps),
Encoders.STRING())
-.mapGroups(new ProfileBuilderFunction(profilerProps, globals),
Encoders.kryo(ProfileMeasurementAdapter.class));
+.mapGroups(new ProfileBuilderFunction(profilerProps, globals),
Encoders.kryo(ProfileMeasurement.class));
LOG.debug("Produced {} profile measurement(s)",
measurements.cache().count());
// write the profile measurements to HBase
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/ProfileMeasurementAdapter.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/ProfileMeasurementAdapter.java
deleted file mode 100644
index 5da7d04..000
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/ProfileMeasurementAdapter.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- *
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- *
- */
-package org.apache.metron.profiler.spark;
-
-import org.apache.metron.common.utils.SerDeUtils;
-import org.apache.metron.profiler.ProfileMeasurement;
-import org.apache.metron.profiler.ProfilePeriod;
-
-import java.io.Serializable;
-import java.util.concurrent.TimeUnit;
-
-/**
- * An adapter for the {@link ProfileMeasurement} class so that the data
- * can be serialized as required by Spark.
- *
- * The `Encoders.bean(Class)` encoder does not handle serialization of
type `Object` well. This
- * adapter encodes the profile's result as byte[] rather than an Object to
work around this.
- */
-public class ProfileMeasurementAdapter implements Serializable {
-
- /**
- * The name of the profile that this measurement is associated with.
- */
- private String profileName;
-
- /**
- * The name of the entity be
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (69163fa -> 92034de)
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 69163fa Merge remote-tracking branch 'upstream/master' into feature/METRON-2088-support-hdp-3.1 add fca9ebf METRON-2239 Metron Automated backup and restore (mmiklavc) closes apache/metron#1546 new 92034de Merge branch 'master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: Upgrade_steps.md | 40 +++ .../packaging/docker/rpm-docker/SPECS/metron.spec | 1 + metron-platform/metron-common/README.md| 30 ++ .../src/main/scripts/upgrade_helper.sh | 120 + 4 files changed, 191 insertions(+) create mode 100644 Upgrade_steps.md create mode 100755 metron-platform/metron-common/src/main/scripts/upgrade_helper.sh
[metron] 01/01: Merge branch 'master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit 92034de2f6fc5657c43a996ac4784fbe4717a010 Merge: 69163fa fca9ebf Author: Michael Miklavcic AuthorDate: Wed Nov 13 11:37:03 2019 -0700 Merge branch 'master' into feature/METRON-2088-support-hdp-3.1 Upgrade_steps.md | 40 +++ .../packaging/docker/rpm-docker/SPECS/metron.spec | 1 + metron-platform/metron-common/README.md| 30 ++ .../src/main/scripts/upgrade_helper.sh | 120 + 4 files changed, 191 insertions(+)
[metron] branch master updated: METRON-2239 Metron Automated backup and restore (mmiklavc) closes apache/metron#1546
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new fca9ebf METRON-2239 Metron Automated backup and restore (mmiklavc)
closes apache/metron#1546
fca9ebf is described below
commit fca9ebf0dc945bea485be334e4fcfcbe7ed179cf
Author: mmiklavc
AuthorDate: Wed Nov 13 11:36:07 2019 -0700
METRON-2239 Metron Automated backup and restore (mmiklavc) closes
apache/metron#1546
---
Upgrade_steps.md | 40 +++
.../packaging/docker/rpm-docker/SPECS/metron.spec | 1 +
metron-platform/metron-common/README.md| 30 ++
.../src/main/scripts/upgrade_helper.sh | 120 +
4 files changed, 191 insertions(+)
diff --git a/Upgrade_steps.md b/Upgrade_steps.md
new file mode 100644
index 000..7d197b8
--- /dev/null
+++ b/Upgrade_steps.md
@@ -0,0 +1,40 @@
+
+# Upgrade Steps
+General guidance for upgrading Metron
+
+1. Cut-off all inputs to Metron
+1. Monitor processing until all in-flight data is indexed and stored
+1. Stop Metron service
+1. Run the [Metron Upgrade
Helper](./metron-platform/metron-common#metron-upgrade-helper) script in backup
mode - does these 2 tasks
+1. Export zookeeper based metron configs
+1. Export ambari-based metron configs
+1. Delete the metron service via Ambari
+1. Upgrade OS, if applicable
+1. Major Hadoop platform upgrade steps (example. using HDP 2.6 to 3.x) -
https://docs.cloudera.com/HDPDocuments/Ambari-2.7.3.0/bk_ambari-upgrade-major/content/ambari_upgrade_guide.html
+ 1. Update Ambari to latest version
+ 1. Update to HDP 3.1 using existing Ambari/HDP update documentation
(including Solr/ES etc).
+1. Reinstall Metron mpack using "--force" option
+1. Reinstall Metron service
+1. Turn off Metron service
+1. Tweak any configs required to support new Metron version
+ 1. Minimally, you should update `metron.home` in `Ambari -> Metron ->
Configs -> Advanced metron-env -> Metron home` to point to the new Metron home
dir.
+1. Run upgrade helper script in restore mode - does these 2 tasks
+ 1. Re-import zookeeper based metron configs
+ 1. Re-import ambari-based zookeeper configs
+1. Turn on Metron service
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index 673ace5..e4b99ca 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -143,6 +143,7 @@ This package installs the Metron common files %{metron_home}
%{metron_home}/bin/stellar
%{metron_home}/bin/cluster_info.py
%{metron_home}/bin/tgt_renew.py
+%{metron_home}/bin/upgrade_helper.sh
%{metron_home}/config/zookeeper/global.json
%attr(0644,root,root) %{metron_home}/lib/metron-common-%{full_version}-uber.jar
diff --git a/metron-platform/metron-common/README.md
b/metron-platform/metron-common/README.md
index f3082a5..2cee5dd 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -25,6 +25,7 @@ limitations under the License.
* [Topology Errors](topology-errors)
* [Performance Logging](#performance-logging)
* [Metron Debugging](#metron-debugging)
+* [Metron Upgrade Helper](#metron-upgrade-helper)
# Stellar Language
@@ -471,3 +472,32 @@ Options:
-p DIRECTORY, --hdp_home=DIRECTORY
HDP home directory
```
+
+# Metron Upgrade Helper
+
+A bash script is provided to assist in performing backup and restore
operations for Metron Ambari configurations and configurations stored in
Zookeeper.
+
+If your Ambari Server is installed on a separate host from Metron, you may
need to scp the upgrade_helper.sh script to the Ambari host along with the file
`/etc/default/metron`.
+There is an optional argument, `directory_base`, that allows you to specify
where you would like backups to be written to and restored from. Be aware that
while it's optional, the
+default is to write the data to the directory from which you're executing the
script, i.e. `./metron-backup`.
+
+```
+# $METRON_HOME/bin/upgrade_helper.sh -h
+5 args required
+Usage:
+ mode: [backup|restore] - backup will save configs to a directory named
"metron-backup". Restore will take those same configs and restore them to
Ambari.
+ ambari_address: host and port for Ambari server, e.g. "node1:8080"
+ username: Ambari admin username
+ password: Ambari admin user password
+ cluster_name: hadoop cluster name. Can be found in Ambari under "Admin >
Manage Ambari"
+ directory_base: (Optional) root directory location where the backup will be
written to and read from. Default is the executing directory, ".", with backup
data stored to a subdir
[metron] branch master updated: METRON-2293 Fix some inaccuracies in the MaaS README (mmiklavc) closes apache/metron#1536
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 6c064b7 METRON-2293 Fix some inaccuracies in the MaaS README
(mmiklavc) closes apache/metron#1536
6c064b7 is described below
commit 6c064b76e2252776cde64280b9ee15091339cf5a
Author: mmiklavc
AuthorDate: Mon Nov 4 12:43:35 2019 -0700
METRON-2293 Fix some inaccuracies in the MaaS README (mmiklavc) closes
apache/metron#1536
---
metron-analytics/metron-maas-service/README.md | 209 +++--
1 file changed, 199 insertions(+), 10 deletions(-)
diff --git a/metron-analytics/metron-maas-service/README.md
b/metron-analytics/metron-maas-service/README.md
index bd1d76c..93c798a 100644
--- a/metron-analytics/metron-maas-service/README.md
+++ b/metron-analytics/metron-maas-service/README.md
@@ -135,10 +135,9 @@ Let's augment the `squid` proxy sensor to use a model that
will determine if the
## Install Prerequisites and Mock DGA Service
Now let's install some prerequisites:
-* Flask via `yum install python-flask`
-* Jinja2 via `yum install python-jinja2`
-* Squid client via `yum install squid`
-* ES Head plugin via `/usr/share/elasticsearch/bin/plugin install
mobz/elasticsearch-head`
+* Flask via `yum -y install python-flask`
+* Jinja2 via `yum -y install python-jinja2`
+* Squid client via `yum -y install squid`
Start Squid via `service squid start`
@@ -154,13 +153,13 @@ The following presumes that you are a logged in as a user
who has a
home directory in HDFS under `/user/$USER`. If you do not, please create one
and ensure the permissions are set appropriate:
```
-su - hdfs -c "hadoop fs -mkdir /user/$USER"
-su - hdfs -c "hadoop fs -chown $USER:$USER /user/$USER"
+su - hdfs -c "hdfs dfs -mkdir /user/$USER"
+su - hdfs -c "hdfs dfs -chown $USER:$USER /user/$USER"
```
-Or, in the common case for the `metron` user:
+Or, in the common case for the `metron` user (if the user does not already
exist):
```
-su - hdfs -c "hadoop fs -mkdir /user/metron"
-su - hdfs -c "hadoop fs -chown metron:metron /user/metron"
+su - hdfs -c "hdfs dfs -mkdir /user/metron"
+su - hdfs -c "hdfs dfs -chown metron:metron /user/metron"
```
Now let's start MaaS and deploy the Mock DGA Service:
@@ -173,6 +172,10 @@ Now let's start MaaS and deploy the Mock DGA Service:
## Adjust Configurations for Squid to Call Model
Now that we have a deployed model, let's adjust the configurations for the
Squid topology to annotate the messages with the output of the model.
+* First pull down the latest configuration from Zookeeper
+```
+$METRON_HOME/bin/zk_load_configs.sh -m PULL -o ${METRON_HOME}/config/zookeeper
-z $ZOOKEEPER -f
+```
* Edit the squid parser configuration at
`$METRON_HOME/config/zookeeper/parsers/squid.json` in your favorite text editor
and add a new FieldTransformation to indicate a threat alert based on the model
(note the addition of `is_malicious` and `is_alert`):
```
{
@@ -217,8 +220,185 @@ Now that we have a deployed model, let's adjust the
configurations for the Squid
}
}
```
+* Setup an indexing configuration here
`${METRON_HOME}/config/zookeeper/indexing/squid.json` with the following
contents:
+```
+{
+"hdfs" : {
+"index": "squid",
+"batchSize": 5,
+"enabled" : true
+},
+"elasticsearch" : {
+"index": "squid",
+"batchSize": 5,
+"enabled" : true
+},
+"solr" : {
+"index": "squid",
+"batchSize": 5,
+"enabled" : true
+}
+}
+```
* Upload new configs via `$METRON_HOME/bin/zk_load_configs.sh --mode PUSH -i
$METRON_HOME/config/zookeeper -z node1:2181`
* Make the Squid topic in kafka via
`/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181
--create --topic squid --partitions 1 --replication-factor 1`
+* Setup your squid indexing template for Elasticsearch (if using Elasticsearch)
+```
+curl -XPUT 'http://node1:9200/_template/squid_index' -d '
+{
+ "template": "squid_index*",
+ "mappings": {
+"squid_doc": {
+ "dynamic_templates": [
+ {
+"geo_location_point": {
+ "match": "enrichments:geo:*:location_point",
+ "match_mapping_type": "*",
+ "mapping": {
+"type": "geo_point"
+ }
+}
+ },
+ {
+"geo_country": {
+ "match": "enrichments:geo:*:country",
+ &q
[metron] 01/01: Merge branch 'master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit b1091fefcbe52a5c9b59a191ee350654072d0bdf Merge: 018e90f eb1dda6 Author: Michael Miklavcic AuthorDate: Mon Nov 4 11:22:56 2019 -0700 Merge branch 'master' into feature/METRON-2088-support-hdp-3.1 .../src/app/alerts/alerts-list/alerts-list.component.ts | 10 +- .../auto-polling/auto-polling.service.spec.ts | 17 ++--- .../alerts-list/auto-polling/auto-polling.service.ts| 3 ++- .../alerts-list/table-view/table-view.component.html| 7 +++ .../alerts-list/table-view/table-view.component.scss| 17 + 5 files changed, 41 insertions(+), 13 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (018e90f -> b1091fe)
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 018e90f METRON-2305 Unable to Add Enrichment Coprocessor with Kerberos (MohanDV via nickwallen) closes apache/metron#1549 add d0d64d4 METRON-2291 [UI] Fixing and rephrasing warning messages on Alerts UI (tiborm via sardell) closes apache/metron#1535 add 581a43f METRON-2294 [UI] Fixing Stale mode issue in Alert UI Manual Query Mode (subhashjha35 via sardell) closes apache/metron#1540 add df6e181 METRON-2295 [UI] Displaying "No Data" message in the Alerts UI screen (subhashjha35 via sardell) closes apache/metron#1543 add eb1dda6 METRON-2302 [UI] Change the default polling interval for Alerts UI to longer time (tiborm via sardell) closes apache/metron#1547 new b1091fe Merge branch 'master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/app/alerts/alerts-list/alerts-list.component.ts | 10 +- .../auto-polling/auto-polling.service.spec.ts | 17 ++--- .../alerts-list/auto-polling/auto-polling.service.ts| 3 ++- .../alerts-list/table-view/table-view.component.html| 7 +++ .../alerts-list/table-view/table-view.component.scss| 17 + 5 files changed, 41 insertions(+), 13 deletions(-)
[metron] branch asf-site updated: METRON-2300 Fix Brad Kolarov's Apache ID
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/asf-site by this push:
new dd58f23 METRON-2300 Fix Brad Kolarov's Apache ID
dd58f23 is described below
commit dd58f23fca2acadbf951c78797be184162f451bc
Author: Michael Miklavcic
AuthorDate: Mon Oct 28 12:30:25 2019 -0600
METRON-2300 Fix Brad Kolarov's Apache ID
---
community/index.html |2 +-
css/screen.css | 1801 ++
css/screen.css.map | 22 +
feed.xml | 12 +-
4 files changed, 543 insertions(+), 1294 deletions(-)
diff --git a/community/index.html b/community/index.html
index cd3f4e7..052a21e 100644
--- a/community/index.html
+++ b/community/index.html
@@ -197,7 +197,7 @@
Mark Bittmann mbittmann PMC
- Brad Kolarov billie PMC
+ Brad Kolarov bjkolly PMC
Dave Hirko dbhirko PMC
diff --git a/css/screen.css b/css/screen.css
index c940f33..ced7f60 100644
--- a/css/screen.css
+++ b/css/screen.css
@@ -1,1411 +1,638 @@
@charset "UTF-8";
/*! normalize.css v3.0.2 | MIT License | git.io/normalize */
-html {
- font-family: sans-serif;
- -ms-text-size-adjust: 100%;
- -webkit-text-size-adjust: 100%; }
+html { font-family: sans-serif; -ms-text-size-adjust: 100%;
-webkit-text-size-adjust: 100%; }
-body {
- margin: 0; }
+body { margin: 0; }
-article, aside, details, figcaption, figure, footer, header, hgroup, main,
menu, nav, section, summary {
- display: block; }
+article, aside, details, figcaption, figure, footer, header, hgroup, main,
menu, nav, section, summary { display: block; }
-audio, canvas, progress, video {
- display: inline-block;
- vertical-align: baseline; }
+audio, canvas, progress, video { display: inline-block; vertical-align:
baseline; }
-audio:not([controls]) {
- display: none;
- height: 0; }
+audio:not([controls]) { display: none; height: 0; }
-[hidden], template {
- display: none; }
+[hidden], template { display: none; }
-a {
- background-color: transparent; }
+a { background-color: transparent; }
-a:active, a:hover {
- outline: 0; }
+a:active, a:hover { outline: 0; }
-abbr[title] {
- border-bottom: 1px dotted; }
+abbr[title] { border-bottom: 1px dotted; }
-b, strong {
- font-weight: 700; }
+b, strong { font-weight: 700; }
-dfn {
- font-style: italic; }
+dfn { font-style: italic; }
-h1 {
- font-size: 2em;
- margin: .67em 0; }
+h1 { font-size: 2em; margin: .67em 0; }
-mark {
- background: #ff0;
- color: #000; }
+mark { background: #ff0; color: #000; }
-small {
- font-size: 80%; }
+small { font-size: 80%; }
-sub, sup {
- font-size: 75%;
- line-height: 0;
- position: relative;
- vertical-align: baseline; }
+sub, sup { font-size: 75%; line-height: 0; position: relative; vertical-align:
baseline; }
-sup {
- top: -.5em; }
+sup { top: -.5em; }
-sub {
- bottom: -.25em; }
+sub { bottom: -.25em; }
-img {
- border: 0; }
+img { border: 0; }
-svg:not(:root) {
- overflow: hidden; }
+svg:not(:root) { overflow: hidden; }
-figure {
- margin: 1em 40px; }
+figure { margin: 1em 40px; }
-hr {
- -moz-box-sizing: content-box;
- box-sizing: content-box;
- height: 0; }
+hr { -moz-box-sizing: content-box; box-sizing: content-box; height: 0; }
-pre {
- overflow: auto; }
+pre { overflow: auto; }
-code, kbd, pre, samp {
- font-family: monospace,monospace;
- font-size: 1em; }
+code, kbd, pre, samp { font-family: monospace,monospace; font-size: 1em; }
-button, input, optgroup, select, textarea {
- color: inherit;
- font: inherit;
- margin: 0; }
+button, input, optgroup, select, textarea { color: inherit; font: inherit;
margin: 0; }
-button {
- overflow: visible; }
+button { overflow: visible; }
-button, select {
- text-transform: none; }
+button, select { text-transform: none; }
-button, html input[type=button], input[type=reset], input[type=submit] {
- -webkit-appearance: button;
- cursor: pointer; }
+button, html input[type=button], input[type=reset], input[type=submit] {
-webkit-appearance: button; cursor: pointer; }
-button[disabled], html input[disabled] {
- cursor: default; }
+button[disabled], html input[disabled] { cursor: default; }
-button::-moz-focus-inner, input::-moz-focus-inner {
- border: 0;
- padding: 0; }
+button::-moz-focus-inner, input::-moz-focus-inner { border: 0; padding: 0; }
-input {
- line-height: normal; }
+input { line-height: normal; }
-input[type=checkbox], input[type=radio] {
- box-sizing: border-box;
- padding: 0; }
+input[type=checkbox], input[type=radio] { box-sizing: border-box; padding: 0; }
-input[type=number]::-webkit-inner-spin-button,
input[type=number]::-webkit-outer-spin-button {
- height: auto; }
+input[type=number]::-webkit-inner-spin-button,
input[type=number]::-webkit-outer-spin-b
[metron] branch master updated: METRON-2300 Fix Brad Kolarov's Apache ID (billierinaldi via mmiklavc) closes apache/metron#1541
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new e645ffe METRON-2300 Fix Brad Kolarov's Apache ID (billierinaldi via mmiklavc) closes apache/metron#1541 e645ffe is described below commit e645ffe34a98617b91261738477358c760bb60cf Author: billierinaldi AuthorDate: Mon Oct 28 09:03:24 2019 -0600 METRON-2300 Fix Brad Kolarov's Apache ID (billierinaldi via mmiklavc) closes apache/metron#1541 --- site/community/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/community/index.md b/site/community/index.md index b222ba3..c11dcb8 100644 --- a/site/community/index.md +++ b/site/community/index.md @@ -60,7 +60,7 @@ title: Apache Metron Community Mark Bittmann mbittmann PMC - Brad Kolarov billie PMC + Brad Kolarov bjkolly PMC Dave Hirko dbhirko PMC
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (0a619e0 -> 275a6d8)
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a change to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git. from 0a619e0 METRON-2297 Enrichment Topology Unable to Load Geo IP Data from HDFS (nickwallen) closes apache/metron#1539 add 50b5c50 METRON-2278 "Metron on CentOS 6" Documentation is outdated (subhashjha35 via sardell) closes apache/metron#1530 add bb9a244 METRON-2259 [UI] Hide Resolved and Hide Dismissed toggles not works when filtering is in manual mode (tiborm via sardell) closes apache/metron#1532 add 67b4d0b METRON-2280 PCAP queries no longer work (mmiklavc) closes apache/metron#1537 new 275a6d8 Merge branch 'master' into feature/METRON-2088-support-hdp-3.1 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: metron-deployment/development/centos6/README.md| 8 +++--- .../show-hide-alert-entries.component.spec.ts | 32 -- .../show-hide/show-hide-alert-entries.component.ts | 16 +-- .../show-hide/show-hide.service.spec.ts| 12 +++- .../configure-rows/show-hide/show-hide.service.ts | 6 +++- .../src/app/shared/switch/switch.component.html| 4 +-- .../src/app/shared/switch/switch.component.scss| 9 ++ .../src/app/shared/switch/switch.component.ts | 1 + metron-interface/metron-rest/pom.xml | 3 +- metron-platform/metron-pcap-backend/pom.xml| 3 +- 10 files changed, 78 insertions(+), 16 deletions(-)
[metron] 01/01: Merge branch 'master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git
commit 275a6d837c329a0a7b7d1cfa70bc5e8f323a3d6a
Merge: 0a619e0 67b4d0b
Author: Michael Miklavcic
AuthorDate: Wed Oct 23 15:13:50 2019 -0600
Merge branch 'master' into feature/METRON-2088-support-hdp-3.1
metron-deployment/development/centos6/README.md| 8 +++---
.../show-hide-alert-entries.component.spec.ts | 32 --
.../show-hide/show-hide-alert-entries.component.ts | 16 +--
.../show-hide/show-hide.service.spec.ts| 12 +++-
.../configure-rows/show-hide/show-hide.service.ts | 6 +++-
.../src/app/shared/switch/switch.component.html| 4 +--
.../src/app/shared/switch/switch.component.scss| 9 ++
.../src/app/shared/switch/switch.component.ts | 1 +
metron-interface/metron-rest/pom.xml | 3 +-
metron-platform/metron-pcap-backend/pom.xml| 3 +-
10 files changed, 78 insertions(+), 16 deletions(-)
diff --cc metron-platform/metron-pcap-backend/pom.xml
index 361ebaf,719292c..cf49ecf
--- a/metron-platform/metron-pcap-backend/pom.xml
+++ b/metron-platform/metron-pcap-backend/pom.xml
@@@ -62,19 -50,21 +62,20 @@@
org.apache.metron
metron-common
${project.parent.version}
-
-
-org.apache.curator
-curator-client
-
-
+
org.apache.metron
stellar-common
${project.parent.version}
- provided
+org.apache.metron
+metron-common-storm
+${project.parent.version}
+
+
junit
junit
${global_junit_version}
[metron] branch master updated: METRON-2280 PCAP queries no longer work (mmiklavc) closes apache/metron#1537
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 67b4d0b METRON-2280 PCAP queries no longer work (mmiklavc) closes
apache/metron#1537
67b4d0b is described below
commit 67b4d0b2c84f7f370890c1caf639459a4c40b37a
Author: mmiklavc
AuthorDate: Wed Oct 23 15:12:47 2019 -0600
METRON-2280 PCAP queries no longer work (mmiklavc) closes apache/metron#1537
---
metron-interface/metron-rest/pom.xml| 3 ++-
metron-platform/metron-pcap-backend/pom.xml | 3 ++-
2 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/metron-interface/metron-rest/pom.xml
b/metron-interface/metron-rest/pom.xml
index 66dd99e..b4ca507 100644
--- a/metron-interface/metron-rest/pom.xml
+++ b/metron-interface/metron-rest/pom.xml
@@ -192,10 +192,11 @@
+
org.apache.metron
stellar-common
${project.parent.version}
-provided
org.apache.hadoop
diff --git a/metron-platform/metron-pcap-backend/pom.xml
b/metron-platform/metron-pcap-backend/pom.xml
index 5042b70..719292c 100644
--- a/metron-platform/metron-pcap-backend/pom.xml
+++ b/metron-platform/metron-pcap-backend/pom.xml
@@ -58,10 +58,11 @@
+
org.apache.metron
stellar-common
${project.parent.version}
-provided
junit
[metron] 01/01: Merge branch 'master' into feature/METRON-2088-support-hdp-3.1
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch feature/METRON-2088-support-hdp-3.1 in repository https://gitbox.apache.org/repos/asf/metron.git commit 903e6cd4ad579ecbf3ae9f22bad15f03899d486f Merge: 939c4ea 895fe49 Author: Michael Miklavcic AuthorDate: Fri Oct 4 13:22:36 2019 -0600 Merge branch 'master' into feature/METRON-2088-support-hdp-3.1 .../metron-alerts/cypress/fixtures/search-1.1.json | 102 .../metron-alerts/cypress/fixtures/search-1.2.json | 102 .../search/auto-polling.feature.spec.js| 98 .../alerts/alerts-list/alerts-list.component.html | 52 ++- .../alerts/alerts-list/alerts-list.component.scss | 21 - .../alerts-list/alerts-list.component.spec.ts | 513 +--- .../alerts/alerts-list/alerts-list.component.ts| 249 +- .../app/alerts/alerts-list/alerts-list.module.ts | 12 +- .../auto-polling/auto-polling.component.html | 17 + .../auto-polling/auto-polling.component.scss | 73 +++ .../auto-polling/auto-polling.component.spec.ts| 80 .../auto-polling/auto-polling.component.ts}| 29 +- .../auto-polling/auto-polling.service.spec.ts | 519 + .../auto-polling/auto-polling.service.ts | 184 .../app/alerts/alerts-list/query-builder.spec.ts | 124 +++-- .../src/app/alerts/alerts-list/query-builder.ts| 74 +-- .../alerts-list/tree-view/tree-view.component.ts | 1 - .../configure-rows/configure-rows.component.html | 34 +- .../configure-rows/configure-rows.component.ts | 76 ++- .../show-hide-alert-entries.component.spec.ts | 23 +- .../show-hide/show-hide-alert-entries.component.ts | 20 +- .../metron-alerts/src/app/model/search-response.ts | 2 +- .../metron-alerts/src/app/model/table-metadata.ts | 10 +- .../app/service/elasticsearch-localstorage-impl.ts | 6 +- .../src/app/service/search.service.spec.ts | 78 .../src/app/service/search.service.ts | 26 +- .../shared/directives/alert-search.directive.ts| 118 +++-- .../modal-loading-indicator.component.html | 25 + .../modal-loading-indicator.component.scss}| 27 +- .../modal-loading-indicator.component.spec.ts} | 33 +- .../modal-loading-indicator.component.ts | 29 ++ .../metron-alerts/src/app/utils/constants.ts | 2 +- .../metron-alerts/src/app/utils/httpUtil.ts| 20 +- .../src/main/scripts/flatfile_loader.sh| 7 +- .../src/main/scripts/flatfile_summarizer.sh| 2 + 35 files changed, 2283 insertions(+), 505 deletions(-)
[metron] branch feature/METRON-2088-support-hdp-3.1 updated (939c4ea -> 903e6cd)
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a change to branch feature/METRON-2088-support-hdp-3.1
in repository https://gitbox.apache.org/repos/asf/metron.git.
from 939c4ea METRON-2265 Update Kerberos settings (merrimanr via
nickwallen) closes apache/metron#1519
add d693e28 METRON-2190 [UI] Alerts UI: Indicating loading and preventing
parallel requests (tiborm via sardell) closes apache/metron#1514
add 03e0977 METRON-2272 [UI] Performance: Switching manual filtering on
and off multiple times leads slow typing (ruffle1986 via sardell) closes
apache/metron#1524
add 895fe49 METRON-2274 Flatfile loader and summarizer mapreduce mode
broken (mmiklavc) closes apache/metron#1525
new 903e6cd Merge branch 'master' into feature/METRON-2088-support-hdp-3.1
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../metron-alerts/cypress/fixtures/search-1.1.json | 102
.../metron-alerts/cypress/fixtures/search-1.2.json | 102
.../search/auto-polling.feature.spec.js| 98
.../alerts/alerts-list/alerts-list.component.html | 52 ++-
.../alerts/alerts-list/alerts-list.component.scss | 21 -
.../alerts-list/alerts-list.component.spec.ts | 513 +---
.../alerts/alerts-list/alerts-list.component.ts| 249 +-
.../app/alerts/alerts-list/alerts-list.module.ts | 12 +-
.../auto-polling/auto-polling.component.html} | 17 +-
.../auto-polling/auto-polling.component.scss} | 79 ++--
.../auto-polling/auto-polling.component.spec.ts| 80
.../auto-polling/auto-polling.component.ts}| 28 +-
.../auto-polling/auto-polling.service.spec.ts | 519 +
.../auto-polling/auto-polling.service.ts | 184
.../app/alerts/alerts-list/query-builder.spec.ts | 124 +++--
.../src/app/alerts/alerts-list/query-builder.ts| 74 +--
.../alerts-list/tree-view/tree-view.component.ts | 1 -
.../configure-rows/configure-rows.component.html | 34 +-
.../configure-rows/configure-rows.component.ts | 76 ++-
.../show-hide-alert-entries.component.spec.ts | 23 +-
.../show-hide/show-hide-alert-entries.component.ts | 20 +-
.../metron-alerts/src/app/model/search-response.ts | 2 +-
.../metron-alerts/src/app/model/table-metadata.ts | 10 +-
.../app/service/elasticsearch-localstorage-impl.ts | 6 +-
.../src/app/service/search.service.spec.ts | 78
.../src/app/service/search.service.ts | 26 +-
.../shared/directives/alert-search.directive.ts| 118 +++--
.../modal-loading-indicator.component.html}| 27 +-
.../modal-loading-indicator.component.scss | 25 +-
.../modal-loading-indicator.component.spec.ts} | 14 +-
.../modal-loading-indicator.component.ts | 29 ++
.../metron-alerts/src/app/utils/constants.ts | 2 +-
.../metron-alerts/src/app/utils/httpUtil.ts| 20 +-
.../metron-alerts/src/environments/environment.js | 27 --
.../src/environments/environment.prod.js | 21 -
.../src/main/scripts/flatfile_loader.sh| 7 +-
.../src/main/scripts/flatfile_summarizer.sh| 2 +
37 files changed, 2200 insertions(+), 622 deletions(-)
create mode 100644
metron-interface/metron-alerts/cypress/fixtures/search-1.1.json
create mode 100644
metron-interface/metron-alerts/cypress/fixtures/search-1.2.json
create mode 100644
metron-interface/metron-alerts/cypress/integration/search/auto-polling.feature.spec.js
mode change 100644 => 100755
metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
mode change 100644 => 100755
metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
mode change 100644 => 100755
metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.module.ts
copy
metron-interface/{metron-config/src/app/shared/metron-modal/metron-modal.component.html
=>
metron-alerts/src/app/alerts/alerts-list/auto-polling/auto-polling.component.html}
(60%)
mode change 100644 => 100755
copy
metron-interface/metron-alerts/src/app/{pcap/pcap-filters/pcap-filters.component.scss
=> alerts/alerts-list/auto-polling/auto-polling.component.scss} (53%)
create mode 100644
metron-interface/metron-alerts/src/app/alerts/alerts-list/auto-polling/auto-polling.component.spec.ts
copy
metron-interface/{metron-rest-client/src/main/java/org/apache/metron/rest/model/PcapResponse.java
=>
metron-alerts/src/app/alerts/alerts-list/auto-polling/auto-polling.component.ts}
(63%)
mode change 100644 => 100755
create mode 100644
metron-interface/metron-alerts/src/app/alerts/alerts-list/auto-polling/auto-polling.service.spec.
[metron] branch master updated: METRON-2274 Flatfile loader and summarizer mapreduce mode broken (mmiklavc) closes apache/metron#1525
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 895fe49 METRON-2274 Flatfile loader and summarizer mapreduce mode
broken (mmiklavc) closes apache/metron#1525
895fe49 is described below
commit 895fe493f38fd85abd3e0f14df507293b977ccfe
Author: mmiklavc
AuthorDate: Fri Oct 4 13:15:30 2019 -0600
METRON-2274 Flatfile loader and summarizer mapreduce mode broken (mmiklavc)
closes apache/metron#1525
---
.../metron-data-management/src/main/scripts/flatfile_loader.sh | 7 +--
.../metron-data-management/src/main/scripts/flatfile_summarizer.sh | 2 ++
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git
a/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
b/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
index 5453bf4..8baa6c7 100755
--- a/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
+++ b/metron-platform/metron-data-management/src/main/scripts/flatfile_loader.sh
@@ -36,8 +36,11 @@ export HBASE_CONF=${HBASE_CONF:-/etc/hbase/conf}
export HADOOP_OPTS="$HADOOP_OPTS $METRON_JVMFLAGS"
if [ $(which hadoop) ]
then
- export
HADOOP_CLASSPATH="$METRON_HOME/lib/$DM_JAR:$HBASE_CONF:$METRON_HOME/lib/$STELLAR_JAR"
- hadoop jar $METRON_HOME/lib/$DM_JAR $CLASSNAME "$@"
+ # libjars needed to pass the stellar dep to the map and reduce jvm tasks
+ LIBJARS="$METRON_HOME/lib/$STELLAR_JAR,$LIBJARS"
+ # hadoop classpath used in the local jvm for the client application
+ export HADOOP_CLASSPATH="$HBASE_CONF:$METRON_HOME/lib/$STELLAR_JAR"
+ hadoop jar $METRON_HOME/lib/$DM_JAR $CLASSNAME -libjars ${LIBJARS} "$@"
else
echo "Warning: Metron cannot find the hadoop client on this node. This
means that loading via Map Reduce will NOT function."
CP=$METRON_HOME/lib/$DM_JAR:$HBASE_CONF:$METRON_HOME/lib/$STELLAR_JAR
diff --git
a/metron-platform/metron-data-management/src/main/scripts/flatfile_summarizer.sh
b/metron-platform/metron-data-management/src/main/scripts/flatfile_summarizer.sh
index 1279ecc..aa94ebb 100755
---
a/metron-platform/metron-data-management/src/main/scripts/flatfile_summarizer.sh
+++
b/metron-platform/metron-data-management/src/main/scripts/flatfile_summarizer.sh
@@ -43,6 +43,8 @@ then
fi
done
export HADOOP_CLASSPATH
+ # need this bc Stellar is no longer in the uber jar deps for
metron-data-management
+ LIBJARS="$METRON_HOME/lib/$STELLAR_JAR,$LIBJARS"
hadoop jar $METRON_HOME/lib/$DM_JAR $CLASSNAME -libjars ${LIBJARS} "$@"
else
echo "Warning: Metron cannot find the hadoop client on this node. This
means that loading via Map Reduce will NOT function."
[metron] branch master updated: METRON-2235 Increase server startup timeout (tigerquoll via mmiklavc) closes apache/metron#1496
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 3a666d5 METRON-2235 Increase server startup timeout (tigerquoll via
mmiklavc) closes apache/metron#1496
3a666d5 is described below
commit 3a666d5d8d6489882bcc32ded5f53bb5225c3ab1
Author: tigerquoll
AuthorDate: Tue Sep 24 13:09:50 2019 -0600
METRON-2235 Increase server startup timeout (tigerquoll via mmiklavc)
closes apache/metron#1496
---
metron-deployment/ansible/roles/ambari_master/tasks/ambari.yml | 6 ++
1 file changed, 6 insertions(+)
diff --git a/metron-deployment/ansible/roles/ambari_master/tasks/ambari.yml
b/metron-deployment/ansible/roles/ambari_master/tasks/ambari.yml
index ba55f30..0395f36 100644
--- a/metron-deployment/ansible/roles/ambari_master/tasks/ambari.yml
+++ b/metron-deployment/ansible/roles/ambari_master/tasks/ambari.yml
@@ -30,6 +30,12 @@
replace: " -Xmx{{ ambari_server_mem }}m "
backup: no
+- name: Allow 150 seconds of startup time for ambari server
+ lineinfile:
+path: /etc/ambari-server/conf/ambari.properties
+regexp: 'server\.startup\.web\.timeout='
+line: 'server.startup.web.timeout=150'
+
- name: Setup Ambari Server
shell: ambari-server setup -s && touch /etc/ambari-server/configured
creates=/etc/ambari-server/configured
register: ambari_server_setup
[metron] branch master updated: METRON-2217 Migrate current HBase client from HTableInterface to Table (mmiklavc) closes apache/metron#1483
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new c402e64 METRON-2217 Migrate current HBase client from HTableInterface
to Table (mmiklavc) closes apache/metron#1483
c402e64 is described below
commit c402e6442d8d699eef5aa5464cd91347160b082c
Author: mmiklavc
AuthorDate: Wed Sep 4 14:29:46 2019 -0600
METRON-2217 Migrate current HBase client from HTableInterface to Table
(mmiklavc) closes apache/metron#1483
---
.../profiler/client/HBaseProfilerClient.java | 34 +++-
.../metron/profiler/client/stellar/GetProfile.java | 61 --
.../profiler/client/stellar/VerboseProfile.java| 57
.../profiler/client/HBaseProfilerClientTest.java | 30 ++-
.../metron/profiler/client/ProfileWriter.java | 25 +
.../profiler/client/stellar/GetProfileTest.java| 35 ++---
.../client/stellar/VerboseProfileTest.java | 50 +-
.../metron/profiler/spark/BatchProfilerConfig.java | 11 ++--
.../org/apache/metron/rest/config/IndexConfig.java | 2 +-
.../metron/rest/user/UserSettingsClient.java | 8 +--
.../apache/metron/rest/config/HBaseConfigTest.java | 8 ---
.../org/apache/metron/rest/config/TestConfig.java | 4 +-
.../metron/rest/user/UserSettingsClientTest.java | 35 +++--
.../metron/dataloads/hbase/mr/PrunerMapper.java| 3 +-
.../nonbulk/flatfile/HBaseExtractorState.java | 8 +--
.../nonbulk/flatfile/importer/LocalImporter.java | 12 +++--
.../dataloads/nonbulk/taxii/TaxiiHandler.java | 16 +++---
.../mr/LeastRecentlyUsedPrunerIntegrationTest.java | 22
...pleEnrichmentFlatFileLoaderIntegrationTest.java | 35 ++---
.../nonbulk/taxii/TaxiiIntegrationTest.java| 4 +-
.../enrichment/adapters/cif/CIFHbaseAdapter.java | 13 ++---
.../adapters/simplehbase/SimpleHBaseAdapter.java | 2 +
.../adapters/threatintel/ThreatIntelAdapter.java | 2 +
.../enrichment/converter/EnrichmentHelper.java | 7 ++-
.../metron/enrichment/lookup/EnrichmentLookup.java | 18 +++
.../lookup/accesstracker/AccessTrackerUtil.java| 4 +-
.../accesstracker/PersistentAccessTracker.java | 8 +--
.../PersistentBloomTrackerCreator.java | 9 ++--
.../stellar/SimpleHBaseEnrichmentFunctions.java| 20 +++
.../metron/enrichment/utils/EnrichmentUtils.java | 6 +--
.../threatintel/ThreatIntelAdapterTest.java| 4 +-
.../EnrichmentCoprocessorIntegrationTest.java | 11 ++--
.../coprocessor/EnrichmentCoprocessorTest.java | 5 +-
.../org/apache/metron/hbase/HTableProvider.java| 46 ++--
.../org/apache/metron/hbase/TableProvider.java | 8 +--
.../apache/metron/hbase/client/HBaseClient.java| 11 ++--
.../metron/hbase/client/HBaseClientTest.java | 37 +++--
.../metron/hbase/mock/MockHBaseTableProvider.java | 18 +++
.../org/apache/metron/hbase/mock/MockHTable.java | 35 -
.../org/apache/metron/indexing/dao/HBaseDao.java | 6 +--
.../writer/hbase/SimpleHbaseEnrichmentWriter.java | 12 ++---
41 files changed, 372 insertions(+), 370 deletions(-)
diff --git
a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
index 2e537da..f4bff2b 100644
---
a/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
+++
b/metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/HBaseProfilerClient.java
@@ -20,22 +20,22 @@
package org.apache.metron.profiler.client;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Optional;
+import java.util.concurrent.TimeUnit;
+import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.client.Get;
-import org.apache.hadoop.hbase.client.HTableInterface;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.metron.common.utils.SerDeUtils;
+import org.apache.metron.hbase.TableProvider;
import org.apache.metron.profiler.ProfileMeasurement;
import org.apache.metron.profiler.ProfilePeriod;
import org.apache.metron.profiler.hbase.ColumnBuilder;
import org.apache.metron.profiler.hbase.RowKeyBuilder;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Optional;
-import java.util.concurrent.TimeUnit;
-
/**
* The default implementation of a ProfilerClient that fetches profile data
persisted in HBase.
*/
@@ -44,7 +44,7 @@ public class HBaseProfilerClient implements ProfilerClient {
/**
* Used to access the
[metron] branch master updated: METRON-2201 The description for the IS_IP method default behavior needs to corrected as per implementation (MohanDV via mmiklavc) closes apache/metron#1474
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 3b96474 METRON-2201 The description for the IS_IP method default
behavior needs to corrected as per implementation (MohanDV via mmiklavc) closes
apache/metron#1474
3b96474 is described below
commit 3b96474ee3b82e9d254b466d5997321b90a6f8c5
Author: MohanDV
AuthorDate: Wed Sep 4 11:56:31 2019 -0600
METRON-2201 The description for the IS_IP method default behavior needs to
corrected as per implementation (MohanDV via mmiklavc) closes apache/metron#1474
---
.../org/apache/metron/common/field/validation/network/IPValidation.java | 2 +-
metron-stellar/stellar-common/README.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/network/IPValidation.java
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/network/IPValidation.java
index 585b979..35ec4ef 100644
---
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/network/IPValidation.java
+++
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/network/IPValidation.java
@@ -33,7 +33,7 @@ public class IPValidation implements FieldValidation,
Predicate> {
, description = "Determine if an string is an IP or not."
, params = {
"ip - An object which we wish to test is an ip"
- ,"type (optional) - Object of string or collection type (e.g.
list) one of IPV4 or IPV6 or both. The default is IPV4."
+ ,"type (optional) - Object of string or collection type (e.g.
list) one of IPV4 or IPV6 or both. The default is both IPV4 and IPV6."
}
, returns = "True if the string is an IP and false otherwise.")
public static class IS_IP extends Predicate2StellarFunction {
diff --git a/metron-stellar/stellar-common/README.md
b/metron-stellar/stellar-common/README.md
index 9f1634b..3031228 100644
--- a/metron-stellar/stellar-common/README.md
+++ b/metron-stellar/stellar-common/README.md
@@ -686,7 +686,7 @@ Where:
* Description: Determine if an string is an IP or not.
* Input:
* ip - An object which we wish to test is an ip
-* type (optional) - Object of string or collection type (e.g. list) one of
IPV4 or IPV6 or both. The default is IPV4.
+* type (optional) - Object of string or collection type (e.g. list) one of
IPV4 or IPV6 or both. The default is both IPV4 and IPV6.
* Returns: True if the string is an IP and false otherwise.
### `IS_NAN`
[metron] branch master updated: METRON-2227 Increase Kafka test harness timeout (tigerquoll via mmiklavc) closes apache/metron#1493
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 67fa5a4 METRON-2227 Increase Kafka test harness timeout (tigerquoll
via mmiklavc) closes apache/metron#1493
67fa5a4 is described below
commit 67fa5a403b01d0f7c8607c06e63f9d06f8b8cbc1
Author: tigerquoll
AuthorDate: Wed Sep 4 11:47:04 2019 -0600
METRON-2227 Increase Kafka test harness timeout (tigerquoll via mmiklavc)
closes apache/metron#1493
---
.../integration/components/KafkaComponent.java | 22 +-
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java
index 08910be..0fa414b 100644
---
a/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java
+++
b/metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java
@@ -65,6 +65,10 @@ import org.slf4j.LoggerFactory;
public class KafkaComponent implements InMemoryComponent {
protected static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ public static final long KAFKA_PROPAGATE_TIMEOUT_MS = 1l;
+ public static final int ZK_SESSION_TIMEOUT_MS = 3;
+ public static final int ZK_CONNECTION_TIMEOUT_MS = 3;
+ public static final int KAFKA_ZOOKEEPER_TIMEOUT_MS = 100;
public static class Topic {
public int numPartitions;
@@ -159,11 +163,11 @@ public class KafkaComponent implements InMemoryComponent {
// setup Zookeeper
zookeeperConnectString =
topologyProperties.getProperty(ZKServerComponent.ZOOKEEPER_PROPERTY);
-zkClient = new ZkClient(zookeeperConnectString, 3, 3,
ZKStringSerializer$.MODULE$);
+zkClient = new ZkClient(zookeeperConnectString, ZK_SESSION_TIMEOUT_MS,
ZK_CONNECTION_TIMEOUT_MS, ZKStringSerializer$.MODULE$);
// setup Broker
Properties props = TestUtilsWrapper.createBrokerConfig(0,
zookeeperConnectString, brokerPort);
-props.setProperty("zookeeper.connection.timeout.ms","100");
+props.setProperty("zookeeper.connection.timeout.ms",
Integer.toString(KAFKA_ZOOKEEPER_TIMEOUT_MS));
KafkaConfig config = new KafkaConfig(props);
Time mock = new MockTime();
kafkaServer = TestUtils.createServer(config, mock);
@@ -175,7 +179,7 @@ public class KafkaComponent implements InMemoryComponent {
for(Topic topic : getTopics()) {
try {
-createTopic(topic.name, topic.numPartitions, true);
+createTopic(topic.name, topic.numPartitions,
KAFKA_PROPAGATE_TIMEOUT_MS);
} catch (InterruptedException e) {
throw new RuntimeException("Unable to create topic", e);
}
@@ -288,26 +292,26 @@ public class KafkaComponent implements InMemoryComponent {
}
public void createTopic(String name) throws InterruptedException {
-createTopic(name, 1, true);
+createTopic(name, 1, KAFKA_PROPAGATE_TIMEOUT_MS);
}
- public void waitUntilMetadataIsPropagated(String topic, int numPartitions) {
+ public void waitUntilMetadataIsPropagated(String topic, int numPartitions,
long timeOutMS) {
List servers = new ArrayList<>();
servers.add(kafkaServer);
for(int part = 0;part < numPartitions;++part) {
-
TestUtils.waitUntilMetadataIsPropagated(scala.collection.JavaConversions.asScalaBuffer(servers),
topic, part, 5000);
+
TestUtils.waitUntilMetadataIsPropagated(scala.collection.JavaConversions.asScalaBuffer(servers),
topic, part, timeOutMS);
}
}
- public void createTopic(String name, int numPartitions, boolean
waitUntilMetadataIsPropagated) throws InterruptedException {
+ public void createTopic(String name, int numPartitions, long
waitThisLongForMetadataToPropagate) throws InterruptedException {
ZkUtils zkUtils = null;
Level oldLevel = UnitTestHelper.getJavaLoggingLevel();
try {
UnitTestHelper.setJavaLoggingLevel(Level.OFF);
zkUtils = ZkUtils.apply(zookeeperConnectString, 3, 3, false);
AdminUtilsWrapper.createTopic(zkUtils, name, numPartitions, 1, new
Properties());
- if (waitUntilMetadataIsPropagated) {
-waitUntilMetadataIsPropagated(name, numPartitions);
+ if (waitThisLongForMetadataToPropagate > 0) {
+waitUntilMetadataIsPropagated(name, numPartitions,
waitThisLongForMetadataToPropagate);
}
}catch(TopicExistsException tee) {
}finally {
[metron] branch master updated: METRON-2238 Streaming enrichments regression (merrimanr via mmiklavc) closes apache/metron#1498
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 1a6d77f METRON-2238 Streaming enrichments regression (merrimanr via
mmiklavc) closes apache/metron#1498
1a6d77f is described below
commit 1a6d77fd74e37950ced4ae1620d9ee9d80cdc4b1
Author: merrimanr
AuthorDate: Thu Aug 29 10:20:02 2019 -0600
METRON-2238 Streaming enrichments regression (merrimanr via mmiklavc)
closes apache/metron#1498
---
metron-platform/metron-parsing/metron-parsing-storm/pom.xml | 5 +
1 file changed, 5 insertions(+)
diff --git a/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
b/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
index 98bfcff..9bd898e 100644
--- a/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
+++ b/metron-platform/metron-parsing/metron-parsing-storm/pom.xml
@@ -62,6 +62,11 @@
${project.parent.version}
provided
+
+ com.google.guava
+ guava
+ ${global_hbase_guava_version}
+
[metron] branch master updated: METRON-2212 Add debugging developer docs to hbase-server README (mmiklavc) closes apache/metron#1481
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 7219606 METRON-2212 Add debugging developer docs to hbase-server README (mmiklavc) closes apache/metron#1481 7219606 is described below commit 7219606fd56258c3037c73e5709f2144702fe37a Author: mmiklavc AuthorDate: Thu Aug 22 09:44:53 2019 -0600 METRON-2212 Add debugging developer docs to hbase-server README (mmiklavc) closes apache/metron#1481 --- metron-platform/metron-hbase-server/README.md | 27 +++ 1 file changed, 27 insertions(+) diff --git a/metron-platform/metron-hbase-server/README.md b/metron-platform/metron-hbase-server/README.md index a3b83e2..15874a5 100644 --- a/metron-platform/metron-hbase-server/README.md +++ b/metron-platform/metron-hbase-server/README.md @@ -42,3 +42,30 @@ HBase table name for the enrichments list. Defaults to `enrichment_list`. `enrichment.list.hbase.cf` HBase table column family for the enrichments list. Defaults to `t`. + +## Debugging + +If you have trouble with a RegionServer failing to start due to a coprocessor problem, e.g. +``` +2019-08-13 14:37:40,793 ERROR [RS_OPEN_REGION-regionserver/node1:16020-0] regionserver.HRegionServer: * ABORTING region server node1,16020,1565707051425: The coprocessor org.apache.metron.hbase.coprocessor.EnrichmentCoprocessor threw... +``` + +you may need to temporarily disable coprocessor loading while you fix the issue. + +### Disabling coprocessor loading + +* Navigate to HBase > Config in Ambari +* Expand the `Custom hbase-site` subpanel +* Add the property "`hbase.coprocessor.enabled`" and set it to `false`. **Note:** you can also use the property `hbase.coprocessor.user.enabled` instead. From the HBase documentation: +> Enables or disables user (aka. table) coprocessor loading. If 'false' (disabled), any table coprocessor attributes in table descriptors will be ignored. If "hbase.coprocessor.enabled" is 'false' this setting has no effect. +* Restart the HBase regionservers. You should notice a similar message to the following in your regionserver logs. +``` +2019-08-13 15:49:18,859 INFO [regionserver/node1:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is disabled +2019-08-13 15:49:18,859 INFO [regionserver/node1:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is disabled +``` +* HBase should now start successfully + + Reference + +* https://hbase.apache.org/1.1/book.html#load_coprocessor_in_shell +* https://hbase.apache.org/1.1/book.html#hbase_default_configurations
[metron] branch master updated: METRON-2076 Fixed up flakey stellar timezone test (tigerquoll via mmiklavc) closes apache/metron#1487
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 5e93677 METRON-2076 Fixed up flakey stellar timezone test (tigerquoll
via mmiklavc) closes apache/metron#1487
5e93677 is described below
commit 5e93677d38889fdd34d4f6a93ab0e8099bf01c7b
Author: tigerquoll
AuthorDate: Wed Aug 21 10:38:23 2019 -0600
METRON-2076 Fixed up flakey stellar timezone test (tigerquoll via mmiklavc)
closes apache/metron#1487
---
.../stellar/dsl/functions/DateFunctionsTest.java | 29 +-
1 file changed, 17 insertions(+), 12 deletions(-)
diff --git
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java
index 48b2995..ce8d5ce 100644
---
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java
+++
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/DateFunctionsTest.java
@@ -28,7 +28,9 @@ import org.apache.metron.stellar.dsl.StellarFunctions;
import org.junit.Before;
import org.junit.Test;
+import java.time.Instant;
import java.time.LocalDate;
+import java.time.ZoneId;
import java.time.format.DateTimeFormatter;
import java.util.Calendar;
import java.util.HashMap;
@@ -63,13 +65,13 @@ public class DateFunctionsTest {
@Before
public void setup() {
-variables.put("epoch", AUG2016);
+variables.put("test_datetime", AUG2016);
calendar = Calendar.getInstance();
}
@Test
public void testDayOfWeek() {
-Object result = run("DAY_OF_WEEK(epoch)");
+Object result = run("DAY_OF_WEEK(test_datetime)");
assertEquals(Calendar.THURSDAY, result);
}
@@ -92,7 +94,7 @@ public class DateFunctionsTest {
@Test
public void testWeekOfMonth() {
-Object result = run("WEEK_OF_MONTH(epoch)");
+Object result = run("WEEK_OF_MONTH(test_datetime)");
assertEquals(4, result);
}
@@ -115,7 +117,7 @@ public class DateFunctionsTest {
@Test
public void testMonth() {
-Object result = run("MONTH(epoch)");
+Object result = run("MONTH(test_datetime)");
assertEquals(Calendar.AUGUST, result);
}
@@ -138,7 +140,7 @@ public class DateFunctionsTest {
@Test
public void testYear() {
-Object result = run("YEAR(epoch)");
+Object result = run("YEAR(test_datetime)");
assertEquals(2016, result);
}
@@ -161,7 +163,7 @@ public class DateFunctionsTest {
@Test
public void testDayOfMonth() {
-Object result = run("DAY_OF_MONTH(epoch)");
+Object result = run("DAY_OF_MONTH(test_datetime)");
assertEquals(25, result);
}
@@ -184,7 +186,7 @@ public class DateFunctionsTest {
@Test
public void testWeekOfYear() {
-Object result = run("WEEK_OF_YEAR(epoch)");
+Object result = run("WEEK_OF_YEAR(test_datetime)");
calendar.setTimeInMillis(AUG2016);
assertEquals(calendar.get(Calendar.WEEK_OF_YEAR), result);
}
@@ -208,7 +210,7 @@ public class DateFunctionsTest {
@Test
public void testDayOfYear() {
-Object result = run("DAY_OF_YEAR(epoch)");
+Object result = run("DAY_OF_YEAR(test_datetime)");
assertEquals(238, result);
}
@@ -231,7 +233,7 @@ public class DateFunctionsTest {
@Test
public void testDateFormat() {
-Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss zzz', epoch,
'EST')");
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss zzz',
test_datetime, 'EST')");
assertEquals("Thu Aug 25 2016 08:27:10 EST", result);
}
@@ -255,8 +257,11 @@ public class DateFunctionsTest {
@Test
public void testDateFormatDefaultTimezone() {
-Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ', epoch)");
-
assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
1)));
+Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ',
test_datetime)");
+
+boolean inDaylightSavings = ZoneId.of( TimeZone.getDefault().getID() )
+.getRules().isDaylightSavings(Instant.ofEpochMilli(AUG2016) );
+
assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(inDaylightSavings,
1)));
}
/**
@@ -269,6 +274,6 @@ public class DateFunctionsTest {
@Test(expected = ParseException.class)
public void testDateFormatInvalid() {
-Object result = run("DATE_FORMAT('INVALID DATE FORMAT', epoch, 'EST')");
+Object result = run("DATE_FORMAT('INVALID DATE FORMAT', test_datetime,
'EST')");
}
}
[metron] branch master updated: METRON-2195 Add defensive log level checks when constructing logs is expensive (tigerquoll via mmiklavc) closes apache/metron#1473
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 771d66b METRON-2195 Add defensive log level checks when constructing
logs is expensive (tigerquoll via mmiklavc) closes apache/metron#1473
771d66b is described below
commit 771d66b5a806f9c0f1c53c1782977e00fa26917a
Author: tigerquoll
AuthorDate: Mon Aug 5 10:16:19 2019 -0600
METRON-2195 Add defensive log level checks when constructing logs is
expensive (tigerquoll via mmiklavc) closes apache/metron#1473
---
.../spark/function/MessageRouterFunction.java | 9 +-
.../metron/rest/generator/SampleDataGenerator.java | 2 +-
.../enrichment/SensorEnrichmentUpdateConfig.java | 8 +-
.../metron/common/hadoop/SequenceFileIterable.java | 10 +-
.../org/apache/metron/common/utils/LazyLogger.java | 94 +++
.../metron/common/utils/LazyLoggerFactory.java | 71 ++
.../apache/metron/common/utils/LazyLoggerImpl.java | 572
.../configurations/ConfigurationsUpdater.java | 10 +-
.../metron/common/utils/LazyLoggerImplTest.java| 759 +
.../common/utils/LazzyLoggerImplPerfTest.java | 118
.../adapters/simplehbase/SimpleHBaseAdapter.java | 8 +-
.../adapters/stellar/StellarAdapter.java | 2 +-
.../java/org/apache/metron/parsers/GrokParser.java | 11 +-
.../apache/metron/parsers/asa/BasicAsaParser.java | 12 +-
.../apache/metron/parsers/bro/BasicBroParser.java | 8 +-
.../java/org/apache/metron/pcap/mr/PcapJob.java| 9 +-
.../apache/metron/pcap/utils/FileFilterUtil.java | 12 +-
.../writer/hbase/SimpleHbaseEnrichmentWriter.java | 19 +-
.../org/apache/metron/writer/hdfs/HdfsWriter.java | 13 +-
.../apache/metron/writer/hdfs/SourceHandler.java | 8 +-
20 files changed, 1686 insertions(+), 69 deletions(-)
diff --git
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/MessageRouterFunction.java
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/MessageRouterFunction.java
index 31734d0..f38ed3b 100644
---
a/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/MessageRouterFunction.java
+++
b/metron-analytics/metron-profiler-spark/src/main/java/org/apache/metron/profiler/spark/function/MessageRouterFunction.java
@@ -20,6 +20,8 @@
package org.apache.metron.profiler.spark.function;
import org.apache.metron.common.configuration.profiler.ProfilerConfig;
+import org.apache.metron.common.utils.LazyLogger;
+import org.apache.metron.common.utils.LazyLoggerFactory;
import org.apache.metron.profiler.DefaultMessageRouter;
import org.apache.metron.profiler.MessageRoute;
import org.apache.metron.profiler.MessageRouter;
@@ -27,12 +29,9 @@ import org.apache.metron.profiler.clock.Clock;
import org.apache.metron.profiler.clock.ClockFactory;
import org.apache.metron.profiler.clock.EventTimeOnlyClockFactory;
import org.apache.metron.stellar.dsl.Context;
-import org.apache.metron.stellar.dsl.StellarFunctions;
import org.apache.spark.api.java.function.FlatMapFunction;
import org.json.simple.JSONObject;
import org.json.simple.parser.JSONParser;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
import java.lang.invoke.MethodHandles;
import java.util.Collections;
@@ -46,7 +45,7 @@ import java.util.Optional;
*/
public class MessageRouterFunction implements FlatMapFunction {
- protected static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ protected static final LazyLogger LOG =
LazyLoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
/**
* The global configuration used for the execution of Stellar.
@@ -112,7 +111,7 @@ public class MessageRouterFunction implements
FlatMapFunction
timestamp, () -> prettyPrint(begin), () -> prettyPrint(end));
}
} else {
diff --git
a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/generator/SampleDataGenerator.java
b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/generator/SampleDataGenerator.java
index 8557035..2a5db30 100644
---
a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/generator/SampleDataGenerator.java
+++
b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/generator/SampleDataGenerator.java
@@ -138,7 +138,7 @@ public class SampleDataGenerator {
} catch (InterruptedException e) {
e.printStackTrace();
}
-LOG.info("Emitting " + sensorType + " message " + message);
+LOG.info("Emitting {} message {}", sensorType, message);
emitToKafka(sensorType, message);
}
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/me
[metron] branch master updated: METRON-2202 Add parameter validation for the stellar field validation functions (MohanDV via mmiklavc) closes apache/metron#1476
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 0239432 METRON-2202 Add parameter validation for the stellar field
validation functions (MohanDV via mmiklavc) closes apache/metron#1476
0239432 is described below
commit 0239432b967bc176d20d6965544be582f54ae0e9
Author: MohanDV
AuthorDate: Mon Aug 5 08:48:57 2019 -0600
METRON-2202 Add parameter validation for the stellar field validation
functions (MohanDV via mmiklavc) closes apache/metron#1476
---
.../apache/metron/common/field/validation/SimpleValidation.java | 3 +++
.../common/field/validation/network/DomainValidationTest.java | 8
.../common/field/validation/network/EmailValidationTest.java | 6 ++
.../metron/common/field/validation/network/URLValidationTest.java | 8 ++--
.../common/field/validation/primitive/IntegerValidationTest.java | 6 ++
5 files changed, 29 insertions(+), 2 deletions(-)
diff --git
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/SimpleValidation.java
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/SimpleValidation.java
index 3a0b4f8..2909df3 100644
---
a/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/SimpleValidation.java
+++
b/metron-platform/metron-common/src/main/java/org/apache/metron/common/field/validation/SimpleValidation.java
@@ -52,6 +52,9 @@ public abstract class SimpleValidation implements
FieldValidation, Predicate input) {
+if(input.isEmpty()) {
+ return false;
+}
Predicate predicate = getPredicate();
for(Object o : input) {
if(o == null || !predicate.test(o)){
diff --git
a/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/DomainValidationTest.java
b/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/DomainValidationTest.java
index ee57efa..5277c3f 100644
---
a/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/DomainValidationTest.java
+++
b/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/DomainValidationTest.java
@@ -25,6 +25,7 @@ import org.junit.Assert;
import org.junit.Test;
import java.io.IOException;
+import java.util.Collections;
import static
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
@@ -64,6 +65,13 @@ public class DomainValidationTest extends BaseValidationTest{
Assert.assertTrue(execute(validWithSingleField, ImmutableMap.of("field1",
"www.hotmail.co.uk")));
Assert.assertTrue(runPredicate(validWithSingleField_MQL,
ImmutableMap.of("field1", "www.hotmail.co.uk")));
}
+
+ @Test
+ public void negativeTest_empty() throws IOException {
+Assert.assertFalse(runPredicate("IS_DOMAIN()", Collections.emptyMap()));
+Assert.assertFalse(runPredicate("IS_DOMAIN('')", Collections.emptyMap()));
+ }
+
@Test
public void negativeTest_single() throws IOException {
Assert.assertFalse(execute(validWithSingleField, ImmutableMap.of("field1",
"foo")));
diff --git
a/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/EmailValidationTest.java
b/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/EmailValidationTest.java
index e45c467..6360ac1 100644
---
a/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/EmailValidationTest.java
+++
b/metron-platform/metron-common/src/test/java/org/apache/metron/common/field/validation/network/EmailValidationTest.java
@@ -25,6 +25,7 @@ import org.junit.Assert;
import org.junit.Test;
import java.io.IOException;
+import java.util.Collections;
import static
org.apache.metron.stellar.common.utils.StellarProcessorUtils.runPredicate;
@@ -76,6 +77,11 @@ public class EmailValidationTest extends BaseValidationTest {
Assert.assertFalse(runPredicate(validWithSingleField_MQL,
ImmutableMap.of("field1", 2.7f)));
}
@Test
+ public void negativeTest_empty() throws IOException {
+Assert.assertFalse(runPredicate("IS_EMAIL()", Collections.emptyMap()));
+Assert.assertFalse(runPredicate("IS_EMAIL('')", Collections.emptyMap()));
+ }
+ @Test
public void positiveTest_multiple() throws IOException {
Assert.assertTrue(execute(validWithMultipleFields,
ImmutableMap.of("field1", "m...@www.gmail.com", "field2",
"m...@www.hotmail.com")));
Assert.assertTrue(runPredicate(validWithMultipleFields_MQL,
ImmutableMap.of("f
[metron] branch master updated: METRON-2197 Add debugging info output for Solr queries (mmiklavc) closes apache/metron#1475
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 8c4b53d METRON-2197 Add debugging info output for Solr queries
(mmiklavc) closes apache/metron#1475
8c4b53d is described below
commit 8c4b53dae73067e05cfcdac42ceb9c562bf4c2fd
Author: mmiklavc
AuthorDate: Mon Aug 5 08:42:03 2019 -0600
METRON-2197 Add debugging info output for Solr queries (mmiklavc) closes
apache/metron#1475
---
.../org/apache/metron/solr/dao/SolrSearchDao.java | 50 --
.../integration/SolrSearchIntegrationTest.java | 1 -
2 files changed, 37 insertions(+), 14 deletions(-)
diff --git
a/metron-platform/metron-solr/metron-solr-common/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
b/metron-platform/metron-solr/metron-solr-common/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
index 134d28c..885132d 100644
---
a/metron-platform/metron-solr/metron-solr-common/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
+++
b/metron-platform/metron-solr/metron-solr-common/src/main/java/org/apache/metron/solr/dao/SolrSearchDao.java
@@ -81,6 +81,20 @@ public class SolrSearchDao implements SearchDao {
// If null, use whatever the searchRequest defines.
public SearchResponse search(SearchRequest searchRequest, String fieldList)
throws InvalidSearchException {
+validateSearchRequest(searchRequest);
+try {
+ SolrQuery query = buildSearchRequest(searchRequest, fieldList);
+ QueryResponse response = client.query(query);
+ logQueryDebugDetail(query, response);
+ return buildSearchResponse(searchRequest, response);
+} catch (SolrException | IOException | SolrServerException e) {
+ String msg = e.getMessage();
+ LOG.error(msg, e);
+ throw new InvalidSearchException(msg, e);
+}
+ }
+
+ private void validateSearchRequest(SearchRequest searchRequest) throws
InvalidSearchException {
if (searchRequest.getQuery() == null) {
throw new InvalidSearchException("Search query is invalid: null");
}
@@ -91,29 +105,31 @@ public class SolrSearchDao implements SearchDao {
throw new InvalidSearchException(
"Search result size must be less than " +
accessConfig.getMaxSearchResults());
}
-try {
- SolrQuery query = buildSearchRequest(searchRequest, fieldList);
- QueryResponse response = client.query(query);
- return buildSearchResponse(searchRequest, response);
-} catch (SolrException | IOException | SolrServerException e) {
- String msg = e.getMessage();
- LOG.error(msg, e);
- throw new InvalidSearchException(msg, e);
+ }
+
+ private void logQueryDebugDetail(SolrQuery query, QueryResponse response) {
+if (LOG.isDebugEnabled()) {
+ final String ls = System.lineSeparator();
+ LOG.debug(
+ "Solr query string: {}{} Solr query debug map: {}{} Solr query
elapsed time: {}{} Solr query Q time: {}",
+ query.toQueryString(), ls,
+ response.getDebugMap(), ls,
+ response.getElapsedTime(), ls,
+ response.getQTime());
}
}
@Override
public GroupResponse group(GroupRequest groupRequest) throws
InvalidSearchException {
try {
- if (groupRequest.getGroups() == null || groupRequest.getGroups().size()
== 0) {
-throw new InvalidSearchException("At least 1 group must be provided.");
- }
+ validateGroupRequest(groupRequest);
String groupNames =
groupRequest.getGroups().stream().map(Group::getField).collect(
Collectors.joining(","));
SolrQuery query = new SolrQuery()
.setStart(0)
.setRows(0)
- .setQuery(groupRequest.getQuery());
+ .setQuery(groupRequest.getQuery())
+ .setShowDebugInfo(LOG.isDebugEnabled()); // tie Solr query debug
output to our log level
query.set("collection", getCollections(groupRequest.getIndices()));
Optional scoreField = groupRequest.getScoreField();
@@ -124,6 +140,7 @@ public class SolrSearchDao implements SearchDao {
query.set("facet", true);
query.set("facet.pivot", String.format("{!stats=piv1}%s", groupNames));
QueryResponse response = client.query(query);
+ logQueryDebugDetail(query, response);
return buildGroupResponse(groupRequest, response);
} catch (IOException | SolrServerException e) {
String msg = e.getMessage();
@@ -132,6 +149,12 @@ public class SolrSearchDao implements SearchDao {
}
}
+ private void validateGroupRequest(GroupRequest groupRequest) throws
InvalidSearchException {
+if (groupRequest.getGroups() == null || groupRequest.getGroups().size() ==
0) {
+ throw new InvalidSearchException("At
[metron] branch master updated: METRON-2189 Optimize imports in mpack python scripts (mmiklavc) closes apache/metron#1467
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 340827f METRON-2189 Optimize imports in mpack python scripts (mmiklavc) closes apache/metron#1467 340827f is described below commit 340827f2569767056698ac38cf4517dc866c4e06 Author: mmiklavc AuthorDate: Thu Aug 1 13:49:24 2019 -0600 METRON-2189 Optimize imports in mpack python scripts (mmiklavc) closes apache/metron#1467 --- .../5.6.14/package/scripts/elastic_commands.py| 14 ++ .../5.6.14/package/scripts/elastic_master.py | 7 --- .../ELASTICSEARCH/5.6.14/package/scripts/elastic_slave.py | 3 ++- .../ELASTICSEARCH/5.6.14/package/scripts/service_check.py | 4 ++-- .../KIBANA/5.6.14/package/scripts/common.py | 8 +--- .../KIBANA/5.6.14/package/scripts/kibana_master.py| 2 +- .../METRON/CURRENT/package/scripts/alerts_ui_commands.py | 4 ++-- .../METRON/CURRENT/package/scripts/alerts_ui_master.py| 10 +- .../CURRENT/package/scripts/dashboard/dashboardindex.py | 10 ++ .../METRON/CURRENT/package/scripts/enrichment_commands.py | 6 +++--- .../METRON/CURRENT/package/scripts/enrichment_master.py | 10 +- .../METRON/CURRENT/package/scripts/indexing_commands.py | 8 .../METRON/CURRENT/package/scripts/indexing_master.py | 15 +++ .../CURRENT/package/scripts/management_ui_commands.py | 6 ++ .../CURRENT/package/scripts/management_ui_master.py | 7 ++- .../METRON/CURRENT/package/scripts/metron_client.py | 11 +-- .../METRON/CURRENT/package/scripts/metron_security.py | 7 --- .../METRON/CURRENT/package/scripts/metron_service.py | 12 ++-- .../METRON/CURRENT/package/scripts/params/params.py | 2 +- .../METRON/CURRENT/package/scripts/params/params_linux.py | 9 +++-- .../CURRENT/package/scripts/params/status_params.py | 4 ++-- .../METRON/CURRENT/package/scripts/parser_commands.py | 9 - .../METRON/CURRENT/package/scripts/parser_master.py | 7 +++ .../METRON/CURRENT/package/scripts/pcap_commands.py | 11 --- .../METRON/CURRENT/package/scripts/pcap_master.py | 13 - .../METRON/CURRENT/package/scripts/profiler_commands.py | 6 +++--- .../METRON/CURRENT/package/scripts/profiler_master.py | 13 - .../METRON/CURRENT/package/scripts/rest_commands.py | 6 ++ .../METRON/CURRENT/package/scripts/rest_master.py | 8 .../METRON/CURRENT/package/scripts/service_check.py | 14 +++--- .../common-services/METRON/CURRENT/service_advisor.py | 6 ++ 31 files changed, 110 insertions(+), 142 deletions(-) diff --git a/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/package/scripts/elastic_commands.py b/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/package/scripts/elastic_commands.py index 618d10a..277c3c9 100644 --- a/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/package/scripts/elastic_commands.py +++ b/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/package/scripts/elastic_commands.py @@ -18,20 +18,18 @@ limitations under the License. """ -import os - from ambari_commons.os_check import OSCheck -from resource_management.core.exceptions import ExecutionFailed from resource_management.core.exceptions import ComponentIsNotRunning -from resource_management.core.resources.system import Execute +from resource_management.core.exceptions import ExecutionFailed +from resource_management.core.logger import Logger +from resource_management.core.resources import User from resource_management.core.resources.system import Directory +from resource_management.core.resources.system import Execute from resource_management.core.resources.system import File from resource_management.core.source import InlineTemplate from resource_management.core.source import Template -from resource_management.core.resources import User -from resource_management.core.logger import Logger -from resource_management.libraries.functions import format as ambari_format -from resource_management.libraries.functions.get_user_call_output import get_user_call_output +from resource_management.libraries.functions.get_user_call_output import \ + get_user_call_output def service_check(cmd, user, label): diff --git a/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/package/scripts/elastic_master.py b/metron-deployment/packaging/ambari/elastics
[metron] branch master updated: METRON-2194 Update Ambari tooltip to specify single quotes for parser names with hyphens (mmiklavc) closes apache/metron#1471
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 855de4a METRON-2194 Update Ambari tooltip to specify single quotes for parser names with hyphens (mmiklavc) closes apache/metron#1471 855de4a is described below commit 855de4a21ceb33bb608d7eb8100397eae294d81a Author: mmiklavc AuthorDate: Mon Jul 29 11:15:44 2019 -0600 METRON-2194 Update Ambari tooltip to specify single quotes for parser names with hyphens (mmiklavc) closes apache/metron#1471 --- .../common-services/METRON/CURRENT/configuration/metron-parsers-env.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-parsers-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-parsers-env.xml index 03a2594..85efec5 100644 --- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-parsers-env.xml +++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-parsers-env.xml @@ -21,7 +21,7 @@ parsers bro,snort,yaf -Metron parsers to deploy. You can also specify an aggregated parser list by grouping them with double quotes. For example: "parserA,parserB",parserC,parserD +Metron parsers to deploy. You can also specify an aggregated parser list by grouping them with double quotes. For example: "parserA,parserB",parserC,parserD. Parser names with hyphens need to be wrapped in single quotes, e.g. 'my-hyphenated-parser-name' Metron Parsers
[metron] branch master updated: METRON-2130 [UI] Numeric steppers on the Management UI seems broken (ruffle1986 via mmiklavc) closes apache/metron#1421
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 6306c93 METRON-2130 [UI] Numeric steppers on the Management UI seems
broken (ruffle1986 via mmiklavc) closes apache/metron#1421
6306c93 is described below
commit 6306c935d71150456c2adbd0702001ae6b88fe38
Author: ruffle1986
AuthorDate: Tue Jul 23 14:05:16 2019 -0600
METRON-2130 [UI] Numeric steppers on the Management UI seems broken
(ruffle1986 via mmiklavc) closes apache/metron#1421
---
.../src/app/shared/number-spinner/number-spinner.component.html | 2 +-
.../src/app/shared/number-spinner/number-spinner.component.scss | 9 -
2 files changed, 9 insertions(+), 2 deletions(-)
diff --git
a/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.html
b/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.html
index 60e453b..b63908f 100644
---
a/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.html
+++
b/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.html
@@ -17,6 +17,6 @@
-
+
diff --git
a/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.scss
b/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.scss
index b4352c4..cf8214f 100644
---
a/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.scss
+++
b/metron-interface/metron-config/src/app/shared/number-spinner/number-spinner.component.scss
@@ -34,6 +34,13 @@
margin-left: -1px;
position: relative;
border-radius: 0;
+ border-left: solid 1px #4d4d4d;
+}
+
+.input-group-btn-vertical > .btn,
+.input-group-btn-vertical > .btn:focus,
+.input-group-btn-vertical > .btn:active {
+ box-shadow: none;
}
.input-group-btn-vertical > .btn:first-child
@@ -43,8 +50,8 @@
.input-group-btn-vertical > .btn:last-child
{
- margin-top: -1px;
border-bottom-right-radius: 0.25em;
+ border-top: none;
}
.input-group-btn-vertical i
[metron] branch master updated: METRON-2129 [UI] Clearing the search bar resets alert filter range to 'All Time' (ruffle1986 via mmiklavc) closes apache/metron#1420
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 65681a0 METRON-2129 [UI] Clearing the search bar resets alert filter
range to 'All Time' (ruffle1986 via mmiklavc) closes apache/metron#1420
65681a0 is described below
commit 65681a0a0888576725663347e52deafe8ca94a1a
Author: ruffle1986
AuthorDate: Tue Jul 23 13:53:18 2019 -0600
METRON-2129 [UI] Clearing the search bar resets alert filter range to 'All
Time' (ruffle1986 via mmiklavc) closes apache/metron#1420
---
.../metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts| 1 -
1 file changed, 1 deletion(-)
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
index 47e777f..4cf1886 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
+++
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
@@ -221,7 +221,6 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
onClear() {
this.timeStampFilterPresent = false;
this.queryBuilder.clearSearch();
-this.selectedTimeRange = new Filter(TIMESTAMP_FIELD_NAME, ALL_TIME, false);
this.search();
}
[metron] branch master updated: METRON-2140: [UI] Implement logic behind show/hide RESOLVE and DISMISS items in Alerts UI (tiborm via mmiklavc) closes apache/metron#1459
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 5e1e3bd METRON-2140: [UI] Implement logic behind show/hide RESOLVE
and DISMISS items in Alerts UI (tiborm via mmiklavc) closes apache/metron#1459
5e1e3bd is described below
commit 5e1e3bd2e2876a6774b9c50d72115332f0bc15ee
Author: tiborm
AuthorDate: Tue Jul 23 13:35:21 2019 -0600
METRON-2140: [UI] Implement logic behind show/hide RESOLVE and DISMISS
items in Alerts UI (tiborm via mmiklavc) closes apache/metron#1459
---
.../alerts/alerts-list/alerts-list.component.html | 13 +-
.../alerts-list/alerts-list.component.spec.ts | 4 +
.../alerts/alerts-list/alerts-list.component.ts| 52
.../app/alerts/alerts-list/alerts-list.module.ts | 89 --
.../src/app/alerts/alerts-list/query-builder.ts| 20 ++--
.../alerts-list/table-view/table-view.component.ts | 29 +++--
.../alerts-list/tree-view/tree-view.component.ts | 52
.../configure-rows/configure-rows.component.html | 7 +-
.../configure-rows/configure-rows.component.scss | 2 +-
.../configure-rows.component.spec.ts | 14 ++-
.../configure-rows/configure-rows.component.ts | 5 +-
.../alerts/configure-rows/configure-rows.module.ts | 22 ++--
.../show-hide-alert-entries.component.spec.ts | 132 +
.../show-hide/show-hide-alert-entries.component.ts | 51
.../show-hide/show-hide.service.spec.ts| 125 +++
.../configure-rows/show-hide/show-hide.service.ts | 70 +++
.../metron-alerts/src/app/app.module.ts| 2 -
.../src/app/service/search.service.ts | 5 +-
.../src/app/shared/switch/switch.component.html| 2 +-
.../src/app/shared/switch/switch.component.ts | 12 +-
.../src/app/shared/switch/switch.module.ts | 15 ++-
21 files changed, 558 insertions(+), 165 deletions(-)
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
index 26a38cb..4ed3951 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
+++
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
@@ -24,7 +24,7 @@
-
+
@@ -82,26 +82,27 @@
+
(onSelectedAlertsChange)="onSelectedAlertsChange($event)"
+(onSortChanged)="onSortChanged($event)"
+
(onPageChanged)="onPageChanged($event)">
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.spec.ts
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.spec.ts
index e922984..6779baa 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.spec.ts
+++
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.spec.ts
@@ -30,6 +30,7 @@ import { GlobalConfigService } from
'app/service/global-config.service';
import { DialogService } from 'app/service/dialog.service';
import { Observable } from 'rxjs';
import { Filter } from 'app/model/filter';
+import { QueryBuilder } from './query-builder';
describe('AlertsListComponent', () => {
@@ -68,6 +69,9 @@ describe('AlertsListComponent', () => {
get: () => new Observable(),
} } },
{ provide: DialogService, useClass: () => { return {} } },
+{ provide: QueryBuilder, useClass: () => { return {
+ addOrUpdateFilter: () => {}
+} } },
]
})
.compileComponents();
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
index 7fd69ba..47e777f 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
+++
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
@@ -37,16 +37,15 @@ import {SearchResponse} from '../../model/search-response';
import {ElasticsearchUtils} from '../../utils/elasticsearch-utils';
import {Filter} from '../../model/filter';
import { TIMESTAMP_FIELD_NAME, ALL_TIME, POLLING_DEFAULT_STATE } from
'../../utils/constants';
-import {TableViewComponent} from './t
[metron] branch master updated: METRON-2079 Fix documentation for installing Ansible for fulldev Centos 6 (mmiklavc) closes apache/metron#1450
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 7e39143 METRON-2079 Fix documentation for installing Ansible for fulldev Centos 6 (mmiklavc) closes apache/metron#1450 7e39143 is described below commit 7e39143caf841fdb576caa417ce33110ba123270 Author: mmiklavc AuthorDate: Thu Jul 18 15:36:29 2019 -0600 METRON-2079 Fix documentation for installing Ansible for fulldev Centos 6 (mmiklavc) closes apache/metron#1450 --- metron-deployment/development/centos6/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/metron-deployment/development/centos6/README.md b/metron-deployment/development/centos6/README.md index 99ec967..5ca3ed5 100644 --- a/metron-deployment/development/centos6/README.md +++ b/metron-deployment/development/centos6/README.md @@ -29,7 +29,7 @@ Getting Started The computer used to deploy Apache Metron will need to have the following components installed. - - [Ansible](https://github.com/ansible/ansible) 2.4.0+ + - [Ansible](https://github.com/ansible/ansible) 2.6.5 - [Docker](https://www.docker.com/community-edition) - [Vagrant](https://www.vagrantup.com) 2.0+ - [Vagrant Hostmanager Plugin](https://github.com/devopsgroup-io/vagrant-hostmanager) @@ -53,9 +53,10 @@ Any platform that supports these tools is suitable, but the following instructio 1. Run the following command in a terminal to install all of the required tools. ``` -brew cask install vagrant virtualbox docker ansible +brew cask install vagrant virtualbox docker brew cask install caskroom/versions/java8 brew install maven@3.3 git +sudo pip install ansible=2.6.5 vagrant plugin install vagrant-hostmanager open /Applications/Docker.app ```
[metron] branch master updated: METRON-2084 Add documentation notice for MacOS Mojave users for new security permissions (mmiklavc) closes apache/metron#1449
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new c8f2c9a METRON-2084 Add documentation notice for MacOS Mojave users for new security permissions (mmiklavc) closes apache/metron#1449 c8f2c9a is described below commit c8f2c9affd8e3b2ed9b99cf7eb9ac7b61e4d37a2 Author: mmiklavc AuthorDate: Tue Jul 2 11:47:15 2019 -0600 METRON-2084 Add documentation notice for MacOS Mojave users for new security permissions (mmiklavc) closes apache/metron#1449 --- metron-deployment/development/README.md | 9 + 1 file changed, 9 insertions(+) diff --git a/metron-deployment/development/README.md b/metron-deployment/development/README.md index b86a5c4..448b53f 100644 --- a/metron-deployment/development/README.md +++ b/metron-deployment/development/README.md @@ -61,3 +61,12 @@ To setup this up, start full dev. Now, when you go to Swagger or the UIs, you should be able to give a user and password. "admin" will have the roles ROLE_ADMIN and ROLE_USER, which can be verified via the "/whoami/roles" endpoint in Swagger. Similarly, there is a user "sam" that only has ROLE_USER. A third user, "tom" has neither role. + +## Common Problems + +### Mac Mojave - Operation Not Permitted Error + +`tee: /etc/exports: Operation not permitted on macOS 10.14 Mojave with nfs exports` + +If you have the Mojave OS or newer, you may run into this issue when running `vagrant up`. In order to correct this you will need to grant permissions to your relevant terminal application. +Navigate to `System Preferences -> Security & Privacy -> Privacy` and add your terminal application to "Full Disk Access". See [https://github.com/hashicorp/vagrant/issues/10234](https://github.com/hashicorp/vagrant/issues/10234) for more details.
[metron] branch master updated: METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but was:<2> (mmiklavc) closes apache/metron#1452
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new c9604c2 METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116
expected:<1> but was:<2> (mmiklavc) closes apache/metron#1452
c9604c2 is described below
commit c9604c255aa596202c6fb348fca3aa9816c0b297
Author: mmiklavc
AuthorDate: Fri Jun 28 16:37:45 2019 -0600
METRON-2166 FileFilterUtilTest.test_getPaths_leftEdge:116 expected:<1> but
was:<2> (mmiklavc) closes apache/metron#1452
---
.../apache/metron/pcap/mr/FileFilterUtilTest.java | 37 +-
1 file changed, 22 insertions(+), 15 deletions(-)
diff --git
a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
index cc05a9a..72e2b93 100644
---
a/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
+++
b/metron-platform/metron-pcap/src/test/java/org/apache/metron/pcap/mr/FileFilterUtilTest.java
@@ -29,7 +29,6 @@ import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
import org.apache.hadoop.fs.Path;
-import org.apache.metron.common.utils.timestamp.TimestampConverters;
import org.apache.metron.pcap.utils.FileFilterUtil;
import org.junit.Assert;
import org.junit.Before;
@@ -108,43 +107,51 @@ public class FileFilterUtilTest {
@Test
public void test_getPaths_leftEdge() throws Exception {
+final long firstFileTSNanos = 1461589332993573000L;
+final long secondFileTSNanos = 1561589332993573000L;
final List inputFiles = new ArrayList() {{
- add(new
Path("/apps/metron/pcap/pcap_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
- add(new
Path("/apps/metron/pcap/pcap_pcap_1561589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap_pcap_" + firstFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+ add(new Path("/apps/metron/pcap/pcap_pcap_" + secondFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
}};
-Iterable paths = FileFilterUtil.getPathsInTimeRange(0,
TimestampConverters.MILLISECONDS.toNanoseconds(System.currentTimeMillis()),
inputFiles);
+Iterable paths = FileFilterUtil.getPathsInTimeRange(0,
secondFileTSNanos - 1L, inputFiles);
Assert.assertEquals(1, Iterables.size(paths));
}
@Test
public void test_getPaths_rightEdge() throws Exception {
+final long firstFileTSNanos = 1461589332993573000L;
+final long secondFileTSNanos = 1461589333993573000L;
+final long thirdFileTSNanos = 1461589334993573000L;
{
final List inputFiles = new ArrayList() {{
-add(new
Path("/apps/metron/pcap/pcap0_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
-add(new
Path("/apps/metron/pcap/pcap1_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+add(new Path("/apps/metron/pcap/pcap0_pcap_" + firstFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+add(new Path("/apps/metron/pcap/pcap1_pcap_" + secondFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
}};
- Iterable paths =
FileFilterUtil.getPathsInTimeRange(1461589333993573000L - 1L,
1461589333993573000L + 1L, inputFiles);
+ Iterable paths =
FileFilterUtil.getPathsInTimeRange(secondFileTSNanos - 1L, secondFileTSNanos +
1L, inputFiles);
Assert.assertEquals(2, Iterables.size(paths));
}
{
final List inputFiles = new ArrayList() {{
-add(new
Path("/apps/metron/pcap/pcap0_pcap_1461589332993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
-add(new
Path("/apps/metron/pcap/pcap1_pcap_1461589333993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
-add(new
Path("/apps/metron/pcap/pcap1_pcap_1461589334993573000_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+add(new Path("/apps/metron/pcap/pcap0_pcap_" + firstFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+add(new Path("/apps/metron/pcap/pcap1_pcap_" + secondFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
+add(new Path("/apps/metron/pcap/pcap1_pcap_" + thirdFileTSNanos +
"_0_73686171-64a1-46e5-9e67-66cf603fb094"));
}};
- Iterable paths =
FileFilterUtil.getPathsInTimeRange(1461589334993573000L - 1L,
1461589334993573000L + 1L, inputFiles);
+ Iterable paths =
FileFilterUtil.getPathsInTimeRange(thirdFileTSNanos - 1L, thirdFileTSNanos +
1L, inputFiles);
Assert.assertEquals(2, Ite
[metron] branch master updated: METRON-2141 Cache REST API status update calls to the Storm UI (mmiklavc) closes apache/metron#1439
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 3c13173 METRON-2141 Cache REST API status update calls to the Storm
UI (mmiklavc) closes apache/metron#1439
3c13173 is described below
commit 3c1317360243398eef8097edcea5f7c1a0bb2f59
Author: mmiklavc
AuthorDate: Wed Jun 12 17:13:16 2019 -0600
METRON-2141 Cache REST API status update calls to the Storm UI (mmiklavc)
closes apache/metron#1439
---
.../CURRENT/configuration/metron-rest-env.xml | 13 +-
.../CURRENT/package/scripts/params/params_linux.py | 2 +
.../METRON/CURRENT/package/templates/metron.j2 | 3 +
.../METRON/CURRENT/themes/metron_theme.json| 20 +++
.../apache/metron/rest/model/TopologySummary.java | 11 +-
metron-interface/metron-rest/README.md | 24 ++--
.../src/main/config/rest_application.yml | 4 +
.../apache/metron/rest/MetronRestConstants.java| 2 +
.../org/apache/metron/rest/config/StormConfig.java | 21 ++-
.../service/impl/CachedStormStatusServiceImpl.java | 127 +
.../rest/service/impl/StormStatusServiceImpl.java | 14 +-
.../org/apache/metron/rest/config/TestConfig.java | 12 ++
.../impl/CachedStormStatusServiceImplTest.java | 158 +
13 files changed, 385 insertions(+), 26 deletions(-)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml
index 68b1140..145b64e 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-rest-env.xml
@@ -181,5 +181,16 @@
The field name where the threat triage score can be found
in the search indices. This setting primarily affects the Alerts
UI.
threat:triage:score
-
+
+storm_status_cache_max_size
+1
+The maximum size for the cache that fronts calls to the
Storm API for topology status.
+Storm Status Cache Max Size
+
+
+storm_status_cache_timeout_seconds
+5
+Duration in seconds for cache entries to timeout. Note
that the higher the value, the more stale the returned value will
be.
+Storm Status Cache Timeout Seconds
+
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index de6b8bc..a7f20fc 100755
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -64,6 +64,8 @@ metron_alerts_ui_host = status_params.metron_alerts_ui_host
metron_alerts_ui_port = status_params.metron_alerts_ui_port
metron_alerts_ui_path = metron_home + '/web/alerts-ui/'
metron_jvm_flags =
config['configurations']['metron-rest-env']['metron_jvm_flags']
+storm_status_cache_max_size =
config['configurations']['metron-rest-env']['storm_status_cache_max_size']
+storm_status_cache_timeout_seconds =
config['configurations']['metron-rest-env']['storm_status_cache_timeout_seconds']
# Construct the profiles as a temp variable first. Only the first time it's
set will carry through
metron_spring_profiles_active =
config['configurations']['metron-rest-env']['metron_spring_profiles_active']
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
index 936118c..5c43bbd 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
@@ -66,3 +66,6 @@ PCAP_FINAL_OUTPUT_PATH="{{pcap_final_output_path}}"
PCAP_PAGE_SIZE="{{pcap_page_size}}"
PCAP_YARN_QUEUE="{{pcap_yarn_queue}}"
PCAP_FINALIZER_THREADPOOL_SIZE="{{pcap_finalizer_thre
[metron] branch master updated: METRON-2102 [UI] Adding click-through navigation to Alerts table (tiborm via mmiklavc) closes apache/metron#1431
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 5bd7e01 METRON-2102 [UI] Adding click-through navigation to Alerts
table (tiborm via mmiklavc) closes apache/metron#1431
5bd7e01 is described below
commit 5bd7e010c2f389c2c14476337df982f2e781af8d
Author: tiborm
AuthorDate: Wed Jun 12 07:56:34 2019 -0600
METRON-2102 [UI] Adding click-through navigation to Alerts table (tiborm
via mmiklavc) closes apache/metron#1431
---
.../CURRENT/package/scripts/params/params_linux.py | 1 +
.../package/templates/alerts-ui-app-config.json.j2 | 3 +-
.../packaging/docker/rpm-docker/SPECS/metron.spec | 1 +
metron-interface/metron-alerts/README.md | 4 +
.../cypress/fixtures/context-menu.conf.json| 49
.../integration/alert-list/context-menu.spec.js| 89 +++
.../alerts/alerts-list/alerts-list.component.html | 2 +-
.../alerts/alerts-list/alerts-list.component.ts| 3 +-
.../table-view/table-view.component.html | 89 +--
.../table-view/table-view.component.spec.ts| 2 +
.../alerts-list/table-view/table-view.component.ts | 9 +-
.../app-config.service.ts => app.module.spec.ts} | 33 +--
.../src/app/service/app-config.service.spec.ts | 154 +++
.../src/app/service/app-config.service.ts | 21 +-
.../src/app/shared/context-menu/README.md | 203 +++
.../context-menu/context-menu.component.html | 25 ++
.../context-menu/context-menu.component.scss} | 55 ++--
.../context-menu/context-menu.component.spec.ts| 281 +
.../shared/context-menu/context-menu.component.ts | 164
.../context-menu/context-menu.module.spec.ts} | 39 +--
.../context-menu/context-menu.module.ts} | 52 ++--
.../context-menu/context-menu.service.spec.ts | 229 +
.../shared/context-menu/context-menu.service.ts| 94 +++
.../context-menu/context-menu.util.spec.ts}| 57 ++---
.../context-menu/context-menu.util.ts} | 37 +--
.../shared/context-menu/dynamic-item.model.spec.ts | 42 +++
.../context-menu/dynamic-item.model.ts}| 54 ++--
.../metron-alerts/src/app/shared/shared.module.ts | 5 +-
.../metron-alerts/src/assets/app-config.json | 3 +-
.../src/assets/context-menu.conf.json | 49
30 files changed, 1614 insertions(+), 235 deletions(-)
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
index 64105e3..de6b8bc 100755
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/params/params_linux.py
@@ -465,6 +465,7 @@ knox_group =
config['configurations']['knox-env']['knox_group']
metron_knox_root_path = '/gateway/metron'
metron_rest_path = '/api/v1'
metron_alerts_ui_login_path = '/login'
+metron_alerts_ui_context_menu_config_url = '/assets/context-menu.conf.json'
metron_management_ui_login_path = '/login'
metron_knox_enabled =
config['configurations']['metron-security-env']['metron.knox.enabled']
metron_knox_sso_pubkey =
config['configurations']['metron-security-env']['metron.knox.sso.pubkey']
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/alerts-ui-app-config.json.j2
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/alerts-ui-app-config.json.j2
index edbc1b6..cdc064e 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/alerts-ui-app-config.json.j2
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/alerts-ui-app-config.json.j2
@@ -1,4 +1,5 @@
{
"apiRoot": "{{metron_rest_path}}",
- "loginPath": "{{metron_alerts_ui_login_path}}"
+ "loginPath": "{{metron_alerts_ui_login_path}}",
+ "contextMenuConfigURL": "{{metron_alerts_ui_context_menu_config_url}}"
}
\ No newline at end of file
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
b/metron-deployment/packaging/docker/rpm-dock
[metron] branch master updated: METRON-2127 Update Maven repositories to https (justinleet via mmiklavc) closes apache/metron#1417
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new e16ff31 METRON-2127 Update Maven repositories to https (justinleet via mmiklavc) closes apache/metron#1417 e16ff31 is described below commit e16ff316ae1a05f88a1f515ba2040d6cddab1d3f Author: justinleet AuthorDate: Wed Jun 5 10:50:14 2019 -0600 METRON-2127 Update Maven repositories to https (justinleet via mmiklavc) closes apache/metron#1417 --- pom.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/pom.xml b/pom.xml index 239c89d..66a935e 100644 --- a/pom.xml +++ b/pom.xml @@ -59,7 +59,7 @@ clojars.org -http://clojars.org/repo +https://clojars.org/repo jcenter @@ -78,7 +78,7 @@ HDPReleases HDP Releases - http://repo.hortonworks.com/content/repositories/releases/ + https://repo.hortonworks.com/content/repositories/releases/ default
[metron] branch master updated: METRON-2145 Clarify RPM build documentation (mmiklavc) closes apache/metron#1434
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new dbf8bf9 METRON-2145 Clarify RPM build documentation (mmiklavc) closes apache/metron#1434 dbf8bf9 is described below commit dbf8bf9dd0f7140092ba789ff048c2f6beeb79c6 Author: mmiklavc AuthorDate: Tue Jun 4 10:03:05 2019 -0600 METRON-2145 Clarify RPM build documentation (mmiklavc) closes apache/metron#1434 --- metron-deployment/packaging/docker/deb-docker/README.md | 2 +- metron-deployment/packaging/docker/rpm-docker/README.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/metron-deployment/packaging/docker/deb-docker/README.md b/metron-deployment/packaging/docker/deb-docker/README.md index a0ac0b3..3f91b3f 100644 --- a/metron-deployment/packaging/docker/deb-docker/README.md +++ b/metron-deployment/packaging/docker/deb-docker/README.md @@ -28,7 +28,7 @@ If you are installing Metron using Ambari, these packages are necessary prerequi ### Quick Start -1. Execute the following command from the project's root directory. +1. Execute the following command from the project's root directory. This will build/package **all** of Metron prior to building the DEBs. See [Build Packages](#build-packages) below to only build the DEBs. ``` mvn clean package -DskipTests -Pbuild-debs ``` diff --git a/metron-deployment/packaging/docker/rpm-docker/README.md b/metron-deployment/packaging/docker/rpm-docker/README.md index 7667ed1..4f2bc08 100644 --- a/metron-deployment/packaging/docker/rpm-docker/README.md +++ b/metron-deployment/packaging/docker/rpm-docker/README.md @@ -26,7 +26,7 @@ If you are installing Metron using Ambari, these packages are necessary prerequi ### Quick Start -1. Execute the following command from the project's root directory. +1. Execute the following command from the project's root directory. This will build/package **all** of Metron prior to building the RPMs. See [Build Packages](#build-packages) below to only build the RPMs. ``` mvn clean package -DskipTests -Pbuild-rpms ``` @@ -41,7 +41,7 @@ If you are installing Metron using Ambari, these packages are necessary prerequi If Metron has already been built, just the RPM packages can be built by executing the following commands. ``` cd metron-deployment - mvn clean package -Pbuild-debs + mvn clean package -Pbuild-rpms ``` ### How does this work?
[metron] branch master updated: METRON-2083 Fix broken links in root metron README (mmiklavc) closes apache/metron#1435
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new a8f2d3a METRON-2083 Fix broken links in root metron README (mmiklavc) closes apache/metron#1435 a8f2d3a is described below commit a8f2d3a2814315da950f24ec74bacc3b8f44b1f2 Author: mmiklavc AuthorDate: Tue Jun 4 10:01:35 2019 -0600 METRON-2083 Fix broken links in root metron README (mmiklavc) closes apache/metron#1435 --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 3390e55..1f4e030 100644 --- a/README.md +++ b/README.md @@ -132,7 +132,7 @@ component and Apache Kafka as the unified data bus. Some high level links to the relevant subparts of the architecture, for more information: -* [Parsers](metron-platform/metron-parsers-common) : Parsing data from kafka into the Metron data model and passing it downstream to Enrichment. +* [Parsers](metron-platform/metron-parsing) : Parsing data from kafka into the Metron data model and passing it downstream to Enrichment. * [Enrichment](metron-platform/metron-enrichment) : Enriching data post-parsing and providing the ability to tag a message as an alert and assign a risk triage level via a custom rule language. * [Indexing](metron-platform/metron-indexing) : Indexing the data post-enrichment into HDFS, Elasticsearch or Solr. @@ -146,4 +146,4 @@ Some useful utilities that cross all of these parts of the architecture: In order to allow for meta alerts to be queries alongside regular alerts in Elasticsearch 2.x, it is necessary to add an additional field to the templates and mapping for existing sensors. -Please see a description of the steps necessary to make this change in the metron-elasticsearch [Using Metron with Elasticsearch 2.x](./metron-platform/metron-elasticsearch#using-metron-with-elasticsearch-2x) +Please see a description of the steps necessary to make this change in metron-elasticsearch [Using Metron with Elasticsearch 5.6](./metron-platform/metron-elasticsearch/metron-elasticsearch-common#using-metron-with-elasticsearch-56)
[metron] branch master updated: METRON-2152 Add debug logging for when sensor batchTimeout exceeds the calculated maximum (mmiklavc) closes apache/metron#1437
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 2dd753e METRON-2152 Add debug logging for when sensor batchTimeout
exceeds the calculated maximum (mmiklavc) closes apache/metron#1437
2dd753e is described below
commit 2dd753e4b69fe1b85b7604a8d0e20af8ecea0ae8
Author: mmiklavc
AuthorDate: Tue Jun 4 09:29:19 2019 -0600
METRON-2152 Add debug logging for when sensor batchTimeout exceeds the
calculated maximum (mmiklavc) closes apache/metron#1437
---
.../src/main/java/org/apache/metron/writer/BatchTimeoutPolicy.java | 1 +
1 file changed, 1 insertion(+)
diff --git
a/metron-platform/metron-writer/metron-writer-common/src/main/java/org/apache/metron/writer/BatchTimeoutPolicy.java
b/metron-platform/metron-writer/metron-writer-common/src/main/java/org/apache/metron/writer/BatchTimeoutPolicy.java
index 8edd8ff..4c167d5 100644
---
a/metron-platform/metron-writer/metron-writer-common/src/main/java/org/apache/metron/writer/BatchTimeoutPolicy.java
+++
b/metron-platform/metron-writer/metron-writer-common/src/main/java/org/apache/metron/writer/BatchTimeoutPolicy.java
@@ -100,6 +100,7 @@ public class BatchTimeoutPolicy implements
FlushPolicy {
protected long getBatchTimeout(String sensorType, WriterConfiguration
configurations) {
int batchTimeoutSecs = configurations.getBatchTimeout(sensorType);
if (batchTimeoutSecs <= 0 || batchTimeoutSecs > maxBatchTimeout) {
+ LOG.debug("The configured batch timeout '{}' for sensor type '{}' is <=0
or > the maximum allowable batch timeout '{}'. Setting the batch timeout to the
maximum allowable.", batchTimeoutSecs, sensorType, maxBatchTimeout);
batchTimeoutSecs = maxBatchTimeout;
}
return TimeUnit.SECONDS.toMillis(batchTimeoutSecs);
[metron] branch master updated: METRON-2112 Normalize parser original_string handling (mmiklavc) closes apache/metron#1409
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 3754ff3 METRON-2112 Normalize parser original_string handling (mmiklavc) closes apache/metron#1409 3754ff3 is described below commit 3754ff33f6cd149ffca57474d744e0298d4c172a Author: mmiklavc AuthorDate: Thu May 30 14:54:17 2019 -0600 METRON-2112 Normalize parser original_string handling (mmiklavc) closes apache/metron#1409 --- metron-platform/metron-common/README.md| 4 +- .../java/org/apache/metron/common/Constants.java | 6 ++- .../data/jsonMapQuery/parsed/jsonMapExampleParsed | 20 +++ .../parsed/jsonMapExampleParsed| 12 ++--- metron-platform/metron-parsing/README.md | 18 +-- .../org/apache/metron/parsers/ParserComponent.java | 3 ++ .../apache/metron/parsers/ParserRunnerImpl.java| 2 + .../apache/metron/parsers/json/JSONMapParser.java | 21 +--- .../metron/parsers/ParserRunnerImplTest.java | 45 +++- .../parsers/json/JSONMapParserQueryTest.java | 61 +++--- .../metron/parsers/json/JSONMapParserTest.java | 14 +++-- .../json/JSONMapParserWrappedQueryTest.java| 12 ++--- .../metron-parsing/metron-parsing-storm/README.md | 8 ++- 13 files changed, 169 insertions(+), 57 deletions(-) diff --git a/metron-platform/metron-common/README.md b/metron-platform/metron-common/README.md index 4d19769..f3082a5 100644 --- a/metron-platform/metron-common/README.md +++ b/metron-platform/metron-common/README.md @@ -87,7 +87,7 @@ but a convenient index is provided here: | [`es.port`](../metron-elasticsearch#esport) | Indexing | String | N/A | | [`es.date.format`](../metron-elasticsearch#esdateformat) | Indexing | String | `es_date_format`| | [`es.client.settings`](../metron-elasticsearch#esclientsettings) | Indexing | Object | N/A | -| [`indexing.writer.elasticsearch.setDocumentId`](../metron-indexing#elasticsearch) | Indexing | Boolean| N/A | +| [`indexing.writer.elasticsearch.setDocumentId`](../metron-indexing#elasticsearch) | Indexing | Boolean| N/A | | [`solr.zookeeper`](../metron-solr#configuration) | Indexing | String | `solr_zookeeper_url`| | [`solr.commitPerBatch`](../metron-solr#configuration) | Indexing | String | N/A | | [`solr.commit.soft`](../metron-solr#configuration) | Indexing | String | N/A | @@ -96,7 +96,7 @@ but a convenient index is provided here: | [`solr.collection`](../metron-solr#configuration) | Indexing | String | N/A | | [`solr.http.config`](../metron-solr#configuration) | Indexing | String | N/A | | [`fieldValidations`](#validation-framework) | Parsing | Object | N/A | -| [`parser.error.topic`](../metron-parsers#parsererrortopic) | Parsing | String | `parser_error_topic`| +| [`parser.error.topic`](../metron-parsing#parsererrortopic) | Parsing | String | `parser_error_topic`| | [`stellar.function.paths`](../../metron-stellar/stellar-common#stellarfunctionpaths) | Stellar | CSV String | N/A | | [`stellar.function.resolver.includes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes) | Stellar | CSV String | N/A | | [`stellar.function.resolver.excludes`](../../metron-stellar/stellar-common#stellarfunctionresolverincludesexcludes) | Stellar | CSV String | N/A | diff --git a
[metron] branch master updated: METRON-2128 LEEF config file is missing in RPM spec file (simonellistonball via mmiklavc) closes apache/metron#1419
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new bfe662d METRON-2128 LEEF config file is missing in RPM spec file
(simonellistonball via mmiklavc) closes apache/metron#1419
bfe662d is described below
commit bfe662d6e7b07f8340067d7aaeb976068617b6b1
Author: simonellistonball
AuthorDate: Tue May 28 10:29:39 2019 -0600
METRON-2128 LEEF config file is missing in RPM spec file (simonellistonball
via mmiklavc) closes apache/metron#1419
---
.../packaging/docker/rpm-docker/SPECS/metron.spec | 1 +
.../src/main/config/zookeeper/parsers/leef.json| 0
.../org/apache/metron/parsers/leef/LEEFParser.java | 458 ++---
.../apache/metron/parsers/leef/LEEFParserTest.java | 396 +-
4 files changed, 428 insertions(+), 427 deletions(-)
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index dc40967..8b68b6f 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -192,6 +192,7 @@ This package installs the Metron Bundled Parser files
%{metron_home}/config/zookeeper/parsers/websphere.json
%{metron_home}/config/zookeeper/parsers/yaf.json
%{metron_home}/config/zookeeper/parsers/asa.json
+%{metron_home}/config/zookeeper/parsers/leef.json
%{metron_home}/patterns/asa
%{metron_home}/patterns/fireeye
%{metron_home}/patterns/sourcefire
diff --git
a/metron-platform/metron-parsing/metron-parsers-common/src/main/config/zookeeper/parsers/leef.json
b/metron-platform/metron-parsing/metron-parsers/src/main/config/zookeeper/parsers/leef.json
similarity index 100%
rename from
metron-platform/metron-parsing/metron-parsers-common/src/main/config/zookeeper/parsers/leef.json
rename to
metron-platform/metron-parsing/metron-parsers/src/main/config/zookeeper/parsers/leef.json
diff --git
a/metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
b/metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
index db19ebd..ea09714 100644
---
a/metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
+++
b/metron-platform/metron-parsing/metron-parsers/src/main/java/org/apache/metron/parsers/leef/LEEFParser.java
@@ -53,235 +53,235 @@ import org.slf4j.LoggerFactory;
*
*/
public class LEEFParser extends BasicParser {
- private static final long serialVersionUID = 1L;
-
- public enum HeaderFields {
- DEVICE_VENDOR("DeviceVendor"),
- DEVICE_PRODUCT("DeviceProduct"),
- DEVICE_VERSION("DeviceVersion"),
- DEVICE_EVENT("DeviceEvent"),
- DELIMITER("Delimiter"),
- VERSION("Version")
- ;
-
- private String name;
-
- HeaderFields(String name) {
- this.name = name;
- }
-
- public String getName() {
- return name;
- }
- }
-
- // Field name for custom device time in LEEF
- private static final String DEV_TIME = "devTime";
- private static final String DEV_TIME_FORMAT = "devTimeFormat";
-
- protected static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
- private static final String HEADER_CAPTURE_PATTERN = "[^\\|]*";
- private static final Charset UTF_8 = StandardCharsets.UTF_8;
-
- private Pattern pattern;
-
- public void init() {
-
- // LEEF Headers: Version|Device Vendor|Device Product|Device
Version|Device Event|Delimiter
- String syslogTime =
"(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\\b
+(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9])
(?!<[0-9])(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9])(?::(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?))(?![0-9])?";
- String syslogTime5424 =
"(?:\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}(?:\\.\\d+)?(?:Z|[+-]\\d{2}:\\d{2}))";
- String syslogPriority = "<(?:[0-9]+)>";
- String syslogHost = "[a-z0-9\\.-_]+";
-
- StringBuilder sb = new StringBuilder("");
- sb.append("(?");
- sb.append(syslogPriority);
- sb.append(")?");
- sb.append("(?");
- sb.append(syslogTime);
- sb.append(&q
[metron] branch master updated: METRON-2123 Expand Stellar JOIN to work on all Iterables (mmiklavc) closes apache/metron#1416
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 9b70adf METRON-2123 Expand Stellar JOIN to work on all Iterables
(mmiklavc) closes apache/metron#1416
9b70adf is described below
commit 9b70adf1d7f716c4a301af10c6726c40ecb1f3dc
Author: mmiklavc
AuthorDate: Fri May 24 11:51:47 2019 -0600
METRON-2123 Expand Stellar JOIN to work on all Iterables (mmiklavc) closes
apache/metron#1416
---
metron-stellar/stellar-common/README.md | 4 ++--
.../stellar/dsl/functions/StringFunctions.java | 20 +---
.../stellar/dsl/functions/BasicStellarTest.java | 2 ++
3 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/metron-stellar/stellar-common/README.md
b/metron-stellar/stellar-common/README.md
index 551a52a..5e48b1c 100644
--- a/metron-stellar/stellar-common/README.md
+++ b/metron-stellar/stellar-common/README.md
@@ -702,9 +702,9 @@ Where:
* Returns: True if the string is a valid URL and false if otherwise.
### `JOIN`
- * Description: Joins the components in the list of strings with the
specified delimiter.
+ * Description: Joins the non-null items in the iterable as strings with the
specified delimiter. Null items are dropped.
* Input:
-* list - List of strings
+* iterable - Java iterable (e.g. List, LinkedHashSet, etc.) of items
treated as strings
* delim - String delimiter
* Returns: String
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
index 5019852..89e5f61 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/StringFunctions.java
@@ -19,22 +19,20 @@
package org.apache.metron.stellar.dsl.functions;
import com.fasterxml.jackson.core.JsonProcessingException;
-import com.fasterxml.jackson.databind.ObjectMapper;
import com.google.common.base.Joiner;
import com.google.common.base.Splitter;
import com.google.common.collect.Iterables;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.metron.stellar.common.utils.JSONUtils;
-import org.apache.metron.stellar.dsl.BaseStellarFunction;
-import org.apache.metron.stellar.dsl.ParseException;
-import org.apache.metron.stellar.dsl.Stellar;
-import org.apache.metron.stellar.common.utils.ConversionUtils;
-
import java.io.IOException;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.stellar.common.utils.JSONUtils;
+import org.apache.metron.stellar.dsl.BaseStellarFunction;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.Stellar;
public class StringFunctions {
@@ -133,15 +131,15 @@ public class StringFunctions {
}
@Stellar( name="JOIN"
- , description="Joins the components in the list of strings with the
specified delimiter."
- , params = { "list - List of strings", "delim - String delimiter"}
+ , description="Joins the non-null items in the iterable as strings
with the specified delimiter. Null items are dropped."
+ , params = { "iterable - Java iterable (e.g. List, LinkedHashSet,
etc.) of items treated as strings", "delim - String delimiter"}
, returns = "String"
)
public static class JoinFunction extends BaseStellarFunction {
@Override
@SuppressWarnings("unchecked")
public Object apply(List args) {
- List arg1 = (List) args.get(0);
+ Iterable arg1 = (Iterable) args.get(0);
String delim = (String) args.get(1);
return Joiner.on(delim).join(Iterables.filter(arg1, x -> x != null));
}
diff --git
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
index 4b64f72..c2f5ca0 100644
---
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
+++
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/BasicStellarTest.java
@@ -620,6 +620,8 @@ public class BasicStellarTest {
public void testJoin() {
String query = "JOIN( [ TO_UPPER(TRIM(foo)), 'bar' ], ',')";
Assert.assertEquals("CASEY,bar", run(query, Immutable
[metron] branch master updated: METRON-1788 Batch profiler pull profile information from zookeeper (tigerquoll via mmiklavc) closes apache/metron#1383
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 3b04460 METRON-1788 Batch profiler pull profile information from
zookeeper (tigerquoll via mmiklavc) closes apache/metron#1383
3b04460 is described below
commit 3b0446006c327b7dea494b3a0bcbb9de4f662d5a
Author: tigerquoll
AuthorDate: Mon May 20 18:05:50 2019 -0600
METRON-1788 Batch profiler pull profile information from zookeeper
(tigerquoll via mmiklavc) closes apache/metron#1383
---
metron-analytics/metron-profiler-spark/README.md | 61 --
metron-analytics/metron-profiler-spark/pom.xml | 6 +
.../profiler/spark/cli/BatchProfilerCLI.java | 126 +++--
.../spark/cli/BatchProfilerCLIOptions.java | 15 ++-
.../src/main/scripts/start_batch_profiler.sh | 18 ++-
.../profiler/spark/cli/BatchProfilerCLITest.java | 28 -
.../spark/cli/BatchProfilerZKIntegrationTest.java | 82 ++
.../apache/metron/integration/TestZKServer.java| 79 +
8 files changed, 395 insertions(+), 20 deletions(-)
diff --git a/metron-analytics/metron-profiler-spark/README.md
b/metron-analytics/metron-profiler-spark/README.md
index 5ee8510..8750550 100644
--- a/metron-analytics/metron-profiler-spark/README.md
+++ b/metron-analytics/metron-profiler-spark/README.md
@@ -42,8 +42,7 @@ The portion of a profile produced by the Batch Profiler
should be indistinguisha
For an introduction to the Profiler, see the [Profiler
README](../metron-profiler-common/README.md).
## Getting Started
-
-1. Create a profile definition by editing
`$METRON_HOME/config/zookeeper/profiler.json` as follows.
+1. If a profile file does not already exist, you can create a profile
definition by editing `$METRON_HOME/config/zookeeper/profiler.json` as follows.
```
cat $METRON_HOME/config/zookeeper/profiler.json
@@ -60,7 +59,8 @@ For an introduction to the Profiler, see the [Profiler
README](../metron-profile
"timestampField": "timestamp"
}
```
-
+See [Specifying profiles](#specifying-profiles) for information on how to
load profile definitions from zookeeper.
+
1. Ensure that you have archived telemetry available for the Batch Profiler to
consume. By default, Metron will store this in HDFS at
`/apps/metron/indexing/indexed/*/*`.
```
@@ -80,7 +80,6 @@ For an introduction to the Profiler, see the [Profiler
README](../metron-profile
```
log4j.logger.org.apache.metron.profiler.spark=DEBUG
```
-
1. Run the Batch Profiler.
```
@@ -91,6 +90,41 @@ For an introduction to the Profiler, see the [Profiler
README](../metron-profile
1. Query for the profile data using the [Profiler
Client](../metron-profiler-client/README.md).
+## Specifying profiles
+
+The profile to use for batch processing can be specified as either a JSON file
on disk
+or by utilizing a profile already loaded into zookeeper for use by the
streaming profiler.
+
+### Loading a profile from disk
+
+1. If a profile file does not already exist, you can create a profile
definition by editing `$METRON_HOME/config/zookeeper/profiler.json` as follows.
+
+```
+cat $METRON_HOME/config/zookeeper/profiler.json
+{
+ "profiles": [
+{
+ "profile": "hello-world",
+ "foreach": "'global'",
+ "init":{ "count": "0" },
+ "update": { "count": "count + 1" },
+ "result": "count"
+}
+ ],
+ "timestampField": "timestamp"
+}
+```
+1. When launching the batch profiler directly, use the `--profiles ` option.
+If using the wrapper script to launch the batch profiler, it will
automatically add the command argument
+`--profiles $METRON_HOME/config/zookeeper/profiler.json ` to the batch
launching process if `$SPARK_PROFILER_USE_ZOOKEEPER` is not defined.
+
+### Loading a profile from zookeeper
+
+Choose to use profiles already loaded into zookeeper (e.g. for use by the
streaming profiler) by setting the environment variable
`$SPARK_PROFILER_USE_ZOOKEEPER`.
+This will cause the wrapper script to add `--zookeeper $ZOOKEEPER` to the
batch launching process,
+which will cause the spark profiler to extract profiles from the zookeeper
quorum located at `$ZOOKEEPER`.
+
+
## Installation
The Batch Profiler package is installed automatically when installing Metron
using the Ambari MPack. See the following notes when installing the Batch
Profiler without the Ambari MPack.
@@ -147,9 +181,11 @@ The Batch Profiler requires Spark version 2.3.0+.
A script located at `$METRON_HOME/bin/start_batch_profiler.sh` has be
[metron] branch master updated: METRON-2118 Added a LEEF parser (simonellistonball via mmiklavc) closes apache/metron#1408
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 22cc622 METRON-2118 Added a LEEF parser (simonellistonball via
mmiklavc) closes apache/metron#1408
22cc622 is described below
commit 22cc622d6f08cfa77c06dea2df7f4c56edc1862e
Author: simonellistonball
AuthorDate: Mon May 20 14:25:41 2019 -0600
METRON-2118 Added a LEEF parser (simonellistonball via mmiklavc) closes
apache/metron#1408
---
.../src/main/sample/data/leef/parsed/leefParsed| 4 +
.../src/main/sample/data/leef/raw/leefOutput | 4 +
metron-platform/metron-parsing/README.md | 2 +
.../src/main/config/zookeeper/parsers/leef.json| 4 +
.../org/apache/metron/parsers/utils/DateUtils.java | 9 +
.../parsers/integration/ParserIntegrationTest.java | 3 +-
.../org/apache/metron/parsers/cef/CEFParser.java | 97 ---
.../org/apache/metron/parsers/leef/LEEFParser.java | 287 +
.../apache/metron/parsers/cef/CEFParserTest.java | 57 ++--
.../apache/metron/parsers/leef/LEEFParserTest.java | 243 +
.../org/apache/metron/parsers/leef/sample.leef | 4 +
.../org/apache/metron/parsers/leef/sample.schema | 27 ++
12 files changed, 659 insertions(+), 82 deletions(-)
diff --git
a/metron-platform/metron-integration-test/src/main/sample/data/leef/parsed/leefParsed
b/metron-platform/metron-integration-test/src/main/sample/data/leef/parsed/leefParsed
new file mode 100644
index 000..0eb5949
--- /dev/null
+++
b/metron-platform/metron-integration-test/src/main/sample/data/leef/parsed/leefParsed
@@ -0,0 +1,4 @@
+{"msg":"Alert: CPUWarning Threshold Exceeded\\nSubject:
10.201.114.164\\nSeverity:Warning","DeviceEvent":"192","sev":"3","TrendMicroDsTenant":"Primary","DeviceVersion":"","original_string":"LEEF:2.0|Trend Micro|Deep Security Manager||192|cat=System\tname=Alert Ended\tdesc=Alert: CPU Warning Threshold
Exceeded\\nSubject: 10.201.114.164\\nSeverity:
Warning\tsev=3\tsrc=10.201.114.164\tusrName=System\tmsg=Alert: CPUWarning
Threshold Exceeded\\nSubject: 10.201.114.16 [...]
+{"DeviceEvent":"2002779","TrendMicroDsTenantId":"0","sev":"8
cn1","TrendMicroDsTenant":"Primary","DeviceVersion":"","act":"updated","original_string":"LEEF:2.0|Trend Micro|Deep Security
Agent||2002779|cat=Integrity Monitor\tname=Microsoft Windows -
System file modified\tdesc=Microsoft Windows - System file modified\tsev=8
cn1=37 cn1Label=Host
ID\tdvchost=www.example.com\tTrendMicroDsTenant=Primary\tTrendMicroDsTenantId=0\tact=updated\tsuser=admin","suser":"admin
[...]
+{"msg":"Realtime","DeviceEvent":"430","TrendMicroDsTenantId":"0","sev":"6
cn1","TrendMicroDsTenant":"Primary","filePath":"C:WindowsSystem32virus.exe","DeviceVersion":"","TrendMicroDsTags":"FS","TrendMicroDsMalwareTarget":"Multiple","dvc":"10.0.0.1","TrendMicroDsRelevantDetectionNames":"Ransom_CERBER.BZC;Ransom_CERBER.C;Ransom_CRYPNISCA.SM","act":"Terminate","original_string":"LEEF:2.0|Trend
Micro|Deep Security Agent||430|cat=Anti-Malware\tnam [...]
+{"msg":"Suspicious","DeviceEvent":"500","request":"http:\/\/yw.olx5x9ny.org.it\/HvuauRH\/eighgSS.htm","TrendMicroDsTenantId":"0","sev":"6
cn1","TrendMicroDsTenant":"Primary","DeviceVersion":"","original_string":"LEEF:2.0|Trend Micro|Deep Security Agent||500|cat=Web
Reputation\tname=WebReputation\tdesc=WebReputation\tsev=6 cn1=3 cn1Label=Host
ID\tdvchost=exch01.example.com\tTrendMicroDsTenant=Primary\tTrendMicroDsTenantId=0\trequest=http:\/\/yw.olx5x9ny.o
[...]
\ No newline at end of file
diff --git
a/metron-platform/metron-integration-test/src/main/sample/data/leef/raw/leefOutput
b/metron-platform/metron-integration-test/src/main/sample/data/leef/raw/leefOutput
new file mode 100644
index 000..41af31c
--- /dev/null
+++
b/metron-platform/metron-integration-test/src/main/sample/data/leef/raw/leefOutput
@@ -0,0 +1,4 @@
+LEEF:2.0|Trend Micro|Deep Security Manager||192|cat=System
name=Alert Endeddesc=Alert: CPU Warning Threshold Exceeded\nSubject:
10.201.114.164\nSeverity: Warning sev=3 src=10.201.114.164
usrName=System msg=Alert: CPUWarning Thr
[metron] branch master updated: METRON-2107 Add architecture diagram item to PR checklist (mmiklavc) closes apache/metron#1401
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new b8df052 METRON-2107 Add architecture diagram item to PR checklist (mmiklavc) closes apache/metron#1401 b8df052 is described below commit b8df052fabe4e00a1a30e20f958f6e07c4fb4fef Author: mmiklavc AuthorDate: Thu May 16 08:00:41 2019 -0600 METRON-2107 Add architecture diagram item to PR checklist (mmiklavc) closes apache/metron#1401 --- .github/PULL_REQUEST_TEMPLATE.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index af6c1e7..4c999a7 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -37,6 +37,8 @@ In order to streamline the review of the contribution we ask you follow these gu mvn site ``` +- [ ] Have you ensured that any documentation diagrams have been updated, along with their source files, using [draw.io](https://www.draw.io/)? See [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) for instructions. + Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.
[metron] branch master updated: METRON-1989 Tooltip for ES mpack path_data is incorrect (JonZeolla via mmiklavc) closes apache/metron#1329
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new fb80e32 METRON-1989 Tooltip for ES mpack path_data is incorrect (JonZeolla via mmiklavc) closes apache/metron#1329 fb80e32 is described below commit fb80e32e04f5e02ba1b10bd1a915047e42480819 Author: JonZeolla AuthorDate: Mon May 13 17:03:41 2019 -0600 METRON-1989 Tooltip for ES mpack path_data is incorrect (JonZeolla via mmiklavc) closes apache/metron#1329 --- .../common-services/ELASTICSEARCH/5.6.14/configuration/elastic-site.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/configuration/elastic-site.xml b/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/configuration/elastic-site.xml index 440c66d..bb6773c 100755 --- a/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/configuration/elastic-site.xml +++ b/metron-deployment/packaging/ambari/elasticsearch-mpack/src/main/resources/common-services/ELASTICSEARCH/5.6.14/configuration/elastic-site.xml @@ -54,7 +54,7 @@ path_data "/opt/lmm/es_data" -Comma-separated list of directories where to store index data allocated for each node: "/mnt/first","/mnt/second". Number of paths should relate to number of shards, and preferably should be on separate physical volumes. +Comma-separated list of directories where to store index data allocated for each node: [ "/mnt/first", "/mnt/second" ]. Number of paths should relate to number of shards, and preferably should be on separate physical volumes. http_cors_enabled
[metron] branch master updated: METRON-2075 Site book build support for MacOS that has GNU sed installed (tigerquoll via mmiklavc) closes apache/metron#1384
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new d92e0a2 METRON-2075 Site book build support for MacOS that has GNU
sed installed (tigerquoll via mmiklavc) closes apache/metron#1384
d92e0a2 is described below
commit d92e0a2102306ce6d6b6b2a8c986590b030d4a88
Author: tigerquoll
AuthorDate: Mon May 13 16:59:01 2019 -0600
METRON-2075 Site book build support for MacOS that has GNU sed installed
(tigerquoll via mmiklavc) closes apache/metron#1384
---
site-book/bin/generate-md.sh | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/site-book/bin/generate-md.sh b/site-book/bin/generate-md.sh
index 1d1e6f8..219e2b5 100755
--- a/site-book/bin/generate-md.sh
+++ b/site-book/bin/generate-md.sh
@@ -264,8 +264,9 @@ for (( i=0; i<${#HREF_REWRITE_LIST[@]} ; i+=2 )) ; do
sed -i -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
;;
darwin*)
+ # Use absolute path to ensure that MacOS sed is being used
# MacOS sed needs an empty-string argument after -i option to get
the same result
-sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
+/usr/bin/sed -i '' -e "${HREF_REWRITE_LIST[ $(( i + 1 )) ]}"
"${HREF_REWRITE_LIST[$i]}"
;;
*)
echo "ERROR: Unable to determine 'sed' argument list for OS
${OSTYPE}" > /dev/stderr
[metron] branch master updated: METRON-2100 Update developer documentation for full dev management UI parser aggregation feature gap (mmiklavc) closes apache/metron#1398
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new e1d1901 METRON-2100 Update developer documentation for full dev management UI parser aggregation feature gap (mmiklavc) closes apache/metron#1398 e1d1901 is described below commit e1d1901395d7b135deb48c3c4af5a94ed35fbaa4 Author: mmiklavc AuthorDate: Wed May 8 10:10:21 2019 -0600 METRON-2100 Update developer documentation for full dev management UI parser aggregation feature gap (mmiklavc) closes apache/metron#1398 --- Upgrading.md| 7 --- metron-deployment/README.md | 35 +++ 2 files changed, 39 insertions(+), 3 deletions(-) diff --git a/Upgrading.md b/Upgrading.md index b3ab8d2..d59aa57 100644 --- a/Upgrading.md +++ b/Upgrading.md @@ -19,13 +19,14 @@ limitations under the License. This document constitutes a per-version listing of changes of configuration which are non-backwards compatible. -## 0.7.1 to 0.7.2 +## 0.7.0 to 0.7.1 + +### [METRON-2100: Update developer documentation for full dev management UI parser aggregation feature gap](https://issues.apache.org/jira/browse/METRON-2100) +The original full_dev environment change was actually introduced in Metron 0.7.0. This Jira addresses missing user documentation for the Management UI feature gap for parser aggregation. See [Parser Aggregation Feature](metron-deployment#parser-aggregation-feature) for more details on how to work with and configure parsers with this feature change enabled in full_dev. ### [METRON-2053: Refactor metron-enrichment to decouple Storm dependencies](https://issues.apache.org/jira/browse/METRON-2053) `org.apache.metron.enrichment.writer.SimpleHbaseEnrichmentWriter` has had its packaged changed to `org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter`. It has also been moved from the `metron-platform/metron-enrichment` module to a more appropriate home in `metron-platform/metron-writer`. -## 0.7.0 to 0.7.1 - ### [METRON-1929: Build GET_ASN Stellar function](https://issues.apache.org/jira/browse/METRON-1929) The script for `geo_enrichment_load.sh` has been renamed, and now is `maxmind_enrichment_load.sh`. A couple changes should happen for users who are upgrading. diff --git a/metron-deployment/README.md b/metron-deployment/README.md index b78a3e2..73cf602 100644 --- a/metron-deployment/README.md +++ b/metron-deployment/README.md @@ -73,6 +73,41 @@ To deploy Metron in a VM running on your computer, follow the instructions at [d We recommend looking at Ambari and shutting down any services you may not be using. For example, we recommend turning off Metron Profiler, as this commonly causes REST services to crash when running on a single VM. +### Parser Aggregation Feature + +The [Parser Aggregation](../metron-platform/metron-parsing/metron-parsing-storm#parser-aggregation) feature does not currently exist in the management UI. In order to address resource limitations in the full dev development environments, bro, yaf, and snort have been aggregated into a single parser +topology. However, the Management UI is not currently able to display its status until the feature is added. Aggregated parsers can still be created via Ambari and the command line scripts. + +Here are some tips for working with parser aggregation while the UI feature is being developed. + +* **How are parsers picked up by the UI?:** This is based entirely on what is currently stored in the Zookeeper configs. See [Management Utility](../metron-platform/metron-common#management-utility) "DUMP" option with "-c PARSER" to see all of what is currently loaded. The management UI does not +update the configurations stored locally on disk, so Zookeeper is the source of truth. + +* **Removing an existing aggregation:** In the [Ambari UI](http://node1:8080) click on the Metron serice and select "Metron Parsers." Select "stop" from the dropdown for the parser component. Click "back," "configs," and then navigate to "Parsers." In the text field option labeled "parsers". +Remove the double quotes from around the listed parsers. Save and choose "Restart" when prompted. This will deploy three individual parsers rather than a single aggregated parser: bro, snort, and yaf. Be aware, you may need to shut down other topologies to free up resources so that you can +run the parsers without aggregation. Stopping the profiler, pcap, or batch_indexing are a few options that will still allow data to pass through the system end-to-end. + +* **Managing parser lifecycle:** Starting and stopping parsers in the management UI will in no way affect a parser running as aggregated. The exception to this is if you create a parser via
[metron] branch master updated: METRON-2067 Maven pom file duplicate dependency fixes (mmiklavc) closes apache/metron#1379
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 7f6ad42 METRON-2067 Maven pom file duplicate dependency fixes
(mmiklavc) closes apache/metron#1379
7f6ad42 is described below
commit 7f6ad42e534eeaf9423f33c3a6ea1d3d131049f5
Author: mmiklavc
AuthorDate: Wed Apr 17 12:49:36 2019 -0600
METRON-2067 Maven pom file duplicate dependency fixes (mmiklavc) closes
apache/metron#1379
---
metron-analytics/metron-profiler-storm/pom.xml| 16
metron-platform/metron-parsing/metron-parsers/pom.xml | 3 ++-
metron-platform/metron-writer/pom.xml | 6 --
3 files changed, 6 insertions(+), 19 deletions(-)
diff --git a/metron-analytics/metron-profiler-storm/pom.xml
b/metron-analytics/metron-profiler-storm/pom.xml
index 30566fd..66deb4d 100644
--- a/metron-analytics/metron-profiler-storm/pom.xml
+++ b/metron-analytics/metron-profiler-storm/pom.xml
@@ -170,6 +170,10 @@
org.hamcrest
hamcrest-core
+
+org.slf4j
+slf4j-log4j12
+
@@ -313,18 +317,6 @@
org.apache.metron
-metron-integration-test
-${project.parent.version}
-test
-
-
-org.slf4j
-slf4j-log4j12
-
-
-
-
-org.apache.metron
metron-test-utilities
${project.parent.version}
test
diff --git a/metron-platform/metron-parsing/metron-parsers/pom.xml
b/metron-platform/metron-parsing/metron-parsers/pom.xml
index 9645fb9..9b82754 100644
--- a/metron-platform/metron-parsing/metron-parsers/pom.xml
+++ b/metron-platform/metron-parsing/metron-parsers/pom.xml
@@ -143,7 +143,6 @@
-
@@ -164,6 +163,8 @@
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
+
+
diff --git a/metron-platform/metron-writer/pom.xml
b/metron-platform/metron-writer/pom.xml
index 2205cec..f8ae4c4 100644
--- a/metron-platform/metron-writer/pom.xml
+++ b/metron-platform/metron-writer/pom.xml
@@ -245,12 +245,6 @@
stellar-common
${project.parent.version}
-
-org.apache.httpcomponents
-httpclient
-${global_httpclient_version}
-test
-
[metron] branch master updated: METRON-2074 Script to handle TGT renewal with Storm and Kerberos enabled (mmiklavc) closes apache/metron#1382
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new b6d8cad METRON-2074 Script to handle TGT renewal with Storm and
Kerberos enabled (mmiklavc) closes apache/metron#1382
b6d8cad is described below
commit b6d8cade6aec7fe6290d4fd5660ff205de06bd64
Author: mmiklavc
AuthorDate: Wed Apr 17 12:09:14 2019 -0600
METRON-2074 Script to handle TGT renewal with Storm and Kerberos enabled
(mmiklavc) closes apache/metron#1382
---
metron-deployment/Kerberos-manual-setup.md | 45 +++
.../packaging/docker/rpm-docker/SPECS/metron.spec | 1 +
.../metron-common/src/main/scripts/tgt_renew.py| 90 ++
3 files changed, 136 insertions(+)
diff --git a/metron-deployment/Kerberos-manual-setup.md
b/metron-deployment/Kerberos-manual-setup.md
index ecb0cb7..d50da34 100644
--- a/metron-deployment/Kerberos-manual-setup.md
+++ b/metron-deployment/Kerberos-manual-setup.md
@@ -31,6 +31,7 @@ This document provides instructions for kerberizing Metron's
Vagrant-based devel
* [Push Data](#push-data)
* [More Information](#more-information)
* [Elasticseach X-Pack](#X-Pack)
+* [TGT Ticket Renew](#tgt-ticket-renew)
Setup
-
@@ -636,3 +637,47 @@ The random access indexer topology fails with the
following exception. This exc
Solution
This can occur when an HDFS Client is not installed on the Storm worker nodes.
This might occur on any Storm worker node where an HDFS Client is not
installed. Installing the HDFS Client on all Storm worker nodes should resolve
the problem.
+
+## TGT Ticket Renew
+
+Apache Storm doesn't handle automatic TGT ticket renewal for their running
topologies. Instead, it is left up to the operations team deploying the Storm
topologies
+in a Kerberized environment to manage this themselves. We've included a Python
script that can be setup with a cron process to automatically manage the renewal
+process for you. The script should be run on an interval that is shorter than
the renew_lifetime configured for your TGT.
+
+### Setup Instructions
+
+Run the following on a node with a Storm and Metron client installed. We need
python 2.7 via virtualenv for this to work correctly.
+
+```
+# run yum commands as root
+for item in epel-release centos-release-scl "@Development tools" python27
python27-scldevel python27-python-virtualenv libselinux-python; do yum install
-y $item; done
+sudo yum install -y gcc krb5-devel python-devel
+sudo yum install -y libffi libffi-devel
+sudo yum install -y python-cffi
+sudo yum install -y openssl-devel
+# setup python with metron user
+su - metron
+export PYTHON27_HOME=/opt/rh/python27/root
+export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
+mkdir project_dir
+cd project_dir
+${PYTHON27_HOME}/usr/bin/virtualenv venv
+source venv/bin/activate
+pip install --upgrade setuptools==18.5
+pip install requests-kerberos
+```
+
+The script `$METRON_HOME/bin/tgt_renew.py` takes two arguments:
+
+* arg1 = host:port for Storm UI server
+* arg2 = topology owner - typically "metron" for a kerberized cluster with
metron topologies
+
+Execute it like the following example:
+
+```
+# run as the metron user
+su - metron
+python $METRON_HOME/bin/tgt_renew.py node1:8744 metron
+```
+
+
diff --git a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
index fbffe28..dc40967 100644
--- a/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
+++ b/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec
@@ -136,6 +136,7 @@ This package installs the Metron common files %{metron_home}
%{metron_home}/bin/zk_load_configs.sh
%{metron_home}/bin/stellar
%{metron_home}/bin/cluster_info.py
+%{metron_home}/bin/tgt_renew.py
%{metron_home}/config/zookeeper/global.json
%attr(0644,root,root) %{metron_home}/lib/metron-common-%{full_version}.jar
diff --git a/metron-platform/metron-common/src/main/scripts/tgt_renew.py
b/metron-platform/metron-common/src/main/scripts/tgt_renew.py
new file mode 100755
index 000..b0a8d0f
--- /dev/null
+++ b/metron-platform/metron-common/src/main/scripts/tgt_renew.py
@@ -0,0 +1,90 @@
+"""
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distribu
[metron] branch master updated: METRON-2071 Add MAP_PUT and MAP_MERGE to Stellar (mmiklavc) closes apache/metron#1385
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new dc273a5 METRON-2071 Add MAP_PUT and MAP_MERGE to Stellar (mmiklavc) closes apache/metron#1385 dc273a5 is described below commit dc273a5be628c458ad0b25380d55536cd37b7a4f Author: mmiklavc AuthorDate: Tue Apr 16 12:37:16 2019 -0600 METRON-2071 Add MAP_PUT and MAP_MERGE to Stellar (mmiklavc) closes apache/metron#1385 --- metron-stellar/stellar-common/README.md| 26 -- .../metron/stellar/dsl/functions/MapFunctions.java | 93 +--- .../stellar/dsl/functions/BasicStellarTest.java| 99 ++ 3 files changed, 201 insertions(+), 17 deletions(-) diff --git a/metron-stellar/stellar-common/README.md b/metron-stellar/stellar-common/README.md index 8301f6d..261be59 100644 --- a/metron-stellar/stellar-common/README.md +++ b/metron-stellar/stellar-common/README.md @@ -165,7 +165,7 @@ Where: | -- | | [ `ABS`](../../metron-analytics/metron-statistics#abs) | | [ `APPEND_IF_MISSING`](#append_if_missing) | -| [ `ASN_GET`](#asn_get) | +| [ `ASN_GET`](#asn_get) | | [ `BIN`](../../metron-analytics/metron-statistics#bin) | | [ `BLOOM_ADD`](#bloom_add) | | [ `BLOOM_EXISTS`](#bloom_exists) | @@ -176,7 +176,7 @@ Where: | [ `CHOP`](#chop) | | [ `CHOMP`](#chomp) | | [ `COUNT_MATCHES`](#count_matches) | -| [ `DATE_FORMAT`](#date_format) +| [ `DATE_FORMAT`](#date_format) | | [ `DAY_OF_MONTH`](#day_of_month) | | [ `DAY_OF_WEEK`](#day_of_week) | | [ `DAY_OF_YEAR`](#day_of_year) | @@ -238,6 +238,8 @@ Where: | [ `MAP`](#map) | | [ `MAP_EXISTS`](#map_exists) | | [ `MAP_GET`](#map_get) | +| [ `MAP_MERGE`](#map_merge) | +| [ `MAP_PUT`](#map_put) | | [ `MAX`](#MAX) | | [ `MIN`](#MIN) | | [ `MONTH`](#month) | @@ -249,15 +251,15 @@ Where: | [ `OBJECT_GET`](#object_get) | | [ `PREPEND_IF_MISSING`](#prepend_if_missing) | | [ `PROFILE_GET`](#profile_get) | -| [ `PROFILE_VERBOSE`](#profile_verbose) | +| [ `PROFILE_VERBOSE`](#profile_verbose) | | [ `PROFILE_FIXED`](#profile_fixed) | | [ `PROFILE_WINDOW`](#profile_window) | | [ `PROTOCOL_TO_NAME`](#protocol_to_name) | | [ `REDUCE`](#reduce) | | [ `REGEXP_MATCH`](#regexp_match) | | [ `REGEXP_GROUP_VAL`](#regexp_group_val) | -| [ `REGEXP_REPLACE`](#regexp_replace) -| [ `REST_GET`](#rest_get) +| [ `REGEXP_REPLACE`](#regexp_replace) | +| [ `REST_GET`](#rest_get
[metron] branch master updated: METRON-2062 Metron Alerts: Accidentally commited 'fdescribe' in unit tests (ruffle1986 via mmiklavc) closes apache/metron#1372
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 80ae71f METRON-2062 Metron Alerts: Accidentally commited 'fdescribe'
in unit tests (ruffle1986 via mmiklavc) closes apache/metron#1372
80ae71f is described below
commit 80ae71f5ebeb3c6761951a70871fee208448f3e1
Author: ruffle1986
AuthorDate: Mon Apr 15 14:58:03 2019 -0600
METRON-2062 Metron Alerts: Accidentally commited 'fdescribe' in unit tests
(ruffle1986 via mmiklavc) closes apache/metron#1372
---
.../app/shared/time-range/time-range.component.spec.ts | 18 +++---
.../metron-alerts/src/app/utils/utils.spec.ts | 2 +-
2 files changed, 12 insertions(+), 8 deletions(-)
diff --git
a/metron-interface/metron-alerts/src/app/shared/time-range/time-range.component.spec.ts
b/metron-interface/metron-alerts/src/app/shared/time-range/time-range.component.spec.ts
index 51c64b2..15a12c1 100644
---
a/metron-interface/metron-alerts/src/app/shared/time-range/time-range.component.spec.ts
+++
b/metron-interface/metron-alerts/src/app/shared/time-range/time-range.component.spec.ts
@@ -61,31 +61,35 @@ describe('TimeRangeComponent', () => {
component.datePickerFromDate = '2000-01-31 00:00:00';
component.datePickerToDate = '2000-02-28 00:00:00';
+ const fromTS = new Date(component.datePickerFromDate).getTime();
+ const toTS = new Date(component.datePickerToDate).getTime();
+
spyOn(component.timeRangeChange, 'emit');
component.applyCustomDate();
- const filter = new Filter(TIMESTAMP_FIELD_NAME, '[94927320 TO
95169240]', false);
- filter.dateFilterValue = new DateFilterValue(94927320, 95169240);
+ const filter = new Filter(TIMESTAMP_FIELD_NAME, `[${fromTS} TO
${toTS}]`, false);
+ filter.dateFilterValue = new DateFilterValue(fromTS, toTS);
expect(component.timeRangeChange.emit).toHaveBeenCalledWith(filter);
});
it('should apply current date and time if To field empty', () => {
- const currentTs = new Date().getTime();
+ jasmine.clock().mockDate(new Date('2000-02-01T12:00:01'));
component.datePickerFromDate = '2000-01-31 00:00:00';
component.datePickerToDate = '';
+ const fromTS = new Date(component.datePickerFromDate).getTime();
+ const currentTs = new Date().getTime();
+
spyOn(component.timeRangeChange, 'emit');
component.applyCustomDate();
- const filter = new Filter(TIMESTAMP_FIELD_NAME, `[94927320 TO
${currentTs}]`, false);
- filter.dateFilterValue = new DateFilterValue(94927320, currentTs);
+ const filter = new Filter(TIMESTAMP_FIELD_NAME, `[${fromTS} TO
${currentTs}]`, false);
+ filter.dateFilterValue = new DateFilterValue(fromTS, currentTs);
expect(component.timeRangeChange.emit).toHaveBeenCalledWith(filter);
});
-
-
});
describe('Quick Ranges', () => {
diff --git a/metron-interface/metron-alerts/src/app/utils/utils.spec.ts
b/metron-interface/metron-alerts/src/app/utils/utils.spec.ts
index b64c461..ef68685 100644
--- a/metron-interface/metron-alerts/src/app/utils/utils.spec.ts
+++ b/metron-interface/metron-alerts/src/app/utils/utils.spec.ts
@@ -18,7 +18,7 @@
import { Utils } from './utils';
import { DateFilterValue } from 'app/model/date-filter-value';
-fdescribe('utils.Utils', () => {
+describe('utils.Utils', () => {
it('Converting time range based on From/To', () => {
expect(Utils.timeRangeToDateObj('94927320 TO
95169240')).toEqual(new DateFilterValue(94927320, 95169240));
[metron] branch master updated: METRON-2050 Automatically populate a list of enrichments from HBase (mmiklavc) closes apache/metron#1365
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 5709548 METRON-2050 Automatically populate a list of enrichments from HBase (mmiklavc) closes apache/metron#1365 5709548 is described below commit 57095488f28a017e290e021fea65412e7240e8da Author: mmiklavc AuthorDate: Fri Apr 12 12:50:00 2019 -0600 METRON-2050 Automatically populate a list of enrichments from HBase (mmiklavc) closes apache/metron#1365 --- metron-deployment/packaging/ambari/README.md | 2 +- .../configuration/metron-enrichment-env.xml| 24 ++ .../common-services/METRON/CURRENT/metainfo.xml| 4 +- .../CURRENT/package/scripts/enrichment_commands.py | 117 ++-- .../CURRENT/package/scripts/enrichment_master.py | 6 +- .../CURRENT/package/scripts/metron_service.py | 16 +- .../CURRENT/package/scripts/params/params_linux.py | 20 +- .../package/scripts/params/status_params.py| 13 +- .../packaging/docker/rpm-docker/SPECS/metron.spec | 23 ++ .../packaging/docker/rpm-docker/pom.xml| 6 + metron-interface/metron-rest/README.md | 2 +- metron-interface/metron-rest/pom.xml | 19 ++ .../org/apache/metron/rest/config/HBaseConfig.java | 33 ++- .../service/SensorEnrichmentConfigService.java | 8 +- .../rest/service/impl/AlertsUIServiceImpl.java | 23 +- .../impl/SensorEnrichmentConfigServiceImpl.java| 40 ++- .../metron/rest/user}/UserSettingsClient.java | 19 +- .../apache/metron/rest/config/HBaseConfigTest.java | 23 +- .../org/apache/metron/rest/config/TestConfig.java | 29 +- ...rEnrichmentConfigControllerIntegrationTest.java | 33 ++- .../rest/service/impl/AlertsUIServiceImplTest.java | 11 +- .../SensorEnrichmentConfigServiceImplTest.java | 43 +-- .../metron/rest/user}/UserSettingsClientTest.java | 30 +- .../src/test/resources/zookeeper/global.json | 2 +- metron-platform/metron-common/README.md| 73 ++--- .../configuration/EnrichmentConfigurations.java| 5 + metron-platform/metron-data-management/pom.xml | 12 + .../metron/dataloads/hbase/mr/HBaseUtil.java | 54 ++-- metron-platform/metron-enrichment/README.md| 14 + metron-platform/metron-hbase-server/README.md | 44 +++ metron-platform/metron-hbase-server/pom.xml| 326 + .../src/main/assembly/assembly.xml | 54 .../hbase/coprocessor/EnrichmentCoprocessor.java | 198 + .../hbase/coprocessor/GlobalConfigService.java}| 16 +- .../metron/hbase/coprocessor/HBaseCacheWriter.java | 78 + .../main/scripts/load_enrichment_coprocessor.sh| 38 +++ .../EnrichmentCoprocessorIntegrationTest.java | 208 + .../coprocessor/EnrichmentCoprocessorTest.java | 167 +++ .../org/apache/metron/hbase/helper/HelperDao.java | 58 .../org/apache/metron/hbase/HTableProvider.java| 3 +- .../org/apache/metron/hbase/TableProvider.java | 8 + .../apache/metron/hbase/client/HBaseClient.java| 34 +++ .../src/test/resources/log4j.properties| 7 +- .../apache/metron/test/utils/UnitTestHelper.java | 31 +- metron-platform/pom.xml| 1 + 45 files changed, 1761 insertions(+), 214 deletions(-) diff --git a/metron-deployment/packaging/ambari/README.md b/metron-deployment/packaging/ambari/README.md index 14dd5ca..4b0aace 100644 --- a/metron-deployment/packaging/ambari/README.md +++ b/metron-deployment/packaging/ambari/README.md @@ -61,7 +61,7 @@ The layout of `/common-services/METRON/CURRENT` is * `kerberos.json` * Defines the keytabs and other Kerberos configuration to be used when Kerberizing a cluster * `metainfo.xml` - * Defines the METRON service, along with required packages, services, etc. + * Defines the METRON service, along with required packages, services, etc. If you need to have the MPack install a new package (e.g. RPM, DEB), add it here. * `service_advisor.py` * Handles component layout and validation, along with handling some configurations for other services or that needs configs from other services. diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml index 950db6a..1fd4702 100644 --- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml +++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml @@ -135,6 +135,30 @@ The HBase
[metron] branch master updated: METRON-2060 Improving Alerts table config pane (tiborm via mmiklavc) closes apache/metron#1375
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 46e8625 METRON-2060 Improving Alerts table config pane (tiborm via
mmiklavc) closes apache/metron#1375
46e8625 is described below
commit 46e8625865100b35ee69fe6499e8bda56197fbcd
Author: tiborm
AuthorDate: Fri Apr 12 11:43:16 2019 -0600
METRON-2060 Improving Alerts table config pane (tiborm via mmiklavc) closes
apache/metron#1375
---
.../alert-details/alert-details.component.html | 59
.../alert-details/alert-details.component.scss | 15 +++
.../alerts-list/alerts-list.component.spec.ts | 2 +-
.../configure-table/configure-table.component.html | 148 +
.../configure-table/configure-table.component.scss | 39 ++
.../configure-table.component.spec.ts | 147 +---
.../configure-table/configure-table.component.ts | 110 +--
metron-interface/metron-alerts/src/slider.scss | 28 ++--
metron-interface/metron-alerts/src/styles.scss | 17 +++
metron-interface/metron-alerts/src/vendor.scss | 1 +
10 files changed, 399 insertions(+), 167 deletions(-)
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.html
b/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.html
index abc01ca..c4bcc88 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.html
+++
b/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.html
@@ -12,7 +12,7 @@
the specific language governing permissions and limitations under the
License.
-->
-
+
@@ -81,33 +81,38 @@
-
-
- Alert {{ i
+ 1 }} of {{ alertSources.length }}
-
-{{ field }}
{{ alert[field] }}
-
-
-
+
+
+
+ Alert
{{ i + 1 }} of {{ alertSources.length }}
+
+
+{{ field }}
+ {{ alert[field] }}
+
+
+
+
-
- Comments ({{alertCommentsWrapper.length}})
-
-ADD
COMMENT
-
-
-
-
-
- {{
alertCommentWrapper.alertComment.comment }}
- -
{{ alertCommentWrapper.alertComment.username }} -
{{alertCommentWrapper.displayTime}}
-
-
+
+ Comments ({{alertCommentsWrapper.length}})
+
+ADD
COMMENT
+
+
+
+
+
+ {{
alertCommentWrapper.alertComment.comment }}
+ - {{ alertCommentWrapper.alertComment.username }} -
{{alertCommentWrapper.displayTime}}
+
+
+
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss
b/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss
index 3b10c8f..3373292 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss
+++
b/metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.scss
@@ -176,3 +176,18 @@ textarea {
opacity: 0.5;
cursor: not-allowed;
}
+
+.tabContainer {
+ max-height: 100%;
+ height: 100%;
+ overflow: scroll;
+
+ ul {
+padding-inline-start: 20px;
+padding-bottom: 1rem;
+
+li {
+ margin-bottom: 1rem;
+}
+ }
+}
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.spec.ts
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.spec.ts
index 7adbbe9..fe838b3 100644
---
a/metron-interface/met
[metron] branch master updated: METRON-2066 Documentation and logging corrections (mmiklavc) closes apache/metron#1378
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 54aa46e METRON-2066 Documentation and logging corrections (mmiklavc)
closes apache/metron#1378
54aa46e is described below
commit 54aa46ee44a329504559f417790324c175f5af6a
Author: mmiklavc
AuthorDate: Wed Apr 10 13:04:03 2019 -0600
METRON-2066 Documentation and logging corrections (mmiklavc) closes
apache/metron#1378
---
metron-platform/Performance-tuning-guide.md| 2 +-
metron-platform/README.md | 2 +-
metron-platform/metron-common/README.md| 18 +-
metron-platform/metron-parsing/README.md | 35 ++-
.../java/org/apache/metron/parsers/GrokParser.java | 39 +++---
5 files changed, 64 insertions(+), 32 deletions(-)
diff --git a/metron-platform/Performance-tuning-guide.md
b/metron-platform/Performance-tuning-guide.md
index bd5c126..fe1b01b 100644
--- a/metron-platform/Performance-tuning-guide.md
+++ b/metron-platform/Performance-tuning-guide.md
@@ -412,7 +412,7 @@ And we ran our bro parser topology with the following
options. We did not need t
though you could certainly do so if necessary. Notice that we only needed 1
worker.
```
-/usr/metron/0.7.1/bin/start_parser_topology.sh \
+$METRON_HOME/bin/start_parser_topology.sh \
-e ~metron/.storm/storm-bro.config \
-esc ~/.storm/spout-bro.config \
-k $BROKERLIST \
diff --git a/metron-platform/README.md b/metron-platform/README.md
index feb30e5..e5a7e6a 100644
--- a/metron-platform/README.md
+++ b/metron-platform/README.md
@@ -27,4 +27,4 @@ Extensible set of Storm topologies and topology attributes
for streaming, enrich
# Documentation
-Please see documentation within each individual module for description and
usage instructions. Sample topologies are provided under Metron_Topologies to
get you started with the framework. We pre-assume knowledge of Hadoop, Storm,
Kafka, and HBase.
+Please see documentation within each individual module for description and
usage instructions. Sample topologies are provided under Metron_Topologies to
get you started with the framework. We pre-assume knowledge of Hadoop, Storm,
Kafka, Zookeeper, and HBase.
diff --git a/metron-platform/metron-common/README.md
b/metron-platform/metron-common/README.md
index 20f0eef..cbea9dd 100644
--- a/metron-platform/metron-common/README.md
+++ b/metron-platform/metron-common/README.md
@@ -18,6 +18,7 @@ limitations under the License.
# Contents
* [Stellar Language](#stellar-language)
+* [High Level Architecture](#high-level-architecture)
* [Global Configuration](#global-configuration)
* [Validation Framework](#validation-framework)
* [Management Utility](#management-utility)
@@ -109,6 +110,20 @@ If a field is managed via ambari, you should change the
field via
ambari. Otherwise, upon service restarts, you may find your update
overwritten.
+# High Level Architecture
+
+As already pointed out in the main project README, Apache Metron is a Kappa
architecture (see [Navigating the
Architecture](../../#navigating-the-architecture)) primarily backed by Storm
and Kafka. We additionally leverage:
+* Zookeeper for dynamic configuration updates to running Storm topologies.
This enables us to push updates to our Storm topologies without restarting them.
+* HBase primarily for enrichments. But we also use it to store user state for
our UI's.
+* HDFS for long term storage. Our parsed and enriched messages land here,
along with any reported exceptions or errors encountered along the way.
+* Solr and Elasticsearch (plus Kibana) for real-time access. We provide out of
the box compatibility with both Solr and Elasticsearch, and custom dashboards
for data exploration in Kibana.
+* Zeppelin for providing dashboards to do custom analytics.
+
+Getting data "into" Metron is accomplished by setting up a Kafka topic for
parsers to read from. There are a variety of options, including, but not
limited to:
+* [Bro Kafka plugin](https://github.com/apache/metron-bro-plugin-kafka)
+* [Fastcapa](../../metron-sensors/fastcapa)
+* [NiFi](https://nifi.apache.org)
+
# Validation Framework
Inside of the global configuration, there is a validation framework in
@@ -336,7 +351,8 @@ Errors generated in Metron topologies are transformed into
JSON format and follo
"error_hash":
"f7baf053f2d3c801a01d196f40f3468e87eea81788b2567423030100865c5061",
"error_type": "parser_error",
"message": "Unable to parse Message: {\"http\":
{\"ts\":1488809627.00.31915,\"uid\":\"C9JpSd2vFAWo3mXKz1\", ...",
- "timestamp": 1488809630698
+ "timestamp": 1488809630698,
+ "guid": "b
[metron] branch master updated: METRON-2051 Improve stellar-zeppelin documentation (mmiklavc) closes apache/metron#1366
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 8bb7a27 METRON-2051 Improve stellar-zeppelin documentation (mmiklavc) closes apache/metron#1366 8bb7a27 is described below commit 8bb7a27cd97a4d6e513f073463b9e62e96387f3d Author: mmiklavc AuthorDate: Wed Mar 27 11:28:46 2019 -0600 METRON-2051 Improve stellar-zeppelin documentation (mmiklavc) closes apache/metron#1366 --- metron-stellar/stellar-zeppelin/README.md | 4 1 file changed, 4 insertions(+) diff --git a/metron-stellar/stellar-zeppelin/README.md b/metron-stellar/stellar-zeppelin/README.md index 1720257..b79e2d3 100644 --- a/metron-stellar/stellar-zeppelin/README.md +++ b/metron-stellar/stellar-zeppelin/README.md @@ -62,6 +62,10 @@ To install the Stellar Interpreter in your Apache Zeppelin installation, follow bin/install-interpreter.sh --name stellar --artifact org.apache.metron:stellar-zeppelin:0.7.1 ``` +**Note:** The above command will download maven artifact groupId1:artifact1:version1 (org.apache.metron:stellar-zeppelin:0.7.1) and all of its transitive dependencies into the $ZEPPELIN_HOME/interpreter/stellar directory. `stellar-common`, which contains many of the [Stellar Core Functions](../stellar-common#stellar-core-functions), will be included transitively because `stellar-zeppelin` declares it as a direct dependency in its Maven pom.xml. + +* [3rd Party Zeppelin Interpreter Installation Documentation](https://zeppelin.apache.org/docs/0.7.3/manual/interpreterinstallation.html#3rd-party-interpreters) + 1. Start Zeppelin. ```
[metron] branch master updated: METRON-2041 RegularExpressionsParser in wrong source folder (mmiklavc) closes apache/metron#1361
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 0c774ab METRON-2041 RegularExpressionsParser in wrong source folder (mmiklavc) closes apache/metron#1361 0c774ab is described below commit 0c774ab921ff4271611bdf9d3bb7ad3978d3f7b0 Author: mmiklavc AuthorDate: Tue Mar 19 09:58:38 2019 -0600 METRON-2041 RegularExpressionsParser in wrong source folder (mmiklavc) closes apache/metron#1361 --- .../java/org/apache/metron/parsers/regex/RegularExpressionsParser.java| 0 1 file changed, 0 insertions(+), 0 deletions(-) diff --git a/metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParser.java b/metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/regex/RegularExpressionsParser.java similarity index 100% rename from metron-platform/metron-parsing/metron-parsers-common/src/test/java/org/apache/metron/parsers/regex/RegularExpressionsParser.java rename to metron-platform/metron-parsing/metron-parsers-common/src/main/java/org/apache/metron/parsers/regex/RegularExpressionsParser.java
[metron] branch master updated: METRON-2036 Maven builds fail locally in HDFSWriterTest (mmiklavc) closes apache/metron#1357
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 9c69ffc METRON-2036 Maven builds fail locally in HDFSWriterTest
(mmiklavc) closes apache/metron#1357
9c69ffc is described below
commit 9c69ffc8559830096a6138a5ef8fb3c31ea5bca3
Author: mmiklavc
AuthorDate: Thu Mar 14 14:06:44 2019 -0600
METRON-2036 Maven builds fail locally in HDFSWriterTest (mmiklavc) closes
apache/metron#1357
---
metron-platform/metron-parsing/metron-parsers/pom.xml | 6 --
metron-platform/metron-writer/pom.xml | 6 ++
.../org/apache/metron/writer/hdfs/HdfsWriterTest.java | 8
pom.xml| 18 ++
4 files changed, 32 insertions(+), 6 deletions(-)
diff --git a/metron-platform/metron-parsing/metron-parsers/pom.xml
b/metron-platform/metron-parsing/metron-parsers/pom.xml
index 83825f6..9645fb9 100644
--- a/metron-platform/metron-parsing/metron-parsers/pom.xml
+++ b/metron-platform/metron-parsing/metron-parsers/pom.xml
@@ -63,12 +63,6 @@
-
- org.apache.metron
- metron-parsers-common
- ${project.parent.version}
- compile
-
diff --git a/metron-platform/metron-writer/pom.xml
b/metron-platform/metron-writer/pom.xml
index 818e66d..45cba9d 100644
--- a/metron-platform/metron-writer/pom.xml
+++ b/metron-platform/metron-writer/pom.xml
@@ -228,6 +228,12 @@
stellar-common
${project.parent.version}
+
+org.apache.httpcomponents
+httpclient
+${global_httpclient_version}
+test
+
diff --git
a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
index 1d71c8a..ed78bf8 100644
---
a/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
+++
b/metron-platform/metron-writer/src/test/java/org/apache/metron/writer/hdfs/HdfsWriterTest.java
@@ -37,6 +37,7 @@ import org.apache.storm.task.TopologyContext;
import org.json.simple.JSONObject;
import org.junit.Assert;
import org.junit.Before;
+import org.junit.BeforeClass;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
@@ -53,6 +54,13 @@ public class HdfsWriterTest {
private File folder;
private FileNameFormat testFormat;
+ @BeforeClass
+ public static void beforeAll() throws Exception {
+// See https://issues.apache.org/jira/browse/METRON-2036
+// The need for this should go away when JUnit 4.13 is released and we can
upgrade.
+Thread.interrupted();
+ }
+
@Before
public void setup() throws IOException {
// Ensure each test has a unique folder to work with.
diff --git a/pom.xml b/pom.xml
index 8b660c6..7e886aa 100644
--- a/pom.xml
+++ b/pom.xml
@@ -38,6 +38,24 @@
metron-stellar
+
+
clojars.org
[metron] branch master updated: METRON-2030 SensorParserGroupControllerIntegrationTest intermittent errors (merrimanr via mmiklavc) closes apache/metron#1352
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 91368e1 METRON-2030 SensorParserGroupControllerIntegrationTest
intermittent errors (merrimanr via mmiklavc) closes apache/metron#1352
91368e1 is described below
commit 91368e1ee3565a91a46ad8d2d3639fecce320e84
Author: merrimanr
AuthorDate: Fri Mar 8 13:30:13 2019 -0700
METRON-2030 SensorParserGroupControllerIntegrationTest intermittent errors
(merrimanr via mmiklavc) closes apache/metron#1352
---
...SensorParserGroupControllerIntegrationTest.java | 31 +-
1 file changed, 24 insertions(+), 7 deletions(-)
diff --git
a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SensorParserGroupControllerIntegrationTest.java
b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SensorParserGroupControllerIntegrationTest.java
index 8106573..f74bf58 100644
---
a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SensorParserGroupControllerIntegrationTest.java
+++
b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/controller/SensorParserGroupControllerIntegrationTest.java
@@ -135,6 +135,11 @@ public class SensorParserGroupControllerIntegrationTest {
this.sensorParserConfigService.save("squid", new SensorParserConfig());
this.sensorParserConfigService.save("yaf", new SensorParserConfig());
this.sensorParserConfigService.save("jsonMap", new SensorParserConfig());
+TestUtils.assertEventually(() ->
Assert.assertNotNull(sensorParserConfigService.findOne("bro")));
+TestUtils.assertEventually(() ->
Assert.assertNotNull(sensorParserConfigService.findOne("snort")));
+TestUtils.assertEventually(() ->
Assert.assertNotNull(sensorParserConfigService.findOne("squid")));
+TestUtils.assertEventually(() ->
Assert.assertNotNull(sensorParserConfigService.findOne("yaf")));
+TestUtils.assertEventually(() ->
Assert.assertNotNull(sensorParserConfigService.findOne("jsonMap")));
}
@Test
@@ -175,7 +180,9 @@ public class SensorParserGroupControllerIntegrationTest {
@Test
public void testUpdate() throws Exception {
-this.sensorParserGroupService.save(JSONUtils.INSTANCE.load(group1BroSquid,
SensorParserGroup.class));
+SensorParserGroup group1 = JSONUtils.INSTANCE.load(group1BroSquid,
SensorParserGroup.class);
+this.sensorParserGroupService.save(group1);
+TestUtils.assertEventually(() -> Assert.assertEquals(group1,
this.sensorParserGroupService.findOne("group1")));
this.mockMvc.perform(post(sensorParserGroupUrl).with(httpBasic(user,
password)).with(csrf()).contentType(MediaType.parseMediaType("application/json;charset=UTF-8")).content(group1BroSquid))
.andExpect(status().isOk())
@@ -189,7 +196,9 @@ public class SensorParserGroupControllerIntegrationTest {
@Test
public void testFindOne() throws Exception {
-this.sensorParserGroupService.save(JSONUtils.INSTANCE.load(group1BroSquid,
SensorParserGroup.class));
+SensorParserGroup group1 = JSONUtils.INSTANCE.load(group1BroSquid,
SensorParserGroup.class);
+this.sensorParserGroupService.save(group1);
+TestUtils.assertEventually(() -> Assert.assertEquals(group1,
this.sensorParserGroupService.findOne("group1")));
this.mockMvc.perform(get(sensorParserGroupUrl +
"/group1").with(httpBasic(user,password)))
.andExpect(status().isOk())
@@ -206,8 +215,12 @@ public class SensorParserGroupControllerIntegrationTest {
@Test
public void testGetAll() throws Exception {
-this.sensorParserGroupService.save(JSONUtils.INSTANCE.load(group1BroSquid,
SensorParserGroup.class));
-
this.sensorParserGroupService.save(JSONUtils.INSTANCE.load(group2YafJsonMap,
SensorParserGroup.class));
+SensorParserGroup group1 = JSONUtils.INSTANCE.load(group1BroSquid,
SensorParserGroup.class);
+this.sensorParserGroupService.save(group1);
+TestUtils.assertEventually(() -> Assert.assertEquals(group1,
this.sensorParserGroupService.findOne("group1")));
+SensorParserGroup group2 = JSONUtils.INSTANCE.load(group2YafJsonMap,
SensorParserGroup.class);
+this.sensorParserGroupService.save(group2);
+TestUtils.assertEventually(() -> Assert.assertEquals(group2,
this.sensorParserGroupService.findOne("group2")));
this.mockMvc.perform(get(sensorParserGroupUrl).with(httpBasic(user,password)))
.andExpect(status().isOk())
@@ -227,7 +240,9 @@ public class SensorParserGroupControllerIntegrationTest {
@Test
public void testError() throws Exception {
-this.sensorParserGroupService.sa
[metron] branch master updated: METRON-2031 [UI] Turning off initial search request and polling by default on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 143901b METRON-2031 [UI] Turning off initial search request and
polling by default on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
143901b is described below
commit 143901b018fff7008415e08351f32a63095b357e
Author: tiborm
AuthorDate: Fri Mar 8 13:14:16 2019 -0700
METRON-2031 [UI] Turning off initial search request and polling by default
on Alerts UI (tiborm via mmiklavc) closes apache/metron#1353
---
.../alerts/alerts-list/alerts-list.component.html | 4 +--
.../alerts/alerts-list/alerts-list.component.ts| 33 +-
.../metron-alerts/src/app/model/table-metadata.ts | 2 +-
.../metron-alerts/src/app/utils/constants.ts | 2 +-
4 files changed, 23 insertions(+), 18 deletions(-)
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
index 0b6f36d..a02147f 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
+++
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.html
@@ -51,8 +51,8 @@
-
-
+
+
ACTIONS
diff --git
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
index 20c3a19..342f44e 100644
---
a/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
+++
b/metron-interface/metron-alerts/src/app/alerts/alerts-list/alerts-list.component.ts
@@ -61,10 +61,10 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
alerts: Alert[] = [];
searchResponse: SearchResponse = new SearchResponse();
colNumberTimerId: number;
- refreshInterval = RefreshInterval.ONE_MIN;
+ refreshInterval = RefreshInterval.TEN_MIN;
refreshTimer: Subscription;
- pauseRefresh = POLLING_DEFAULT_STATE;
- lastPauseRefreshValue = false;
+ isRefreshPaused = POLLING_DEFAULT_STATE;
+ lastIsRefreshPausedValue = false;
isMetaAlertPresentInSelectedAlerts = false;
timeStampfilterPresent = false;
selectedTimeRange = new Filter(TIMESTAMP_FIELD_NAME, ALL_TIME, false);
@@ -164,10 +164,17 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
this.configureTableService.getTableMetadata(),
this.clusterMetaDataService.getDefaultColumns()
).subscribe((response: any) => {
- this.prepareData(response[0], response[1], resetPaginationForSearch);
+ this.prepareData(response[0], response[1]);
+ this.refreshAlertData(resetPaginationForSearch);
});
}
+ private refreshAlertData(resetPaginationForSearch: boolean) {
+if (this.alerts.length) {
+ this.search(resetPaginationForSearch);
+}
+ }
+
getColumnNamesForQuery() {
let fieldNames = this.alertsColumns.map(columnMetadata =>
columnMetadata.name);
fieldNames = fieldNames.filter(name => !(name === 'id' || name ===
'alert_status'));
@@ -249,8 +256,8 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
}
onPausePlay() {
-this.pauseRefresh = !this.pauseRefresh;
-if (this.pauseRefresh) {
+this.isRefreshPaused = !this.isRefreshPaused;
+if (this.isRefreshPaused) {
this.tryStopPolling();
} else {
this.search(false);
@@ -278,14 +285,12 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
this.calcColumnsToDisplay();
}
- prepareData(tableMetaData: TableMetadata, defaultColumns: ColumnMetadata[],
resetPagination: boolean) {
+ prepareData(tableMetaData: TableMetadata, defaultColumns: ColumnMetadata[]) {
this.tableMetaData = tableMetaData;
this.refreshInterval = this.tableMetaData.refreshInterval;
this.updateConfigRowsSettings();
this.prepareColumnData(tableMetaData.tableColumns, defaultColumns);
-
-this.search(resetPagination);
}
processEscalate() {
@@ -326,7 +331,7 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
}
restoreRefreshState() {
-this.pauseRefresh = this.lastPauseRefreshValue;
+this.isRefreshPaused = this.lastIsRefreshPausedValue;
this.tryStartPolling();
}
@@ -412,17 +417,17 @@ export class AlertsListComponent implements OnInit,
OnDestroy {
}
saveRefreshState() {
-this.lastPauseRefreshValue = this.pauseRefresh;
+this.lastIsRefreshPausedValue = this.isRefreshPaused;
this.tryStopPolling();
}
pause() {
-this.pauseRefresh = tr
[metron] branch master updated: METRON-1951: Add site-book generation to Travis build (mmiklavc) closes apache/metron#1310
This is an automated email from the ASF dual-hosted git repository. mmiklavcic pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/metron.git The following commit(s) were added to refs/heads/master by this push: new 7f073b8 METRON-1951: Add site-book generation to Travis build (mmiklavc) closes apache/metron#1310 7f073b8 is described below commit 7f073b863c5ab5a420b34f83f27297a7674640bf Author: mmiklavc AuthorDate: Fri Dec 21 13:00:37 2018 -0700 METRON-1951: Add site-book generation to Travis build (mmiklavc) closes apache/metron#1310 --- .travis.yml | 1 + 1 file changed, 1 insertion(+) diff --git a/.travis.yml b/.travis.yml index 3a73d7e..58f8861 100644 --- a/.travis.yml +++ b/.travis.yml @@ -22,6 +22,7 @@ addons: env: - SCRIPT="mvn surefire:test@unit-tests -T 2C" - SCRIPT="mvn surefire:test@integration-tests" + - SCRIPT="mvn clean site --projects site-book" - SCRIPT="mvn test --projects metron-interface/metron-config,metron-interface/metron-alerts" - SCRIPT="./dev-utilities/build-utils/verify_licenses.sh"
[metron] branch master updated: METRON-1950: Site-book generation broken in master (mmiklavc) closes apache/metron#1309
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new 9e026e3 METRON-1950: Site-book generation broken in master (mmiklavc)
closes apache/metron#1309
9e026e3 is described below
commit 9e026e3e902769dae364b4c4acf64e00839d24f5
Author: mmiklavc
AuthorDate: Thu Dec 20 12:22:19 2018 -0700
METRON-1950: Site-book generation broken in master (mmiklavc) closes
apache/metron#1309
---
metron-platform/metron-parsing/README.md | 536 +++--
.../{metron-parsers-common => }/parser_arch.png| Bin
site-book/bin/generate-md.sh | 6 +-
3 files changed, 276 insertions(+), 266 deletions(-)
diff --git a/metron-platform/metron-parsing/README.md
b/metron-platform/metron-parsing/README.md
index 76b6168..9a46532 100644
--- a/metron-platform/metron-parsing/README.md
+++ b/metron-platform/metron-parsing/README.md
@@ -21,127 +21,129 @@ limitations under the License.
Parsers are pluggable components which are used to transform raw data
(textual or raw bytes) into JSON messages suitable for downstream
-enrichment and indexing.
+enrichment and indexing.
There are two general types types of parsers:
* A parser written in Java which conforms to the `MessageParser` interface.
This kind of parser is optimized for speed and performance and is built for use
with higher velocity topologies. These parsers are not easily modifiable and
in order to make changes to them the entire topology need to be recompiled.
* A general purpose parser. This type of parser is primarily designed for
lower-velocity topologies or for quickly standing up a parser for a new
telemetry before a permanent Java parser can be written for it. As of the time
of this writing, we have:
- * Grok parser: `org.apache.metron.parsers.GrokParser` with possible
`parserConfig` entries of
-* `grokPath` : The path in HDFS (or in the Jar) to the grok statement
-* `patternLabel` : The pattern label to use from the grok statement
-* `multiLine` : The raw data passed in should be handled as a long with
multiple lines, with each line to be parsed separately. This setting's valid
values are 'true' or 'false'. The default if unset is 'false'. When set the
parser will handle multiple lines with successfully processed lines emitted
normally, and lines with errors sent to the error topic.
-* `timestampField` : The field to use for timestamp
-* `timeFields` : A list of fields to be treated as time
-* `dateFormat` : The date format to use to parse the time fields
-* `timezone` : The timezone to use. `UTC` is default.
-* The Grok parser supports either 1 line to parse per incoming message, or
incoming messages with multiple log lines, and will produce a json message per
line
- * CSV Parser: `org.apache.metron.parsers.csv.CSVParser` with possible
`parserConfig` entries of
-* `timestampFormat` : The date format of the timestamp to use. If
unspecified, the parser assumes the timestamp is ms since unix epoch.
-* `columns` : A map of column names you wish to extract from the CSV to
their offsets (e.g. `{ 'name' : 1, 'profession' : 3}` would be a column map
for extracting the 2nd and 4th columns from a CSV)
-* `separator` : The column separator, `,` by default.
- * JSON Map Parser: `org.apache.metron.parsers.json.JSONMapParser` with
possible `parserConfig` entries of
-* `mapStrategy` : A strategy to indicate how to handle multi-dimensional
Maps. This is one of
- * `DROP` : Drop fields which contain maps
- * `UNFOLD` : Unfold inner maps. So `{ "foo" : { "bar" : 1} }` would
turn into `{"foo.bar" : 1}`
- * `ALLOW` : Allow multidimensional maps
- * `ERROR` : Throw an error when a multidimensional map is encountered
-* `jsonpQuery` : A [JSON Path](#json_path) query string. If present, the
result of the JSON Path query should be a list of messages. This is useful if
you have a JSON document which contains a list or array of messages embedded in
it, and you do not have another means of splitting the message.
-* `wrapInEntityArray` : `"true" or "false"`. If `jsonQuery` is present and
this flag is present and set to `"true"`, the incoming message will be wrapped
in a JSON entity and array.
- for example:
- `{"name":"value"},{"name2","value2}` will be wrapped as `{"message" :
[{"name":"value"},{"name2","value2}]}`.
- This is using the default value for `wrapEntityName` if that property
is not set.
-* `wrapEntityName` : Sets the name to use when wrapping JSON using
`wrapInEntityArray`. The `jsonpQuery` sh
[metron] branch master updated: METRON-1795: General Purpose Regex Parser (jadeepsinh2 via mmiklavc) closes apache/metron#1245
This is an automated email from the ASF dual-hosted git repository.
mmiklavcic pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/metron.git
The following commit(s) were added to refs/heads/master by this push:
new b8e426c METRON-1795: General Purpose Regex Parser (jadeepsinh2 via
mmiklavc) closes apache/metron#1245
b8e426c is described below
commit b8e426c755a5969e24dba50f5d8fa81d1ccb472d
Author: jagdeepsingh2
AuthorDate: Mon Dec 17 09:44:50 2018 -0700
METRON-1795: General Purpose Regex Parser (jadeepsinh2 via mmiklavc) closes
apache/metron#1245
---
metron-platform/metron-parsers/README.md | 90 +
.../parsers/regex/RegularExpressionsParser.java| 435 +
.../regex/RegularExpressionsParserTest.java| 275 +
3 files changed, 800 insertions(+)
diff --git a/metron-platform/metron-parsers/README.md
b/metron-platform/metron-parsers/README.md
index cfcf6ed..5aff84a 100644
--- a/metron-platform/metron-parsers/README.md
+++ b/metron-platform/metron-parsers/README.md
@@ -52,6 +52,96 @@ There are two general types types of parsers:
This is using the default value for `wrapEntityName` if that property
is not set.
* `wrapEntityName` : Sets the name to use when wrapping JSON using
`wrapInEntityArray`. The `jsonpQuery` should reference this name.
* A field called `timestamp` is expected to exist and, if it does not,
then current time is inserted.
+ * Regular Expressions Parser
+ * `recordTypeRegex` : A regular expression to uniquely identify a record
type.
+ * `messageHeaderRegex` : A regular expression used to extract fields
from a message part which is common across all the messages.
+ * `convertCamelCaseToUnderScore` : If this property is set to true, this
parser will automatically convert all the camel case property names to
underscore seperated.
+ For example, following convertions will automatically happen:
+
+ ```
+ ipSrcAddr -> ip_src_addr
+ ipDstAddr -> ip_dst_addr
+ ipSrcPort -> ip_src_port
+ ```
+ Note this property may be necessary, because java does not support
underscores in the named group names. So in case your property naming
conventions requires underscores in property names, use this property.
+
+ * `fields` : A json list of maps contaning a record type to regular
expression mapping.
+
+ A complete configuration example would look like:
+
+ ```json
+ "convertCamelCaseToUnderScore": true,
+ "recordTypeRegex": "kernel|syslog",
+ "messageHeaderRegex":
"((<=^<)\\d{1,4}(?=>)).*?((<=>)[A-Za-z]
{3}\\s{1,2}\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2}(?=\\s)).*?((<=\\s).*?(?=\\s))",
+ "fields": [
+{
+ "recordType": "kernel",
+ "regex": ".*((<=\\]|\\w\\:).*?(?=$))"
+},
+{
+ "recordType": "syslog",
+ "regex":
".*((<=PID\\s=\\s).*?(?=\\sLine)).*((<=64\\s)\/([A-Za-z0-9_-]+\/)+(?=\\w))
(.*?(?=\")).*((<=\").*?(?=$))"
+}
+ ]
+ ```
+ **Note**: messageHeaderRegex and regex (withing fields) could be
specified as lists also e.g.
+ ```json
+ "messageHeaderRegex": [
+ "regular expression 1",
+ "regular expression 2"
+ ]
+ ```
+ Where **regular expression 1** are valid regular expressions and may
have named
+ groups, which would be extracted into fields. This list will be
evaluated in order until a
+ matching regular expression is found.
+
+ **messageHeaderRegex** is run on all the messages.
+ Yes, all the messages are expected to contain the fields which are being
extracted using the **messageHeaderRegex**.
+ **messageHeaderRegex** is a sort of HCF (highest common factor) in all
messages.
+
+ **recordTypeRegex** can be a more advanced regular expression containing
named goups. For example
+
+ "recordTypeRegex":
"(<process>(<=\\s)\\b(kernel|syslog)\\b(?=\\[|:))"
+
+ Here all the named goups (process in above example) will be extracted as
fields.
+
+ Though having named group in recordType is completely optional, still
one could want extract named groups in recordType for following reasons:
+
+ 1. Since **recordType** regular expression is already getting matched
and we are paying the price for a regular expression match already,
+ we can extract certain fields as a by product of this match.
+ 2. Most likely the **recordType** field is common across all the
messages. Hence having it extracted in the recordType (or messageHeaderRegex)
would
+ reduce the overall complexity of
metron git commit: METRON-1889: Add any missing timestamp fields to unified enrichment topology (mmiklavc via mmiklavc) closes apache/metron#1286
Repository: metron
Updated Branches:
refs/heads/master b4d76f98e -> 4ef65e09e
METRON-1889: Add any missing timestamp fields to unified enrichment topology
(mmiklavc via mmiklavc) closes apache/metron#1286
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/4ef65e09
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/4ef65e09
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/4ef65e09
Branch: refs/heads/master
Commit: 4ef65e09ea4a1eac8abf89521e5a999faeca1f37
Parents: b4d76f9
Author: mmiklavc
Authored: Tue Dec 4 07:27:33 2018 -0700
Committer: Michael Miklavcic
Committed: Tue Dec 4 07:27:33 2018 -0700
--
.../enrichment/parallel/ParallelEnricher.java | 10 +-
.../enrichment/utils/EnrichmentUtils.java | 13 +--
.../parallel/ParallelEnricherTest.java | 104 ---
3 files changed, 77 insertions(+), 50 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/4ef65e09/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
--
diff --git
a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
index b10c148..1de8945 100644
---
a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
+++
b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
@@ -157,6 +157,7 @@ public class ParallelEnricher {
throw new IllegalStateException("Unable to find an adapter for " +
task.getKey()
+ ", possible adapters are: " +
Joiner.on(",").join(enrichmentsByType.keySet()));
}
+ message.put("adapter." +
adapter.getClass().getSimpleName().toLowerCase() + ".begin.ts", "" +
System.currentTimeMillis());
for(JSONObject m : task.getValue()) {
/* now for each unit of work (each of these only has one element in
them)
* the key is the field name and the value is value associated with
that field.
@@ -171,6 +172,7 @@ public class ParallelEnricher {
String field = (String) o;
Object value = m.get(o);
if(value == null) {
+message.put("adapter." +
adapter.getClass().getSimpleName().toLowerCase() + ".end.ts", "" +
System.currentTimeMillis());
continue;
}
CacheKey cacheKey = new CacheKey(field, value, config);
@@ -182,7 +184,10 @@ public class ParallelEnricher {
ret = new JSONObject();
}
//each enrichment has their own unique prefix to use to adjust
the keys for the enriched fields.
- return EnrichmentUtils.adjustKeys(new JSONObject(), ret,
cacheKey.getField(), prefix);
+ JSONObject adjustedKeys = EnrichmentUtils
+ .adjustKeys(new JSONObject(), ret, cacheKey.getField(),
prefix);
+ adjustedKeys.put("adapter." +
adapter.getClass().getSimpleName().toLowerCase() + ".end.ts", "" +
System.currentTimeMillis());
+ return adjustedKeys;
} catch (Throwable e) {
JSONObject errorMessage = new JSONObject();
errorMessage.putAll(m);
@@ -197,11 +202,12 @@ public class ParallelEnricher {
}
}
if(taskList.isEmpty()) {
+ message.put(getClass().getSimpleName().toLowerCase() + ".enrich.end.ts",
"" + System.currentTimeMillis());
return new EnrichmentResult(message, errors);
}
EnrichmentResult ret = new EnrichmentResult(all(taskList, message, (left,
right) -> join(left, right)).get(), errors);
-message.put(getClass().getSimpleName().toLowerCase() + ".enrich.end.ts",
"" + System.currentTimeMillis());
+ret.getResult().put(getClass().getSimpleName().toLowerCase() +
".enrich.end.ts", "" + System.currentTimeMillis());
if(perfLog != null) {
String key = message.get(Constants.GUID) + "";
perfLog.log("enrich", "key={}, elapsed time to enrich", key);
http://git-wip-us.apache.org/repos/asf/metron/blob/4ef65e09/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/utils/EnrichmentUtils.java
--
diff --git
a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/utils/EnrichmentUtils.java
b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/utils/EnrichmentUtils.java
index 63d39c5..9a36a87 100644
---
a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/utils/EnrichmentUtils.java
+++
b/met
metron git commit: METRON-1887: Add logging to the ClasspathFunctionResolver (mmiklavc via mmiklavc) closes apache/metron#1274
Repository: metron
Updated Branches:
refs/heads/master e0f9b48e0 -> e1a957be3
METRON-1887: Add logging to the ClasspathFunctionResolver (mmiklavc via
mmiklavc) closes apache/metron#1274
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e1a957be
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e1a957be
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e1a957be
Branch: refs/heads/master
Commit: e1a957be3b44986799fb9672536bce43705bd6fa
Parents: e0f9b48
Author: mmiklavc
Authored: Mon Nov 26 14:21:15 2018 -0700
Committer: Michael Miklavcic
Committed: Mon Nov 26 14:21:15 2018 -0700
--
.../common/utils/VFSClassloaderUtil.java| 4 +++
.../resolver/ClasspathFunctionResolver.java | 32 +++-
2 files changed, 22 insertions(+), 14 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/e1a957be/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/VFSClassloaderUtil.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/VFSClassloaderUtil.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/VFSClassloaderUtil.java
index 803398c..5690682 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/VFSClassloaderUtil.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/utils/VFSClassloaderUtil.java
@@ -112,14 +112,18 @@ public class VFSClassloaderUtil {
* @throws FileSystemException
*/
public static Optional configureClassloader(String paths)
throws FileSystemException {
+LOG.debug("Configuring class loader with paths = {}", paths);
if(paths.trim().isEmpty()) {
+ LOG.debug("No paths provided. Not returning a ClassLoader.");
return Optional.empty();
}
FileSystemManager vfs = generateVfs();
FileObject[] objects = resolve(vfs, paths);
if(objects == null || objects.length == 0) {
+ LOG.debug("No Classloader able to be resolved from provided paths. Not
returning a ClassLoader.");
return Optional.empty();
}
+LOG.debug("vfs = {}, objects = {}", vfs, objects);
return Optional.of(new VFSClassLoader(objects, vfs,
vfs.getClass().getClassLoader()));
}
http://git-wip-us.apache.org/repos/asf/metron/blob/e1a957be/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
index b17233a..7b75009 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/ClasspathFunctionResolver.java
@@ -18,6 +18,11 @@
package org.apache.metron.stellar.dsl.functions.resolver;
+import static
org.apache.metron.stellar.dsl.Context.Capabilities.STELLAR_CONFIG;
+import static
org.apache.metron.stellar.dsl.functions.resolver.ClasspathFunctionResolver.Config.STELLAR_SEARCH_EXCLUDES_KEY;
+import static
org.apache.metron.stellar.dsl.functions.resolver.ClasspathFunctionResolver.Config.STELLAR_SEARCH_INCLUDES_KEY;
+import static
org.apache.metron.stellar.dsl.functions.resolver.ClasspathFunctionResolver.Config.STELLAR_VFS_PATHS;
+
import java.util.ArrayList;
import java.util.Arrays;
import java.util.HashSet;
@@ -33,16 +38,9 @@ import
org.apache.metron.stellar.common.utils.VFSClassloaderUtil;
import org.apache.metron.stellar.dsl.Context;
import org.apache.metron.stellar.dsl.Stellar;
import org.apache.metron.stellar.dsl.StellarFunction;
-
-import org.atteo.classindex.ClassFilter;
import org.atteo.classindex.ClassIndex;
import org.reflections.util.FilterBuilder;
-import static
org.apache.metron.stellar.dsl.Context.Capabilities.STELLAR_CONFIG;
-import static
org.apache.metron.stellar.dsl.functions.resolver.ClasspathFunctionResolver.Config.STELLAR_SEARCH_EXCLUDES_KEY;
-import static
org.apache.metron.stellar.dsl.functions.resolver.ClasspathFunctionResolver.Config.STELLAR_SEARCH_INCLUDES_KEY;
-import static
org.apache.metron.stellar.dsl.functions.resolver.ClasspathFunctionResolver.Config.STELLAR_VFS_PATHS;
-
/**
* Performs function resolution for Stellar by searching the classpath.
*
@@ -239,17 +237,23 @@ public class ClasspathFunctionResolver extends
BaseFunction
[17/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.spec.ts
--
diff --git
a/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.spec.ts
b/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.spec.ts
index 899a2b1..a953b32 100644
---
a/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.spec.ts
+++
b/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.spec.ts
@@ -16,7 +16,6 @@
* limitations under the License.
*/
import { async, ComponentFixture, TestBed } from '@angular/core/testing';
-import { Component, Input } from '@angular/core';
import { SavedSearchesComponent } from './saved-searches.component';
import { CollapseComponent } from '../../shared/collapse/collapse.component';
@@ -24,8 +23,8 @@ import { CenterEllipsesPipe } from
'../../shared/pipes/center-ellipses.pipe';
import { ColumnNameTranslatePipe } from
'../../shared/pipes/column-name-translate.pipe';
import { Router } from '@angular/router';
import { SaveSearchService } from '../../service/save-search.service';
-import { MetronDialogBox } from '../../shared/metron-dialog-box';
import { of } from 'rxjs';
+import { DialogService } from 'app/service/dialog.service';
describe('SavedSearchesComponent', () => {
@@ -40,7 +39,7 @@ describe('SavedSearchesComponent', () => {
listSavedSearches:
jasmine.createSpy('listSavedSearches').and.returnValue(of([])),
listRecentSearches:
jasmine.createSpy('listRecentSearches').and.returnValue(of([])),
} },
-MetronDialogBox
+DialogService
],
declarations: [
SavedSearchesComponent,
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.ts
--
diff --git
a/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.ts
b/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.ts
index 2204179..ab182c0 100644
---
a/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.ts
+++
b/metron-interface/metron-alerts/src/app/alerts/saved-searches/saved-searches.component.ts
@@ -22,9 +22,10 @@ import {forkJoin as observableForkJoin} from 'rxjs';
import {SaveSearchService} from '../../service/save-search.service';
import {SaveSearch} from '../../model/save-search';
-import {MetronDialogBox} from '../../shared/metron-dialog-box';
import {NUM_SAVED_SEARCH} from '../../utils/constants';
import {CollapseComponentData, CollapseComponentDataItems} from
'../../shared/collapse/collapse-component-data';
+import { DialogService } from 'app/service/dialog.service';
+import { ConfirmationType } from 'app/model/confirmation-type';
@Component({
selector: 'app-saved-searches',
@@ -39,7 +40,7 @@ export class SavedSearchesComponent implements OnInit {
recentSearches: CollapseComponentData = new CollapseComponentData();
constructor(private router: Router,
private saveSearchService: SaveSearchService,
- private metronDialog: MetronDialogBox) {
+ private dialogService: DialogService) {
}
doDeleteRecentSearch(selectedSearch: SaveSearch) {
@@ -61,21 +62,33 @@ export class SavedSearchesComponent implements OnInit {
}
deleteRecentSearch($event) {
-let selectedSearch = this.recentSearcheObj.find(savedSearch =>
savedSearch.name === $event.key);
-this.metronDialog.showConfirmationMessage('Do you wish to delete recent
search ' + selectedSearch.name).subscribe((result: boolean) => {
- if (result) {
-this.doDeleteRecentSearch(selectedSearch);
- }
-});
+let selectedSearch = this.recentSearcheObj.find(
+ savedSearch => savedSearch.name === $event.key
+);
+const confirmedSubscription = this.dialogService
+ .launchDialog(
+'Do you wish to delete recent search ' + selectedSearch.name
+ )
+ .subscribe(action => {
+if (action === ConfirmationType.Confirmed) {
+ this.doDeleteRecentSearch(selectedSearch);
+}
+confirmedSubscription.unsubscribe();
+ });
}
deleteSearch($event) {
-let selectedSearch = this.searches.find(savedSearch => savedSearch.name
=== $event.key);
-this.metronDialog.showConfirmationMessage('Do you wish to delete saved
search ' + selectedSearch.name).subscribe((result: boolean) => {
- if (result) {
-this.doDeleteSearch(selectedSearch);
- }
-});
+let selectedSearch = this.searches.find(
+ savedSearch => savedSearch.name === $event.key
+);
+const confirmedSubscription = this.dialogService
[20/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
--
diff --git
a/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
new file mode 100644
index 000..2a3cdcc
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
@@ -0,0 +1,7 @@
+{
+ "jobId":"job_1537878471649_0001",
+ "jobStatus":"RUNNING",
+ "description":"map: 0.0%, reduce: 0.0%",
+ "percentComplete":0.0,
+ "pageTotal":0
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
--
diff --git
a/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
new file mode 100644
index 000..1505f71
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
@@ -0,0 +1,7 @@
+{
+ "jobId":"job_1537878471649_0001",
+ "jobStatus":"RUNNING",
+ "description":"map: 100.0%, reduce: 100.0%",
+ "percentComplete":75.0,
+ "pageTotal":0
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
--
diff --git
a/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
new file mode 100644
index 000..662c27a
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
@@ -0,0 +1,7 @@
+{
+ "jobId":"job_1537878471649_0001",
+ "jobStatus":"SUCCEEDED",
+ "description":"Job completed.",
+ "percentComplete":100.0,
+ "pageTotal":2
+}
\ No newline at end of file
[05/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
--
diff --git
a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
new file mode 100644
index 000..e24a39d
--- /dev/null
+++
b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/MultiLineGrokParserTest.java
@@ -0,0 +1,149 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.parsers;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.metron.parsers.interfaces.MessageParserResult;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+
+public class MultiLineGrokParserTest {
+
+ /**
+ * Test that if a byte[] with multiple lines of log is passed in
+ * it will be parsed into the correct number of messages.
+ * @throws IOException if we can't read from disk
+ * @throws ParseException if we can't parse
+ */
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testLegacyInterfaceReturnsMultiline() throws IOException,
ParseException {
+
+Map parserConfig = new HashMap<>();
+parserConfig.put("grokPath", getGrokPath());
+parserConfig.put("patternLabel", getGrokPatternLabel());
+parserConfig.put("timestampField", getTimestampField());
+parserConfig.put("dateFormat", getDateFormat());
+parserConfig.put("timeFields", getTimeFields());
+parserConfig.put("multiLine", getMultiLine());
+GrokParser grokParser = new GrokParser();
+grokParser.configure(parserConfig);
+grokParser.init();
+
+JSONParser jsonParser = new JSONParser();
+Map testData = getTestData();
+for (Map.Entry e : testData.entrySet()) {
+ byte[] rawMessage = e.getKey().getBytes();
+ Optional> resultOptional =
grokParser.parseOptionalResult(rawMessage);
+ Assert.assertNotNull(resultOptional);
+ Assert.assertTrue(resultOptional.isPresent());
+ List parsedList = resultOptional.get().getMessages();
+ Assert.assertEquals(10, parsedList.size());
+}
+ }
+
+ /**
+ * Test that if a byte[] with multiple lines of log is passed in
+ * it will be parsed into the correct number of messages using the
+ * parseOptionalResult call.
+ * @throws IOException if we can't read from disk
+ * @throws ParseException if we can't parse
+ */
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testOptionalResultReturnsMultiline() throws IOException,
ParseException {
+
+Map parserConfig = new HashMap<>();
+parserConfig.put("grokPath", getGrokPath());
+parserConfig.put("patternLabel", getGrokPatternLabel());
+parserConfig.put("timestampField", getTimestampField());
+parserConfig.put("dateFormat", getDateFormat());
+parserConfig.put("timeFields", getTimeFields());
+parserConfig.put("multiLine", getMultiLine());
+
+GrokParser grokParser = new GrokParser();
+grokParser.configure(parserConfig);
+grokParser.init();
+
+JSONParser jsonParser = new JSONParser();
+Map testData = getTestData();
+for (Map.Entry e : testData.entrySet()) {
+ byte[] rawMessage = e.getKey().getBytes();
+ Optional> resultOptional =
grokParser.parseOptionalResult(rawMessage);
+ Assert.assertTrue(resultOptional.isPresent());
+ Optional throwableOptional =
resultOptional.get().getMasterThrowable();
+ List resultList = resultOptional.get().getMessages();
+ Map errorMap =
resultOptional.get().getMessageThrowables();
+ Assert.assertFalse(throwableOptional.isPresent());
+ Assert.assertEquals(0, errorMap.size());
+ Assert.assertEquals(10, resultList.size());
+}
+ }
+
+ @Suppres
[15/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/package.json
--
diff --git a/metron-interface/metron-config/package.json
b/metron-interface/metron-config/package.json
index 6e196d5..ffc92ff 100644
--- a/metron-interface/metron-config/package.json
+++ b/metron-interface/metron-config/package.json
@@ -7,10 +7,11 @@
},
"angular-cli": {},
"scripts": {
-"build": "./node_modules/angular-cli/bin/ng build -prod",
+"build": "ng build --prod",
"start": "ng serve",
"lint": "tslint \"src/**/*.ts\"",
-"test": "./node_modules/angular-cli/bin/ng test --watch=false",
+"test": "ng test --browsers=Chrome",
+"testCI": "ng test --watch=false --browsers=ChromeHeadless",
"pree2e": "webdriver-manager update",
"e2e": "./node_modules/.bin/protractor",
"e2e-all": "./node_modules/.bin/protractor --suite=all",
@@ -20,47 +21,54 @@
},
"private": true,
"dependencies": {
+"@angular-devkit/schematics": "^0.7.5",
+"@angular/cli": "^6.2.3",
+"@angular/common": "^6.1.8",
+"@angular/compiler": "^6.1.8",
+"@angular/core": "^6.1.8",
+"@angular/forms": "^6.1.8",
+"@angular/http": "^6.1.8",
+"@angular/platform-browser": "^6.1.8",
+"@angular/platform-browser-dynamic": "^6.1.8",
+"@angular/platform-server": "^6.1.8",
+"@angular/router": "^6.1.8",
"@types/ace": "0.0.32",
-"@types/bootstrap": "^3.3.32",
-"@types/jasmine": "2.2.30",
-"@types/jquery": "^2.0.32",
+"@types/bootstrap": "^4.1.2",
+"@types/jasmine": "~2.8.6",
+"@types/jasminewd2": "~2.0.3",
+"@types/jquery": "^3.3.6",
+"@types/node": "^10.9.4",
"@types/tether": "^1.1.27",
-"@angular/common": "2.0.0",
-"@angular/compiler": "2.0.0",
-"@angular/core": "2.0.0",
-"@angular/forms": "2.0.0",
-"@angular/http": "2.0.0",
-"@angular/platform-browser": "2.0.0",
-"@angular/platform-browser-dynamic": "2.0.0",
-"@angular/router": "3.0.0",
"ace-builds": "^1.2.5",
"bootstrap": "4.0.0-alpha.5",
-"core-js": "^2.4.1",
+"core-js": "^2.5.7",
"font-awesome": "^4.6.3",
"jquery": "^3.3.1",
-"rxjs": "5.0.0-beta.12",
+"karma-phantomjs-launcher": "^1.0.4",
+"puppeteer": "^1.8.0",
+"rxjs": "6.2.2",
"tether": "^1.3.4",
"ts-helpers": "^1.1.1",
-"zone.js": "^0.6.23"
+"zone.js": "^0.8.26"
},
"devDependencies": {
-"angular-cli": "1.0.0-beta.15",
+"@angular-devkit/build-angular": "^0.8.3",
+"@angular/compiler-cli": "^6.1.8",
+"@types/request": "2.0.3",
"buffer-shims": "^1.0.0",
-"codelyzer": "~0.0.26",
+"codelyzer": "~4.2.1",
"copy": "^0.3.0",
-"jasmine-core": "2.4.1",
-"jasmine-spec-reporter": "2.5.0",
-"karma": "1.2.0",
-"karma-chrome-launcher": "^2.0.0",
-"karma-cli": "^1.0.1",
-"karma-jasmine": "^1.0.2",
-"karma-phantomjs-launcher": "^1.0.4",
-"karma-remap-istanbul": "0.6.0",
+"jasmine-core": "~2.99.1",
+"jasmine-spec-reporter": "~4.2.1",
+"karma": "^3.0.0",
+"karma-chrome-launcher": "~2.2.0",
+"karma-coverage-istanbul-reporter": "~2.0.0",
+"karma-jasmine": "^1.1.2",
+"karma-jasmine-html-reporter": "^0.2.2",
"phantomjs-prebuilt": "^2.1.14",
-"protractor": "4.0.5",
-"ts-node": "1.2.1",
-"tslint": "3.13.0",
-"typescript": "~2.0.3",
-"@types/request": "2.0.3"
+"protractor": "^5.4.1",
+"ts-node": "~5.0.1",
+"tslint": "^5.11.0",
+"typescript": "~2.7.2"
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/pom.xml
--
diff --git a/metron-interface/metron-config/pom.xml
b/metron-interface/metron-config/pom.xml
index fd76447..c6ef5b5 100644
--- a/metron-interface/metron-config/pom.xml
+++ b/metron-interface/metron-config/pom.xml
@@ -63,7 +63,7 @@
generate-resources
-ng build
+npm run build
npm
@@ -72,13 +72,13 @@
-npm test
+test
+npm testCI
- npm
+npm
-test
- test
+run testCI
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/scripts/prepend_license_header.sh
--
diff --git a/metron-interface/metron-config/scripts/prepend_license_header.sh
b/metron-interface/metron-config/scripts/prepend_license_header.sh
index 1957cd6..6bf004f 100755
--- a/metron-interface/metron-config/scripts/prepend_license_header.sh
+
[18/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
--
diff --git
a/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
b/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
new file mode 100644
index 000..58f7d26
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
@@ -0,0 +1,228 @@
+///
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+context('PCAP Tab', () => {
+
+ beforeEach(() => {
+cy.server();
+cy.route({
+ method: 'GET',
+ url: '/api/v1/user',
+ response: 'user'
+});
+
+cy.route('GET', 'config', 'fixture:config.json');
+cy.route('POST', 'search', 'fixture:search.json');
+
+cy.route({
+ method: 'GET',
+ url: '/api/v1/pcap?state=*',
+ response: []
+}).as('runningJobs');
+
+cy.visit('http://localhost:4200/login');
+cy.get('[name="user"]').type('user');
+cy.get('[name="password"]').type('password');
+cy.contains('LOG IN').click();
+ });
+
+ afterEach(() => {
+cy.get('.logout-link').click();
+ });
+
+ it('checking running jobs on navigating to PCAP tab', () => {
+cy.contains('PCAP').click();
+cy.wait('@runningJobs').its('url').should('include', '?state=RUNNING');
+ });
+
+ it('submitting PCAP job request', () => {
+cy.contains('PCAP').click();
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json')
+ .as('postingPcapJob');
+
+cy.get('[data-qe-id="ip-src-addr"]').type('222.123.111.000');
+cy.get('[data-qe-id="ip-dst-addr"]').type('111.123.222.000');
+cy.get('[data-qe-id="ip-src-port"]').type('');
+cy.get('[data-qe-id="ip-dst-port"]').type('');
+cy.get('[data-qe-id="protocol"]').type('24');
+cy.get('[data-qe-id="include-reverse"]').check();
+cy.get('[data-qe-id="packet-filter"]').type('filter');
+
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@postingPcapJob').then((xhr) => {
+ expect(xhr.request.body.ipSrcAddr).to.equal('222.123.111.000');
+ expect(xhr.request.body.ipDstAddr).to.equal('111.123.222.000');
+ expect(xhr.request.body.ipSrcPort).to.equal('');
+ expect(xhr.request.body.ipDstPort).to.equal('');
+ expect(xhr.request.body.protocol).to.equal('24');
+ expect(xhr.request.body.includeReverse).to.equal(true);
+ expect(xhr.request.body.packetFilter).to.equal('filter');
+});
+ });
+
+ it('requesting job status', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-01.json').as('jobStatusCheck');
+
+cy.contains('PCAP').click();
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@jobStatusCheck').its('url').should('include',
'/api/v1/pcap/job_1537878471649_0001');
+ });
+
+ it('process status in percentage', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-01.json').as('jobStatusCheck');
+
+cy.contains('PCAP').click();
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@jobStatusCheck');
+
+cy.contains('75%').should('be.visible');
+ });
+
+ it('getting pcap json', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-02.json').as('statusCheck');
+cy.route('GET', '/api/v1/pcap/*/pdml*',
'fixture:pcap.page-01.json').as('gettingPdml');
+
+cy.contains('PCAP').click();
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@statusCheck');
+
+cy.wait('@gettingPdml').its('url').should('include',
'/api/v1/pcap/job_1537878471649_0001/pdml?page=1');
+ });
+
+
+ it('rendering pcap table', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-02.json').as('statusCheck');
+cy.route('GET', '/api/v1/pcap/*/pdml*',
'fixture:pcap.page-01.json').as('g
[07/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-platform/metron-enrichment/README.md
--
diff --git a/metron-platform/metron-enrichment/README.md
b/metron-platform/metron-enrichment/README.md
index 8a53e71..c72970f 100644
--- a/metron-platform/metron-enrichment/README.md
+++ b/metron-platform/metron-enrichment/README.md
@@ -31,36 +31,22 @@ data format (e.g. a JSON Map structure with
`original_message` and
## Enrichment Architecture
-
+
### Unified Enrichment Topology
-There is an experimental unified enrichment topology which is shipped.
-Currently the architecture, as described above, has a split/join in
-order to perform enrichments in parallel. This poses some issues in
-terms of ease of tuning and reasoning about performance.
-
-In order to deal with these issues, there is an alternative enrichment
topology which
-uses data parallelism as opposed to the split/join task parallelism.
-This architecture uses a worker pool to fully enrich any message within
-a worker. This results in
+The unified enrichment topology uses data parallelism as opposed to the
deprecated
+split/join topology's task parallelism. This architecture uses a worker pool
to fully
+enrich any message within a worker. This results in
* Fewer bolts in the topology
* Each bolt fully operates on a message.
* Fewer network hops
-
-
-This architecture is fully backwards compatible; the only difference is
-how the enrichment will operate on each message (in one bolt where the
-split/join is done in a threadpool as opposed
+This architecture is fully backwards compatible with the old split-join
+topology; the only difference is how the enrichment will operate on each
+message (in one bolt where the split/join is done in a threadpool as opposed
to split across multiple bolts).
- Using It
-
-In order to use this, you will need to
-* Edit `$METRON_HOME/bin/start_enrichment_topology.sh` and adjust it to use
`remote-unified.yaml` instead of `remote.yaml`
-* Restart the enrichment topology.
-
Configuring It
There are two parameters which you might want to tune in this topology.
@@ -76,6 +62,19 @@ intel bolt, the configurations will be taken from the
respective join bolt
parallelism. When proper ambari support for this is added, we will add
its own property.
+### Split-Join Enrichment Topology
+
+The now-deprecated split/join topology is also available and performs
enrichments in parallel.
+This poses some issues in terms of ease of tuning and reasoning about
performance.
+
+
+
+ Using It
+
+In order to use the older, deprecated topology, you will need to
+* Edit `$METRON_HOME/bin/start_enrichment_topology.sh` and adjust it to use
`remote-splitjoin.yaml` instead of `remote-unified.yaml`
+* Restart the enrichment topology.
+
## Enrichment Configuration
The configuration for the `enrichment` topology, the topology primarily
@@ -85,7 +84,6 @@ defined by JSON documents stored in zookeeper.
There are two types of configurations at the moment, `global` and
`sensor` specific.
-
## Global Configuration
There are a few enrichments which have independent configurations, such
@@ -134,7 +132,6 @@ The configuration is a complex JSON object with the
following top level fields:
### The `enrichment` Configuration
-
| Field| Description
| Example |
|--|---|--|
| `fieldToTypeMap` | In the case of a simple HBase enrichment (i.e. a
key/value lookup), the mapping between fields and the enrichment types
associated with those fields must be known. This enrichment type is used as
part of the HBase key. Note: applies to hbaseEnrichment only. |
`"fieldToTypeMap" : { "ip_src_addr" : [ "asset_enrichment" ] }` |
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
--
diff --git
a/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEnrichmentBolt.java
b/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/GenericEn
[22/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
--
diff --git
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
index c33644f..83800af 100644
---
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
+++
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/BatchProfilerIntegrationTest.java
@@ -58,8 +58,11 @@ import static
org.apache.metron.profiler.spark.BatchProfilerConfig.TELEMETRY_INP
import static
org.apache.metron.profiler.spark.BatchProfilerConfig.TELEMETRY_INPUT_END;
import static
org.apache.metron.profiler.spark.BatchProfilerConfig.TELEMETRY_INPUT_FORMAT;
import static
org.apache.metron.profiler.spark.BatchProfilerConfig.TELEMETRY_INPUT_PATH;
+import static
org.apache.metron.profiler.spark.BatchProfilerConfig.TELEMETRY_INPUT_READER;
import static org.junit.Assert.assertTrue;
+import static org.apache.metron.profiler.spark.reader.TelemetryReaders.*;
+
/**
* An integration test for the {@link BatchProfiler}.
*/
@@ -159,8 +162,8 @@ public class BatchProfilerIntegrationTest {
@Test
public void testBatchProfilerWithJSON() throws Exception {
// the input telemetry is text/json stored in the local filesystem
+profilerProperties.put(TELEMETRY_INPUT_READER.getKey(), JSON.toString());
profilerProperties.put(TELEMETRY_INPUT_PATH.getKey(),
"src/test/resources/telemetry.json");
-profilerProperties.put(TELEMETRY_INPUT_FORMAT.getKey(), "text");
BatchProfiler profiler = new BatchProfiler();
profiler.run(spark, profilerProperties, getGlobals(), readerProperties,
getProfile());
@@ -170,20 +173,41 @@ public class BatchProfilerIntegrationTest {
@Test
public void testBatchProfilerWithORC() throws Exception {
-// re-write the test data as ORC
+// re-write the test data as column-oriented ORC
String pathToORC = tempFolder.getRoot().getAbsolutePath();
spark.read()
-.format("text")
+.format("json")
.load("src/test/resources/telemetry.json")
-.as(Encoders.STRING())
.write()
.mode("overwrite")
.format("org.apache.spark.sql.execution.datasources.orc")
.save(pathToORC);
// tell the profiler to use the ORC input data
+profilerProperties.put(TELEMETRY_INPUT_READER.getKey(), ORC.toString());
profilerProperties.put(TELEMETRY_INPUT_PATH.getKey(), pathToORC);
-profilerProperties.put(TELEMETRY_INPUT_FORMAT.getKey(),
"org.apache.spark.sql.execution.datasources.orc");
+
+BatchProfiler profiler = new BatchProfiler();
+profiler.run(spark, profilerProperties, getGlobals(), readerProperties,
getProfile());
+
+validateProfiles();
+ }
+
+ @Test
+ public void testBatchProfilerWithParquet() throws Exception {
+// re-write the test data as column-oriented ORC
+String inputPath = tempFolder.getRoot().getAbsolutePath();
+spark.read()
+.format("json")
+.load("src/test/resources/telemetry.json")
+.write()
+.mode("overwrite")
+.format("parquet")
+.save(inputPath);
+
+// tell the profiler to use the ORC input data
+profilerProperties.put(TELEMETRY_INPUT_READER.getKey(),
PARQUET.toString());
+profilerProperties.put(TELEMETRY_INPUT_PATH.getKey(), inputPath);
BatchProfiler profiler = new BatchProfiler();
profiler.run(spark, profilerProperties, getGlobals(), readerProperties,
getProfile());
@@ -206,7 +230,9 @@ public class BatchProfilerIntegrationTest {
.save(pathToCSV);
// tell the profiler to use the CSV input data
+// CSV is an example of needing to define both the reader and the input
format
profilerProperties.put(TELEMETRY_INPUT_PATH.getKey(), pathToCSV);
+profilerProperties.put(TELEMETRY_INPUT_READER.getKey(), "text");
profilerProperties.put(TELEMETRY_INPUT_FORMAT.getKey(), "csv");
// set a reader property; tell the reader to expect a header
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/function/reader/ColumnEncodedTelemetryReaderTest.java
--
diff --git
a/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/function/reader/ColumnEncodedTelemetryReaderTest.java
b/metron-analytics/metron-profiler-spark/src/test/java/org/apache/metron/profiler/spark/function/reader/ColumnEncodedTelemetryRead
[23/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/8bf3b6ec Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/8bf3b6ec Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/8bf3b6ec Branch: refs/heads/master Commit: 8bf3b6ec9c6871daadcaa2341d01082d2584e341 Parents: e7e19fb Author: mmiklavc Authored: Thu Nov 15 14:16:15 2018 -0700 Committer: Michael Miklavcic Committed: Thu Nov 15 16:51:22 2018 -0700 -- .gitignore | 1 + Upgrading.md|24 + dependencies_with_url.csv | 5 +- .../committer-utils/metron-committer-common | 358 + dev-utilities/committer-utils/prepare-commit| 241 +- dev-utilities/release-utils/README.md | 129 + dev-utilities/release-utils/metron-rc-check | 6 +- .../release-utils/prepare-release-candidate | 334 + metron-analytics/metron-maas-common/pom.xml | 2 +- .../metron-profiler-spark/README.md |53 +- .../metron/profiler/spark/BatchProfiler.java|21 +- .../profiler/spark/BatchProfilerConfig.java | 9 +- .../reader/ColumnEncodedTelemetryReader.java|84 + .../profiler/spark/reader/TelemetryReader.java |43 + .../profiler/spark/reader/TelemetryReaders.java | 110 + .../reader/TextEncodedTelemetryReader.java |83 + .../spark/BatchProfilerIntegrationTest.java |36 +- .../ColumnEncodedTelemetryReaderTest.java | 118 + .../spark/function/reader/IsValidJSON.java |38 + .../function/reader/TelemetryReadersTest.java |89 + .../reader/TextEncodedTelemetryReaderTest.java | 114 + metron-deployment/Kerberos-manual-setup.md | 154 +- .../roles/ambari_master/defaults/main.yml | 1 + .../ambari_master/tasks/elasticsearch_mpack.yml | 4 +- .../ansible/roles/bro/tasks/bro.yml | 4 +- .../ansible/roles/bro/tasks/dependencies.yml| 4 +- .../roles/bro/tasks/metron-bro-plugin-kafka.yml | 4 +- .../ansible/roles/librdkafka/defaults/main.yml | 4 +- metron-deployment/development/README.md |34 + metron-deployment/development/centos6/README.md |26 +- .../development/knox-demo-ldap.ldif | 101 + .../development/ubuntu14/README.md |26 +- .../configuration/metron-enrichment-env.xml | 8 +- .../configuration/metron-indexing-env.xml | 4 +- .../configuration/metron-profiler-env.xml | 2 +- .../CURRENT/configuration/metron-rest-env.xml |10 +- .../configuration/metron-security-env.xml | 186 + .../common-services/METRON/CURRENT/metainfo.xml | 3 + .../CURRENT/package/files/bro_index.template| 3 + .../package/files/metaalert_index.template | 4 +- .../CURRENT/package/files/snort_index.template | 3 + .../CURRENT/package/files/yaf_index.template| 3 + .../package/scripts/params/params_linux.py |27 +- .../CURRENT/package/scripts/rest_commands.py| 4 + .../METRON/CURRENT/package/templates/metron.j2 |16 + .../METRON/CURRENT/themes/metron_theme.json | 171 +- .../docker/rpm-docker/SPECS/metron.spec | 5 +- metron-interface/metron-alerts/cypress.json | 7 + .../metron-alerts/cypress/fixtures/config.json |23 + .../cypress/fixtures/pcap.page-01.json | 12383 +++ .../cypress/fixtures/pcap.status-00.json| 7 + .../cypress/fixtures/pcap.status-01.json| 7 + .../cypress/fixtures/pcap.status-02.json| 7 + .../metron-alerts/cypress/fixtures/search.json | 5647 + .../cypress/integration/pcap/pcap.spec.js | 228 + .../metron-alerts/package-lock.json | 1134 + metron-interface/metron-alerts/package.json |11 +- .../alert-details/alert-details.component.ts|10 +- .../alerts/alerts-list/alerts-list.component.ts | 9 +- .../table-view/table-view.component.spec.ts | 4 +- .../table-view/table-view.component.ts |49 +- .../tree-view/tree-view.component.spec.ts | 4 +- .../tree-view/tree-view.component.ts|20 +- .../alerts/meta-alerts/meta-alerts.component.ts | 2 - .../save-search/save-search.component.spec.ts | 4 +- .../alerts/save-search/save-search.component.ts |12 +- .../saved-searches.component.spec.ts| 5 +- .../saved-searches/saved-searches.component.ts |41 +- .../metron-alerts/src/app/app.component.html| 1 + .../metron-alerts/src/app/app.component.spec.ts | 4 + .../metron-alerts/src/app/app.module.ts |12 +- .../src/app/model/confirmation-type.ts |21 + .../metron-a
[11/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
b/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
index 02f1fd9..6549b4b 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-storm-settings/sensor-storm-settings.component.spec.ts
@@ -16,153 +16,161 @@
* limitations under the License.
*/
-import {async, TestBed, ComponentFixture} from '@angular/core/testing';
-import {SensorStormSettingsComponent} from './sensor-storm-settings.component';
-import {SharedModule} from '../../shared/shared.module';
-import {SimpleChanges, SimpleChange} from '@angular/core';
-import {SensorParserConfig} from '../../model/sensor-parser-config';
-import {SensorStormSettingsModule} from './sensor-storm-settings.module';
-import '../../rxjs-operators';
+import { async, TestBed, ComponentFixture } from '@angular/core/testing';
+import { SensorStormSettingsComponent } from
'./sensor-storm-settings.component';
+import { SharedModule } from '../../shared/shared.module';
+import { SimpleChanges, SimpleChange } from '@angular/core';
+import { SensorParserConfig } from '../../model/sensor-parser-config';
+import { SensorStormSettingsModule } from './sensor-storm-settings.module';
describe('Component: SensorStormSettingsComponent', () => {
-
-let fixture: ComponentFixture;
-let component: SensorStormSettingsComponent;
-let sensorParserConfig: SensorParserConfig = new SensorParserConfig();
-sensorParserConfig.sensorTopic = 'bro';
-sensorParserConfig.parserClassName =
'org.apache.metron.parsers.bro.BasicBroParser';
-sensorParserConfig.parserConfig = {};
-sensorParserConfig.numWorkers = 2;
-sensorParserConfig.numAckers = 2;
-sensorParserConfig.spoutParallelism = 2;
-sensorParserConfig.spoutNumTasks = 2;
-sensorParserConfig.parserParallelism = 2;
-sensorParserConfig.parserNumTasks = 2;
-sensorParserConfig.errorWriterParallelism = 2;
-sensorParserConfig.errorWriterNumTasks = 2;
-sensorParserConfig.spoutConfig = {'spoutConfigProp': 'spoutConfigValue1'};
-sensorParserConfig.stormConfig = {'stormConfigProp': 'stormConfigValue1'};
-
-beforeEach(async(() => {
-TestBed.configureTestingModule({
-imports: [SharedModule, SensorStormSettingsModule],
-});
-
-fixture = TestBed.createComponent(SensorStormSettingsComponent);
-component = fixture.componentInstance;
-}));
-
-it('should create an instance', () => {
-expect(component).toBeDefined();
-});
-
-it('should create an instance', () => {
-spyOn(component, 'init');
-let changes: SimpleChanges = {'showStormSettings': new
SimpleChange(false, true)};
-
-component.ngOnChanges(changes);
-expect(component.init).toHaveBeenCalled();
-
-changes = {'showStormSettings': new SimpleChange(true, false)};
-component.ngOnChanges(changes);
-expect(component.init['calls'].count()).toEqual(1);
-
-fixture.destroy();
+ let fixture: ComponentFixture;
+ let component: SensorStormSettingsComponent;
+ let sensorParserConfig: SensorParserConfig = new SensorParserConfig();
+ sensorParserConfig.sensorTopic = 'bro';
+ sensorParserConfig.parserClassName =
+'org.apache.metron.parsers.bro.BasicBroParser';
+ sensorParserConfig.parserConfig = {};
+ sensorParserConfig.numWorkers = 2;
+ sensorParserConfig.numAckers = 2;
+ sensorParserConfig.spoutParallelism = 2;
+ sensorParserConfig.spoutNumTasks = 2;
+ sensorParserConfig.parserParallelism = 2;
+ sensorParserConfig.parserNumTasks = 2;
+ sensorParserConfig.errorWriterParallelism = 2;
+ sensorParserConfig.errorWriterNumTasks = 2;
+ sensorParserConfig.spoutConfig = { spoutConfigProp: 'spoutConfigValue1' };
+ sensorParserConfig.stormConfig = { stormConfigProp: 'stormConfigValue1' };
+
+ beforeEach(async(() => {
+TestBed.configureTestingModule({
+ imports: [SharedModule, SensorStormSettingsModule]
});
-it('should initialise the fields', () => {
-
-component.init();
-expect(component.newSensorParserConfig).toEqual(new
SensorParserConfig());
-
-component.sensorParserConfig = sensorParserConfig;
-component.init();
-expect(component.newSensorParserConfig).toEqual(sensorParserConfig);
-expect(component.newSpoutConfig).toEqual('{\n\t"spoutConfigProp":
"spoutConfigValue1"\n}');
-expect(component.newStormConfig).toEqual('{\n\t"stormConfigProp":
"stormConfigValue1"\n}');
-
-fixture.destroy()
[02/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (cstella via mmiklavc)
METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (cstella via mmiklavc) Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e7e19fbb Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e7e19fbb Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e7e19fbb Branch: refs/heads/master Commit: e7e19fbb6491fa47d3794aebdac0280164afeb29 Parents: 5bfc08c Author: cstella Authored: Mon Oct 8 18:06:52 2018 -0600 Committer: Michael Miklavcic Committed: Thu Nov 15 16:51:13 2018 -0700 -- dependencies_with_url.csv | 33 ++-- .../METRON/CURRENT/configuration/metron-env.xml | 9 -- .../CURRENT/package/scripts/metron_service.py | 2 - .../package/scripts/params/params_linux.py | 3 +- .../METRON/CURRENT/themes/metron_theme.json | 10 -- .../rest/service/impl/MetaAlertServiceImpl.java | 2 +- metron-platform/elasticsearch-shaded/pom.xml| 28 +++- .../META-INF/log4j-provider.properties | 18 --- metron-platform/metron-elasticsearch/pom.xml| 29 +++- .../dao/ElasticsearchColumnMetadataDao.java | 82 +- .../elasticsearch/dao/ElasticsearchDao.java | 17 +- .../dao/ElasticsearchMetaAlertDao.java | 2 +- .../dao/ElasticsearchMetaAlertSearchDao.java| 6 +- .../dao/ElasticsearchMetaAlertUpdateDao.java| 4 +- .../dao/ElasticsearchRequestSubmitter.java | 13 +- .../dao/ElasticsearchRetrieveLatestDao.java | 27 ++-- .../dao/ElasticsearchSearchDao.java | 7 +- .../dao/ElasticsearchUpdateDao.java | 18 ++- .../utils/ElasticsearchClient.java | 156 +++ .../elasticsearch/utils/ElasticsearchUtils.java | 95 --- .../elasticsearch/utils/FieldMapping.java | 29 .../elasticsearch/utils/FieldProperties.java| 33 .../writer/ElasticsearchWriter.java | 22 +-- .../dao/ElasticsearchColumnMetadataDaoTest.java | 50 +++--- .../elasticsearch/dao/ElasticsearchDaoTest.java | 7 +- .../dao/ElasticsearchRequestSubmitterTest.java | 20 ++- .../ElasticsearchMetaAlertIntegrationTest.java | 9 +- .../ElasticsearchSearchIntegrationTest.java | 15 +- .../ElasticsearchUpdateIntegrationTest.java | 2 +- .../components/ElasticSearchComponent.java | 6 +- .../dao/metaalert/MetaAlertSearchDao.java | 4 +- .../dao/metaalert/MetaAlertIntegrationTest.java | 2 +- .../src/main/config/zookeeper/global.json | 2 +- pom.xml | 2 +- 34 files changed, 532 insertions(+), 232 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/e7e19fbb/dependencies_with_url.csv -- diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv index 53977f3..66497c3 100644 --- a/dependencies_with_url.csv +++ b/dependencies_with_url.csv @@ -256,12 +256,8 @@ io.dropwizard.metrics:metrics-json:jar:3.1.5:compile,ASLv2,https://github.com/dr io.dropwizard.metrics:metrics-jvm:jar:3.1.5:compile,ASLv2,https://github.com/dropwizard/metrics io.netty:netty-all:jar:4.0.23.Final:compile,ASLv2, io.netty:netty-all:jar:4.0.23.Final:provided,ASLv2, -<<< HEAD io.netty:netty-all:jar:4.1.17.Final:compile,ASLv2, -=== io.netty:netty-all:jar:4.1.23.Final:compile,ASLv2, -io.netty:netty:jar:3.10.5.Final:compile,Apache License, Version 2.0,http://netty.io/ ->>> apache/master io.netty:netty:jar:3.6.2.Final:compile,Apache License, Version 2.0,http://netty.io/ io.netty:netty:jar:3.7.0.Final:compile,Apache License, Version 2.0,http://netty.io/ io.netty:netty:jar:3.9.9.Final:compile,Apache License, Version 2.0,http://netty.io/ @@ -472,20 +468,21 @@ org.eclipse.persistence:org.eclipse.persistence.jpa:jar:2.6.4:compile,EPL 1.0,ht com.github.ben-manes.caffeine:caffeine:jar:2.6.2:compile,ASLv2,https://github.com/ben-manes/caffeine/blob/v2.6.2/LICENSE com.google.code.gson:gson:jar:2.2:compile,ASLv2,https://github.com/google/gson com.google.code.gson:gson:jar:2.8.2:compile,ASLv2,https://github.com/google/gson - org.codehaus.plexus:plexus-classworlds:jar:2.4:compile - org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile - org.codehaus.plexus:plexus-interpolation:jar:1.14:compile - org.codehaus.plexus:plexus-utils:jar:2.0.7:compile - org.jsoup:jsoup:jar:1.6.1:compile - org.sonatype.aether:aether-api:jar:1.12:compile - org.sonatype.aether:aether-connector-file:jar:1.12:compile - org.sonatype.aether:aether-connector-wagon:jar:1.12:compile - org.sonatype.aether:aether-impl:jar:1.12:compile - org.sonatype.aether:aether-spi:jar:1.12:compile - org.sonatype.aether:aether-util:jar:1.12:compile - org.sonatype.sisu:sisu-guice:jar:no_aop:3.0.2:compile - org.son
[04/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
--
diff --git
a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
index 2c90b1e..cc6191c 100644
---
a/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
+++
b/metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/paloalto/BasicPaloAltoFirewallParserTest.java
@@ -18,6 +18,7 @@
package org.apache.metron.parsers.paloalto;
import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
import org.apache.metron.parsers.AbstractParserConfigTest;
import org.json.simple.JSONObject;
@@ -25,6 +26,8 @@ import org.json.simple.parser.ParseException;
import org.junit.Before;
import org.junit.Test;
+import java.util.List;
+
public class BasicPaloAltoFirewallParserTest extends AbstractParserConfigTest {
@Before
@@ -32,6 +35,221 @@ public class BasicPaloAltoFirewallParserTest extends
AbstractParserConfigTest {
parser = new BasicPaloAltoFirewallParser();
}
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testParseSystem61() throws ParseException {
+final String SYSTEM_61 = "1,2017/08/11
12:37:58,00898659,SYSTEM,general,1,2017/08/11
11:37:58,vsys1,eventId_test,object_test,Futureuse1_test,futureuse2_test,management,high,Description_test,1354,0x0";
+
+JSONObject actual = parser.parse(SYSTEM_61.getBytes()).get(0);
+
+JSONObject expected = new JSONObject();
+expected.put(BasicPaloAltoFirewallParser.PaloAltoDomain, "1");
+expected.put(BasicPaloAltoFirewallParser.ReceiveTime, "2017/08/11
12:37:58");
+expected.put(BasicPaloAltoFirewallParser.SerialNum, "00898659");
+expected.put(BasicPaloAltoFirewallParser.Type, "SYSTEM");
+expected.put(BasicPaloAltoFirewallParser.ThreatContentType, "general");
+expected.put(BasicPaloAltoFirewallParser.ConfigVersion, "1");
+expected.put(BasicPaloAltoFirewallParser.GenerateTime, "2017/08/11
11:37:58");
+expected.put(BasicPaloAltoFirewallParser.VirtualSystem, "vsys1");
+expected.put(BasicPaloAltoFirewallParser.EventId, "eventId_test");
+expected.put(BasicPaloAltoFirewallParser.Object, "object_test");
+expected.put(BasicPaloAltoFirewallParser.Module, "management");
+expected.put(BasicPaloAltoFirewallParser.Severity, "high");
+expected.put(BasicPaloAltoFirewallParser.Description, "Description_test");
+expected.put(BasicPaloAltoFirewallParser.Seqno, "1354");
+expected.put(BasicPaloAltoFirewallParser.ActionFlags, "0x0");
+expected.put(BasicPaloAltoFirewallParser.ParserVersion, 61);
+expected.put("original_string", SYSTEM_61);
+expected.put("timestamp", actual.get("timestamp"));
+
+assertEquals(expected, actual);
+ }
+
+ @SuppressWarnings("unchecked")
+ @Test
+ public void testParseSystem80() throws ParseException {
+final String SYSTEM_80 = "1,2017/08/11
12:37:58,00898659,SYSTEM,general,1,2017/08/11
11:37:58,vsys1,eventId_test,object_test,Futureuse1_test,futureuse2_test,management,high,Description_test,1354,0x0,12,34,45,0,virSys1,dev-something200-01";
+
+JSONObject actual = parser.parse(SYSTEM_80.getBytes()).get(0);
+
+JSONObject expected = new JSONObject();
+expected.put(BasicPaloAltoFirewallParser.PaloAltoDomain, "1");
+expected.put(BasicPaloAltoFirewallParser.ReceiveTime, "2017/08/11
12:37:58");
+expected.put(BasicPaloAltoFirewallParser.SerialNum, "00898659");
+expected.put(BasicPaloAltoFirewallParser.Type, "SYSTEM");
+expected.put(BasicPaloAltoFirewallParser.ThreatContentType, "general");
+expected.put(BasicPaloAltoFirewallParser.ConfigVersion, "1");
+expected.put(BasicPaloAltoFirewallParser.GenerateTime, "2017/08/11
11:37:58");
+expected.put(BasicPaloAltoFirewallParser.VirtualSystem, "vsys1");
+expected.put(BasicPaloAltoFirewallParser.EventId, "eventId_test");
+expected.put(BasicPaloAltoFirewallParser.Object, "object_test");
+expected.put(BasicPaloAltoFirewallParser.Module, "management");
+expected.put(BasicPaloAltoFirewallParser.Severity, "high");
+expected.put(BasicPaloAltoFirewallParser.Description, "Description_test");
+expected.put(BasicPaloAltoFirewallParser.Seqno, "1354");
+expected.put(BasicPaloAltoFirewallParser.ActionFlags, "0x0");
+expected.put(BasicPaloAltoFirewallParser.DGH1, "12");
+expected.put(BasicPaloAltoFirewallParser.DGH2, "34");
+expected.put(BasicPaloAltoFirewallParser.DGH3, "45");
+expected.put(BasicPaloAltoFirewallParser.DGH4, "0");
+expected.put(BasicPaloAltoFirewallParser.VSYSName, "virSys1");
+
[12/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
b/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
index 647e02f..1ba297c 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-parser-config/sensor-parser-config.component.ts
@@ -15,30 +15,36 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import {Component, OnInit, ViewChild} from '@angular/core';
-import {FormGroup, Validators, FormControl} from '@angular/forms';
-import {SensorParserConfig} from '../../model/sensor-parser-config';
-import {SensorParserConfigService} from
'../../service/sensor-parser-config.service';
-import {Router, ActivatedRoute} from '@angular/router';
-import {MetronAlerts} from '../../shared/metron-alerts';
-import {SensorParserContext} from '../../model/sensor-parser-context';
-import {SensorEnrichmentConfigService} from
'../../service/sensor-enrichment-config.service';
-import {SensorEnrichmentConfig} from '../../model/sensor-enrichment-config';
-import {SensorFieldSchemaComponent} from
'../sensor-field-schema/sensor-field-schema.component';
-import {SensorRawJsonComponent} from
'../sensor-raw-json/sensor-raw-json.component';
-import {KafkaService} from '../../service/kafka.service';
-import {SensorIndexingConfigService} from
'../../service/sensor-indexing-config.service';
-import {IndexingConfigurations} from '../../model/sensor-indexing-config';
-import {RestError} from '../../model/rest-error';
-import {HdfsService} from '../../service/hdfs.service';
-import {GrokValidationService} from '../../service/grok-validation.service';
+import { Component, OnInit, ViewChild } from '@angular/core';
+import { FormGroup, Validators, FormControl } from '@angular/forms';
+import { SensorParserConfig } from '../../model/sensor-parser-config';
+import { SensorParserConfigService } from
'../../service/sensor-parser-config.service';
+import { Router, ActivatedRoute } from '@angular/router';
+import { MetronAlerts } from '../../shared/metron-alerts';
+import { SensorParserContext } from '../../model/sensor-parser-context';
+import { SensorEnrichmentConfigService } from
'../../service/sensor-enrichment-config.service';
+import { SensorEnrichmentConfig } from '../../model/sensor-enrichment-config';
+import { SensorFieldSchemaComponent } from
'../sensor-field-schema/sensor-field-schema.component';
+import { SensorRawJsonComponent } from
'../sensor-raw-json/sensor-raw-json.component';
+import { KafkaService } from '../../service/kafka.service';
+import { SensorIndexingConfigService } from
'../../service/sensor-indexing-config.service';
+import { IndexingConfigurations } from '../../model/sensor-indexing-config';
+import { RestError } from '../../model/rest-error';
+import { HdfsService } from '../../service/hdfs.service';
+import { GrokValidationService } from '../../service/grok-validation.service';
export enum Pane {
- GROK, RAWJSON, FIELDSCHEMA, THREATTRIAGE, STORMSETTINGS
+ GROK,
+ RAWJSON,
+ FIELDSCHEMA,
+ THREATTRIAGE,
+ STORMSETTINGS
}
export enum KafkaStatus {
- NO_TOPIC, NOT_EMITTING, EMITTING
+ NO_TOPIC,
+ NOT_EMITTING,
+ EMITTING
}
@Component({
@@ -46,9 +52,7 @@ export enum KafkaStatus {
templateUrl: 'sensor-parser-config.component.html',
styleUrls: ['sensor-parser-config.component.scss']
})
-
export class SensorParserConfigComponent implements OnInit {
-
sensorConfigForm: FormGroup;
transformsValidationForm: FormGroup;
@@ -73,7 +77,7 @@ export class SensorParserConfigComponent implements OnInit {
grokStatementValid = false;
availableParsers = {};
availableParserNames = [];
- grokStatement = '';
+ grokStatement = {};
patternLabel = '';
currentSensors = [];
@@ -81,7 +85,10 @@ export class SensorParserConfigComponent implements OnInit {
topicExists: boolean = false;
- transformsValidationResult: {map: any, keys: string[]} = {map: {}, keys: []};
+ transformsValidationResult: { map: any; keys: string[] } = {
+map: {},
+keys: []
+ };
transformsValidation: SensorParserContext = new SensorParserContext();
pane = Pane;
@@ -90,65 +97,94 @@ export class SensorParserConfigComponent implements OnInit {
kafkaStatus = KafkaStatus;
currentKafkaStatus = null;
- @ViewChild(SensorFieldSchemaComponent) sensorFieldSchema:
SensorFieldSchemaComponent;
- @ViewChild(SensorRawJsonComponent) sensorRawJson: SensorRawJsonComponent;
-
- constructor(private sensorParserConfigService: SensorParserConfigService,
private metronAlerts
[01/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (cstella via mmiklavc)
Repository: metron
Updated Branches:
refs/heads/master 0c4c622b9 -> fcd644ca7
http://git-wip-us.apache.org/repos/asf/metron/blob/e7e19fbb/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
index 8071e68..61dd0f6 100644
---
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
+++
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
@@ -39,6 +39,7 @@ import org.apache.metron.indexing.dao.search.SearchRequest;
import org.apache.metron.indexing.dao.search.SearchResponse;
import org.apache.metron.indexing.dao.search.SearchResult;
import org.apache.metron.integration.InMemoryComponent;
+import org.apache.metron.integration.utils.TestUtils;
import org.elasticsearch.action.bulk.BulkRequestBuilder;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequestBuilder;
@@ -200,7 +201,7 @@ public class ElasticsearchSearchIntegrationTest extends
SearchIntegrationTest {
config.setGlobalConfigSupplier( () ->
new HashMap() {{
put("es.clustername", "metron");
- put("es.port", "9300");
+ put("es.port", "9200");
put("es.ip", "localhost");
put("es.date.format", dateFormat);
}}
@@ -272,8 +273,10 @@ public class ElasticsearchSearchIntegrationTest extends
SearchIntegrationTest {
public void returns_column_metadata_for_specified_indices() throws Exception
{
// getColumnMetadata with only bro
{
+ //TODO: It shouldn't require an assertEventually() here as it should be
synchronous.
+ // Before merging, please figure out why.
+ TestUtils.assertEventually(() -> Assert.assertEquals(13,
dao.getColumnMetadata(Collections.singletonList("bro")).size()));
Map fieldTypes =
dao.getColumnMetadata(Collections.singletonList("bro"));
- Assert.assertEquals(13, fieldTypes.size());
Assert.assertEquals(FieldType.TEXT, fieldTypes.get("bro_field"));
Assert.assertEquals(FieldType.TEXT, fieldTypes.get("ttl"));
Assert.assertEquals(FieldType.KEYWORD, fieldTypes.get("guid"));
@@ -292,8 +295,10 @@ public class ElasticsearchSearchIntegrationTest extends
SearchIntegrationTest {
}
// getColumnMetadata with only snort
{
+ //TODO: It shouldn't require an assertEventually() here as it should be
synchronous.
+ // Before merging, please figure out why.
+ TestUtils.assertEventually(() -> Assert.assertEquals(14,
dao.getColumnMetadata(Collections.singletonList("snort")).size()));
Map fieldTypes =
dao.getColumnMetadata(Collections.singletonList("snort"));
- Assert.assertEquals(14, fieldTypes.size());
Assert.assertEquals(FieldType.INTEGER, fieldTypes.get("snort_field"));
Assert.assertEquals(FieldType.INTEGER, fieldTypes.get("ttl"));
Assert.assertEquals(FieldType.KEYWORD, fieldTypes.get("guid"));
@@ -313,8 +318,10 @@ public class ElasticsearchSearchIntegrationTest extends
SearchIntegrationTest {
@Override
public void returns_column_data_for_multiple_indices() throws Exception {
+//TODO: It shouldn't require an assertEventually() here as it should be
synchronous.
+// Before merging, please figure out why.
+TestUtils.assertEventually(() -> Assert.assertEquals(15,
dao.getColumnMetadata(Arrays.asList("bro", "snort")).size()));
Map fieldTypes =
dao.getColumnMetadata(Arrays.asList("bro", "snort"));
-Assert.assertEquals(15, fieldTypes.size());
Assert.assertEquals(FieldType.KEYWORD, fieldTypes.get("guid"));
Assert.assertEquals(FieldType.TEXT, fieldTypes.get("source:type"));
Assert.assertEquals(FieldType.IP, fieldTypes.get("ip_src_addr"));
http://git-wip-us.apache.org/repos/asf/metron/blob/e7e19fbb/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
index c5c0bc1..6f36790 100644
---
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchUpdateIntegrationTest.java
[19/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/cypress/fixtures/search.json
--
diff --git a/metron-interface/metron-alerts/cypress/fixtures/search.json
b/metron-interface/metron-alerts/cypress/fixtures/search.json
new file mode 100644
index 000..e2e03e4
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/search.json
@@ -0,0 +1,5647 @@
+{
+ "total":104593,
+ "results":[
+ {
+"id":"ad5cc7ea-5954-479f-8589-51f94b1c2f02",
+"source":{
+ "average":10.0,
+ "max":10.0,
+ "metron_alert":[
+ {
+ "msg":"'snort test alert'",
+ "sig_rev":"0",
+ "ip_dst_port":"49195",
+ "threatinteljoinbolt:joiner:ts":"1537279364136",
+ "ethsrc":"00:00:00:00:00:00",
+ "threat:triage:rules:0:comment":null,
+ "tcpseq":"0xC88832BC",
+ "enrichments:geo:ip_src_addr:longitude":"2.33870002",
+ "dgmlen":"44",
+ "enrichmentsplitterbolt:splitter:begin:ts":"1537279364122",
+ "enrichmentjoinbolt:joiner:ts":"1537279364128",
+ "adapter:geoadapter:begin:ts":"1537279364125",
+ "tcpwindow":"0xFAF0",
+ "threat:triage:rules:0:name":null,
+ "tcpack":"0x522C98B4",
+ "protocol":"TCP",
+ "source:type":"snort",
+ "adapter:threatinteladapter:end:ts":"1537279364133",
+ "ip_dst_addr":"192.168.138.158",
+ "original_string":"09/18/18-14:02:39.00
,1,999158,0,\"'snort test
alert'\",TCP,188.165.164.184,80,192.168.138.158,49195,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A**S*,0xC88832BC,0x522C98B4,,0xFAF0,128,0,1899,44,45056",
+ "adapter:hostfromjsonlistadapter:end:ts":"1537279364125",
+ "tos":"0",
+ "adapter:geoadapter:end:ts":"1537279364125",
+ "id":"1899",
+ "enrichments:geo:ip_src_addr:latitude":"48.8582",
+ "ip_src_addr":"188.165.164.184",
+ "threatintelsplitterbolt:splitter:end:ts":"1537279364130",
+ "threat:triage:rules:0:score":10,
+ "timestamp":1537279359000,
+ "ethdst":"00:00:00:00:00:00",
+
"enrichments:geo:ip_src_addr:location_point":"48.8582,2.33870002",
+ "threat:triage:rules:0:reason":null,
+ "enrichmentsplitterbolt:splitter:end:ts":"1537279364122",
+ "threat:triage:score":10.0,
+ "is_alert":"true",
+ "adapter:hostfromjsonlistadapter:begin:ts":"1537279364125",
+ "enrichments:geo:ip_src_addr:country":"FR",
+ "ttl":"128",
+ "metaalerts":[
+"ad5cc7ea-5954-479f-8589-51f94b1c2f02"
+ ],
+ "ethlen":"0x3C",
+ "iplen":"45056",
+ "ip_src_port":"80",
+ "threatintelsplitterbolt:splitter:begin:ts":"1537279364130",
+ "adapter:threatinteladapter:begin:ts":"1537279364133",
+ "tcpflags":"***A**S*",
+ "guid":"c6843745-203c-49e1-80ad-f060eb88c9b1",
+ "sig_id":"999158",
+ "sig_generator":"1"
+ },
+ {
+ "msg":"'snort test alert'",
+ "sig_rev":"0",
+ "ip_dst_port":"49195",
+ "threatinteljoinbolt:joiner:ts":"1537280091506",
+ "ethsrc":"00:00:00:00:00:00",
+ "threat:triage:rules:0:comment":null,
+ "tcpseq":"0xC88832BD",
+ "enrichments:geo:ip_src_addr:longitude":"2.33870002",
+ "dgmlen":"40",
+ "enrichmentsplitterbolt:splitter:begin:ts":"1537280091491",
+ "enrichmentjoinbolt:joiner:ts":"1537280091498",
+ "adapter:geoadapter:begin:ts":"1537280091493",
+ "tcpwindow":"0xFAF0",
+ "threat:triage:rules:0:name":null,
+ "tcpack":"0x522C999D",
+ "protocol":"TCP",
+ "source:type":"snort",
+ "adapter:threatinteladapter:end:ts":"1537280091503",
+ "ip_dst_addr":"192.168.138.158",
+ "original_string":"09/18/18-14:14:47.00
,1,999158,0,\"'snort test
alert'\",TCP,188.165.164.184,80,192.168.138.158,49195,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A,0xC88832BD,0x522C999D,,0xFAF0,128,0,1900,40,40960",
+ "adapter:hostfromjsonlistadapter:end:ts":"1537280091493",
+ "tos":"0",
+ "adapter:geoadapter:end:ts":"1537280091493",
+ "id":"1900",
+ "enrichments
[21/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
--
diff --git a/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
new file mode 100644
index 000..61082ed
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
@@ -0,0 +1,12383 @@
+{
+ "version":"0",
+ "creator":"wireshark/1.8.10",
+ "time":"Mon Sep 24 14:16:26 2018",
+ "captureFile":"",
+ "packets":[
+ {
+"protos":[
+ {
+ "name":"geninfo",
+ "pos":"0",
+ "showname":"General information",
+ "size":"722",
+ "hide":null,
+ "fields":[
+ {
+"name":"num",
+"pos":"0",
+"showname":"Number",
+"size":"722",
+"value":"1",
+"show":"1",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"len",
+"pos":"0",
+"showname":"Frame Length",
+"size":"722",
+"value":"2d2",
+"show":"722",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"caplen",
+"pos":"0",
+"showname":"Captured Length",
+"size":"722",
+"value":"2d2",
+"show":"722",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"timestamp",
+"pos":"0",
+"showname":"Captured Time",
+"size":"722",
+"value":"1458240269.373968000",
+"show":"Mar 17, 2016 18:44:29.373968000 UTC",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ }
+ ]
+ },
+ {
+ "name":"frame",
+ "pos":"0",
+ "showname":"Frame 1: 722 bytes on wire (5776 bits), 722 bytes
captured (5776 bits) on interface 0",
+ "size":"722",
+ "hide":null,
+ "fields":[
+ {
+"name":"frame.interface_id",
+"pos":"0",
+"showname":"Interface id: 0",
+"size":"0",
+"value":null,
+"show":"0",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.dlt",
+"pos":"0",
+"showname":"WTAP_ENCAP: 1",
+"size":"0",
+"value":null,
+"show":"1",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.time",
+"pos":"0",
+"showname":"Arrival Time: Mar 17, 2016 18:44:29.373968000
UTC",
+"size":"0",
+"value":null,
+"show":"Mar 17, 2016 18:44:29.373968000",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.offset_shift",
+"pos":"0",
+"showname":"Time shift for this packet: 0.0
seconds",
+"size":"0",
+"value":null,
+"show":"0.0",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.time_epoch",
+"pos":"0",
+"showname":"Epoch Time: 1458240269.373968000 seconds",
+"size":"0",
+"value":null,
+"show":"1458240269.373968000",
+"unmaskedvalue":null,
+"hide":nu
[10/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/src/app/service/sensor-enrichment-config.service.ts
--
diff --git
a/metron-interface/metron-config/src/app/service/sensor-enrichment-config.service.ts
b/metron-interface/metron-config/src/app/service/sensor-enrichment-config.service.ts
index 90c314b..bc26581 100644
---
a/metron-interface/metron-config/src/app/service/sensor-enrichment-config.service.ts
+++
b/metron-interface/metron-config/src/app/service/sensor-enrichment-config.service.ts
@@ -15,57 +15,69 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import {Injectable, Inject} from '@angular/core';
-import {Http, Headers, RequestOptions, Response} from '@angular/http';
-import {Observable} from 'rxjs/Observable';
-import {SensorEnrichmentConfig} from '../model/sensor-enrichment-config';
-import {HttpUtil} from '../util/httpUtil';
-import {IAppConfig} from '../app.config.interface';
-import {APP_CONFIG} from '../app.config';
+import { Injectable, Inject } from '@angular/core';
+import { HttpClient, HttpResponse } from '@angular/common/http';
+import { Observable } from 'rxjs';
+import { map, catchError } from 'rxjs/operators';
+import { SensorEnrichmentConfig } from '../model/sensor-enrichment-config';
+import { HttpUtil } from '../util/httpUtil';
+import { IAppConfig } from '../app.config.interface';
+import { APP_CONFIG } from '../app.config';
@Injectable()
export class SensorEnrichmentConfigService {
url = this.config.apiEndpoint + '/sensor/enrichment/config';
- defaultHeaders = {'Content-Type': 'application/json', 'X-Requested-With':
'XMLHttpRequest'};
- constructor(private http: Http, @Inject(APP_CONFIG) private config:
IAppConfig) {
- }
+ constructor(
+private http: HttpClient,
+@Inject(APP_CONFIG) private config: IAppConfig
+ ) {}
- public post(name: string, sensorEnrichmentConfig: SensorEnrichmentConfig):
Observable {
-return this.http.post(this.url + '/' + name,
JSON.stringify(sensorEnrichmentConfig),
- new RequestOptions({headers: new
Headers(this.defaultHeaders)}))
- .map(HttpUtil.extractData)
- .catch(HttpUtil.handleError);
+ public post(
+name: string,
+sensorEnrichmentConfig: SensorEnrichmentConfig
+ ): Observable {
+return this.http
+ .post(this.url + '/' + name, JSON.stringify(sensorEnrichmentConfig))
+ .pipe(
+map(HttpUtil.extractData),
+catchError(HttpUtil.handleError)
+ );
}
public get(name: string): Observable {
-return this.http.get(this.url + '/' + name, new RequestOptions({headers:
new Headers(this.defaultHeaders)}))
- .map(HttpUtil.extractData)
- .catch(HttpUtil.handleError);
+return this.http.get(this.url + '/' + name).pipe(
+ map(HttpUtil.extractData),
+ catchError(HttpUtil.handleError)
+);
}
public getAll(): Observable {
-return this.http.get(this.url, new RequestOptions({headers: new
Headers(this.defaultHeaders)}))
- .map(HttpUtil.extractData)
- .catch(HttpUtil.handleError);
+return this.http.get(this.url).pipe(
+ map(HttpUtil.extractData),
+ catchError(HttpUtil.handleError)
+);
}
- public deleteSensorEnrichments(name: string): Observable {
-return this.http.delete(this.url + '/' + name, new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
- .catch(HttpUtil.handleError);
+ public deleteSensorEnrichments(name: string) {
+return this.http
+ .delete>(this.url + '/' + name)
+ .pipe>(catchError(HttpUtil.handleError));
}
public getAvailableEnrichments(): Observable {
-return this.http.get(this.url + '/list/available/enrichments', new
RequestOptions({headers: new Headers(this.defaultHeaders)}))
-.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError);
+return this.http.get(this.url + '/list/available/enrichments').pipe(
+ map(HttpUtil.extractData),
+ catchError(HttpUtil.handleError)
+);
}
public getAvailableThreatTriageAggregators(): Observable {
-return this.http.get(this.url +
'/list/available/threat/triage/aggregators',
-new RequestOptions({headers: new Headers(this.defaultHeaders)}))
-.map(HttpUtil.extractData)
-.catch(HttpUtil.handleError);
+return this.http
+ .get(this.url + '/list/available/threat/triage/aggregators')
+ .pipe(
+map(HttpUtil.extractData),
+catchError(HttpUtil.handleError)
+ );
}
-
}
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/src/app/service/sensor-indexing-config.service.spec.ts
--
diff --git
a/metron-interface/metron-config/src/app/service/sensor-indexing-config.service.spec.ts
b/metron-interface/metron-conf
[16/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/package-lock.json
--
diff --git a/metron-interface/metron-config/package-lock.json
b/metron-interface/metron-config/package-lock.json
index fff8cb7..f2a264d 100644
--- a/metron-interface/metron-config/package-lock.json
+++ b/metron-interface/metron-config/package-lock.json
@@ -4,180 +4,530 @@
"lockfileVersion": 1,
"requires": true,
"dependencies": {
-"@angular-cli/ast-tools": {
- "version": "1.0.16",
- "resolved":
"https://registry.npmjs.org/@angular-cli/ast-tools/-/ast-tools-1.0.16.tgz";,
- "integrity": "sha1-YxmULBol+4TjKUID6fejJmMvzlA=",
+"@angular-devkit/architect": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/architect/-/architect-0.8.3.tgz";,
+ "integrity":
"sha512-cFku50grgEJPg1CZZ0DXt4CkA6WnV6zN3hCXzpWbOfc/Id923Mml/jsEaoByeXHsRqb5rIZKZAhz7R509ya8OQ==",
"dev": true,
"requires": {
-"@angular/tsc-wrapped": "0.5.2",
-"denodeify": "1.2.1",
-"rxjs": "5.1.0",
-"typescript": "2.0.10"
+"@angular-devkit/core": "0.8.3",
+"rxjs": "6.2.2"
},
"dependencies": {
-"@angular/tsc-wrapped": {
- "version": "0.5.2",
- "resolved":
"https://registry.npmjs.org/@angular/tsc-wrapped/-/tsc-wrapped-0.5.2.tgz";,
- "integrity": "sha1-Lt30csRn/LM06pTe3aqnGZDFpII=",
+"@angular-devkit/core": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/core/-/core-0.8.3.tgz";,
+ "integrity":
"sha512-2KHt5osMs3zACYXev20ZU5SXdWoinoKwZkj2caj2LCj9W7QNHmsz34QvaygNq7YdJzF3jkXkdy0GSUgUgDke0w==",
"dev": true,
"requires": {
-"tsickle": "0.2.5"
+"ajv": "6.4.0",
+"chokidar": "2.0.4",
+"rxjs": "6.2.2",
+"source-map": "0.5.7"
}
-},
-"denodeify": {
- "version": "1.2.1",
- "resolved":
"https://registry.npmjs.org/denodeify/-/denodeify-1.2.1.tgz";,
- "integrity": "sha1-OjYof1A05pnnV3kBBSwubJQlFjE=",
+}
+ }
+},
+"@angular-devkit/build-angular": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/build-angular/-/build-angular-0.8.3.tgz";,
+ "integrity":
"sha512-NWwWV+6apvCGmllWjwwy9Pmj5uK5tVGL/xIVQgSGC5waLmW/vFWNRXCI50ji5UPP+vAeRi/pWdXWMxuoVA08FA==",
+ "dev": true,
+ "requires": {
+"@angular-devkit/architect": "0.8.3",
+"@angular-devkit/build-optimizer": "0.8.3",
+"@angular-devkit/build-webpack": "0.8.3",
+"@angular-devkit/core": "0.8.3",
+"@ngtools/webpack": "6.2.3",
+"ajv": "6.4.0",
+"autoprefixer": "8.6.5",
+"circular-dependency-plugin": "5.0.2",
+"clean-css": "4.2.1",
+"copy-webpack-plugin": "4.5.2",
+"file-loader": "1.1.11",
+"glob": "7.1.3",
+"html-webpack-plugin": "3.2.0",
+"istanbul": "0.4.5",
+"istanbul-instrumenter-loader": "3.0.1",
+"karma-source-map-support": "1.3.0",
+"less": "3.8.1",
+"less-loader": "4.1.0",
+"license-webpack-plugin": "1.5.0",
+"loader-utils": "1.1.0",
+"mini-css-extract-plugin": "0.4.3",
+"minimatch": "3.0.4",
+"node-sass": "4.9.3",
+"opn": "5.3.0",
+"parse5": "4.0.0",
+"portfinder": "1.0.17",
+"postcss": "6.0.23",
+"postcss-import": "11.1.0",
+"postcss-loader": "2.1.6",
+"postcss-url": "7.3.2",
+"raw-loader": "0.5.1",
+"rxjs": "6.2.2",
+"sass-loader": "7.1.0",
+"semver": "5.5.1",
+"source-map-loader": "0.2.4",
+"source-map-support": "0.5.9",
+"stats-webpack-plugin": "0.6.2",
+"style-loader": "0.21.0",
+"stylus": "0.54.5",
+"stylus-loader": "3.0.2",
+"tree-kill": "1.2.0",
+"uglifyjs-webpack-plugin": "1.3.0",
+"url-loader": "1.1.1",
+"webpack": "4.19.1",
+"webpack-dev-middleware": "3.3.0",
+"webpack-dev-server": "3.1.8",
+"webpack-merge": "4.1.4",
+"webpack-sources": "1.3.0",
+"webpack-subresource-integrity": "1.1.0-rc.6"
+ },
+ "dependencies": {
+"@angular-devkit/core": {
+ "version": "0.8.3",
+ "resolved":
"https://registry.npmjs.org/@angular-devkit/core/-/core-0.8.3.tgz";,
+ "integrity":
"sha512-2KHt5osMs3zACYXev20ZU5SXdWoinoKwZkj2caj2LCj9W7QNHmsz34QvaygNq7YdJzF3jkXkdy0GSUgUgDke0w==",
+ "dev": true,
+ "requires": {
+"ajv": "6.4.0",
+"chokidar": "2.0.4",
+"rxjs": "6.2.2",
+"source-map": "0.5.7"
+ }
+}
+ }
+},
+"@angular-devkit/build-optimizer": {
+ "version": "0.8.3",
+
[08/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java
new file mode 100644
index 000..4e0b2fe
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/main/java/org/apache/metron/elasticsearch/client/ElasticsearchClientFactory.java
@@ -0,0 +1,189 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.client;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.security.KeyManagementException;
+import java.security.KeyStore;
+import java.security.KeyStoreException;
+import java.security.NoSuchAlgorithmException;
+import java.security.cert.CertificateException;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Optional;
+import javax.net.ssl.SSLContext;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.nio.reactor.IOReactorConfig;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.SSLContexts;
+import org.apache.metron.elasticsearch.config.ElasticsearchClientConfig;
+import org.apache.metron.elasticsearch.utils.ElasticsearchUtils;
+import org.apache.metron.elasticsearch.utils.ElasticsearchUtils.HostnamePort;
+import org.elasticsearch.client.RestClient;
+import org.elasticsearch.client.RestClientBuilder;
+import org.elasticsearch.client.RestHighLevelClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Main entry point to create the ES client.
+ */
+public class ElasticsearchClientFactory {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+ private static final String ES_SETTINGS_KEY = "es.client.settings"; // es
config key in global config
+
+ /**
+ * Creates an Elasticsearch client from settings provided via the global
config.
+ *
+ * @return new client
+ */
+ public static ElasticsearchClient create(Map globalConfig) {
+ElasticsearchClientConfig esClientConfig = new ElasticsearchClientConfig(
+getEsSettings(globalConfig));
+HttpHost[] httpHosts = getHttpHosts(globalConfig,
esClientConfig.getConnectionScheme());
+RestClientBuilder builder = RestClient.builder(httpHosts);
+
+builder.setRequestConfigCallback(reqConfigBuilder -> {
+ // Modifies request config builder with connection and socket timeouts.
+ //
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/_timeouts.html
+
reqConfigBuilder.setConnectTimeout(esClientConfig.getConnectTimeoutMillis());
+
reqConfigBuilder.setSocketTimeout(esClientConfig.getSocketTimeoutMillis());
+ return reqConfigBuilder;
+});
+
builder.setMaxRetryTimeoutMillis(esClientConfig.getMaxRetryTimeoutMillis());
+
+builder.setHttpClientConfigCallback(clientBuilder -> {
+
clientBuilder.setDefaultIOReactorConfig(getIOReactorConfig(esClientConfig));
+
clientBuilder.setDefaultCredentialsProvider(getCredentialsProvider(esClientConfig));
+ clientBuilder.setSSLContext(getSSLContext(esClientConfig));
+ return clientBuilder;
+});
+
+RestClient lowLevelClient = builder.build();
+RestHighLevelClient client = new RestHighLevelClient(lowLevelClient);
+return new ElasticsearchClient(lowLevelClient, client);
+ }
+
+ private static Map getEsSettings(Map
globalConfig) {
+return (Map) globalConfig.getOrDefault(ES_SETTINGS_KEY,
new HashMap<>());
+ }
+
+ private static HttpHost[] getHttpHosts(Map
glo
[09/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
--
diff --git
a/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
b/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
index 5488209..b8cdf1f 100644
---
a/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
+++
b/metron-interface/metron-config/src/app/shared/sample-data/sample-data.component.spec.ts
@@ -15,17 +15,15 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-import {async, TestBed, ComponentFixture} from '@angular/core/testing';
-import {KafkaService} from '../../service/kafka.service';
-import {Observable} from 'rxjs/Observable';
-import {SampleDataComponent} from './sample-data.component';
-import {SharedModule} from '../shared.module';
-import '../../rxjs-operators';
+import { async, TestBed, ComponentFixture } from '@angular/core/testing';
+import { KafkaService } from '../../service/kafka.service';
+import { Observable, throwError } from 'rxjs';
+import { SampleDataComponent } from './sample-data.component';
+import { SharedModule } from '../shared.module';
class MockKafkaService {
_sample: string[];
- _sampleCounter: number = 0;
-
+ _sampleCounter = 0;
public setSample(sampleMessages: string[]): void {
this._sample = sampleMessages;
@@ -33,7 +31,6 @@ class MockKafkaService {
}
public sample(name: string): Observable {
-
if (this._sampleCounter < this._sample.length) {
return Observable.create(observer => {
observer.next(this._sample[this._sampleCounter++]);
@@ -41,7 +38,7 @@ class MockKafkaService {
});
}
-return Observable.throw('Error');
+return throwError('Error');
}
}
@@ -58,24 +55,21 @@ describe('SampleDataComponent', () => {
beforeEach(async(() => {
TestBed.configureTestingModule({
imports: [SharedModule],
- declarations: [ SampleDataComponent],
+ declarations: [SampleDataComponent],
providers: [
SampleDataComponent,
-{provide: KafkaService, useClass: MockKafkaService}
+{ provide: KafkaService, useClass: MockKafkaService }
]
});
-
fixture = TestBed.createComponent(SampleDataComponent);
sampleDataComponent = fixture.componentInstance;
-kafkaService = fixture.debugElement.injector.get(KafkaService);
-
+kafkaService = TestBed.get(KafkaService);
}));
it('can instantiate SampleDataComponent', async(() => {
expect(sampleDataComponent instanceof SampleDataComponent).toBe(true);
}));
-
it('should emmit messages', async(() => {
let expectedMessage;
let successCount = 0;
@@ -140,11 +134,9 @@ describe('SampleDataComponent', () => {
sampleDataComponent.getPreviousSample();
expect(successCount).toEqual(7);
expect(failureCount).toEqual(1);
-
}));
it('should emmit messages on blur', async(() => {
-
let expectedMessage;
let successCount = 0;
@@ -155,9 +147,10 @@ describe('SampleDataComponent', () => {
expect(message).toEqual(expectedMessage);
});
-
expectedMessage = 'This is a simple message';
-fixture.debugElement.nativeElement.querySelector('textarea').value =
expectedMessage;
+fixture.debugElement.nativeElement.querySelector(
+ 'textarea'
+).value = expectedMessage;
sampleDataComponent.onBlur();
expect(successCount).toEqual(1);
@@ -165,16 +158,16 @@ describe('SampleDataComponent', () => {
expect(sampleDataComponent.sampleData.length).toEqual(1);
expect(sampleDataComponent.sampleData[0]).toEqual(expectedMessage);
-
expectedMessage = '';
-fixture.debugElement.nativeElement.querySelector('textarea').value =
expectedMessage;
+fixture.debugElement.nativeElement.querySelector(
+ 'textarea'
+).value = expectedMessage;
sampleDataComponent.onBlur();
expect(successCount).toEqual(2);
expect(sampleDataComponent.sampleDataIndex).toEqual(0);
expect(sampleDataComponent.sampleData.length).toEqual(1);
-
expectedMessage = sampleMessages[0];
sampleDataComponent.getNextSample();
@@ -182,7 +175,5 @@ describe('SampleDataComponent', () => {
expect(sampleDataComponent.sampleDataIndex).toEqual(1);
expect(sampleDataComponent.sampleData.length).toEqual(2);
expect(sampleDataComponent.sampleData[1]).toEqual(sampleMessages[0]);
-
}));
-
});
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-interface/metron-config/src/app/util/httpUtil.ts
--
diff --git a/metron-interface/metron-config/src/app/util/httpUtil.ts
b/metron-interface/metron-config/src/app/util/httpUtil.ts
index dfcb61f..d8a21a5 10064
[24/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc)
METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/fcd644ca Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/fcd644ca Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/fcd644ca Branch: refs/heads/master Commit: fcd644ca77394d48d460c460b672a23d6594f49b Parents: 0c4c622 8bf3b6e Author: mmiklavc Authored: Thu Nov 15 17:03:18 2018 -0700 Committer: Michael Miklavcic Committed: Thu Nov 15 17:03:55 2018 -0700 -- Upgrading.md| 7 + dependencies_with_url.csv | 2 + metron-deployment/Kerberos-manual-setup.md | 154 +--- .../METRON/CURRENT/configuration/metron-env.xml | 9 - .../CURRENT/package/scripts/metron_service.py | 2 - .../package/scripts/params/params_linux.py | 3 +- .../METRON/CURRENT/themes/metron_theme.json | 10 - .../rest/service/impl/MetaAlertServiceImpl.java | 2 +- metron-platform/elasticsearch-shaded/pom.xml| 47 +--- .../META-INF/log4j-provider.properties | 18 -- metron-platform/metron-common/README.md | 48 ++-- .../src/main/config/zookeeper/global.json | 1 - .../common/configuration/ConfigOption.java | 7 + metron-platform/metron-elasticsearch/README.md | 45 +++- metron-platform/metron-elasticsearch/pom.xml| 32 ++- .../client/ElasticsearchClient.java | 245 +++ .../client/ElasticsearchClientFactory.java | 189 ++ .../config/ElasticsearchClientConfig.java | 187 ++ .../config/ElasticsearchClientOptions.java | 60 + .../dao/ElasticsearchColumnMetadataDao.java | 101 +++- .../elasticsearch/dao/ElasticsearchDao.java | 21 +- .../dao/ElasticsearchMetaAlertDao.java | 2 +- .../dao/ElasticsearchMetaAlertSearchDao.java| 6 +- .../dao/ElasticsearchMetaAlertUpdateDao.java| 4 +- .../dao/ElasticsearchRequestSubmitter.java | 13 +- .../dao/ElasticsearchRetrieveLatestDao.java | 28 ++- .../dao/ElasticsearchSearchDao.java | 19 +- .../dao/ElasticsearchUpdateDao.java | 19 +- .../elasticsearch/utils/ElasticsearchUtils.java | 182 ++ .../elasticsearch/utils/FieldMapping.java | 32 +++ .../elasticsearch/utils/FieldProperties.java| 36 +++ .../writer/ElasticsearchWriter.java | 26 +- .../dao/ElasticsearchColumnMetadataDaoTest.java | 59 ++--- .../elasticsearch/dao/ElasticsearchDaoTest.java | 8 +- .../dao/ElasticsearchRequestSubmitterTest.java | 23 +- .../dao/ElasticsearchUpdateDaoTest.java | 12 +- .../ElasticsearchMetaAlertIntegrationTest.java | 9 +- .../ElasticsearchSearchIntegrationTest.java | 144 ++- .../ElasticsearchUpdateIntegrationTest.java | 2 +- .../components/ElasticSearchComponent.java | 6 +- .../dao/metaalert/MetaAlertSearchDao.java | 4 +- .../dao/metaalert/MetaAlertIntegrationTest.java | 2 +- .../src/main/config/zookeeper/global.json | 2 +- 43 files changed, 1134 insertions(+), 694 deletions(-) --
[03/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
new file mode 100644
index 000..354322a
--- /dev/null
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/RestFunctions.java
@@ -0,0 +1,388 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.stellar.dsl.functions;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.http.HttpEntity;
+import org.apache.http.HttpHost;
+import org.apache.http.auth.AuthScope;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.CredentialsProvider;
+import org.apache.http.client.config.RequestConfig;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.protocol.HttpClientContext;
+import org.apache.http.impl.client.BasicCredentialsProvider;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
+import org.apache.http.util.EntityUtils;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.stellar.common.utils.JSONUtils;
+import org.apache.metron.stellar.dsl.Context;
+import org.apache.metron.stellar.dsl.ParseException;
+import org.apache.metron.stellar.dsl.Stellar;
+import org.apache.metron.stellar.dsl.StellarFunction;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.TimeUnit;
+
+import static java.lang.String.format;
+import static org.apache.metron.stellar.dsl.Context.Capabilities.GLOBAL_CONFIG;
+import static
org.apache.metron.stellar.dsl.functions.RestConfig.POOLING_DEFAULT_MAX_PER_RUOTE;
+import static
org.apache.metron.stellar.dsl.functions.RestConfig.POOLING_MAX_TOTAL;
+import static
org.apache.metron.stellar.dsl.functions.RestConfig.STELLAR_REST_SETTINGS;
+
+/**
+ * Defines functions that enable REST requests with proper result and error
handling. Depends on an
+ * Apache HttpComponents client being supplied as a Stellar HTTP_CLIENT
capability. Exposes various Http settings
+ * including authentication, proxy and timeouts through the global config with
the option to override any settings
+ * through a config object supplied in the expression.
+ */
+public class RestFunctions {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ /**
+ * Get an argument from a list of arguments.
+ *
+ * @param index The index within the list of arguments.
+ * @param clazz The type expected.
+ * @param args All of the arguments.
+ * @param The type of the argument expected.
+ */
+ public static T getArg(int index, Class clazz, List args) {
+
+if(index >= args.size()) {
+ throw new IllegalArgumentException(format("Expected at least %d
argument(s), found %d", index+1, args.size()));
+}
+
+return ConversionUtils.convert(args.get(index), clazz);
+ }
+
+ @Stellar(
+ namespace = "REST",
+ name = "GET",
+ description = "Performs a REST GET request and parses the JSON
results into a map.",
+ params = {
+
[06/24] metron git commit: METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API (mmiklavc via mmiklavc) closes apache/metron#1242
http://git-wip-us.apache.org/repos/asf/metron/blob/8bf3b6ec/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
--
diff --git
a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
index 489eb00..1cf9fb7 100644
---
a/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
+++
b/metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/fireeye/BasicFireEyeParser.java
@@ -1,4 +1,4 @@
-/**
+/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
@@ -15,18 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
+
package org.apache.metron.parsers.fireeye;
import com.google.common.base.Joiner;
import com.google.common.collect.ArrayListMultimap;
import com.google.common.collect.Multimap;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.metron.parsers.utils.ParserUtils;
-import org.apache.metron.parsers.BasicParser;
-import org.json.simple.JSONObject;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
+import java.lang.invoke.MethodHandles;
+import java.nio.charset.StandardCharsets;
import java.text.ParseException;
import java.util.ArrayList;
import java.util.Arrays;
@@ -34,189 +30,162 @@ import java.util.List;
import java.util.Map;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.metron.parsers.BasicParser;
+import org.apache.metron.parsers.utils.ParserUtils;
+import org.json.simple.JSONObject;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
-public class BasicFireEyeParser extends BasicParser {
-
- private static final long serialVersionUID = 6328907550159134550L;
- protected static final Logger LOG = LoggerFactory
- .getLogger(BasicFireEyeParser.class);
-
-
- String tsRegex
="([a-zA-Z]{3})\\s+(\\d+)\\s+(\\d+\\:\\d+\\:\\d+)\\s+(\\d+\\.\\d+\\.\\d+\\.\\d+)";
-
-
- Pattern tsPattern = Pattern.compile(tsRegex);
- // private transient static MetronGrok grok;
- // private transient static InputStream pattern_url;
-
- public BasicFireEyeParser() throws Exception {
- // pattern_url =
getClass().getClassLoader().getResourceAsStream(
- // "patterns/fireeye");
- //
- // File file = ParserUtils.stream2file(pattern_url);
- // grok = MetronGrok.create(file.getPath());
- //
- // grok.compile("%{FIREEYE_BASE}");
- }
-
- @Override
- public void configure(Map parserConfig) {
-
- }
-
- @Override
- public void init() {
-
- }
-
- @Override
- public List parse(byte[] raw_message) {
- String toParse = "";
- List messages = new ArrayList<>();
- try {
-
- toParse = new String(raw_message, "UTF-8");
-
- // String[] mTokens = toParse.split(" ");
-
- String positveIntPattern = "<[1-9][0-9]*>";
- Pattern p = Pattern.compile(positveIntPattern);
- Matcher m = p.matcher(toParse);
-
- String delimiter = "";
-
- while (m.find()) {
- delimiter = m.group();
-
- }
-
- if (!StringUtils.isBlank(delimiter)) {
- String[] tokens = toParse.split(delimiter);
-
- if (tokens.length > 1)
- toParse = delimiter + tokens[1];
-
- }
-
- JSONObject toReturn = parseMessage(toParse);
-
- toReturn.put("timestamp",
getTimeStamp(toParse,delimiter));
- messages.add(toReturn);
- return messages;
-
- } catch (Exception e) {
- e.printStackTrace();
- return null;
- }
-
- }
-
- private long getTimeStamp(String toParse,String delimiter) throws
ParseException {
-
- long ts = 0;
- String month = null;
- String day = null;
- String time = null;
- Matcher tsMatcher = tsPattern.matcher(toParse);
- if (tsMatcher.find()) {
- month = tsMatcher.group(1);
- day = tsMatcher.group(2);
-
metron git commit: METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver (mmiklavc via mmiklavc) closes apache/metron#1251
Repository: metron
Updated Branches:
refs/heads/master b9461e765 -> 85cd21aa0
METRON-1853: Add shutdown hook to Stellar BaseFunctionResolver (mmiklavc via
mmiklavc) closes apache/metron#1251
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/85cd21aa
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/85cd21aa
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/85cd21aa
Branch: refs/heads/master
Commit: 85cd21aa0f5045184c168248dc2b81c1cfd41ddd
Parents: b9461e7
Author: mmiklavc
Authored: Tue Nov 6 18:09:56 2018 -0700
Committer: Michael Miklavcic
Committed: Tue Nov 6 18:09:56 2018 -0700
--
.../ElasticsearchSearchIntegrationTest.java | 1 -
.../metron/stellar/dsl/StellarFunction.java | 9 +-
.../metron/stellar/dsl/StellarFunctions.java| 5 +
.../resolver/BaseFunctionResolver.java | 44 +
.../functions/resolver/FunctionResolver.java| 14 +-
.../stellar/dsl/functions/BasicStellarTest.java | 20 ++-
.../resolver/BaseFunctionResolverTest.java | 169 +++
7 files changed, 251 insertions(+), 11 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/85cd21aa/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
index 1d2d48e..8187468 100644
---
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
+++
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchSearchIntegrationTest.java
@@ -25,7 +25,6 @@ import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
-import org.adrianwalker.multilinestring.Multiline;
import org.apache.metron.common.Constants;
import org.apache.metron.common.utils.JSONUtils;
import org.apache.metron.elasticsearch.dao.ElasticsearchDao;
http://git-wip-us.apache.org/repos/asf/metron/blob/85cd21aa/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunction.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunction.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunction.java
index efdd185..4fabfaf 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunction.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunction.java
@@ -17,10 +17,17 @@
*/
package org.apache.metron.stellar.dsl;
+import java.io.Closeable;
+import java.io.IOException;
import java.util.List;
-public interface StellarFunction {
+public interface StellarFunction extends Closeable {
Object apply(List args, Context context) throws ParseException;
void initialize(Context context);
boolean isInitialized();
+
+ @Override
+ default void close() throws IOException {
+
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/85cd21aa/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunctions.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunctions.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunctions.java
index dfec90e..73df82f 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunctions.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/StellarFunctions.java
@@ -18,6 +18,7 @@
package org.apache.metron.stellar.dsl;
+import java.io.IOException;
import org.apache.metron.stellar.dsl.functions.resolver.FunctionResolver;
import
org.apache.metron.stellar.dsl.functions.resolver.SingletonFunctionResolver;
@@ -30,4 +31,8 @@ public class StellarFunctions {
public static void initialize(Context context) {
SingletonFunctionResolver.getInstance().initialize(context);
}
+
+ public static void close() throws IOException {
+SingletonFunctionResolver.getInstance().close();
+ }
}
http://git-wip-us.apache.org/repos/asf/metron/blob/85cd21aa/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/dsl/functions/resolver/BaseFunctionResol
metron git commit: METRON-1855: Make unified enrichment topology the default and deprecate split-join (mmiklavc via mmiklavc) closes apache/metron#1252
Repository: metron
Updated Branches:
refs/heads/master fdfca3b26 -> bf6b07f7c
METRON-1855: Make unified enrichment topology the default and deprecate
split-join (mmiklavc via mmiklavc) closes apache/metron#1252
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/bf6b07f7
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/bf6b07f7
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/bf6b07f7
Branch: refs/heads/master
Commit: bf6b07f7cbea3d210878554c7ce7a1bc091b59ee
Parents: fdfca3b
Author: mmiklavc
Authored: Mon Nov 5 16:30:43 2018 -0700
Committer: Michael Miklavcic
Committed: Mon Nov 5 16:30:43 2018 -0700
--
Upgrading.md| 17
.../configuration/metron-enrichment-env.xml | 8 ++--
.../METRON/CURRENT/themes/metron_theme.json | 12 +++---
metron-platform/Performance-tuning-guide.md | 6 ++-
metron-platform/metron-enrichment/README.md | 43 +---
.../main/scripts/start_enrichment_topology.sh | 4 +-
6 files changed, 54 insertions(+), 36 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/bf6b07f7/Upgrading.md
--
diff --git a/Upgrading.md b/Upgrading.md
index 2124ac5..a0dd5d3 100644
--- a/Upgrading.md
+++ b/Upgrading.md
@@ -19,6 +19,23 @@ limitations under the License.
This document constitutes a per-version listing of changes of
configuration which are non-backwards compatible.
+## 0.6.0 to 0.6.1
+
+### [METRON-1855: Make unified enrichment topology the default and deprecate
split-join](https://issues.apache.org/jira/browse/METRON-1855)
+The unified enrichment topology will be the new default in this release,
+and the split-join enrichment topology is now considered deprecated.
+If you wish to keep the deprecated split-join enrichment topology,
+you will need to make the following changes:
+
+* In Ambari > Metron > Config > Enrichment set the enrichment_topology setting
to "Split-Join"
+* If running `start_enrichment_topology.sh` manually, pass in the parameters
to start the Split-Join topology as follows
+
+```
+$METRON_HOME/bin/start_enrichment_topology.sh --remote
$METRON_HOME/flux/enrichment/remote-splitjoin.yaml --filter
$METRON_HOME/config/enrichment-splitjoin.properties
+```
+
+* Restart the enrichment topology
+
## 0.4.2 to 0.5.0
### [METRON-941: native PaloAlto parser corrupts message when having a comma
in the payload](https://issues.apache.org/jira/browse/METRON-941)
http://git-wip-us.apache.org/repos/asf/metron/blob/bf6b07f7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
--
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
index b41c455..69dce3f 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/configuration/metron-enrichment-env.xml
@@ -165,17 +165,17 @@
enrichment_topology
-Which Enrichment topology to execute
-Split-Join
+Which Enrichment topology to execute. Note: Split-Join is
deprecated in favor of the Unified topology.
+Unified
Enrichment Topology
value-list
- Split-Join
+ Unified
- Unified
+ Split-Join
1
http://git-wip-us.apache.org/repos/asf/metron/blob/bf6b07f7/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
--
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
index 1d7b6c5..46c06dd 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/themes/metron_theme.json
@@ -125,7 +125,7 @@
]
},
{
- "name": "section-enrichment-splitjoin",
+
[1/5] metron git commit: METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes apache/metron#1226
Repository: metron
Updated Branches:
refs/heads/master fefbb376f -> 9b6260fd4
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
--
diff --git
a/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
b/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
new file mode 100644
index 000..58f7d26
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/integration/pcap/pcap.spec.js
@@ -0,0 +1,228 @@
+///
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+context('PCAP Tab', () => {
+
+ beforeEach(() => {
+cy.server();
+cy.route({
+ method: 'GET',
+ url: '/api/v1/user',
+ response: 'user'
+});
+
+cy.route('GET', 'config', 'fixture:config.json');
+cy.route('POST', 'search', 'fixture:search.json');
+
+cy.route({
+ method: 'GET',
+ url: '/api/v1/pcap?state=*',
+ response: []
+}).as('runningJobs');
+
+cy.visit('http://localhost:4200/login');
+cy.get('[name="user"]').type('user');
+cy.get('[name="password"]').type('password');
+cy.contains('LOG IN').click();
+ });
+
+ afterEach(() => {
+cy.get('.logout-link').click();
+ });
+
+ it('checking running jobs on navigating to PCAP tab', () => {
+cy.contains('PCAP').click();
+cy.wait('@runningJobs').its('url').should('include', '?state=RUNNING');
+ });
+
+ it('submitting PCAP job request', () => {
+cy.contains('PCAP').click();
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json')
+ .as('postingPcapJob');
+
+cy.get('[data-qe-id="ip-src-addr"]').type('222.123.111.000');
+cy.get('[data-qe-id="ip-dst-addr"]').type('111.123.222.000');
+cy.get('[data-qe-id="ip-src-port"]').type('');
+cy.get('[data-qe-id="ip-dst-port"]').type('');
+cy.get('[data-qe-id="protocol"]').type('24');
+cy.get('[data-qe-id="include-reverse"]').check();
+cy.get('[data-qe-id="packet-filter"]').type('filter');
+
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@postingPcapJob').then((xhr) => {
+ expect(xhr.request.body.ipSrcAddr).to.equal('222.123.111.000');
+ expect(xhr.request.body.ipDstAddr).to.equal('111.123.222.000');
+ expect(xhr.request.body.ipSrcPort).to.equal('');
+ expect(xhr.request.body.ipDstPort).to.equal('');
+ expect(xhr.request.body.protocol).to.equal('24');
+ expect(xhr.request.body.includeReverse).to.equal(true);
+ expect(xhr.request.body.packetFilter).to.equal('filter');
+});
+ });
+
+ it('requesting job status', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-01.json').as('jobStatusCheck');
+
+cy.contains('PCAP').click();
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@jobStatusCheck').its('url').should('include',
'/api/v1/pcap/job_1537878471649_0001');
+ });
+
+ it('process status in percentage', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-01.json').as('jobStatusCheck');
+
+cy.contains('PCAP').click();
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@jobStatusCheck');
+
+cy.contains('75%').should('be.visible');
+ });
+
+ it('getting pcap json', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-02.json').as('statusCheck');
+cy.route('GET', '/api/v1/pcap/*/pdml*',
'fixture:pcap.page-01.json').as('gettingPdml');
+
+cy.contains('PCAP').click();
+cy.get('[data-qe-id="submit-button"]').click();
+
+cy.wait('@statusCheck');
+
+cy.wait('@gettingPdml').its('url').should('include',
'/api/v1/pcap/job_1537878471649_0001/pdml?page=1');
+ });
+
+
+ it('rendering pcap table', () => {
+cy.route('POST', '/api/v1/pcap/fixed', 'fixture:pcap.status-00.json');
+cy.route('GET', '/api/v1/pcap/*',
'fixture:pcap.status-02.json').as('statusCheck')
[5/5] metron git commit: METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes apache/metron#1226
METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes
apache/metron#1226
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/9b6260fd
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/9b6260fd
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/9b6260fd
Branch: refs/heads/master
Commit: 9b6260fd4b9ecf355864b77c8889d27539623381
Parents: fefbb37
Author: tiborm
Authored: Fri Nov 2 10:21:41 2018 -0600
Committer: Michael Miklavcic
Committed: Fri Nov 2 10:21:41 2018 -0600
--
metron-interface/metron-alerts/cypress.json | 7 +
.../metron-alerts/cypress/fixtures/config.json |23 +
.../cypress/fixtures/pcap.page-01.json | 12383 +
.../cypress/fixtures/pcap.status-00.json| 7 +
.../cypress/fixtures/pcap.status-01.json| 7 +
.../cypress/fixtures/pcap.status-02.json| 7 +
.../metron-alerts/cypress/fixtures/search.json | 5647
.../cypress/integration/pcap/pcap.spec.js | 228 +
.../metron-alerts/package-lock.json | 1134 ++
metron-interface/metron-alerts/package.json |11 +-
10 files changed, 19451 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress.json
--
diff --git a/metron-interface/metron-alerts/cypress.json
b/metron-interface/metron-alerts/cypress.json
new file mode 100644
index 000..7c0410d
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress.json
@@ -0,0 +1,7 @@
+{
+ "viewportWidth": 1435,
+ "viewportHeight": 850,
+ "video": false,
+ "supportFile": false,
+ "pluginsFile": false
+}
\ No newline at end of file
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/fixtures/config.json
--
diff --git a/metron-interface/metron-alerts/cypress/fixtures/config.json
b/metron-interface/metron-alerts/cypress/fixtures/config.json
new file mode 100644
index 000..190e514
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/config.json
@@ -0,0 +1,23 @@
+{
+ "es.clustername":"metron",
+ "es.ip":"node1:9300",
+ "es.date.format":".MM.dd.HH",
+ "parser.error.topic":"indexing",
+ "update.hbase.table":"metron_update",
+ "update.hbase.cf":"t",
+ "es.client.settings":{
+ "client.transport.ping_timeout":"500s"
+ },
+ "profiler.client.period.duration":"15",
+ "profiler.client.period.duration.units":"MINUTES",
+ "user.settings.hbase.table":"user_settings",
+ "user.settings.hbase.cf":"cf",
+ "bootstrap.servers":"node1:6667",
+ "source.type.field":"source:type",
+ "threat.triage.score.field":"threat:triage:score",
+ "enrichment.writer.batchSize":"15",
+ "enrichment.writer.batchTimeout":"0",
+ "profiler.writer.batchSize":"15",
+ "profiler.writer.batchTimeout":"0",
+ "geo.hdfs.file":"/apps/metron/geo/default/GeoLite2-City.mmdb.gz"
+}
\ No newline at end of file
[2/5] metron git commit: METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes apache/metron#1226
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/fixtures/search.json
--
diff --git a/metron-interface/metron-alerts/cypress/fixtures/search.json
b/metron-interface/metron-alerts/cypress/fixtures/search.json
new file mode 100644
index 000..e2e03e4
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/search.json
@@ -0,0 +1,5647 @@
+{
+ "total":104593,
+ "results":[
+ {
+"id":"ad5cc7ea-5954-479f-8589-51f94b1c2f02",
+"source":{
+ "average":10.0,
+ "max":10.0,
+ "metron_alert":[
+ {
+ "msg":"'snort test alert'",
+ "sig_rev":"0",
+ "ip_dst_port":"49195",
+ "threatinteljoinbolt:joiner:ts":"1537279364136",
+ "ethsrc":"00:00:00:00:00:00",
+ "threat:triage:rules:0:comment":null,
+ "tcpseq":"0xC88832BC",
+ "enrichments:geo:ip_src_addr:longitude":"2.33870002",
+ "dgmlen":"44",
+ "enrichmentsplitterbolt:splitter:begin:ts":"1537279364122",
+ "enrichmentjoinbolt:joiner:ts":"1537279364128",
+ "adapter:geoadapter:begin:ts":"1537279364125",
+ "tcpwindow":"0xFAF0",
+ "threat:triage:rules:0:name":null,
+ "tcpack":"0x522C98B4",
+ "protocol":"TCP",
+ "source:type":"snort",
+ "adapter:threatinteladapter:end:ts":"1537279364133",
+ "ip_dst_addr":"192.168.138.158",
+ "original_string":"09/18/18-14:02:39.00
,1,999158,0,\"'snort test
alert'\",TCP,188.165.164.184,80,192.168.138.158,49195,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A**S*,0xC88832BC,0x522C98B4,,0xFAF0,128,0,1899,44,45056",
+ "adapter:hostfromjsonlistadapter:end:ts":"1537279364125",
+ "tos":"0",
+ "adapter:geoadapter:end:ts":"1537279364125",
+ "id":"1899",
+ "enrichments:geo:ip_src_addr:latitude":"48.8582",
+ "ip_src_addr":"188.165.164.184",
+ "threatintelsplitterbolt:splitter:end:ts":"1537279364130",
+ "threat:triage:rules:0:score":10,
+ "timestamp":1537279359000,
+ "ethdst":"00:00:00:00:00:00",
+
"enrichments:geo:ip_src_addr:location_point":"48.8582,2.33870002",
+ "threat:triage:rules:0:reason":null,
+ "enrichmentsplitterbolt:splitter:end:ts":"1537279364122",
+ "threat:triage:score":10.0,
+ "is_alert":"true",
+ "adapter:hostfromjsonlistadapter:begin:ts":"1537279364125",
+ "enrichments:geo:ip_src_addr:country":"FR",
+ "ttl":"128",
+ "metaalerts":[
+"ad5cc7ea-5954-479f-8589-51f94b1c2f02"
+ ],
+ "ethlen":"0x3C",
+ "iplen":"45056",
+ "ip_src_port":"80",
+ "threatintelsplitterbolt:splitter:begin:ts":"1537279364130",
+ "adapter:threatinteladapter:begin:ts":"1537279364133",
+ "tcpflags":"***A**S*",
+ "guid":"c6843745-203c-49e1-80ad-f060eb88c9b1",
+ "sig_id":"999158",
+ "sig_generator":"1"
+ },
+ {
+ "msg":"'snort test alert'",
+ "sig_rev":"0",
+ "ip_dst_port":"49195",
+ "threatinteljoinbolt:joiner:ts":"1537280091506",
+ "ethsrc":"00:00:00:00:00:00",
+ "threat:triage:rules:0:comment":null,
+ "tcpseq":"0xC88832BD",
+ "enrichments:geo:ip_src_addr:longitude":"2.33870002",
+ "dgmlen":"40",
+ "enrichmentsplitterbolt:splitter:begin:ts":"1537280091491",
+ "enrichmentjoinbolt:joiner:ts":"1537280091498",
+ "adapter:geoadapter:begin:ts":"1537280091493",
+ "tcpwindow":"0xFAF0",
+ "threat:triage:rules:0:name":null,
+ "tcpack":"0x522C999D",
+ "protocol":"TCP",
+ "source:type":"snort",
+ "adapter:threatinteladapter:end:ts":"1537280091503",
+ "ip_dst_addr":"192.168.138.158",
+ "original_string":"09/18/18-14:14:47.00
,1,999158,0,\"'snort test
alert'\",TCP,188.165.164.184,80,192.168.138.158,49195,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A,0xC88832BD,0x522C999D,,0xFAF0,128,0,1900,40,40960",
+ "adapter:hostfromjsonlistadapter:end:ts":"1537280091493",
+ "tos":"0",
+ "adapter:geoadapter:end:ts":"1537280091493",
+ "id":"1900",
+ "enrichments
[4/5] metron git commit: METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes apache/metron#1226
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
--
diff --git a/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
new file mode 100644
index 000..61082ed
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.page-01.json
@@ -0,0 +1,12383 @@
+{
+ "version":"0",
+ "creator":"wireshark/1.8.10",
+ "time":"Mon Sep 24 14:16:26 2018",
+ "captureFile":"",
+ "packets":[
+ {
+"protos":[
+ {
+ "name":"geninfo",
+ "pos":"0",
+ "showname":"General information",
+ "size":"722",
+ "hide":null,
+ "fields":[
+ {
+"name":"num",
+"pos":"0",
+"showname":"Number",
+"size":"722",
+"value":"1",
+"show":"1",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"len",
+"pos":"0",
+"showname":"Frame Length",
+"size":"722",
+"value":"2d2",
+"show":"722",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"caplen",
+"pos":"0",
+"showname":"Captured Length",
+"size":"722",
+"value":"2d2",
+"show":"722",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"timestamp",
+"pos":"0",
+"showname":"Captured Time",
+"size":"722",
+"value":"1458240269.373968000",
+"show":"Mar 17, 2016 18:44:29.373968000 UTC",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ }
+ ]
+ },
+ {
+ "name":"frame",
+ "pos":"0",
+ "showname":"Frame 1: 722 bytes on wire (5776 bits), 722 bytes
captured (5776 bits) on interface 0",
+ "size":"722",
+ "hide":null,
+ "fields":[
+ {
+"name":"frame.interface_id",
+"pos":"0",
+"showname":"Interface id: 0",
+"size":"0",
+"value":null,
+"show":"0",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.dlt",
+"pos":"0",
+"showname":"WTAP_ENCAP: 1",
+"size":"0",
+"value":null,
+"show":"1",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.time",
+"pos":"0",
+"showname":"Arrival Time: Mar 17, 2016 18:44:29.373968000
UTC",
+"size":"0",
+"value":null,
+"show":"Mar 17, 2016 18:44:29.373968000",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.offset_shift",
+"pos":"0",
+"showname":"Time shift for this packet: 0.0
seconds",
+"size":"0",
+"value":null,
+"show":"0.0",
+"unmaskedvalue":null,
+"hide":null,
+"fields":null,
+"protos":null
+ },
+ {
+"name":"frame.time_epoch",
+"pos":"0",
+"showname":"Epoch Time: 1458240269.373968000 seconds",
+"size":"0",
+"value":null,
+"show":"1458240269.373968000",
+"unmaskedvalue":null,
+"hide":nu
[3/5] metron git commit: METRON-1803: Integrate Cypress with Travis (tiborm via mmiklavc) closes apache/metron#1226
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
--
diff --git
a/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
new file mode 100644
index 000..2a3cdcc
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-00.json
@@ -0,0 +1,7 @@
+{
+ "jobId":"job_1537878471649_0001",
+ "jobStatus":"RUNNING",
+ "description":"map: 0.0%, reduce: 0.0%",
+ "percentComplete":0.0,
+ "pageTotal":0
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
--
diff --git
a/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
new file mode 100644
index 000..1505f71
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-01.json
@@ -0,0 +1,7 @@
+{
+ "jobId":"job_1537878471649_0001",
+ "jobStatus":"RUNNING",
+ "description":"map: 100.0%, reduce: 100.0%",
+ "percentComplete":75.0,
+ "pageTotal":0
+}
http://git-wip-us.apache.org/repos/asf/metron/blob/9b6260fd/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
--
diff --git
a/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
new file mode 100644
index 000..662c27a
--- /dev/null
+++ b/metron-interface/metron-alerts/cypress/fixtures/pcap.status-02.json
@@ -0,0 +1,7 @@
+{
+ "jobId":"job_1537878471649_0001",
+ "jobStatus":"SUCCEEDED",
+ "description":"Job completed.",
+ "percentComplete":100.0,
+ "pageTotal":2
+}
\ No newline at end of file
metron git commit: METRON-1833: Management UI incorrectly displaying sensor topology latency units as seconds instead of millis (mmiklavc via mmiklavc) closes apache/metron#1241
Repository: metron
Updated Branches:
refs/heads/master d44a39256 -> 0c0602c75
METRON-1833: Management UI incorrectly displaying sensor topology latency units
as seconds instead of millis (mmiklavc via mmiklavc) closes apache/metron#1241
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/0c0602c7
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/0c0602c7
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/0c0602c7
Branch: refs/heads/master
Commit: 0c0602c75d9660717920a547bcb9e76d58b6571e
Parents: d44a392
Author: mmiklavc
Authored: Mon Oct 22 12:54:26 2018 -0600
Committer: Michael Miklavcic
Committed: Mon Oct 22 12:54:26 2018 -0600
--
.../sensor-parser-config-readonly.component.ts | 2 +-
.../sensor-parser-list/sensor-parser-list.component.spec.ts| 2 +-
.../app/sensors/sensor-parser-list/sensor-parser-list.component.ts | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/0c0602c7/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts
b/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts
index 5db6d45..7d41003 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-parser-config-readonly/sensor-parser-config-readonly.component.ts
@@ -156,7 +156,7 @@ export class SensorParserConfigReadonlyComponent implements
OnInit {
getTopologyStatus(key: string): string {
if (key === 'latency') {
- return this.topologyStatus.latency >= 0 ? (this.topologyStatus.latency +
's') : '-';
+ return this.topologyStatus.latency >= 0 ? (this.topologyStatus.latency +
'ms') : '-';
} else if (key === 'throughput') {
return this.topologyStatus.throughput >= 0 ?
((Math.round(this.topologyStatus.throughput * 100) / 100) + 'kb/s') : '-';
} else if (key === 'emitted') {
http://git-wip-us.apache.org/repos/asf/metron/blob/0c0602c7/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.spec.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.spec.ts
b/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.spec.ts
index 205d885..fb2a175 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.spec.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.spec.ts
@@ -720,7 +720,7 @@ describe('Component: SensorParserList', () => {
component.updateSensorStatus();
expect(component.sensors[0]['status']).toEqual('Running');
-expect(component.sensors[0]['latency']).toEqual('10s');
+expect(component.sensors[0]['latency']).toEqual('10ms');
expect(component.sensors[0]['throughput']).toEqual('23kb/s');
component.sensorsStatus[0].status = 'KILLED';
http://git-wip-us.apache.org/repos/asf/metron/blob/0c0602c7/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.ts
--
diff --git
a/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.ts
b/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.ts
index 1129914..2694ab4 100644
---
a/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.ts
+++
b/metron-interface/metron-config/src/app/sensors/sensor-parser-list/sensor-parser-list.component.ts
@@ -163,7 +163,7 @@ export class SensorParserListComponent implements OnInit {
sensor['status'] = 'Stopped';
}
-sensor['latency'] = status && status.status === 'ACTIVE' ?
(status.latency + 's') : '-';
+sensor['latency'] = status && status.status === 'ACTIVE' ?
(status.latency + 'ms') : '-';
sensor['throughput'] = status && status.status === 'ACTIVE' ?
(Math.round(status.throughput * 100) / 100) + 'kb/s' : '-';
}
}
metron git commit: METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc) closes apache/metron#1230
Repository: metron Updated Branches: refs/heads/master 747220f00 -> 1f1b9cdd3 METRON-1812: Fix dependencies_with_url.csv (mmiklavc via mmiklavc) closes apache/metron#1230 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/1f1b9cdd Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/1f1b9cdd Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/1f1b9cdd Branch: refs/heads/master Commit: 1f1b9cdd3cef7557ff68b7ab33693161b142ef1d Parents: 747220f Author: mmiklavc Authored: Tue Oct 9 14:51:57 2018 -0600 Committer: Michael Miklavcic Committed: Tue Oct 9 14:51:57 2018 -0600 -- dependencies_with_url.csv | 32 ++-- 1 file changed, 14 insertions(+), 18 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/1f1b9cdd/dependencies_with_url.csv -- diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv index 53977f3..fb6c03c 100644 --- a/dependencies_with_url.csv +++ b/dependencies_with_url.csv @@ -256,12 +256,8 @@ io.dropwizard.metrics:metrics-json:jar:3.1.5:compile,ASLv2,https://github.com/dr io.dropwizard.metrics:metrics-jvm:jar:3.1.5:compile,ASLv2,https://github.com/dropwizard/metrics io.netty:netty-all:jar:4.0.23.Final:compile,ASLv2, io.netty:netty-all:jar:4.0.23.Final:provided,ASLv2, -<<< HEAD io.netty:netty-all:jar:4.1.17.Final:compile,ASLv2, -=== io.netty:netty-all:jar:4.1.23.Final:compile,ASLv2, -io.netty:netty:jar:3.10.5.Final:compile,Apache License, Version 2.0,http://netty.io/ ->>> apache/master io.netty:netty:jar:3.6.2.Final:compile,Apache License, Version 2.0,http://netty.io/ io.netty:netty:jar:3.7.0.Final:compile,Apache License, Version 2.0,http://netty.io/ io.netty:netty:jar:3.9.9.Final:compile,Apache License, Version 2.0,http://netty.io/ @@ -472,20 +468,20 @@ org.eclipse.persistence:org.eclipse.persistence.jpa:jar:2.6.4:compile,EPL 1.0,ht com.github.ben-manes.caffeine:caffeine:jar:2.6.2:compile,ASLv2,https://github.com/ben-manes/caffeine/blob/v2.6.2/LICENSE com.google.code.gson:gson:jar:2.2:compile,ASLv2,https://github.com/google/gson com.google.code.gson:gson:jar:2.8.2:compile,ASLv2,https://github.com/google/gson - org.codehaus.plexus:plexus-classworlds:jar:2.4:compile - org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile - org.codehaus.plexus:plexus-interpolation:jar:1.14:compile - org.codehaus.plexus:plexus-utils:jar:2.0.7:compile - org.jsoup:jsoup:jar:1.6.1:compile - org.sonatype.aether:aether-api:jar:1.12:compile - org.sonatype.aether:aether-connector-file:jar:1.12:compile - org.sonatype.aether:aether-connector-wagon:jar:1.12:compile - org.sonatype.aether:aether-impl:jar:1.12:compile - org.sonatype.aether:aether-spi:jar:1.12:compile - org.sonatype.aether:aether-util:jar:1.12:compile - org.sonatype.sisu:sisu-guice:jar:no_aop:3.0.2:compile - org.sonatype.sisu:sisu-inject-bean:jar:2.2.2:compile - org.sonatype.sisu:sisu-inject-plexus:jar:2.2.2:compile +org.codehaus.plexus:plexus-classworlds:jar:2.4:compile +org.codehaus.plexus:plexus-component-annotations:jar:1.5.5:compile +org.codehaus.plexus:plexus-interpolation:jar:1.14:compile +org.codehaus.plexus:plexus-utils:jar:2.0.7:compile +org.jsoup:jsoup:jar:1.6.1:compile +org.sonatype.aether:aether-api:jar:1.12:compile +org.sonatype.aether:aether-connector-file:jar:1.12:compile +org.sonatype.aether:aether-connector-wagon:jar:1.12:compile +org.sonatype.aether:aether-impl:jar:1.12:compile +org.sonatype.aether:aether-spi:jar:1.12:compile +org.sonatype.aether:aether-util:jar:1.12:compile +org.sonatype.sisu:sisu-guice:jar:no_aop:3.0.2:compile +org.sonatype.sisu:sisu-inject-bean:jar:2.2.2:compile +org.sonatype.sisu:sisu-inject-plexus:jar:2.2.2:compile com.zaxxer:HikariCP:jar:2.7.8:compile,ASLv2,https://github.com/brettwooldridge/HikariCP org.hibernate.validator:hibernate-validator:jar:6.0.9.Final:compile,ASLv2,https://github.com/hibernate/hibernate-validator com.github.palindromicity:simple-syslog-5424:jar:0.0.8:compile,ASLv2,https://github.com/palindromicity/simple-syslog-5424
metron git commit: METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc) closes apache/metron#1224
Repository: metron Updated Branches: refs/heads/master 5bfc08c57 -> e48236672 METRON-1806: Upgrade Maven Shade Plugin version (mmiklavc via mmiklavc) closes apache/metron#1224 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/e4823667 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/e4823667 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/e4823667 Branch: refs/heads/master Commit: e482366726b07173fbc9ff9b084b851596e13005 Parents: 5bfc08c Author: mmiklavc Authored: Tue Oct 9 09:22:22 2018 -0600 Committer: Michael Miklavcic Committed: Tue Oct 9 09:22:22 2018 -0600 -- .gitignore | 1 + pom.xml| 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/e4823667/.gitignore -- diff --git a/.gitignore b/.gitignore index a0b9691..df3f030 100644 --- a/.gitignore +++ b/.gitignore @@ -16,6 +16,7 @@ target/ *.tmp *.bak *.class +*.pyc tmp/** tmp/**/* http://git-wip-us.apache.org/repos/asf/metron/blob/e4823667/pom.xml -- diff --git a/pom.xml b/pom.xml index a98de72..1e6adb0 100644 --- a/pom.xml +++ b/pom.xml @@ -126,7 +126,7 @@ 6.6.2 1.10.19 1.7.0 -2.4.3 +3.2.0 2.7.4 2.0.14 3.0.2
