metron git commit: METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746
Repository: metron
Updated Branches:
refs/heads/master 2c56a13e5 -> 2ae1f5adf
METRON-1173: Fix pointers to old stellar docs closes apache/incubator-metron#746
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/2ae1f5ad
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/2ae1f5ad
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/2ae1f5ad
Branch: refs/heads/master
Commit: 2ae1f5adf8428832550f99bcc4cc76645597851a
Parents: 2c56a13
Author: JonZeolla
Authored: Wed Sep 13 16:58:11 2017 -0700
Committer: cstella
Committed: Wed Sep 13 16:58:11 2017 -0700
--
metron-analytics/metron-profiler-client/README.md| 4 ++--
.../src/main/config/zeppelin/metron/metron-pcap.json | 2 +-
metron-sensors/bro-plugin-kafka/README.md| 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/2ae1f5ad/metron-analytics/metron-profiler-client/README.md
--
diff --git a/metron-analytics/metron-profiler-client/README.md
b/metron-analytics/metron-profiler-client/README.md
index 27aa3f4..63d0cd4 100644
--- a/metron-analytics/metron-profiler-client/README.md
+++ b/metron-analytics/metron-profiler-client/README.md
@@ -305,7 +305,7 @@ The person doing the querying must carry forward the
knowledge of the Profiler c
### Examples
-The following are usage examples that show how the Stellar API can be used to
read profiles generated by the [Metron Profiler](../metron-profiler). This API
would be used in conjunction with other Stellar functions like
[`MAAS_MODEL_APPLY`](../../metron-platform/metron-common#maas_model_apply) to
perform model scoring on streaming data.
+The following are usage examples that show how the Stellar API can be used to
read profiles generated by the [Metron Profiler](../metron-profiler). This API
would be used in conjunction with other Stellar functions like
[`MAAS_MODEL_APPLY`](../../metron-stellar/stellar-common#maas_model_apply) to
perform model scoring on streaming data.
These examples assume a profile has been defined called 'snort-alerts' that
tracks the number of Snort alerts associated with an IP address over time. The
profile definition might look similar to the following.
@@ -462,4 +462,4 @@ Follow these steps in the Stellar REPL to see how it can be
used to help create
This profile simply counts the number of messages by IP source address.
Notice that the value is '3' for the entity '10.0.0.1' as we applied 3 messages
with an 'ip_src_addr' of '10.0.0.1'. There will always be one measurement for
each [profile, entity] pair.
-1. If you are unhappy with the data that has been generated, then 'wash, rinse
and repeat' this process. Once you are happy with the profile that was
created, follow the [Getting Started](../metron-profiler#getting-started) guide
to use the profile against your live, streaming data in a Metron cluster.
\ No newline at end of file
+1. If you are unhappy with the data that has been generated, then 'wash, rinse
and repeat' this process. Once you are happy with the profile that was
created, follow the [Getting Started](../metron-profiler#getting-started) guide
to use the profile against your live, streaming data in a Metron cluster.
http://git-wip-us.apache.org/repos/asf/metron/blob/2ae1f5ad/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
--
diff --git
a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
index c18b5fb..447056f 100644
---
a/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
+++
b/metron-platform/metron-pcap-backend/src/main/config/zeppelin/metron/metron-pcap.json
@@ -1 +1 @@
-{"paragraphs":[{"text":"%md\n# Execute Packet Capture Queries\n\nSpecify
the following to filter the packet capture query:\n* *end time* - The ending
time of the query in MMdd format (e.g. 20170428)\n* *start time* - The
starting time of the query in MMdd format (e.g. 20170428)\n* *query* - The
[Stellar](https://github.com/apache/metron/tree/master/metron-platform/metron-common#stellar-language)
query (i.e. a Stellar expression that returns `true` or `false`) to specify
the packets.\n\nThe available fields to use in the queries are as follows:\n*
`ip_src_addr` - The source IP address of the packets filtered\n* `ip_src_port`
- The source port of the packets filtered\n* `ip_dst_addr` - The destination IP
address of the packets filtered\n* `ip_dst_port` - The destination port of the
p
metron git commit: METRON-1179: Make STATS_ADD to take a list closes apache/incubator-metron#750
Repository: metron
Updated Branches:
refs/heads/master 6017c5974 -> 2c56a13e5
METRON-1179: Make STATS_ADD to take a list closes apache/incubator-metron#750
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/2c56a13e
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/2c56a13e
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/2c56a13e
Branch: refs/heads/master
Commit: 2c56a13e53065954a853cc33a19b569b686cb70d
Parents: 6017c59
Author: cstella
Authored: Wed Sep 13 16:53:54 2017 -0700
Committer: cstella
Committed: Wed Sep 13 16:53:54 2017 -0700
--
metron-analytics/metron-statistics/README.md| 2 +-
.../statistics/StellarStatisticsFunctions.java | 19 +---
.../StellarStatisticsFunctionsTest.java | 24
3 files changed, 41 insertions(+), 4 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/2c56a13e/metron-analytics/metron-statistics/README.md
--
diff --git a/metron-analytics/metron-statistics/README.md
b/metron-analytics/metron-statistics/README.md
index f822845..982132a 100644
--- a/metron-analytics/metron-statistics/README.md
+++ b/metron-analytics/metron-statistics/README.md
@@ -60,7 +60,7 @@ functions can be used from everywhere where Stellar is used.
* Description: Adds one or more input values to those that are used to
calculate the summary statistics.
* Input:
* stats - The Stellar statistics object. If null, then a new one is
initialized.
-* value+ - One or more numbers to add
+* value+ - One or more items to add. Each item may be a number or a list
of numbers. If an item is a list, each number in the list will be added.
* Returns: A Stellar statistics object
`STATS_BIN`
http://git-wip-us.apache.org/repos/asf/metron/blob/2c56a13e/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
--
diff --git
a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
index d02734e..a6cf605 100644
---
a/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
+++
b/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/StellarStatisticsFunctions.java
@@ -125,7 +125,7 @@ public class StellarStatisticsFunctions {
, description = "Adds one or more input values to those that are
used to calculate the summary statistics."
, params = {
"stats - The Stellar statistics object. If null, then a
new one is initialized."
- , "value+ - One or more numbers to add"
+ , "value+ - One or more items to add. Each item may be a
number or a list of numbers. If an item is a list, each number in the list will
be added."
}
, returns = "A Stellar statistics object"
)
@@ -141,8 +141,21 @@ public class StellarStatisticsFunctions {
// add each of the numeric values
for(int i=1; i)n) {
+ if(num != null) {
+Double value = convert(num, Double.class);
+stats.addValue(value);
+ }
+}
+ }
+ else {
+Double value = convert(args.get(i), Double.class);
+stats.addValue(value);
+ }
+}
}
return stats;
http://git-wip-us.apache.org/repos/asf/metron/blob/2c56a13e/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
--
diff --git
a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
index 42b57bd..a026bf2 100644
---
a/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
+++
b/metron-analytics/metron-statistics/src/test/java/org/apache/metron/statistics/StellarStatisticsFunctionsTest.java
@@ -224,6 +224,18 @@ public class StellarStatisticsFunctionsTest {
}
@Test
+ public void testAddAllManyIntegers() throws Exception {
+statsInit(windowSize);
+Object result = run("STATS_COUNT(stats)", variables);
+double countAtStart = (double) result;
+
+run("STATS_ADD(stats, [10, 20, 30, 40, 50])", variables);
+
+Object actual = run("STATS_COU
metron git commit: METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and not require a port closes apache/incubator-metron#751
Repository: metron
Updated Branches:
refs/heads/master 76947ad52 -> 6017c5974
METRON-1180: Make Stellar Shell accept zookeeper quorum as a CSV list and not
require a port closes apache/incubator-metron#751
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/6017c597
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/6017c597
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/6017c597
Branch: refs/heads/master
Commit: 6017c5974a868f94ef86b9493ee0d244ffd2df65
Parents: 76947ad
Author: cstella
Authored: Wed Sep 13 13:09:51 2017 -0700
Committer: cstella
Committed: Wed Sep 13 13:09:51 2017 -0700
--
.../shell/StellarShellOptionsValidator.java | 50 +++-
.../shell/StellarShellOptionsValidatorTest.java | 30 +---
2 files changed, 38 insertions(+), 42 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/6017c597/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidator.java
--
diff --git
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidator.java
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidator.java
index 97f5b70..ab92401 100644
---
a/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidator.java
+++
b/metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidator.java
@@ -26,6 +26,8 @@ import java.net.UnknownHostException;
import java.util.function.Predicate;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
+
+import com.google.common.base.Splitter;
import org.apache.commons.cli.CommandLine;
import org.apache.commons.lang3.StringUtils;
import org.apache.commons.validator.routines.InetAddressValidator;
@@ -73,30 +75,32 @@ public class StellarShellOptionsValidator {
/**
* Zookeeper argument should be in the form [HOST|IP]:PORT.
*
- * @param z the zookeeper url fragment
+ * @param zMulti the zookeeper url fragment
*/
- private static void validateZookeeperOption(String z) throws
IllegalArgumentException {
-
-Matcher matcher = validPortPattern.matcher(z);
-if (!matcher.matches()) {
- throw new IllegalArgumentException(String.format("Zookeeper option must
have port: %s", z));
-}
-
-if (matcher.groupCount() != 2) {
- throw new IllegalArgumentException(
- String.format("Zookeeper Option must be in the form of
[HOST|IP]:PORT %s", z));
-}
-String name = matcher.group(1);
-Integer port = Integer.parseInt(matcher.group(2));
-
-if (!hostnameValidator.test(name) && !inetAddressValidator.isValid(name)) {
- throw new IllegalArgumentException(
- String.format("Zookeeper Option %s is not a valid host name or ip
address %s", name, z));
-}
-
-if(port == 0 || port > 65535){
- throw new IllegalArgumentException(
- String.format("Zookeeper Option %s port is not valid",z));
+ private static void validateZookeeperOption(String zMulti) throws
IllegalArgumentException {
+for(String z : Splitter.on(",").split(zMulti)) {
+ Matcher matcher = validPortPattern.matcher(z);
+ boolean hasPort = z.contains(":");
+ if (hasPort && !matcher.matches()) {
+throw new IllegalArgumentException(String.format("Zookeeper option
must have valid port: %s", z));
+ }
+
+ if (hasPort && matcher.groupCount() != 2) {
+throw new IllegalArgumentException(
+String.format("Zookeeper Option must be in the form of
[HOST|IP]:PORT %s", z));
+ }
+ String name = hasPort?matcher.group(1):z;
+ Integer port = hasPort?Integer.parseInt(matcher.group(2)):null;
+
+ if (!hostnameValidator.test(name) &&
!inetAddressValidator.isValid(name)) {
+throw new IllegalArgumentException(
+String.format("Zookeeper Option %s is not a valid host name or
ip address %s", name, z));
+ }
+
+ if (hasPort && (port == 0 || port > 65535)) {
+throw new IllegalArgumentException(
+String.format("Zookeeper Option %s port is not valid", z));
+ }
}
}
http://git-wip-us.apache.org/repos/asf/metron/blob/6017c597/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidatorTest.java
--
diff --git
a/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/common/shell/StellarShellOptionsValidatorTest.java
b/metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/com
metron git commit: METRON-1183 Improve KDC Setup Instructions (nickwallen) closes apache/metron#753
Repository: metron Updated Branches: refs/heads/master ba9532de3 -> 76947ad52 METRON-1183 Improve KDC Setup Instructions (nickwallen) closes apache/metron#753 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/76947ad5 Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/76947ad5 Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/76947ad5 Branch: refs/heads/master Commit: 76947ad528d79fc0c6b924b8337948ed916f3e12 Parents: ba9532d Author: nickwallen Authored: Wed Sep 13 14:39:39 2017 -0400 Committer: nickallen Committed: Wed Sep 13 14:39:39 2017 -0400 -- metron-deployment/Kerberos-manual-setup.md | 66 - 1 file changed, 42 insertions(+), 24 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/76947ad5/metron-deployment/Kerberos-manual-setup.md -- diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md index 615e6e8..9c4ed1d 100644 --- a/metron-deployment/Kerberos-manual-setup.md +++ b/metron-deployment/Kerberos-manual-setup.md @@ -65,33 +65,51 @@ Setup a KDC yum -y install krb5-server krb5-libs krb5-workstation ``` -1. Define the host, `node1`, as the KDC. +1. Define the current host as the KDC. ``` -sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf +KDC=`hostname` +sed -i.orig 's/kerberos.example.com/'"$KDC"'/g' /etc/krb5.conf cp -f /etc/krb5.conf /var/lib/ambari-server/resources/scripts ``` -1. Ensure the KDC can issue renewable tickets. This can be necessary on a real cluster, but should not be on full-dev. In /var/kerberos/krb5kdc/kdc.conf ensure the following is in the realm section +1. Ensure that the KDC can issue renewable tickets. This may be necessary on a real cluster, but should not be on [Full Dev](vagrant/full-dev-platform/README.md). + +Edit `/var/kerberos/krb5kdc/kdc.conf` and ensure the following is added to the `realm` section ``` max_renewable_life = 7d ``` -1. Do not copy/paste this full set of commands as the `kdb5_util` command will not run as expected. Run the commands individually to ensure they all execute. This step takes a moment. It creates the kerberos database. +1. Create the KDC principal database. You will be prompted for a password. This step takes a moment. ``` kdb5_util create -s +``` + +1. Start the KDC and ensure that it starts on boot. + +``` /etc/rc.d/init.d/krb5kdc start -chkconfig krb5kdc on +chkconfig krb5kdc on +``` + +1. Start the Kerberos Admin service and ensure that it starts on boot. + +``` /etc/rc.d/init.d/kadmin start chkconfig kadmin on ``` -1. Setup the `admin` and `metron` principals. You'll `kinit` as the `metron` principal when running topologies. Make sure to remember the passwords. +1. Setup the `admin` principal. You will be prompted for a password; do not forget it. ``` kadmin.local -q "addprinc admin/admin" +``` + +1. Setup the `metron` principal. You will `kinit` as the `metron` principal when running topologies. You will be prompted for a password; do not forget it. + +``` kadmin.local -q "addprinc metron" ``` @@ -99,30 +117,30 @@ Verify KDC -- -Ticket renewal is by default disallowed in many linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies: - -``` -Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: The TGT found is not renewable -``` +1. Ticket renewal is disallowed by default in many Linux distributions. If the KDC cannot issue renewable tickets, an error will be thrown when starting Metron's Storm topologies: +``` +Exception in thread "main" java.lang.RuntimeException: +java.lang.RuntimeException: The TGT found is not renewable +``` -Ensure the Metron keytab is renewable. Look for the 'R' flag from the following command - -``` -klist -f -``` +1. Ensure the Metron keytab is renewable. Look for the 'R' flag in the output of the following command. -If the 'R' flags are present, you may skip to next section. +``` +klist -f +``` -If the 'R' flags are absent, you will need to follow the below steps: -If the KDC is already setup, then editing max_life and max_renewable_life in `/var/kerberos/krb5kdc/kdc.conf`, and restarting kadmin and krb5kdc services will not change the policies for existing users. +* If the 'R' flags are present, you may skip to next section. +* If the 'R' flags are absent, you will need to follow the below steps: + +1. If the KDC is already setup, then editing `max_life` and `max
metron git commit: METRON-1177 Stale running topologies seen post-kerberization and cause exceptions (nickwallen) closes apache/metron#748
Repository: metron Updated Branches: refs/heads/master 40c93527e -> ba9532de3 METRON-1177 Stale running topologies seen post-kerberization and cause exceptions (nickwallen) closes apache/metron#748 Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/ba9532de Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/ba9532de Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/ba9532de Branch: refs/heads/master Commit: ba9532de39bc7fb455050fea977bd4c543dc8c82 Parents: 40c9352 Author: nickwallen Authored: Wed Sep 13 14:27:16 2017 -0400 Committer: nickallen Committed: Wed Sep 13 14:27:16 2017 -0400 -- .../METRON/CURRENT/role_command_order.json | 28 +--- .../packaging/docker/rpm-docker/.gitignore | 1 + 2 files changed, 19 insertions(+), 10 deletions(-) -- http://git-wip-us.apache.org/repos/asf/metron/blob/ba9532de/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json -- diff --git a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json index 015f026..b474202 100755 --- a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json +++ b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/addon-services/METRON/CURRENT/role_command_order.json @@ -5,16 +5,24 @@ "_comment" : "dependencies for all cases", "METRON_INDEXING-INSTALL" : ["METRON_PARSERS-INSTALL"], "METRON_ENRICHMENT-INSTALL": ["METRON_INDEXING-INSTALL"], -"METRON_PROFILER-INSTALL": ["METRON_ENRICHMENT-INSTALL"], -"METRON_REST-INSTALL": ["METRON_PARSERS-INSTALL"], -"METRON_PARSERS-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_ENRICHMENT_MASTER-START"], +"METRON_PROFILER-INSTALL" : ["METRON_ENRICHMENT-INSTALL"], +"METRON_REST-INSTALL" : ["METRON_PARSERS-INSTALL"], +"METRON_PARSERS-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START" ,"METRON_ENRICHMENT_MASTER-START"], "METRON_ENRICHMENT_MASTER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START"], -"METRON_ENRICHMENT_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_ENRICHMENT_MASTER-START"], -"METRON_INDEXING-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START","METRON_PARSERS-START"], -"METRON_PROFILER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START", "METRON_ENRICHMENT-INSTALL"], -"METRON_REST-START": ["KAFKA_BROKER-START","STORM_REST_API-START","ZOOKEEPER_SERVER-START","NAMENODE-START","METRON_PARSERS-INSTALL","METRON_INDEXING-INSTALL","METRON_ENRICHMENT-INSTALL"], -"METRON_MANAGEMENT_UI-START": ["METRON_REST-START"], -"STORM_REST_API-STOP" : ["METRON_ENRICHMENT_MASTER-STOP","METRON_PARSERS-STOP","METRON_INDEXING-STOP","METRON_REST-STOP","METRON_MANAGEMENT_UI-STOP"], -"METRON_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_PARSERS-START","METRON_INDEXING-START"] +"METRON_INDEXING-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "METRON_PARSERS-START"], +"METRON_PROFILER-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "HBASE_MASTER-START", "HBASE_REGIONSERVER-START", "METRON_ENRICHMENT-INSTALL"], +"METRON_REST-START" : ["NAMENODE-START", "ZOOKEEPER_SERVER-START", "KAFKA_BROKER-START", "STORM_REST_API-START", "METRON_PARSERS-INSTALL","METRON_INDEXING-INSTALL","METRON_ENRICHMENT-INSTALL"], +"METRON_MANAGEMENT_UI-START" : ["METRON_REST-START"], + +"STORM_REST_API-STOP" : ["METRON_PARSERS-STOP", "METRON_ENRICHMENT_MASTER-STOP", "METRON_INDEXING-STOP", "METRON_PROFILER-STOP", "METRON_REST-STOP", "METRON_MANAGEMENT_UI-STOP"], +"STORM_UI_SERVER-STOP" : ["METRON_PARSERS-STOP", "METRON_ENRICHMENT_MASTER-STOP", "METRON_INDEXING-STOP", "METRON_PROFILER-STOP", "METRON_REST-STOP", "METRON_MANAGEMENT_UI-STOP"], + +"METRON_SERVICE_CHECK-SERVICE_CHECK" : ["METRON_PARSERS-START", "METRON_ENRICHMENT_MASTER-START", "METRON_INDEXING-START", "METRON_PROFILER-START", "METRON_REST-START", "METRON_MANAGEMENT_UI-START"], +"METRON_PARSERS_SERVICE_CHECK-SERVICE
[1/2] metron git commit: METRON-1158 Build backend for grouping alerts into meta alerts (justinleet) closes apache/metron#734
Repository: metron
Updated Branches:
refs/heads/master 309d3757d -> 40c93527e
http://git-wip-us.apache.org/repos/asf/metron/blob/40c93527/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDaoTest.java
--
diff --git
a/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDaoTest.java
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDaoTest.java
new file mode 100644
index 000..02ea795
--- /dev/null
+++
b/metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/dao/ElasticsearchMetaAlertDaoTest.java
@@ -0,0 +1,427 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.elasticsearch.dao;
+
+import static org.junit.Assert.assertArrayEquals;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.UUID;
+import org.apache.metron.common.Constants;
+import org.apache.metron.common.Constants.Fields;
+import org.apache.metron.indexing.dao.AccessConfig;
+import org.apache.metron.indexing.dao.IndexDao;
+import org.apache.metron.indexing.dao.MetaAlertDao;
+import org.apache.metron.indexing.dao.MultiIndexDao;
+import org.apache.metron.indexing.dao.metaalert.MetaAlertCreateRequest;
+import org.apache.metron.indexing.dao.metaalert.MetaScores;
+import org.apache.metron.indexing.dao.search.FieldType;
+import org.apache.metron.indexing.dao.search.GroupRequest;
+import org.apache.metron.indexing.dao.search.GroupResponse;
+import org.apache.metron.indexing.dao.search.InvalidCreateException;
+import org.apache.metron.indexing.dao.search.InvalidSearchException;
+import org.apache.metron.indexing.dao.search.SearchRequest;
+import org.apache.metron.indexing.dao.search.SearchResponse;
+import org.apache.metron.indexing.dao.update.Document;
+import org.elasticsearch.action.get.GetResponse;
+import org.elasticsearch.action.get.MultiGetItemResponse;
+import org.elasticsearch.action.get.MultiGetResponse;
+import org.elasticsearch.common.xcontent.XContentBuilder;
+import org.elasticsearch.search.SearchHit;
+import org.elasticsearch.search.SearchHitField;
+import org.elasticsearch.search.SearchHits;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.json.simple.parser.ParseException;
+import org.junit.Test;
+
+public class ElasticsearchMetaAlertDaoTest {
+
+ @Test
+ @SuppressWarnings("unchecked")
+ public void testBuildUpdatedMetaAlertSingleAlert() throws IOException,
ParseException {
+// Construct the expected result
+JSONObject expected = new JSONObject();
+expected.put("average", 5.0);
+expected.put("min", 5.0);
+expected.put("median", 5.0);
+expected.put("max", 5.0);
+expected.put("count", 1L);
+expected.put(Constants.GUID, "m1");
+expected.put("sum", 5.0);
+expected.put(MetaAlertDao.STATUS_FIELD,
MetaAlertStatus.ACTIVE.getStatusString());
+JSONArray expectedAlerts = new JSONArray();
+JSONObject expectedAlert = new JSONObject();
+expectedAlert.put(MetaAlertDao.THREAT_FIELD_DEFAULT, 5L);
+expectedAlert.put("fakekey", "fakevalue");
+expectedAlerts.add(expectedAlert);
+expected.put(MetaAlertDao.ALERT_FIELD, expectedAlerts);
+
+// Construct the meta alert object
+Map metaSource = new HashMap<>();
+metaSource.put(Constants.GUID, "m1");
+metaSource.put(MetaAlertDao.STATUS_FIELD,
MetaAlertStatus.ACTIVE.getStatusString());
+List alertScores = new ArrayList<>();
+alertScores.add(10d);
+metaSource.putAll(new MetaScores(alertScores).getMetaScores());
+SearchHit metaHit = mock(SearchHit.class);
+when(metaHit.getSource()).thenReturn(metaSource);
+
+// Construct the inner alert
+Se
[2/2] metron git commit: METRON-1158 Build backend for grouping alerts into meta alerts (justinleet) closes apache/metron#734
METRON-1158 Build backend for grouping alerts into meta alerts (justinleet)
closes apache/metron#734
Project: http://git-wip-us.apache.org/repos/asf/metron/repo
Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/40c93527
Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/40c93527
Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/40c93527
Branch: refs/heads/master
Commit: 40c93527e2a693ec6580dc0d09356dfa3b525aa4
Parents: 309d375
Author: justinleet
Authored: Wed Sep 13 11:38:05 2017 -0400
Committer: leet
Committed: Wed Sep 13 11:38:05 2017 -0400
--
.../CURRENT/package/files/bro_index.template| 3 +
.../CURRENT/package/files/error_index.template | 3 +
.../CURRENT/package/files/meta_index.mapping| 42 ++
.../CURRENT/package/files/snort_index.template | 3 +
.../CURRENT/package/files/yaf_index.template| 3 +
.../CURRENT/package/scripts/indexing_master.py | 8 +
.../package/scripts/params/params_linux.py | 1 +
metron-interface/metron-rest/README.md | 18 +
.../apache/metron/rest/MetronRestConstants.java | 3 +
.../apache/metron/rest/config/IndexConfig.java | 16 +-
.../rest/controller/MetaAlertController.java| 64 +++
.../metron/rest/service/MetaAlertService.java | 31 ++
.../rest/service/impl/MetaAlertServiceImpl.java | 66 +++
.../rest/service/impl/SearchServiceImpl.java| 1 +
.../src/main/resources/application-test.yml | 5 +
.../src/main/resources/application.yml | 4 +
.../rest/controller/DaoControllerTest.java | 20 +-
.../MetaAlertControllerIntegrationTest.java | 174
.../SearchControllerIntegrationTest.java| 8 +-
.../UpdateControllerIntegrationTest.java| 20 +-
.../elasticsearch/dao/ElasticsearchDao.java | 57 ++-
.../dao/ElasticsearchMetaAlertDao.java | 446 +++
.../elasticsearch/dao/MetaAlertStatus.java | 34 ++
.../dao/ElasticsearchMetaAlertDaoTest.java | 427 ++
.../ElasticsearchMetaAlertIntegrationTest.java | 317 +
.../ElasticsearchSearchIntegrationTest.java | 18 +-
.../ElasticsearchUpdateIntegrationTest.java | 3 +
.../components/ElasticSearchComponent.java | 15 +
metron-platform/metron-indexing/README.md | 17 +
.../metron/indexing/dao/MetaAlertDao.java | 72 +++
.../metron/indexing/dao/MultiIndexDao.java | 4 +
.../dao/metaalert/MetaAlertCreateRequest.java | 51 +++
.../dao/metaalert/MetaAlertCreateResponse.java | 31 ++
.../indexing/dao/metaalert/MetaScores.java | 54 +++
.../metron/indexing/dao/search/FieldType.java | 2 +
.../dao/search/InvalidCreateException.java | 28 ++
.../indexing/dao/search/SearchResult.java | 10 +
.../metron/indexing/dao/update/Document.java| 13 +-
.../apache/metron/indexing/dao/InMemoryDao.java | 38 +-
.../indexing/dao/InMemoryMetaAlertDao.java | 198
.../indexing/dao/SearchIntegrationTest.java | 77 +++-
.../stellar/dsl/functions/BasicStellarTest.java | 5 +
pom.xml | 1 +
43 files changed, 2357 insertions(+), 54 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/metron/blob/40c93527/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
--
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
index 18c5d9b..7db006e 100644
---
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
+++
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/bro_index.template
@@ -151,6 +151,9 @@
"type": "string",
"index": "not_analyzed"
},
+"alert": {
+ "type": "nested"
+},
"ip_src_addr": {
"type": "ip"
},
http://git-wip-us.apache.org/repos/asf/metron/blob/40c93527/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/error_index.template
--
diff --git
a/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/error_index.template
b/metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/files/error_index.template
index 3bb4633..e79d482 100644
---
a/metron-deployment/packaging/a
