[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/940#discussion_r172711543
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
 ---
@@ -0,0 +1,281 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.parallel;
+
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import org.apache.metron.common.Constants;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.performance.PerformanceLogger;
+import org.apache.metron.common.utils.MessageUtils;
+import org.apache.metron.enrichment.bolt.CacheKey;
+import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
+import org.apache.metron.enrichment.utils.EnrichmentUtils;
+import org.json.simple.JSONObject;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.function.BinaryOperator;
+import java.util.function.Supplier;
+
+/**
+ * This is an independent component which will accept a message and a set 
of enrichment adapters as well as a config which defines
+ * how those enrichments should be performed and fully enrich the message. 
 The result will be the enriched message
+ * unified together and a list of errors which happened.
+ */
+public class ParallelEnricher {
+
+  private Map enrichmentsByType = new 
HashMap<>();
+  private EnumMap cacheStats = new 
EnumMap<>(EnrichmentStrategies.class);
+
+  /**
+   * The result of an enrichment.
+   */
+  public static class EnrichmentResult {
+private JSONObject result;
+private List> enrichmentErrors;
+
+public EnrichmentResult(JSONObject result, List> enrichmentErrors) {
+  this.result = result;
+  this.enrichmentErrors = enrichmentErrors;
+}
+
+/**
+ * The unified fully enriched result.
+ * @return
+ */
+public JSONObject getResult() {
+  return result;
+}
+
+/**
+ * The errors that happened in the course of enriching.
+ * @return
+ */
+public List> getEnrichmentErrors() {
+  return enrichmentErrors;
+}
+  }
+
+  private ConcurrencyContext concurrencyContext;
+
+  /**
+   * Construct a parallel enricher with a set of enrichment adapters 
associated with their enrichment types.
+   * @param enrichmentsByType
+   */
+  public ParallelEnricher( Map 
enrichmentsByType
+ , ConcurrencyContext concurrencyContext
+ , boolean logStats
+ )
+  {
+this.enrichmentsByType = enrichmentsByType;
+this.concurrencyContext = concurrencyContext;
+if(logStats) {
+  for(EnrichmentStrategies s : EnrichmentStrategies.values()) {
+cacheStats.put(s, null);
+  }
+}
+  }
+
+  /**
+   * Fully enriches a message.  Each enrichment is done in parallel via a 
threadpool.
+   * Each enrichment is fronted with a LRU cache.
+   *
+   * @param message the message to enrich
+   * @param strategy The enrichment strategy to use (e.g. enrichment or 
threat intel)
+   * @param config The sensor enrichment config
+   * @param perfLog The performance logger.  We log the performance for 
this 

[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/940
  
+1 The unified topology works great.


---

[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/940#discussion_r172694248
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
 ---
@@ -0,0 +1,281 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.parallel;
+
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import org.apache.metron.common.Constants;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.performance.PerformanceLogger;
+import org.apache.metron.common.utils.MessageUtils;
+import org.apache.metron.enrichment.bolt.CacheKey;
+import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
+import org.apache.metron.enrichment.utils.EnrichmentUtils;
+import org.json.simple.JSONObject;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.function.BinaryOperator;
+import java.util.function.Supplier;
+
+/**
+ * This is an independent component which will accept a message and a set 
of enrichment adapters as well as a config which defines
+ * how those enrichments should be performed and fully enrich the message. 
 The result will be the enriched message
+ * unified together and a list of errors which happened.
+ */
+public class ParallelEnricher {
+
+  private Map enrichmentsByType = new 
HashMap<>();
+  private EnumMap cacheStats = new 
EnumMap<>(EnrichmentStrategies.class);
+
+  /**
+   * The result of an enrichment.
+   */
+  public static class EnrichmentResult {
+private JSONObject result;
+private List> enrichmentErrors;
+
+public EnrichmentResult(JSONObject result, List> enrichmentErrors) {
+  this.result = result;
+  this.enrichmentErrors = enrichmentErrors;
+}
+
+/**
+ * The unified fully enriched result.
+ * @return
+ */
+public JSONObject getResult() {
+  return result;
+}
+
+/**
+ * The errors that happened in the course of enriching.
+ * @return
+ */
+public List> getEnrichmentErrors() {
+  return enrichmentErrors;
+}
+  }
+
+  private ConcurrencyContext concurrencyContext;
+
+  /**
+   * Construct a parallel enricher with a set of enrichment adapters 
associated with their enrichment types.
+   * @param enrichmentsByType
+   */
+  public ParallelEnricher( Map 
enrichmentsByType
+ , ConcurrencyContext concurrencyContext
+ , boolean logStats
+ )
+  {
+this.enrichmentsByType = enrichmentsByType;
+this.concurrencyContext = concurrencyContext;
+if(logStats) {
+  for(EnrichmentStrategies s : EnrichmentStrategies.values()) {
+cacheStats.put(s, null);
+  }
+}
+  }
+
+  /**
+   * Fully enriches a message.  Each enrichment is done in parallel via a 
threadpool.
+   * Each enrichment is fronted with a LRU cache.
+   *
+   * @param message the message to enrich
+   * @param strategy The enrichment strategy to use (e.g. enrichment or 
threat intel)
+   * @param config The sensor enrichment config
+   * @param perfLog The performance logger.  We log the performance for 

[GitHub] metron pull request #950: METRON-1470: Update jquery to version 3+

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/950


---

[GitHub] metron issue #950: METRON-1470: Update jquery to version 3+

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/950
  
I ran this up in full dev and tested several different features.  I could 
not find any regressions.  Thanks @xyztdanid4! +1 


---

[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/940#discussion_r172656809
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
 ---
@@ -0,0 +1,281 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.parallel;
+
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import org.apache.metron.common.Constants;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.performance.PerformanceLogger;
+import org.apache.metron.common.utils.MessageUtils;
+import org.apache.metron.enrichment.bolt.CacheKey;
+import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
+import org.apache.metron.enrichment.utils.EnrichmentUtils;
+import org.json.simple.JSONObject;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.function.BinaryOperator;
+import java.util.function.Supplier;
+
+/**
+ * This is an independent component which will accept a message and a set 
of enrichment adapters as well as a config which defines
+ * how those enrichments should be performed and fully enrich the message. 
 The result will be the enriched message
+ * unified together and a list of errors which happened.
+ */
+public class ParallelEnricher {
+
+  private Map enrichmentsByType = new 
HashMap<>();
+  private EnumMap cacheStats = new 
EnumMap<>(EnrichmentStrategies.class);
+
+  /**
+   * The result of an enrichment.
+   */
+  public static class EnrichmentResult {
+private JSONObject result;
+private List> enrichmentErrors;
+
+public EnrichmentResult(JSONObject result, List> enrichmentErrors) {
+  this.result = result;
+  this.enrichmentErrors = enrichmentErrors;
+}
+
+/**
+ * The unified fully enriched result.
+ * @return
+ */
+public JSONObject getResult() {
+  return result;
+}
+
+/**
+ * The errors that happened in the course of enriching.
+ * @return
+ */
+public List> getEnrichmentErrors() {
+  return enrichmentErrors;
+}
+  }
+
+  private ConcurrencyContext concurrencyContext;
+
+  /**
+   * Construct a parallel enricher with a set of enrichment adapters 
associated with their enrichment types.
+   * @param enrichmentsByType
+   */
+  public ParallelEnricher( Map 
enrichmentsByType
+ , ConcurrencyContext concurrencyContext
+ , boolean logStats
+ )
+  {
+this.enrichmentsByType = enrichmentsByType;
+this.concurrencyContext = concurrencyContext;
+if(logStats) {
+  for(EnrichmentStrategies s : EnrichmentStrategies.values()) {
+cacheStats.put(s, null);
+  }
+}
+  }
+
+  /**
+   * Fully enriches a message.  Each enrichment is done in parallel via a 
threadpool.
+   * Each enrichment is fronted with a LRU cache.
+   *
+   * @param message the message to enrich
+   * @param strategy The enrichment strategy to use (e.g. enrichment or 
threat intel)
+   * @param config The sensor enrichment config
+   * @param perfLog The performance logger.  We log the performance for 
this 

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/943
  
Also, I will look at moving the Kibana templates to Metron as this is 
something that will need to be done for Solr as well.


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/943
  
@ottobackwards I believe the repercussions are the same as they are for the 
existing MPack since the services are distinct from the MPack itself. We should 
probably consider a separate test for this.


---

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/943
  
What about upgrading.md?  


---

[GitHub] metron pull request #940: METRON-1460: Create a complementary non-split-join...

[nickwallen]

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/940#discussion_r172595029
  
--- Diff: 
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/parallel/ParallelEnricher.java
 ---
@@ -0,0 +1,281 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.enrichment.parallel;
+
+import com.github.benmanes.caffeine.cache.stats.CacheStats;
+import org.apache.metron.common.Constants;
+import 
org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig;
+import 
org.apache.metron.common.configuration.enrichment.handler.ConfigHandler;
+import org.apache.metron.common.performance.PerformanceLogger;
+import org.apache.metron.common.utils.MessageUtils;
+import org.apache.metron.enrichment.bolt.CacheKey;
+import org.apache.metron.enrichment.interfaces.EnrichmentAdapter;
+import org.apache.metron.enrichment.utils.EnrichmentUtils;
+import org.json.simple.JSONObject;
+
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutionException;
+import java.util.function.BinaryOperator;
+import java.util.function.Supplier;
+
+/**
+ * This is an independent component which will accept a message and a set 
of enrichment adapters as well as a config which defines
+ * how those enrichments should be performed and fully enrich the message. 
 The result will be the enriched message
+ * unified together and a list of errors which happened.
+ */
+public class ParallelEnricher {
+
+  private Map enrichmentsByType = new 
HashMap<>();
+  private EnumMap cacheStats = new 
EnumMap<>(EnrichmentStrategies.class);
+
+  /**
+   * The result of an enrichment.
+   */
+  public static class EnrichmentResult {
+private JSONObject result;
+private List> enrichmentErrors;
+
+public EnrichmentResult(JSONObject result, List> enrichmentErrors) {
+  this.result = result;
+  this.enrichmentErrors = enrichmentErrors;
+}
+
+/**
+ * The unified fully enriched result.
+ * @return
+ */
+public JSONObject getResult() {
+  return result;
+}
+
+/**
+ * The errors that happened in the course of enriching.
+ * @return
+ */
+public List> getEnrichmentErrors() {
+  return enrichmentErrors;
+}
+  }
+
+  private ConcurrencyContext concurrencyContext;
+
+  /**
+   * Construct a parallel enricher with a set of enrichment adapters 
associated with their enrichment types.
+   * @param enrichmentsByType
+   */
+  public ParallelEnricher( Map 
enrichmentsByType
+ , ConcurrencyContext concurrencyContext
+ , boolean logStats
+ )
+  {
+this.enrichmentsByType = enrichmentsByType;
+this.concurrencyContext = concurrencyContext;
+if(logStats) {
+  for(EnrichmentStrategies s : EnrichmentStrategies.values()) {
+cacheStats.put(s, null);
+  }
+}
+  }
+
+  /**
+   * Fully enriches a message.  Each enrichment is done in parallel via a 
threadpool.
+   * Each enrichment is fronted with a LRU cache.
+   *
+   * @param message the message to enrich
+   * @param strategy The enrichment strategy to use (e.g. enrichment or 
threat intel)
+   * @param config The sensor enrichment config
+   * @param perfLog The performance logger.  We log the performance for 

[GitHub] metron issue #943: METRON-1462: Separate ES and Kibana from Metron Mpack

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/943
  
I spun this up in full dev and everything worked as expected.  The 
organization looks good to me and I can't find anything wrong with it.  +1


---

[GitHub] metron pull request #950: METRON-1470: Update jquery to version 3+

[xyztdanid4]

GitHub user xyztdanid4 opened a pull request:

https://github.com/apache/metron/pull/950

METRON-1470: Update jquery to version 3+

I updated jquerry dependency of the project due to security reason.

- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at 
[https://issues.apache.org/jira/browse/METRON-1470](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?

### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```
- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [X] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [X] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xyztdanid4/metron METRON-1470

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/950.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #950


commit a25f69c6d85fe51bb53b4731e5747745e15e9041
Author: Daniel Toth 
Date:   2018-03-06T15:49:41Z

Update jquerry to avoid security issue




---

[GitHub] metron issue #948: METRON-1468: Add support for apache/metron-bro-plugin-kaf...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/948
  
+1 


---

[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/942
  
I would say performance trumps complexity of functionality here.


---

[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/942
  
Maybe making this function so generic is going to necessitate it being so 
complicated that it is harder to maintain etc.


---

[GitHub] metron issue #948: METRON-1468: Add support for apache/metron-bro-plugin-kaf...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/948
  
I agree, updated the script and readme.


---

[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/942
  
The first point here is around semantics.  I am assuming the semantics of 
this would be a "max of maxes".  So if I have a list of stats objects, I 
compare the max of each one.  Whichever max is the greatest, that stats object 
gets returned.  The MIN function would just be a "min of mins".

Does that match your use case?  Is this the right approach?

Given those semantics, I think the 'Comparable' approach could work, but 
with a twist.  You can't just make the Stats objects Comparable.  Because how 
do you compare them?  By the average, median, min or max?  There is not one way 
to do it that is broadly applicable.

The means of comparison for the MAX function should use the max of a stats 
object.  The means of comparison for the MIN function should use the min of the 
stats object. 

One way is to create a class that wraps a Stats object and implements the 
Comparable interface.  In the case of MAX, the wrapper will compare using the 
max of the underlying stats object.  In the case of MIN function, the wrapper 
will compare using the min of the underlying stats object. 





---

[GitHub] metron pull request #949: METRON-1471: Migrate shuffle connections to local ...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/949

METRON-1471: Migrate shuffle connections to local or shuffle

## Contributor Comments
Currently, we use shuffle groupings when we do not want to group by field.  
We should, instead, use local or shuffle groupings.

Manual tests should include ensuring data continues to flow as before.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron 
local_or_shuffle_indexing

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/949.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #949


commit 0ab26fa172a90eaf8eb441c8c085308649efdfb9
Author: cstella 
Date:   2018-03-06T16:18:57Z

Updating the rest of the topologies to prefer local or shuffle over shuffle




---

[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/942
  
I would like to address the issue somehow or at least garner more community 
feedback on this change.

As it stands, usage of the function is not very consistent.  For example, I 
can pass a list of numbers, but I can't pass a list of Stat objects.  I can 
have a list of mixed numeric types, but I can't have a Stats object in a mixed 
list.  That is inconsistent IMO.






---

[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/940
  
Ok, README is updated with the new topology diagram.  Let me know if 
there's anything else.


---

[GitHub] metron issue #948: METRON-1468: Add support for apache/metron-bro-plugin-kaf...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/948
  
Other than that little nit, it works great though.  Thanks @JonZeolla 


---

[GitHub] metron issue #948: METRON-1468: Add support for apache/metron-bro-plugin-kaf...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/948
  
When I got to the point of selecting the repo, I was unsure of what to type 
for the Bro Plugin repo.

```
$ ./metron-pr948/dev-utilities/committer-utils/prepare-commit
  ...using settings from /Users/nallen/.metron-prepare-commit
  which repo? [metron]:
```

Do you think we could make this more obvious somehow?  Maybe something like 
the following?

```
$ ./metron-pr948/dev-utilities/committer-utils/prepare-commit
  ...using settings from /Users/nallen/.metron-prepare-commit
  which repo? [1]:
[1] metron
[2] metron-bro-plugin-kafka
```

This would, as it does now, default to `[1] metron`.


---

[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

[MohanDV]

Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/942
  
@nickwallen I dint consider implementation where STATS object can be passed 
in list of mixed object . That requires to change the STATS object to be 
'Comparable' type . Do you think it should be addressed in this pull request ?


---

[GitHub] metron-bro-plugin-kafka issue #6: Configurable JSON timestamps and default a...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/6
  
@dcode 

1. The JIRA created for this is 
https://issues.apache.org/jira/browse/METRON-1469.

1. Please change the PR title to "METRON-1469: Kafka Plugin for Bro - 
Configurable JSON Timestamps".

1. Please update your PR description to remove references to "send all 
logs".  I just want to avoid any future confusion.


---

[GitHub] metron-bro-plugin-kafka issue #7: METRON-1324: Increment metron-bro-plugin-k...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron-bro-plugin-kafka/pull/7
  
+1 Thanks, @JonZeolla !


---

[GitHub] metron pull request #936: METRON-1450:Add rest endpoint documentation for sp...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/936


---

[GitHub] metron issue #936: METRON-1450:Add rest endpoint documentation for splitting...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/936
  
Thanks @MohanDV .  Will merge this now.


---

[GitHub] metron pull request #946: METRON-1465:Support for Elasticsearch X-pack

[mmiklavc]

Github user mmiklavc commented on a diff in the pull request:

https://github.com/apache/metron/pull/946#discussion_r172544606
  
--- Diff: 
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/scripts/metron_service.py
 ---
@@ -70,6 +70,11 @@ def build_global_config_patch(params, patch_file):
 "path": "/es.date.format",
 "value": "{{es_date_format}}"
 },
+{
+"op": "add",
+"path": "/es.xpack.user",
+"value": "{{es_xpack_user}}"
--- End diff --

We can't store passwords like this in plain text because they'll end up in 
Zookeeper as plaintext without permissions locking it down.


---

[GitHub] metron issue #942: METRON-1461: Modify the MIN, MAX Stellar methods to take ...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/942
  
@MohanDV - It looks like some care was taken previously so that getting the 
max of a list of mixed elements will just work.  For example `MAX([1, 2d, 3f]) 
== 3f`.  

Did you consider an implementation such that this would continue to work 
with STATS objects?  I imagine that something like the following could be made 
to work; `MAX[1, 2d, 3f, stats]`?  

What do you think?




---

[GitHub] metron pull request #853: METRON-1337: List of facets should not be hardcode...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/853


---

[GitHub] metron issue #941: METRON-1355: Convert metron-elasticsearch to new infrastr...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/941
  
Sorry I should have documented this better in the PR description.  The 
docker-machine ip address needs to be substituted in ElasticsearchTestUtils 
temporarily if you want to run this on your local Mac OS.  There is a Jira for 
this work here:  https://issues.apache.org/jira/browse/METRON-1356.  Everything 
you've described including the TODO is intended to be handled there.  Do you 
think that needs to come first before you review this PR?  


---

[GitHub] metron pull request #933: METRON-1452 Rebase Dev Environment on Latest CentO...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/933


---

[GitHub] metron issue #940: METRON-1460: Create a complementary non-split-join enrich...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/940
  
That's great @cestella .  Many thanks.  I will run it up in the lab. No 
problem.


---

[GitHub] metron issue #936: METRON-1450:Add rest endpoint documentation for splitting...

[MohanDV]

Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/936
  
@nickwallen  / @JonZeolla  can you please merge this request.


---