[jira] [Commented] (METRON-1487) Define Performance Benchmarks for Enrichment Topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396139#comment-16396139
 ] 

ASF GitHub Bot commented on METRON-1487:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r173963284
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,522 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Enrichment topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java). 
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration 
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise. 
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+   
+## Benchmark Execution
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do this for all 1 million entries in the data set. Doing this for around 10,000 
records is sufficient.
+
+   ```python
+   import dns.resolver
+   import csv
+
+   resolver = dns.resolver.Resolver()
+   resolver.nameservers = ['8.8.8.8', '8.8.4.4']
+
+   with 

[jira] [Commented] (METRON-1487) Define Performance Benchmarks for Enrichment Topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16396135#comment-16396135
 ] 

ASF GitHub Bot commented on METRON-1487:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r173963096
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,522 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Enrichment topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java). 
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration 
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise. 
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+   
+## Benchmark Execution
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do this for all 1 million entries in the data set. Doing this for around 10,000 
records is sufficient.
+
+   ```python
+   import dns.resolver
+   import csv
+
+   resolver = dns.resolver.Resolver()
+   resolver.nameservers = ['8.8.8.8', '8.8.4.4']
+
+   with 

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395877#comment-16395877
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173943133
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/AbstractMonitor.java
 ---
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load.monitor;
+
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public abstract class AbstractMonitor implements Supplier, 
MonitorNaming {
+  private static final double EPSILON = 1e-6;
+  protected Optional kafkaTopic;
+  protected long timestampPrevious = 0;
+  public AbstractMonitor(Optional kafkaTopic) {
+this.kafkaTopic = kafkaTopic;
+  }
+
+  protected abstract Long monitor(double deltaTs);
+
+  @Override
+  public Long get() {
+long timeStarted = System.currentTimeMillis();
--- End diff --

Nope, I'm missing my common sense is all.


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395872#comment-16395872
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173942355
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/AbstractMonitor.java
 ---
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load.monitor;
+
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public abstract class AbstractMonitor implements Supplier, 
MonitorNaming {
+  private static final double EPSILON = 1e-6;
+  protected Optional kafkaTopic;
+  protected long timestampPrevious = 0;
+  public AbstractMonitor(Optional kafkaTopic) {
+this.kafkaTopic = kafkaTopic;
+  }
+
+  protected abstract Long monitor(double deltaTs);
+
+  @Override
+  public Long get() {
+long timeStarted = System.currentTimeMillis();
--- End diff --

To my knowledge, `currentTimeMillis()` is not affected by DST as it is the 
milliseconds since unix epoch UTC.  Cursory googling seems to hold up this 
assertion.  Did I miss something?


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395852#comment-16395852
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173929511
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/Writer.java
 ---
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load.monitor.writers;
+
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.metron.performance.load.monitor.AbstractMonitor;
+import org.apache.metron.performance.load.monitor.Results;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.function.Consumer;
+
+public class Writer {
+
+  private int summaryLookback;
+  private List summaries = new ArrayList<>();
+  private List writers;
+  private List monitors;
+
+  public Writer(List monitors, int summaryLookback, 
List writers) {
+this.summaryLookback = summaryLookback;
+this.writers = writers;
+this.monitors = monitors;
+for(AbstractMonitor m : monitors) {
+  this.summaries.add(new LinkedList<>());
+}
+  }
+
+  public void writeAll() {
+int i = 0;
+Date dateOf = new Date();
+List results = new ArrayList<>();
+for(AbstractMonitor m : monitors) {
+  Long eps = m.get();
+  if(eps != null) {
+if (summaryLookback > 0) {
+  LinkedList summary = summaries.get(i);
+  addToLookback(eps == null ? Double.NaN : eps.doubleValue(), 
summary);
+  results.add(new Results(m.format(), m.name(), eps, 
Optional.of(getStats(summary;
+}
+else {
+  results.add(new Results(m.format(),m.name(), eps, 
Optional.empty()));
+}
+  }
+  else {
--- End diff --

Since the two else's are the same, I think this should simplify down to 
(with the above change):
```
  if (eps != null && summaryLookback > 0) {
LinkedList summary = summaries.get(i);
addToLookback(eps.doubleValue(), summary);
results.add(new Results(m.format(), m.name(), eps, 
Optional.of(getStats(summary;
  } else {
results.add(new Results(m.format(), m.name(), eps, 
Optional.empty()));
  }
```


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395843#comment-16395843
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173918861
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+
+import com.google.common.base.Joiner;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.PosixParser;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.metron.performance.load.monitor.AbstractMonitor;
+import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor;
+import 
org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor;
+import org.apache.metron.performance.load.monitor.MonitorTask;
+import org.apache.metron.performance.load.monitor.writers.CSVWriter;
+import org.apache.metron.performance.load.monitor.writers.ConsoleWriter;
+import org.apache.metron.performance.load.monitor.writers.Writable;
+import org.apache.metron.performance.load.monitor.writers.Writer;
+import org.apache.metron.performance.sampler.BiasedSampler;
+import org.apache.metron.performance.sampler.Sampler;
+import org.apache.metron.performance.sampler.UnbiasedSampler;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+public class LoadGenerator
+{
+  public static String CONSUMER_GROUP = "load.group";
+  public static long SEND_PERIOD_MS = 100;
+  public static long MONITOR_PERIOD_MS = 1000*10;
+  private static ExecutorService pool;
+  private static ThreadLocal kafkaProducer;
+  public static AtomicLong numSent = new AtomicLong(0);
+
+  public static void main( String[] args ) throws Exception {
+CommandLine cli = LoadOptions.parse(new PosixParser(), args);
+EnumMap evaluatedArgs = 
LoadOptions.createConfig(cli);
+Map kafkaConfig = new HashMap<>();
+kafkaConfig.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
+kafkaConfig.put("value.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
+kafkaConfig.put("key.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
+kafkaConfig.put("value.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
+if(LoadOptions.ZK.has(cli)) {
+  String zkQuorum = (String) evaluatedArgs.get(LoadOptions.ZK).get();
+  kafkaConfig.put("bootstrap.servers", 
Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum)));
--- End diff --

Same as before: ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG





> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file 

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395854#comment-16395854
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173933259
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/UnbiasedSampler.java
 ---
@@ -0,0 +1,28 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.sampler;
+
+import java.util.Random;
+
+public class UnbiasedSampler implements Sampler {
+
+  @Override
+  public int sample(Random rng, int limit) {
+return rng.nextInt(limit);
--- End diff --

If the templates file is empty, there will be no bounds here, and this will 
start puking exceptions out.


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395847#comment-16395847
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173925390
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/SendToKafka.java
 ---
@@ -0,0 +1,107 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerRecord;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.TimerTask;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
+
+public class SendToKafka extends TimerTask {
--- End diff --

This class feels like it should be genericized as SendToKafka where 
it takes args as appropriate.  Right now this is just basically a 
SendToKafka, correct?


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395851#comment-16395851
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173928769
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/Writer.java
 ---
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load.monitor.writers;
+
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.metron.performance.load.monitor.AbstractMonitor;
+import org.apache.metron.performance.load.monitor.Results;
+
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.function.Consumer;
+
+public class Writer {
+
+  private int summaryLookback;
+  private List summaries = new ArrayList<>();
+  private List writers;
+  private List monitors;
+
+  public Writer(List monitors, int summaryLookback, 
List writers) {
+this.summaryLookback = summaryLookback;
+this.writers = writers;
+this.monitors = monitors;
+for(AbstractMonitor m : monitors) {
+  this.summaries.add(new LinkedList<>());
+}
+  }
+
+  public void writeAll() {
+int i = 0;
+Date dateOf = new Date();
+List results = new ArrayList<>();
+for(AbstractMonitor m : monitors) {
+  Long eps = m.get();
+  if(eps != null) {
+if (summaryLookback > 0) {
+  LinkedList summary = summaries.get(i);
+  addToLookback(eps == null ? Double.NaN : eps.doubleValue(), 
summary);
--- End diff --

eps is never null here, so this can be just 
`addToLookback(eps.doubleValue(), summary);`


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395849#comment-16395849
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173928192
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/writers/ConsoleWriter.java
 ---
@@ -0,0 +1,67 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load.monitor.writers;
+
+import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics;
+import org.apache.metron.performance.load.monitor.Results;
+
+import java.text.DateFormat;
+import java.text.SimpleDateFormat;
+import java.util.ArrayList;
+import java.util.Date;
+import java.util.List;
+import java.util.function.Consumer;
+
+public class ConsoleWriter implements Consumer {
+
+  private String getSummary(DescriptiveStatistics stats) {
+return String.format("Mean: %d, Std Dev: %d", (int)stats.getMean(), 
(int)Math.sqrt(stats.getVariance()));
+  }
+
+  @Override
+  public void accept(Writable writable) {
+List parts = new ArrayList<>();
+Date date = writable.getDate();
+for(Results r : writable.getResults()) {
+  Long eps = r.getEps();
+  if(eps != null) {
+String part = String.format(r.getFormat(), eps);
+if (r.getHistory().isPresent()) {
+  part += " (" + getSummary(r.getHistory().get()) + ")";
+}
+parts.add(part);
+  }
+}
+if(date != null) {
+  DateFormat dateFormat = new SimpleDateFormat("/MM/dd HH:mm:ss");
+  String header = dateFormat.format(date) + " - ";
+  String emptyHeader = "";
+  for (int i = 0; i < header.length(); ++i) {
--- End diff --

Could use `StringUtils.rightPad` to avoid the awkward append blanks loop.


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395855#comment-16395855
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173930965
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/BiasedSampler.java
 ---
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.sampler;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.TreeMap;
+
+public class BiasedSampler implements Sampler {
+  TreeMap> discreteDistribution;
+  public BiasedSampler(List>  
discreteDistribution, int max) {
+this.discreteDistribution = createDistribution(discreteDistribution, 
max);
+  }
+
+  public static List> readDistribution(File 
distrFile) throws IOException {
+List> ret = new ArrayList<>();
+System.out.println("Using biased sampler with the following biases:");
+try(BufferedReader br = new BufferedReader(new FileReader(distrFile))) 
{
+  int sumLeft = 0;
+  int sumRight = 0;
+  for(String line = null;(line = br.readLine()) != null;) {
+if(line.startsWith("#")) {
+  continue;
+}
+Iterable it = Splitter.on(",").split(line.trim());
+int left = Integer.parseInt(Iterables.getFirst(it, null));
--- End diff --

I wouldn't expect it to happen, but left/right will throw a 
NumberFormatException if the iterables list is empty. Do we need/want to be 
more robust about it?


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395844#comment-16395844
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173924485
  
--- Diff: 
metron-contrib/metron-performance/src/test/java/org/apache/metron/performance/load/SendToKafkaTest.java
 ---
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.ForkJoinPool;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicLong;
+
+public class SendToKafkaTest {
+
+  @Test
+  public void testWritesCorrectNumber() throws InterruptedException {
--- End diff --

InterruptedException isn't thrown here, is it?


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395850#comment-16395850
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173922548
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/MessageGenerator.java
 ---
@@ -0,0 +1,48 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+import org.apache.metron.performance.sampler.Sampler;
+
+import java.util.List;
+import java.util.Random;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Supplier;
+
+public class MessageGenerator implements Supplier {
+  private static ThreadLocal rng = ThreadLocal.withInitial(() -> 
new Random());
+  private static AtomicLong guidOffset = new AtomicLong(0);
+  private static String guidPrefix = "6141faf6-a8ba-4044-ab80-";
--- End diff --

I assume this is being done to avoid the cost of creating an actual UUID?

Why bother though? Why not just use the guidOffset directly? Is it to be at 
least somewhat close in terms of data size / format? In that case, why not just 
have the prefix be all zeroes or something easier to quickly read at a glance?


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395846#comment-16395846
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173921179
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadOptions.java
 ---
@@ -0,0 +1,504 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+import com.google.common.base.Joiner;
+import org.apache.commons.cli.*;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.common.utils.cli.CLIOptions;
+import org.apache.metron.performance.sampler.BiasedSampler;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.common.utils.cli.OptionHandler;
+
+import javax.annotation.Nullable;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Optional;
+
+public enum LoadOptions implements CLIOptions {
+  HELP(new OptionHandler() {
+
+@Override
+public String getShortCode() {
+  return "h";
+}
+
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  return new Option(s, "help", false, "Generate Help screen");
+}
+  }),
+  ZK(new OptionHandler() {
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  Option o = new Option(s, "zk_quorum", true, "zookeeper quorum");
+  o.setArgName("QUORUM");
+  o.setRequired(false);
+  return o;
+}
+
+@Override
+public Optional getValue(LoadOptions option, CommandLine cli) {
+  if(option.has(cli)) {
+return Optional.ofNullable(option.get(cli));
+  }
+  else {
+return Optional.empty();
+  }
+}
+
+@Override
+public String getShortCode() {
+  return "z";
+}
+  }),
+  CONSUMER_GROUP(new OptionHandler() {
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  Option o = new Option(s, "consumer_group", true, "Consumer Group.  
The default is " + LoadGenerator.CONSUMER_GROUP);
+  o.setArgName("GROUP_ID");
+  o.setRequired(false);
+  return o;
+}
+
+@Override
+public Optional getValue(LoadOptions option, CommandLine cli) {
+  if(option.has(cli)) {
+return Optional.ofNullable(option.get(cli));
+  }
+  else {
+return Optional.of(LoadGenerator.CONSUMER_GROUP);
+  }
+}
+
+@Override
+public String getShortCode() {
+  return "cg";
+}
+  }),
+  BIASED_SAMPLE(new OptionHandler() {
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  Option o = new Option(s, "sample_bias", true, "The discrete 
distribution to bias the sampling. " +
+  "This is a CSV of 2 columns.  The first column is the % of 
the templates " +
+  "and the 2nd column is the probability (0-100) that it's 
chosen.  For instance:\n" +
+  "  20,80\n" +
+  "  80,20\n" +
+  "implies that 20% of the templates will comprise 80% of the 
output and the remaining 80% of the templates will comprise 20% of the 
output.");
+  o.setArgName("BIAS_FILE");
+  o.setRequired(false);
+  return o;
+}
+
+@Override
+public Optional getValue(LoadOptions option, CommandLine cli) {
+  if(!option.has(cli)) {
+return Optional.empty();

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395840#comment-16395840
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173917240
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+
+import com.google.common.base.Joiner;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.PosixParser;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.metron.performance.load.monitor.AbstractMonitor;
+import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor;
+import 
org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor;
+import org.apache.metron.performance.load.monitor.MonitorTask;
+import org.apache.metron.performance.load.monitor.writers.CSVWriter;
+import org.apache.metron.performance.load.monitor.writers.ConsoleWriter;
+import org.apache.metron.performance.load.monitor.writers.Writable;
+import org.apache.metron.performance.load.monitor.writers.Writer;
+import org.apache.metron.performance.sampler.BiasedSampler;
+import org.apache.metron.performance.sampler.Sampler;
+import org.apache.metron.performance.sampler.UnbiasedSampler;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+public class LoadGenerator
+{
+  public static String CONSUMER_GROUP = "load.group";
+  public static long SEND_PERIOD_MS = 100;
+  public static long MONITOR_PERIOD_MS = 1000*10;
+  private static ExecutorService pool;
+  private static ThreadLocal kafkaProducer;
+  public static AtomicLong numSent = new AtomicLong(0);
+
+  public static void main( String[] args ) throws Exception {
+CommandLine cli = LoadOptions.parse(new PosixParser(), args);
+EnumMap evaluatedArgs = 
LoadOptions.createConfig(cli);
+Map kafkaConfig = new HashMap<>();
+kafkaConfig.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
--- End diff --

Kafka exposes these constants, rather than having to have them hardcoded.

ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG
ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG
ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG
ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395841#comment-16395841
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173920926
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadOptions.java
 ---
@@ -0,0 +1,504 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+import com.google.common.base.Joiner;
+import org.apache.commons.cli.*;
+import org.apache.metron.common.utils.JSONUtils;
+import org.apache.metron.common.utils.cli.CLIOptions;
+import org.apache.metron.performance.sampler.BiasedSampler;
+import org.apache.metron.stellar.common.utils.ConversionUtils;
+import org.apache.metron.common.utils.cli.OptionHandler;
+
+import javax.annotation.Nullable;
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileNotFoundException;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Optional;
+
+public enum LoadOptions implements CLIOptions {
+  HELP(new OptionHandler() {
+
+@Override
+public String getShortCode() {
+  return "h";
+}
+
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  return new Option(s, "help", false, "Generate Help screen");
+}
+  }),
+  ZK(new OptionHandler() {
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  Option o = new Option(s, "zk_quorum", true, "zookeeper quorum");
+  o.setArgName("QUORUM");
+  o.setRequired(false);
+  return o;
+}
+
+@Override
+public Optional getValue(LoadOptions option, CommandLine cli) {
+  if(option.has(cli)) {
+return Optional.ofNullable(option.get(cli));
+  }
+  else {
+return Optional.empty();
+  }
+}
+
+@Override
+public String getShortCode() {
+  return "z";
+}
+  }),
+  CONSUMER_GROUP(new OptionHandler() {
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  Option o = new Option(s, "consumer_group", true, "Consumer Group.  
The default is " + LoadGenerator.CONSUMER_GROUP);
+  o.setArgName("GROUP_ID");
+  o.setRequired(false);
+  return o;
+}
+
+@Override
+public Optional getValue(LoadOptions option, CommandLine cli) {
+  if(option.has(cli)) {
+return Optional.ofNullable(option.get(cli));
+  }
+  else {
+return Optional.of(LoadGenerator.CONSUMER_GROUP);
+  }
+}
+
+@Override
+public String getShortCode() {
+  return "cg";
+}
+  }),
+  BIASED_SAMPLE(new OptionHandler() {
+@Nullable
+@Override
+public Option apply(@Nullable String s) {
+  Option o = new Option(s, "sample_bias", true, "The discrete 
distribution to bias the sampling. " +
+  "This is a CSV of 2 columns.  The first column is the % of 
the templates " +
+  "and the 2nd column is the probability (0-100) that it's 
chosen.  For instance:\n" +
+  "  20,80\n" +
+  "  80,20\n" +
+  "implies that 20% of the templates will comprise 80% of the 
output and the remaining 80% of the templates will comprise 20% of the 
output.");
+  o.setArgName("BIAS_FILE");
+  o.setRequired(false);
+  return o;
+}
+
+@Override
+public Optional getValue(LoadOptions option, CommandLine cli) {
+  if(!option.has(cli)) {
+return Optional.empty();

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395853#comment-16395853
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173936212
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/sampler/BiasedSampler.java
 ---
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.sampler;
+
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.FileReader;
+import java.io.IOException;
+import java.util.AbstractMap;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Random;
+import java.util.TreeMap;
+
+public class BiasedSampler implements Sampler {
+  TreeMap> discreteDistribution;
+  public BiasedSampler(List>  
discreteDistribution, int max) {
+this.discreteDistribution = createDistribution(discreteDistribution, 
max);
+  }
+
+  public static List> readDistribution(File 
distrFile) throws IOException {
+List> ret = new ArrayList<>();
+System.out.println("Using biased sampler with the following biases:");
+try(BufferedReader br = new BufferedReader(new FileReader(distrFile))) 
{
+  int sumLeft = 0;
+  int sumRight = 0;
+  for(String line = null;(line = br.readLine()) != null;) {
+if(line.startsWith("#")) {
+  continue;
+}
+Iterable it = Splitter.on(",").split(line.trim());
+int left = Integer.parseInt(Iterables.getFirst(it, null));
+int right = Integer.parseInt(Iterables.getLast(it, null));
+System.out.println("\t" + left + "% of templates will comprise 
roughly " + right + "% of sample output");
+ret.add(new AbstractMap.SimpleEntry<>(left, right));
+sumLeft += left;
+sumRight += right;
+  }
+  if(sumLeft > 100 || sumRight > 100 ) {
+throw new IllegalStateException("Neither columns must sum to 
beyond 100.  " +
+"The first column is the % of templates. " +
+"The second column is the % of the sample that % of 
template occupies.");
+  }
+  else if(sumLeft < 100 && sumRight < 100) {
+int left = 100 - sumLeft;
+int right = 100 - sumRight;
+System.out.println("\t" + left + "% of templates will comprise 
roughly " + right + "% of sample output");
+ret.add(new AbstractMap.SimpleEntry<>(left, right));
+  }
+  return ret;
+}
+  }
+
+  private static TreeMap>
+ createDistribution(List>  
discreteDistribution, int max) {
+TreeMap> ret = new TreeMap<>();
+int from = 0;
+double weight = 0.0d;
+for(Map.Entry kv : discreteDistribution) {
+  double pctVals = kv.getKey()/100.0;
+  int to = from + (int)(max*pctVals);
+  double pctWeight = kv.getValue()/100.0;
+  ret.put(weight, new AbstractMap.SimpleEntry<>(from, to));
+  weight += pctWeight;
+  from = to;
+}
+return ret;
+  }
+
+  @Override
+  public int sample(Random rng, int limit) {
+double weight = rng.nextDouble();
+Map.Entry range = 
discreteDistribution.floorEntry(weight).getValue();
+return rng.nextInt(range.getValue() - range.getKey()) + range.getKey();
--- End diff --

Jerks like me who provide negatives or zeroes cause ugly 

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395845#comment-16395845
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173918976
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+
+import com.google.common.base.Joiner;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.PosixParser;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.metron.performance.load.monitor.AbstractMonitor;
+import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor;
+import 
org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor;
+import org.apache.metron.performance.load.monitor.MonitorTask;
+import org.apache.metron.performance.load.monitor.writers.CSVWriter;
+import org.apache.metron.performance.load.monitor.writers.ConsoleWriter;
+import org.apache.metron.performance.load.monitor.writers.Writable;
+import org.apache.metron.performance.load.monitor.writers.Writer;
+import org.apache.metron.performance.sampler.BiasedSampler;
+import org.apache.metron.performance.sampler.Sampler;
+import org.apache.metron.performance.sampler.UnbiasedSampler;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+public class LoadGenerator
+{
+  public static String CONSUMER_GROUP = "load.group";
+  public static long SEND_PERIOD_MS = 100;
+  public static long MONITOR_PERIOD_MS = 1000*10;
+  private static ExecutorService pool;
+  private static ThreadLocal kafkaProducer;
+  public static AtomicLong numSent = new AtomicLong(0);
+
+  public static void main( String[] args ) throws Exception {
+CommandLine cli = LoadOptions.parse(new PosixParser(), args);
+EnumMap evaluatedArgs = 
LoadOptions.createConfig(cli);
+Map kafkaConfig = new HashMap<>();
+kafkaConfig.put("key.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
+kafkaConfig.put("value.serializer", 
"org.apache.kafka.common.serialization.StringSerializer");
+kafkaConfig.put("key.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
+kafkaConfig.put("value.deserializer", 
"org.apache.kafka.common.serialization.StringDeserializer");
+if(LoadOptions.ZK.has(cli)) {
+  String zkQuorum = (String) evaluatedArgs.get(LoadOptions.ZK).get();
+  kafkaConfig.put("bootstrap.servers", 
Joiner.on(",").join(KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum)));
+}
+String groupId = 
evaluatedArgs.get(LoadOptions.CONSUMER_GROUP).get().toString();
+System.out.println("Consumer Group: " + groupId);
+kafkaConfig.put("group.id", groupId);
--- End diff --

Same as before: ConsumerGroup.GROUP_ID_CONFIG


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load 

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395842#comment-16395842
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173917605
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/LoadGenerator.java
 ---
@@ -0,0 +1,165 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load;
+
+
+import com.google.common.base.Joiner;
+import org.apache.commons.cli.CommandLine;
+import org.apache.commons.cli.PosixParser;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.metron.performance.load.monitor.AbstractMonitor;
+import org.apache.metron.performance.load.monitor.EPSGeneratedMonitor;
+import 
org.apache.metron.performance.load.monitor.EPSThroughputWrittenMonitor;
+import org.apache.metron.performance.load.monitor.MonitorTask;
+import org.apache.metron.performance.load.monitor.writers.CSVWriter;
+import org.apache.metron.performance.load.monitor.writers.ConsoleWriter;
+import org.apache.metron.performance.load.monitor.writers.Writable;
+import org.apache.metron.performance.load.monitor.writers.Writer;
+import org.apache.metron.performance.sampler.BiasedSampler;
+import org.apache.metron.performance.sampler.Sampler;
+import org.apache.metron.performance.sampler.UnbiasedSampler;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.EnumMap;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Timer;
+import java.util.TimerTask;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.function.Consumer;
+
+public class LoadGenerator
+{
+  public static String CONSUMER_GROUP = "load.group";
--- End diff --

Should this maybe be "metron.load.group" by default?  This is a little 
generic, and even though it's not terribly likely, could collide with someone 
else on the Kafka cluster.


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1483) Create a tool to monitor performance of the topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395848#comment-16395848
 ] 

ASF GitHub Bot commented on METRON-1483:


Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/958#discussion_r173926217
  
--- Diff: 
metron-contrib/metron-performance/src/main/java/org/apache/metron/performance/load/monitor/AbstractMonitor.java
 ---
@@ -0,0 +1,49 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.performance.load.monitor;
+
+import java.util.Optional;
+import java.util.function.Supplier;
+
+public abstract class AbstractMonitor implements Supplier, 
MonitorNaming {
+  private static final double EPSILON = 1e-6;
+  protected Optional kafkaTopic;
+  protected long timestampPrevious = 0;
+  public AbstractMonitor(Optional kafkaTopic) {
+this.kafkaTopic = kafkaTopic;
+  }
+
+  protected abstract Long monitor(double deltaTs);
+
+  @Override
+  public Long get() {
+long timeStarted = System.currentTimeMillis();
--- End diff --

I'd probably use `System.nanoTime` here (if it's not going to cause perf 
issues).  `System.currentTimeMillis` is, iirc, subject entirely to the system 
clock, which can be hit with daylight savings time and such.


> Create a tool to monitor performance of the topologies
> --
>
> Key: METRON-1483
> URL: https://issues.apache.org/jira/browse/METRON-1483
> Project: Metron
>  Issue Type: New Feature
>Reporter: Casey Stella
>Priority: Major
>
> In performance evaluation, generating synthetic load and monitoring the write 
> throughput of our kafka-to-kafka topologies has required a lot of custom 
> scripting.  We should have a tool that could do the following:
>  * Take a file representing a message template and generate synthetic load at 
> a given events per second
>  * Monitor the kafka offsets of a topic and report throughput numbers in 
> events per second
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1477) Remove jquerry from the the metron project

[James Sirota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Sirota reassigned METRON-1477:


Assignee: Daniel Toth

> Remove jquerry from the the metron project
> --
>
> Key: METRON-1477
> URL: https://issues.apache.org/jira/browse/METRON-1477
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Minor
>
> Remove jquerry from the project. Jquerry is not needed for angular projects.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1479) Add editorconfig - create guideline for code formatting in the angular part of the app

[James Sirota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Sirota reassigned METRON-1479:


Assignee: Daniel Toth

> Add editorconfig - create guideline for code formatting in the angular part 
> of the app
> --
>
> Key: METRON-1479
> URL: https://issues.apache.org/jira/browse/METRON-1479
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1478) Move type definitions to dev-dependencies

[James Sirota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Sirota reassigned METRON-1478:


Assignee: Daniel Toth

> Move type definitions to dev-dependencies
> -
>
> Key: METRON-1478
> URL: https://issues.apache.org/jira/browse/METRON-1478
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1476) Update angular

[James Sirota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Sirota reassigned METRON-1476:


Assignee: Daniel Toth

> Update angular
> --
>
> Key: METRON-1476
> URL: https://issues.apache.org/jira/browse/METRON-1476
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>
> Update angular to speed up development



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1480) Add yarn as default build tool for the frontend

[James Sirota (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Sirota reassigned METRON-1480:


Assignee: Daniel Toth

> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1487) Define Performance Benchmarks for Enrichment Topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395795#comment-16395795
 ] 

ASF GitHub Bot commented on METRON-1487:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/961#discussion_r173928377
  
--- Diff: metron-platform/metron-enrichment/Performance.md ---
@@ -0,0 +1,522 @@
+
+
+# Enrichment Performance
+
+This guide defines a set of benchmarks used to measure the performance of 
the Enrichment topology.  The guide also provides detailed steps on how to 
execute those benchmarks along with advice for tuning the Enrichment topology.
+
+* [Benchmarks](#benchmarks)
+* [Benchmark Execution](#benchmark-execution)
+* [Performance Tuning](#performance-tuning)
+* [Benchmark Results](#benchmark-results)
+
+## Benchmarks
+
+* [Geo IP Enrichment](#geo-ip-enrichment)
+* [HBase Enrichment](#hbase-enrichment)
+* [Stellar Enrichment](#stellar-enrichment)
+
+### Geo IP Enrichment
+
+This benchmark measures the performance of executing a Geo IP enrichment.  
Given a valid IP address the enrichment will append detailed location 
information for that IP.  The location information is sourced from an external 
Geo IP data source like [Maxmind](https://github.com/maxmind/GeoIP2-java). 
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a Geo IP enrichment.
+```
+geo := GEO_GET(ip_dst_addr)
+```
+
+After the enrichment process completes, the  telemetry message will 
contain a set of fields with location information for the given IP address.
+```
+{
+   "ip_dst_addr":"151.101.129.140",
+   ...
+   "geo.city":"San Francisco",
+   "geo.country":"US",
+   "geo.dmaCode":"807",
+   "geo.latitude":"37.7697",
+   "geo.location_point":"37.7697,-122.3933",
+   "geo.locID":"5391959",
+   "geo.longitude":"-122.3933",
+   "geo.postalCode":"94107",
+ }
+```
+
+### HBase Enrichment
+
+This benchmark measures the performance of executing an enrichment that 
retrieves data from an external HBase table. This type of enrichment is useful 
for enriching telemetry from an Asset Database or other source of relatively 
static data.
+
+ Configuration
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define an Hbase enrichment.  This looks up the 'ip_dst_addr' 
within an HBase table 'top-1m' and returns a hostname.
+```
+top1m := ENRICHMENT_GET('top-1m', ip_dst_addr, 'top-1m', 't')
+```
+
+After the telemetry has been enriched, it will contain the host and IP 
elements that were retrieved from the HBase table.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "top1m.host":"earther.com",
+   "top1m.ip":"151.101.2.166"
+}
+```
+
+### Stellar Enrichment
+
+This benchmark measures the performance of executing a basic Stellar 
expression.  In this benchmark, the enrichment is purely a computational task 
that has no dependence on an external system like a database.  
+
+ Configuration 
+
+Adding the following Stellar expression to the Enrichment topology 
configuration will define a basic Stellar enrichment.  The following returns 
true if the IP is in the given subnet and false otherwise. 
+```
+local := IN_SUBNET(ip_dst_addr, '192.168.0.0/24')
+```
+
+After the telemetry has been enriched, it will contain a field with a 
boolean value indicating whether the IP was within the given subnet.
+```
+{
+   "ip_dst_addr":"151.101.2.166",
+   ...
+   "local":false
+}
+```
+   
+## Benchmark Execution
+
+* [Prepare Enrichment Data](#prepare-enrichment-data)
+* [Load HBase with Enrichment Data](#load-hbase-with-enrichment-data)
+* [Configure the Enrichments](#configure-the-enrichments)
+* [Create Input Telemetry](#create-input-telemetry)
+* [Cluster Setup](#cluster-setup)
+* [Monitoring](#monitoring)
+
+### Prepare Enrichment Data
+
+The Alexa Top 1 Million was used as an data source for these benchmarks.
+
+1. Download the [Alexa Top 1 
Million](http://s3.amazonaws.com/alexa-static/top-1m.csv.zip).
+
+2. For each hostname, query DNS to retrieve an associated IP address.  
+
+   A script like the following can be used for this.  There is no need to 
do this for all 1 million entries in the data set. Doing this for around 10,000 
records is sufficient.
+
+   ```python
+   import dns.resolver
+   import csv
+
+   resolver = dns.resolver.Resolver()
+   resolver.nameservers = ['8.8.8.8', '8.8.4.4']
+
+   with 

[jira] [Commented] (METRON-1488) user_settings hbase table does not have acls set up for kerberos

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395788#comment-16395788
 ] 

ASF GitHub Bot commented on METRON-1488:


GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/962

METRON-1488: user_settings hbase table does not have acls set up for 
kerberos

## Contributor Comments
Currently some REST calls will fail because we do not set ACL's on the new 
user_settings table, which is new.

To exercise this, kerberize full-dev (or whatever cluster you please) and 
go to swagger and try to do a search via `api/v1/search/search` with the 
following arg:
```
{
 "indices": [],
 "facetFields":[],
 "query": "*",
 "from": 0,
 "size": 20
}
```
This should succeed now.  Before we got the following exception:
```
{
  "responseCode": 500,
  "message": "org.apache.hadoop.hbase.security.AccessDeniedException: 
Insufficient permissions for user 'metron' (table=user_settings, 
action=READ)\n\tat 
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1616)\n\tat
 
org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1624)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:816)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:812)\n\tat
 org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6917)\n\tat 
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6905)\n\tat 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat
 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)\n\tat
 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)\n\tat 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)\n",
  "fullMessage": "RemoteWithExtrasException: 
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
permissions for user 'metron' (table=user_settings, action=READ)\n\tat 
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1616)\n\tat
 
org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1624)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:816)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:812)\n\tat
 org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6917)\n\tat 
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6905)\n\tat 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat
 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)\n\tat
 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)\n\tat 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)\n"
}
```


## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 

[jira] [Created] (METRON-1488) user_settings hbase table does not have acls set up for kerberos

[Casey Stella (JIRA)]

Casey Stella created METRON-1488:


 Summary: user_settings hbase table does not have acls set up for 
kerberos
 Key: METRON-1488
 URL: https://issues.apache.org/jira/browse/METRON-1488
 Project: Metron
  Issue Type: Improvement
Reporter: Casey Stella


Currently some REST calls will fail because we do not set ACL's on the new 
user_settings table, which is new.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1487) Define Performance Benchmarks for Enrichment Topology

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395513#comment-16395513
 ] 

ASF GitHub Bot commented on METRON-1487:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/961

METRON-1487 Define Performance Benchmarks for Enrichment Topology

I created a markdown document that defines a set of performance benchmarks 
for the Enrichment topology.  These benchmarks should be repeatable to help 
detect performance regressions that might occur over time.

This PR creates a new markdown document under 
`metron-platform/metron-enrichment` that does the following.
(1) Defines performance benchmarks for the Enrichment topology
(2) Describes how the benchmarks can be executed
(3) Describes how to tune the topology when executing the benchmarks 
(4) Describes actual benchmark results and tuned parameters

## Pull Request Checklist

- [ ] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [ ] Has your PR been rebased against the latest commit within the target 
branch (typically master)?
- [ ] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/metron METRON-1487

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/961.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #961


commit 706b06c01d0109fe4b1ce4e332f736864413435f
Author: Nick Allen 
Date:   2018-03-12T16:58:53Z

METRON-1487 Define Performance Benchmarks for Enrichment Topology




> Define Performance Benchmarks for Enrichment Topology
> -
>
> Key: METRON-1487
> URL: https://issues.apache.org/jira/browse/METRON-1487
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Define a set of performance benchmarks for the Enrichment topology.  These 
> benchmarks should be repeatable to help detect performance regressions that 
> might occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (METRON-1487) Define Performance Benchmarks for Enrichment Topology

[Nick Allen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/METRON-1487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen reassigned METRON-1487:
--

Assignee: Nick Allen

> Define Performance Benchmarks for Enrichment Topology
> -
>
> Key: METRON-1487
> URL: https://issues.apache.org/jira/browse/METRON-1487
> Project: Metron
>  Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Define a set of performance benchmarks for the Enrichment topology.  These 
> benchmarks should be repeatable to help detect performance regressions that 
> might occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1487) Define Performance Benchmarks for Enrichment Topology

[Nick Allen (JIRA)]

Nick Allen created METRON-1487:
--

 Summary: Define Performance Benchmarks for Enrichment Topology
 Key: METRON-1487
 URL: https://issues.apache.org/jira/browse/METRON-1487
 Project: Metron
  Issue Type: Improvement
Reporter: Nick Allen


Define a set of performance benchmarks for the Enrichment topology.  These 
benchmarks should be repeatable to help detect performance regressions that 
might occur.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1484) Indexing Topologies Fail Partial Start in Ambari

[Anand Subramanian (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16395310#comment-16395310
 ] 

Anand Subramanian commented on METRON-1484:
---

Are these errors in METRON-1451 seen in the ambar-agent.log?

> Indexing Topologies Fail Partial Start in Ambari
> 
>
> Key: METRON-1484
> URL: https://issues.apache.org/jira/browse/METRON-1484
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Priority: Major
>
> If either topology is down, Ambari shows all of Indexing as dead. Clicking 
> start attempts to start them both and fails if either is still running. 
> Furthermore, it appears to retry 3 times before finally failing the command.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1486) Search using post ' "/api/v1/search/search" Throws unauthorized exception

[Mohan (JIRA)]

Mohan created METRON-1486:
-

 Summary:  Search using post ' "/api/v1/search/search" Throws 
unauthorized exception
 Key: METRON-1486
 URL: https://issues.apache.org/jira/browse/METRON-1486
 Project: Metron
  Issue Type: Bug
Reporter: Mohan


Search Controller API "/api/v1/search/search" Throws unauthorized exception, 
for the below payload

{code:java}
{
 "indices": [],
 "facetFields":[],
 "query": "*",
 "from": 0,
 "size": 20
 }
{code}

Exception is as follows 

{code:java}
{
  "responseCode": 500,
  "message": "org.apache.hadoop.hbase.security.AccessDeniedException: 
Insufficient permissions for user 'metron' (table=user_settings, 
action=READ)\n\tat 
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1616)\n\tat
 
org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1624)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:816)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:812)\n\tat
 org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6917)\n\tat 
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6905)\n\tat 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat
 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)\n\tat
 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)\n\tat 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)\n",
  "fullMessage": "RemoteWithExtrasException: 
org.apache.hadoop.hbase.security.AccessDeniedException: Insufficient 
permissions for user 'metron' (table=user_settings, action=READ)\n\tat 
org.apache.hadoop.hbase.security.access.AccessController.internalPreRead(AccessController.java:1616)\n\tat
 
org.apache.hadoop.hbase.security.access.AccessController.preGetOp(AccessController.java:1624)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$26.call(RegionCoprocessorHost.java:816)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1660)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1734)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1692)\n\tat
 
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.preGet(RegionCoprocessorHost.java:812)\n\tat
 org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6917)\n\tat 
org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6905)\n\tat 
org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2026)\n\tat
 
org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:32381)\n\tat
 org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2150)\n\tat 
org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:187)\n\tat 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:167)\n"
}
{code}





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394908#comment-16394908
 ] 

ASF GitHub Bot commented on METRON-1480:


Github user xyztdanid4 commented on a diff in the pull request:

https://github.com/apache/metron/pull/952#discussion_r173710125
  
--- Diff: metron-interface/metron-alerts/pom.xml ---
@@ -35,36 +35,36 @@
 
 com.github.eirslett
 frontend-maven-plugin
-1.3
+1.6
 
 ./
 ${node.version}
-${npm.version}
-
false
+${yarn.version}
+
false
 
 
 
 generate-resources
-install node and npm
+install node and yarn
 
-install-node-and-npm
+install-node-and-yarn
 
 
 
 generate-resources
-npm install
+yarn install
 
-npm
+yarn
 
 
-install
+ install --mutex network 
--ignore-engines --pure-lockfile
--- End diff --

@merrimanr I would like to thank you for your help with pointing out the 
problem is with the CI. I tried to set the cache folder differently for each 
angular project, but somehow I got some permission errors. The solution is that 
I introduced the 'mutex network' flag on the yarn install, so it will prevent 
multiple installation on the same time by yarn.


> Add yarn as default build tool for the frontend
> ---
>
> Key: METRON-1480
> URL: https://issues.apache.org/jira/browse/METRON-1480
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)