date:20181019



[ 
https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657624#comment-16657624
 ] 

ASF GitHub Bot commented on METRON-1831:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1243
  
Transient integration test failure?
```
Tests in error: 
  ElasticsearchIndexingIntegrationTest>IndexingIntegrationTest.test:137 Â» 
Runtime
```


> Project Version Substitution Not Working
> 
>
> Key: METRON-1831
> URL: https://issues.apache.org/jira/browse/METRON-1831
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> Steps to Replicate:
>  # Spin-up development environment.
>  # Launch REPL.
>  # Notice the warning:
> {code:java}
> bin/stellar: line 36: ${project.version}: bad substitution{code}
>  
> {code:java}
> [root@node1 ~]# source /etc/default/metron
> [root@node1 ~]# cd $METRON_HOME
> [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
> bin/stellar: line 36: ${project.version}: bad substitution
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Stellar, Go!
> Functions are loading lazily in the background and will be unavailable until 
> loaded fully.
> {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
> parser.error.topic=indexing, update.hbase.table=metron_update, 
> update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
> es.document.id=, profiler.client.period.duration=15, 
> profiler.client.period.duration.units=MINUTES, 
> user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
> bootstrap.servers=node1:6667, source.type.field=source:type, 
> threat.triage.score.field=threat:triage:score, 
> enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, 
> profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, 
> geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code}
>  
> This only seems to be a problem with the $METRON_HOME/bin/stellar script.
> {code:java}
> [root@node1 0.6.1]# grep project.version bin/*
> bin/stellar:export 
> METRON_VERSION="${METRON_VERSION:-${project.version}}"{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1243: METRON-1831 Project Version Substitution Not Work...

GitHub user nickwallen reopened a pull request:

https://github.com/apache/metron/pull/1243

METRON-1831 Project Version Substitution Not Working

When running the stellar REPL script, a warning is reported.
```
bin/stellar: line 36: ${project.version}: bad substitution
```
For example...
```
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# cd $METRON_HOME
[root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
bin/stellar: line 36: ${project.version}: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
parser.error.topic=indexing, update.hbase.table=metron_update, 
update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
es.document.id=, profiler.client.period.duration=15, 
profiler.client.period.duration.units=MINUTES, 
user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
bootstrap.servers=node1:6667, source.type.field=source:type, 
threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, 
enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, 
profiler.writer.batchTimeout=0, 
geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}
```
## Problem

In the stellar REPL launch script, the ${project.version} is not getting 
substituted at build time, perhaps because it is wrapped inside another ${...}. 
 This is probably why it is an issue in the stellar script and not the other 
scripts.

## Change 

* The stellar REPL launch script was changed in #1232 , but I did not see 
this warning occur when testing that change.  
* Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` 
automatically. There really is no need to set `METRON_HOME` and 
`METRON_VERSION` in the script any longer.  It is cleaner to just remove this 
(which I almost did in #1232).  
* I use a `set -u` so the script will error if `METRON_HOME` is not defined 
by some odd happenstance.

## Testing

1. Stand-up a development environment.

1. Launch the Stellar REPL.  Ensure there is no warning reported.
```
[vagrant@node1 ~]$ sudo su -
[root@node1 ~]# cd /usr/metron/0.6.1/
[root@node1 0.6.1]# bin/stellar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{}
[Stellar]>>>
```
1. Ensure the standard suite of Stellar functions are discovered.

```
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, BYTEARRAY_MATCHER, CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, 
COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, 
ENRICHMENT_STELLAR_TRANSFORM_ADD, ENRICHMENT_STELLAR_TRANSFORM_PRINT, 
ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, 
GEOHASH_FROM_LATLONG, GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, 
GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE, GET_LAST, 
GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, 
HDFS_READ_LINES, HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, 
HLLP_MERGE, INDEXING_SET_BATCH, INDEXING_SET_ENABLED, INDEXING_SET_INDEX, 
IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_E
 NCODING, IS_INTEGER, IS_IP, IS_NAN, IS_URL, IT_ENTROPY, JOIN, KAFKA_FIND, 
KAFKA_GET, KAFKA_PROPS, KAFKA_PUT, KAFKA_SEEK, KAFKA_TAIL, LENGTH, LIST_ADD, 
LN, LOCAL_LS, LOCAL_READ, LOCAL_READ_LINES, LOCAL_RM, LOCAL_WRITE, LOG10, LOG2, 
MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MAX, MIN, MONTH,

[GitHub] metron issue #1243: METRON-1831 Project Version Substitution Not Working

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1243
  
Transient integration test failure?
```
Tests in error: 
  ElasticsearchIndexingIntegrationTest>IndexingIntegrationTest.test:137 
ÃÂ» Runtime
```


---

[jira] [Commented] (METRON-1831) Project Version Substitution Not Working



[ 
https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657625#comment-16657625
 ] 

ASF GitHub Bot commented on METRON-1831:


Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1243


> Project Version Substitution Not Working
> 
>
> Key: METRON-1831
> URL: https://issues.apache.org/jira/browse/METRON-1831
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> Steps to Replicate:
>  # Spin-up development environment.
>  # Launch REPL.
>  # Notice the warning:
> {code:java}
> bin/stellar: line 36: ${project.version}: bad substitution{code}
>  
> {code:java}
> [root@node1 ~]# source /etc/default/metron
> [root@node1 ~]# cd $METRON_HOME
> [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
> bin/stellar: line 36: ${project.version}: bad substitution
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Stellar, Go!
> Functions are loading lazily in the background and will be unavailable until 
> loaded fully.
> {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
> parser.error.topic=indexing, update.hbase.table=metron_update, 
> update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
> es.document.id=, profiler.client.period.duration=15, 
> profiler.client.period.duration.units=MINUTES, 
> user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
> bootstrap.servers=node1:6667, source.type.field=source:type, 
> threat.triage.score.field=threat:triage:score, 
> enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, 
> profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, 
> geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code}
>  
> This only seems to be a problem with the $METRON_HOME/bin/stellar script.
> {code:java}
> [root@node1 0.6.1]# grep project.version bin/*
> bin/stellar:export 
> METRON_VERSION="${METRON_VERSION:-${project.version}}"{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1243: METRON-1831 Project Version Substitution Not Work...

Github user nickwallen closed the pull request at:

https://github.com/apache/metron/pull/1243


---

[jira] [Commented] (METRON-1831) Project Version Substitution Not Working



[ 
https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657626#comment-16657626
 ] 

ASF GitHub Bot commented on METRON-1831:


GitHub user nickwallen reopened a pull request:

https://github.com/apache/metron/pull/1243

METRON-1831 Project Version Substitution Not Working

When running the stellar REPL script, a warning is reported.
```
bin/stellar: line 36: ${project.version}: bad substitution
```
For example...
```
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# cd $METRON_HOME
[root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
bin/stellar: line 36: ${project.version}: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
parser.error.topic=indexing, update.hbase.table=metron_update, 
update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
es.document.id=, profiler.client.period.duration=15, 
profiler.client.period.duration.units=MINUTES, 
user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
bootstrap.servers=node1:6667, source.type.field=source:type, 
threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, 
enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, 
profiler.writer.batchTimeout=0, 
geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}
```
## Problem

In the stellar REPL launch script, the ${project.version} is not getting 
substituted at build time, perhaps because it is wrapped inside another ${...}. 
 This is probably why it is an issue in the stellar script and not the other 
scripts.

## Change 

* The stellar REPL launch script was changed in #1232 , but I did not see 
this warning occur when testing that change.  
* Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` 
automatically. There really is no need to set `METRON_HOME` and 
`METRON_VERSION` in the script any longer.  It is cleaner to just remove this 
(which I almost did in #1232).  
* I use a `set -u` so the script will error if `METRON_HOME` is not defined 
by some odd happenstance.

## Testing

1. Stand-up a development environment.

1. Launch the Stellar REPL.  Ensure there is no warning reported.
```
[vagrant@node1 ~]$ sudo su -
[root@node1 ~]# cd /usr/metron/0.6.1/
[root@node1 0.6.1]# bin/stellar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{}
[Stellar]>>>
```
1. Ensure the standard suite of Stellar functions are discovered.

```
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, BYTEARRAY_MATCHER, CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, 
COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, 
ENRICHMENT_STELLAR_TRANSFORM_ADD, ENRICHMENT_STELLAR_TRANSFORM_PRINT, 
ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, 
GEOHASH_FROM_LATLONG, GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, 
GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE, GET_LAST, 
GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, 
HDFS_READ_LINES, HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, 
HLLP_MERGE, INDEXING_SET_BATCH, INDEXING_SET_ENABLED, INDEXING_SET_INDEX, 
IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_ENCODING, IS_INTEGER, 
IS_IP, IS_NAN, IS_URL, IT_ENTROPY,

[jira] [Commented] (METRON-1831) Project Version Substitution Not Working



[ 
https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657584#comment-16657584
 ] 

ASF GitHub Bot commented on METRON-1831:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1243
  
+1 by inspection @nickwallen, thanks for the fix.


> Project Version Substitution Not Working
> 
>
> Key: METRON-1831
> URL: https://issues.apache.org/jira/browse/METRON-1831
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
>
> Steps to Replicate:
>  # Spin-up development environment.
>  # Launch REPL.
>  # Notice the warning:
> {code:java}
> bin/stellar: line 36: ${project.version}: bad substitution{code}
>  
> {code:java}
> [root@node1 ~]# source /etc/default/metron
> [root@node1 ~]# cd $METRON_HOME
> [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
> bin/stellar: line 36: ${project.version}: bad substitution
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Stellar, Go!
> Functions are loading lazily in the background and will be unavailable until 
> loaded fully.
> {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
> parser.error.topic=indexing, update.hbase.table=metron_update, 
> update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
> es.document.id=, profiler.client.period.duration=15, 
> profiler.client.period.duration.units=MINUTES, 
> user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
> bootstrap.servers=node1:6667, source.type.field=source:type, 
> threat.triage.score.field=threat:triage:score, 
> enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, 
> profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, 
> geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code}
>  
> This only seems to be a problem with the $METRON_HOME/bin/stellar script.
> {code:java}
> [root@node1 0.6.1]# grep project.version bin/*
> bin/stellar:export 
> METRON_VERSION="${METRON_VERSION:-${project.version}}"{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1243: METRON-1831 Project Version Substitution Not Working

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1243
  
+1 by inspection @nickwallen, thanks for the fix.


---

[jira] [Commented] (METRON-1831) Project Version Substitution Not Working



[ 
https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657516#comment-16657516
 ] 

ASF GitHub Bot commented on METRON-1831:


GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/1243

METRON-1831 Project Version Substitution Not Working

When running the stellar REPL script, a warning is reported.
```
bin/stellar: line 36: ${project.version}: bad substitution
```
For example...
```
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# cd $METRON_HOME
[root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
bin/stellar: line 36: ${project.version}: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
parser.error.topic=indexing, update.hbase.table=metron_update, 
update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
es.document.id=, profiler.client.period.duration=15, 
profiler.client.period.duration.units=MINUTES, 
user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
bootstrap.servers=node1:6667, source.type.field=source:type, 
threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, 
enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, 
profiler.writer.batchTimeout=0, 
geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}
```
## Problem

In the stellar REPL launch script, the ${project.version} is not getting 
substituted at build time, perhaps because it is wrapped inside another ${...}. 
 This is probably why it is an issue in the stellar script and not the other 
scripts.

## Change 

* The stellar REPL launch script was changed in #1232 , but I did not see 
this warning occur when testing that change.  
* Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` 
automatically. There really is no need to set `METRON_HOME` and 
`METRON_VERSION` in the script any longer.  It is cleaner to just remove this 
(which I almost did in #1232).  
* I use a `set -u` so the script will error if `METRON_HOME` is not defined 
by some odd happenstance.

## Testing

1. Stand-up a development environment.

1. Launch the Stellar REPL.  Ensure there is no warning reported.
```
[vagrant@node1 ~]$ sudo su -
[root@node1 ~]# cd /usr/metron/0.6.1/
[root@node1 0.6.1]# bin/stellar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{}
[Stellar]>>>
```
1. Ensure the standard suite of Stellar functions are discovered.

```
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, BYTEARRAY_MATCHER, CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, 
COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, 
ENRICHMENT_STELLAR_TRANSFORM_ADD, ENRICHMENT_STELLAR_TRANSFORM_PRINT, 
ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, 
GEOHASH_FROM_LATLONG, GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, 
GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE, GET_LAST, 
GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, 
HDFS_READ_LINES, HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, 
HLLP_MERGE, INDEXING_SET_BATCH, INDEXING_SET_ENABLED, INDEXING_SET_INDEX, 
IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_ENCODING, IS_INTEGER, 
IS_IP, IS_NAN, IS_URL, IT_ENTROPY, JO

[GitHub] metron pull request #1243: METRON-1831 Project Version Substitution Not Work...

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/1243

METRON-1831 Project Version Substitution Not Working

When running the stellar REPL script, a warning is reported.
```
bin/stellar: line 36: ${project.version}: bad substitution
```
For example...
```
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# cd $METRON_HOME
[root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
bin/stellar: line 36: ${project.version}: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
parser.error.topic=indexing, update.hbase.table=metron_update, 
update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
es.document.id=, profiler.client.period.duration=15, 
profiler.client.period.duration.units=MINUTES, 
user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
bootstrap.servers=node1:6667, source.type.field=source:type, 
threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, 
enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, 
profiler.writer.batchTimeout=0, 
geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}
```
## Problem

In the stellar REPL launch script, the ${project.version} is not getting 
substituted at build time, perhaps because it is wrapped inside another ${...}. 
 This is probably why it is an issue in the stellar script and not the other 
scripts.

## Change 

* The stellar REPL launch script was changed in #1232 , but I did not see 
this warning occur when testing that change.  
* Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` 
automatically. There really is no need to set `METRON_HOME` and 
`METRON_VERSION` in the script any longer.  It is cleaner to just remove this 
(which I almost did in #1232).  
* I use a `set -u` so the script will error if `METRON_HOME` is not defined 
by some odd happenstance.

## Testing

1. Stand-up a development environment.

1. Launch the Stellar REPL.  Ensure there is no warning reported.
```
[vagrant@node1 ~]$ sudo su -
[root@node1 ~]# cd /usr/metron/0.6.1/
[root@node1 0.6.1]# bin/stellar
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable 
until loaded fully.
{}
[Stellar]>>>
```
1. Ensure the standard suite of Stellar functions are discovered.

```
[Stellar]>>> %functions
ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, 
BLOOM_MERGE, BYTEARRAY_MATCHER, CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, 
COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, 
DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, 
DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, 
ENRICHMENT_STELLAR_TRANSFORM_ADD, ENRICHMENT_STELLAR_TRANSFORM_PRINT, 
ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, 
FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, 
GEOHASH_FROM_LATLONG, GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, 
GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE, GET_LAST, 
GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, 
HDFS_READ_LINES, HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, 
HLLP_MERGE, INDEXING_SET_BATCH, INDEXING_SET_ENABLED, INDEXING_SET_INDEX, 
IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_E
 NCODING, IS_INTEGER, IS_IP, IS_NAN, IS_URL, IT_ENTROPY, JOIN, KAFKA_FIND, 
KAFKA_GET, KAFKA_PROPS, KAFKA_PUT, KAFKA_SEEK, KAFKA_TAIL, LENGTH, LIST_ADD, 
LN, LOCAL_LS, LOCAL_READ, LOCAL_READ_LINES, LOCAL_RM, LOCAL_WRITE, LOG10, LOG2, 
MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP, MAP_EXISTS, MAP_GET, MAX, MIN, MONTH, 
MU

[jira] [Created] (METRON-1838) New client SSL settings

Michael Miklavcic created METRON-1838:
-

 Summary: New client SSL settings
 Key: METRON-1838
 URL: https://issues.apache.org/jira/browse/METRON-1838
 Project: Metron
  Issue Type: Sub-task
Reporter: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1837) Full test script

Michael Miklavcic created METRON-1837:
-

 Summary: Full test script
 Key: METRON-1837
 URL: https://issues.apache.org/jira/browse/METRON-1837
 Project: Metron
  Issue Type: Sub-task
Reporter: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1836) Finish updating README documentation including migration guide

Michael Miklavcic created METRON-1836:
-

 Summary: Finish updating README documentation including migration 
guide
 Key: METRON-1836
 URL: https://issues.apache.org/jira/browse/METRON-1836
 Project: Metron
  Issue Type: Sub-task
Reporter: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API



 [ 
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Miklavcic updated METRON-1834:
--
Description: (was: Tasks for completion)

> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1835) Performance regression testing

Michael Miklavcic created METRON-1835:
-

 Summary: Performance regression testing
 Key: METRON-1835
 URL: https://issues.apache.org/jira/browse/METRON-1835
 Project: Metron
  Issue Type: Sub-task
Reporter: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API



 [ 
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Miklavcic updated METRON-1834:
--
Description: Tasks for completion

> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>
> Tasks for completion



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API



[ 
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657428#comment-16657428
 ] 

ASF GitHub Bot commented on METRON-1834:


Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1242
  
A recent merge with https://github.com/apache/metron/pull/1218 has caused a 
number of integration test failures. Looking into it.


> Migrate Elasticsearch from TransportClient to new Java REST API
> ---
>
> Key: METRON-1834
> URL: https://issues.apache.org/jira/browse/METRON-1834
> Project: Metron
>  Issue Type: Improvement
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1242: METRON-1834: Migrate Elasticsearch from TransportClient ...

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/1242
  
A recent merge with https://github.com/apache/metron/pull/1218 has caused a 
number of integration test failures. Looking into it.


---

[jira] [Commented] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API

[
https://issues.apache.org/jira/browse/METRON-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657425#comment-16657425
]

ASF GitHub Bot commented on METRON-1834:

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/1242

METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1834

This task has been a long time coming after having completed the ES upgrade
in https://issues.apache.org/jira/browse/METRON-939. Motivation for completing
this now is that Elasticsearch will be deprecating use of the TransportClient
in v 7.x. This PR migrates the Elasticsearch client from TransportClient to the
newer Java REST API.

1.
https://www.elastic.co/guide/en/elasticsearch/client/java-api/5.6/client.html
2.
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-overview.html
3.
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-high-level-migration.html

This builds off and finishes work started by @cestella here -
https://github.com/cestella/incubator-metron/tree/es_rest_client. I condensed
his branch into 1 flattened commit and built on top of it in order to provide
attribution.

I have a number of tasks I'm still working through, but I wanted to get the
review process started. I've minimally validated X-Pack auth and will have some
follow-up for SSL. Test plans and a breakdown of the changes will be soon to
follow. For starters, full dev should continue to work as normal and you should
see data flowing into indexes for bro, snort, and yaf. There are some
additional changes to how this client will be configured, which I'll be
documenting shortly. The new client does not take a Map of settings any longer
now that it is leverage Apache HTTP Async Client
https://www.elastic.co/guide/en/elasticsearch/client/java-rest/5.6/java-rest-low-usage-dependencies.html
under the hood. This meant choosing a set of properties to expose and doing a
translation to the builder pattern under the hood. Again, I'll have a write-up
of this in the migration guide and update the README's accordingly.

NOTE: This checks off 2 items from this follow-on list
https://github.com/apache/metron/pull/840#issuecomment-347281776

1. Fix Log4j logging problem - classpath issues
2. Migrate to new ES REST client

Per discussion in the Metron Slack channel, I will be updating the Jira
ticket with a series of tasks to be completed prior to acceptance, including
performance regression testing compared with the old API.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?

### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```

- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendere

[GitHub] metron pull request #1242: METRON-1834: Migrate Elasticsearch from Transport...

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/1242

METRON-1834: Migrate Elasticsearch from TransportClient to new Java REST API

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1834

NOTE: This checks off 2 items from this follow-on list
https://github.com/apache/metron/pull/840#issuecomment-347281776

1. Fix Log4j logging problem - classpath issues
2. Migrate to new ES REST client

## Pull Request Checklist

In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:

### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in
which it is rendered by building and verifying the site-book? If not then run
the following commands and the verify changes via
`site-book/target/site/index.html`:

```
cd site-book
mvn site
```

Note:
Please ensure that once the PR

[jira] [Created] (METRON-1834) Migrate Elasticsearch from TransportClient to new Java REST API

Michael Miklavcic created METRON-1834:
-

 Summary: Migrate Elasticsearch from TransportClient to new Java 
REST API
 Key: METRON-1834
 URL: https://issues.apache.org/jira/browse/METRON-1834
 Project: Metron
  Issue Type: Improvement
Reporter: Michael Miklavcic
Assignee: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1833) Management UI incorrectly displaying sensor topology latency units as seconds instead of millis



[ 
https://issues.apache.org/jira/browse/METRON-1833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657326#comment-16657326
 ] 

ASF GitHub Bot commented on METRON-1833:


GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/1241

METRON-1833: Management UI incorrectly displaying sensor topology latency 
units as seconds instead of millis

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1833

Management UI is displaying units for topology latency as seconds instead 
of milliseconds. This value should be identical to that reported in the Storm 
UI. The change is straightforward - spin up full dev and the UI should show 
"ms" instead of "s"

## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- N/A Have you written or updated unit tests and or integration tests to 
verify your changes?
- N/A If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- N/A Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron ui-storm-latency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1241


commit 4b26720e0581721470ea39198abee6bb5fa7bea6
Author: Michael Miklavcic 
Date:   2018-10-19T19:34:49Z

Fix latency measurement.




> Management UI incorrectly displaying sensor topology latency units as seconds 
> instead of millis
> ---
>
> Key: METRON-1833
> URL: https://issues.apache.org/jira/browse/METRON-1833
> Project: Metron
>  Issue Type: Bug
>Reporter: Michael Miklavcic
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1241: METRON-1833: Management UI incorrectly displaying...

GitHub user mmiklavc opened a pull request:

https://github.com/apache/metron/pull/1241

METRON-1833: Management UI incorrectly displaying sensor topology latency 
units as seconds instead of millis

## Contributor Comments

https://issues.apache.org/jira/browse/METRON-1833

Management UI is displaying units for topology latency as seconds instead 
of milliseconds. This value should be identical to that reported in the Storm 
UI. The change is straightforward - spin up full dev and the UI should show 
"ms" instead of "s"

## Pull Request Checklist

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- N/A Have you written or updated unit tests and or integration tests to 
verify your changes?
- N/A If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- N/A Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mmiklavc/metron ui-storm-latency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/1241.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1241


commit 4b26720e0581721470ea39198abee6bb5fa7bea6
Author: Michael Miklavcic 
Date:   2018-10-19T19:34:49Z

Fix latency measurement.




---

[jira] [Created] (METRON-1833) Management UI incorrectly displaying sensor topology latency units as seconds instead of millis

Michael Miklavcic created METRON-1833:
-

 Summary: Management UI incorrectly displaying sensor topology 
latency units as seconds instead of millis
 Key: METRON-1833
 URL: https://issues.apache.org/jira/browse/METRON-1833
 Project: Metron
  Issue Type: Bug
Reporter: Michael Miklavcic






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1829) Large Error Message Causes Slow Search Performance



[ 
https://issues.apache.org/jira/browse/METRON-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657161#comment-16657161
 ] 

ASF GitHub Bot commented on METRON-1829:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1239
  
+1 Works as advertised.  Thanks for the fix


> Large Error Message Causes Slow Search Performance
> --
>
> Key: METRON-1829
> URL: https://issues.apache.org/jira/browse/METRON-1829
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
>
> Errors that occur during batch writes in the index topologies (batch and RA) 
> are written to Elasticsearch as a single, large error message, with a field 
> for each failed message. For example, if the batch size is 5000, a single 
> error message will be created with 5000 fields `raw_message_0`, 
> `raw_message_1`, .., `raw_message_4999`. With such large messages, searching 
> the error index in Elasticsearch is excessively slow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1239: METRON-1829: Large Error Message Causes Slow Search Perf...

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1239
  
+1 Works as advertised.  Thanks for the fix


---

[jira] [Commented] (METRON-1829) Large Error Message Causes Slow Search Performance



[ 
https://issues.apache.org/jira/browse/METRON-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657159#comment-16657159
 ] 

ASF GitHub Bot commented on METRON-1829:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1239#discussion_r226725656
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
 ---
@@ -118,12 +119,18 @@ public void commit(BulkWriterResponse response) {
 
   public void error(String sensorType, Throwable e, Iterable 
tuples, MessageGetStrategy messageGetStrategy) {
 LOG.error(format("Failing %d tuple(s); sensorType=%s", 
Iterables.size(tuples), sensorType), e);
-MetronError error = new MetronError()
-.withSensorType(Collections.singleton(sensorType))
-.withErrorType(Constants.ErrorType.INDEXING_ERROR)
-.withThrowable(e);
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
-handleError(tuples, error);
+tuples.forEach(t -> {
+  MetronError error = new MetronError()
+  .withSensorType(Collections.singleton(sensorType))
+  .withErrorType(Constants.ErrorType.INDEXING_ERROR)
+  .withThrowable(e)
+  .addRawMessage(messageGetStrategy.get(t));
--- End diff --

Thinking more about this some more... I feel that what I have described in 
METRON-1832 is a pre-existing condition.  I don't think this PR makes that 
pre-existing condition any worse.  This PR is a solid, necessary fix.  I am in 
favor of merging this and I will open a discuss thread on a resolution for 
METRON-1832.


> Large Error Message Causes Slow Search Performance
> --
>
> Key: METRON-1829
> URL: https://issues.apache.org/jira/browse/METRON-1829
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
>
> Errors that occur during batch writes in the index topologies (batch and RA) 
> are written to Elasticsearch as a single, large error message, with a field 
> for each failed message. For example, if the batch size is 5000, a single 
> error message will be created with 5000 fields `raw_message_0`, 
> `raw_message_1`, .., `raw_message_4999`. With such large messages, searching 
> the error index in Elasticsearch is excessively slow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1239: METRON-1829: Large Error Message Causes Slow Sear...

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1239#discussion_r226725656
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
 ---
@@ -118,12 +119,18 @@ public void commit(BulkWriterResponse response) {
 
   public void error(String sensorType, Throwable e, Iterable 
tuples, MessageGetStrategy messageGetStrategy) {
 LOG.error(format("Failing %d tuple(s); sensorType=%s", 
Iterables.size(tuples), sensorType), e);
-MetronError error = new MetronError()
-.withSensorType(Collections.singleton(sensorType))
-.withErrorType(Constants.ErrorType.INDEXING_ERROR)
-.withThrowable(e);
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
-handleError(tuples, error);
+tuples.forEach(t -> {
+  MetronError error = new MetronError()
+  .withSensorType(Collections.singleton(sensorType))
+  .withErrorType(Constants.ErrorType.INDEXING_ERROR)
+  .withThrowable(e)
+  .addRawMessage(messageGetStrategy.get(t));
--- End diff --

Thinking more about this some more... I feel that what I have described in 
METRON-1832 is a pre-existing condition.  I don't think this PR makes that 
pre-existing condition any worse.  This PR is a solid, necessary fix.  I am in 
favor of merging this and I will open a discuss thread on a resolution for 
METRON-1832.


---

[jira] [Commented] (METRON-1829) Large Error Message Causes Slow Search Performance



[ 
https://issues.apache.org/jira/browse/METRON-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657116#comment-16657116
 ] 

ASF GitHub Bot commented on METRON-1829:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1239#discussion_r226720064
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
 ---
@@ -118,12 +119,18 @@ public void commit(BulkWriterResponse response) {
 
   public void error(String sensorType, Throwable e, Iterable 
tuples, MessageGetStrategy messageGetStrategy) {
 LOG.error(format("Failing %d tuple(s); sensorType=%s", 
Iterables.size(tuples), sensorType), e);
-MetronError error = new MetronError()
-.withSensorType(Collections.singleton(sensorType))
-.withErrorType(Constants.ErrorType.INDEXING_ERROR)
-.withThrowable(e);
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
-handleError(tuples, error);
+tuples.forEach(t -> {
+  MetronError error = new MetronError()
+  .withSensorType(Collections.singleton(sensorType))
+  .withErrorType(Constants.ErrorType.INDEXING_ERROR)
+  .withThrowable(e)
+  .addRawMessage(messageGetStrategy.get(t));
--- End diff --

While testing this, I am seeing a problem.  I am trying to decide whether 
we should call that a pre-existing condition and fix that problem after this PR 
goes in or whether the change here makes things any worse.  Let me know what 
you think.

I have described the problem in 
[METRON-1832](https://issues.apache.org/jira/browse/METRON-1832).  If any index 
destination goes down, error messages are continually recycled and grow larger 
and larger.  In addition, long sequence of escape characters are created.




> Large Error Message Causes Slow Search Performance
> --
>
> Key: METRON-1829
> URL: https://issues.apache.org/jira/browse/METRON-1829
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
>
> Errors that occur during batch writes in the index topologies (batch and RA) 
> are written to Elasticsearch as a single, large error message, with a field 
> for each failed message. For example, if the batch size is 5000, a single 
> error message will be created with 5000 fields `raw_message_0`, 
> `raw_message_1`, .., `raw_message_4999`. With such large messages, searching 
> the error index in Elasticsearch is excessively slow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1239: METRON-1829: Large Error Message Causes Slow Sear...

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1239#discussion_r226720064
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
 ---
@@ -118,12 +119,18 @@ public void commit(BulkWriterResponse response) {
 
   public void error(String sensorType, Throwable e, Iterable 
tuples, MessageGetStrategy messageGetStrategy) {
 LOG.error(format("Failing %d tuple(s); sensorType=%s", 
Iterables.size(tuples), sensorType), e);
-MetronError error = new MetronError()
-.withSensorType(Collections.singleton(sensorType))
-.withErrorType(Constants.ErrorType.INDEXING_ERROR)
-.withThrowable(e);
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
-handleError(tuples, error);
+tuples.forEach(t -> {
+  MetronError error = new MetronError()
+  .withSensorType(Collections.singleton(sensorType))
+  .withErrorType(Constants.ErrorType.INDEXING_ERROR)
+  .withThrowable(e)
+  .addRawMessage(messageGetStrategy.get(t));
--- End diff --

While testing this, I am seeing a problem.  I am trying to decide whether 
we should call that a pre-existing condition and fix that problem after this PR 
goes in or whether the change here makes things any worse.  Let me know what 
you think.

I have described the problem in 
[METRON-1832](https://issues.apache.org/jira/browse/METRON-1832).  If any index 
destination goes down, error messages are continually recycled and grow larger 
and larger.  In addition, long sequence of escape characters are created.




---

[jira] [Commented] (METRON-1832) Recurrent Large Indexing Error Messages



[ 
https://issues.apache.org/jira/browse/METRON-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657112#comment-16657112
 ] 

Nick Allen commented on METRON-1832:


As you can see from the attached error message, the escape character '\' also 
continually compound one another leading to long strings of '\' characters.
{code:java}
guid\\\":\\
    
\"5ed00199-4417-4b97-a204-f7f83303503e\\\",\\\
    
"message\\\":\
    \\"All 
datanodes DatanodeInfoWithStorage[127.0.0.1:50010,DS-495f88fc-f    
d24-41e1-b55c-fc6ec586c6e4,DISK] are bad. 
Aborting...\\\",
    \\\"{code}

> Recurrent Large Indexing Error Messages
> ---
>
> Key: METRON-1832
> URL: https://issues.apache.org/jira/browse/METRON-1832
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
> Attachments: error-message.json
>
>
> If any index destination like HDFS, Elasticsearch, or Solr goes down while 
> the Indexing topology is running, an error message is created and sent back 
> to the user-defined error topic.  By default, this is defined to also be the 
> 'indexing' topic.
> The Indexing topology then consumes this error message and attempts to write 
> it again. If the index destination is still down, another error occurs and 
> another error message is created that encapsulates the original error 
> message.  That message is then sent to the 'indexing' topic, which is later 
> consumed, yet again, by the Indexing topology.
> These error messages will continue to be recycled and grow larger and larger 
> as each new error message encapsulates all previous error messages in the 
> "raw_message" field.
> Once the index destination recovers, one giant error message will finally be 
> written that contains massively duplicated, useless information which can 
> further negatively impact performance of the index destination.
> See attachment for example of once of these error messages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (METRON-1832) Recurrent Large Indexing Error Messages



 [ 
https://issues.apache.org/jira/browse/METRON-1832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Allen updated METRON-1832:
---
Attachment: error-message.json

> Recurrent Large Indexing Error Messages
> ---
>
> Key: METRON-1832
> URL: https://issues.apache.org/jira/browse/METRON-1832
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Priority: Major
> Attachments: error-message.json
>
>
> If any index destination like HDFS, Elasticsearch, or Solr goes down while 
> the Indexing topology is running, an error message is created and sent back 
> to the user-defined error topic.  By default, this is defined to also be the 
> 'indexing' topic.
> The Indexing topology then consumes this error message and attempts to write 
> it again. If the index destination is still down, another error occurs and 
> another error message is created that encapsulates the original error 
> message.  That message is then sent to the 'indexing' topic, which is later 
> consumed, yet again, by the Indexing topology.
> These error messages will continue to be recycled and grow larger and larger 
> as each new error message encapsulates all previous error messages in the 
> "raw_message" field.
> Once the index destination recovers, one giant error message will finally be 
> written that contains massively duplicated, useless information which can 
> further negatively impact performance of the index destination.
> See attachment for example of once of these error messages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1832) Recurrent Large Indexing Error Messages

Nick Allen created METRON-1832:
--

 Summary: Recurrent Large Indexing Error Messages
 Key: METRON-1832
 URL: https://issues.apache.org/jira/browse/METRON-1832
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen


If any index destination like HDFS, Elasticsearch, or Solr goes down while the 
Indexing topology is running, an error message is created and sent back to the 
user-defined error topic.  By default, this is defined to also be the 
'indexing' topic.

The Indexing topology then consumes this error message and attempts to write it 
again. If the index destination is still down, another error occurs and another 
error message is created that encapsulates the original error message.  That 
message is then sent to the 'indexing' topic, which is later consumed, yet 
again, by the Indexing topology.

These error messages will continue to be recycled and grow larger and larger as 
each new error message encapsulates all previous error messages in the 
"raw_message" field.

Once the index destination recovers, one giant error message will finally be 
written that contains massively duplicated, useless information which can 
further negatively impact performance of the index destination.

See attachment for example of once of these error messages.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (METRON-1831) Project Version Substitution Not Working

Nick Allen created METRON-1831:
--

 Summary: Project Version Substitution Not Working
 Key: METRON-1831
 URL: https://issues.apache.org/jira/browse/METRON-1831
 Project: Metron
  Issue Type: Bug
Reporter: Nick Allen


Steps to Replicate:
 # Spin-up development environment.
 # Launch REPL.
 # Notice the warning:
{code:java}
bin/stellar: line 36: ${project.version}: bad substitution{code}

 
{code:java}
[root@node1 ~]# source /etc/default/metron
[root@node1 ~]# cd $METRON_HOME
[root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER
bin/stellar: line 36: ${project.version}: bad substitution
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Stellar, Go!
Functions are loading lazily in the background and will be unavailable until 
loaded fully.
{es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, 
parser.error.topic=indexing, update.hbase.table=metron_update, 
update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, 
es.document.id=, profiler.client.period.duration=15, 
profiler.client.period.duration.units=MINUTES, 
user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, 
bootstrap.servers=node1:6667, source.type.field=source:type, 
threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, 
enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, 
profiler.writer.batchTimeout=0, 
geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code}
 

This only seems to be a problem with the $METRON_HOME/bin/stellar script.
{code:java}
[root@node1 0.6.1]# grep project.version bin/*
bin/stellar:export METRON_VERSION="${METRON_VERSION:-${project.version}}"{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1239: METRON-1829: Large Error Message Causes Slow Sear...

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1239#discussion_r226675652
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
 ---
@@ -118,12 +118,15 @@ public void commit(BulkWriterResponse response) {
 
   public void error(String sensorType, Throwable e, Iterable 
tuples, MessageGetStrategy messageGetStrategy) {
 LOG.error(format("Failing %d tuple(s); sensorType=%s", 
Iterables.size(tuples), sensorType), e);
-MetronError error = new MetronError()
-.withSensorType(Collections.singleton(sensorType))
-.withErrorType(Constants.ErrorType.INDEXING_ERROR)
-.withThrowable(e);
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
-handleError(tuples, error);
+tuples.forEach(t -> {
--- End diff --

Looks great.  I'm just going to spin it up and do some manual testing now.


---

[jira] [Commented] (METRON-1829) Large Error Message Causes Slow Search Performance



[ 
https://issues.apache.org/jira/browse/METRON-1829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656887#comment-16656887
 ] 

ASF GitHub Bot commented on METRON-1829:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/1239#discussion_r226675652
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
 ---
@@ -118,12 +118,15 @@ public void commit(BulkWriterResponse response) {
 
   public void error(String sensorType, Throwable e, Iterable 
tuples, MessageGetStrategy messageGetStrategy) {
 LOG.error(format("Failing %d tuple(s); sensorType=%s", 
Iterables.size(tuples), sensorType), e);
-MetronError error = new MetronError()
-.withSensorType(Collections.singleton(sensorType))
-.withErrorType(Constants.ErrorType.INDEXING_ERROR)
-.withThrowable(e);
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
-handleError(tuples, error);
+tuples.forEach(t -> {
--- End diff --

Looks great.  I'm just going to spin it up and do some manual testing now.


> Large Error Message Causes Slow Search Performance
> --
>
> Key: METRON-1829
> URL: https://issues.apache.org/jira/browse/METRON-1829
> Project: Metron
>  Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
>
> Errors that occur during batch writes in the index topologies (batch and RA) 
> are written to Elasticsearch as a single, large error message, with a field 
> for each failed message. For example, if the batch size is 5000, a single 
> error message will be created with 5000 fields `raw_message_0`, 
> `raw_message_1`, .., `raw_message_4999`. With such large messages, searching 
> the error index in Elasticsearch is excessively slow.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery



[ 
https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656585#comment-16656585
 ] 

ASF GitHub Bot commented on METRON-1830:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226597603
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
 ---
@@ -259,19 +259,25 @@ export class TableViewComponent implements OnInit, 
OnChanges, OnDestroy {
   }
 
   deleteOneAlertFromMetaAlert($event, alert: Alert, metaAlertIndex: 
number) {
-this.metronDialogBox.showConfirmationMessage('Do you wish to remove 
the alert from the meta alert?').subscribe(response => {
-  if (response) {
+let confirmedSubscription = this.dialogService.confirm('Do you wish to 
remove the alert from the meta alert?').subscribe(r => {
+  if (r === 'Confirmed') {
 this.doDeleteOneAlertFromMetaAlert(alert, metaAlertIndex);
   }
+  if (r !== 'Initial') {
--- End diff --

What if we remove the if statement and simply unsubscribe at the end of 
this function?
With that, I think we could be able to remove 'Initial' value from 
ConfirmationType and the dispatching of 'Initial' from the first line of 
DialogService.confirm function. Could we keep the same functionality with this 
simplification?

Also, we can call variable 'r' like 'action' if we remove the coco egg. 


> Re-implement Alerts dialog box without jQuery
> -
>
> Key: METRON-1830
> URL: https://issues.apache.org/jira/browse/METRON-1830
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Minor
>
> Currently, the dialog box class in both UIs directly manipulates the DOM with 
> jQuery. This is problematic when using a framework like Angular because it 
> causes a disconnect with how Angular handles and tracks DOM changes. We also 
> don't take advantage of Angular's rendering engine.
> The dialog box can and should be implemented as a component and a singleton 
> service (since we would never want to launch more than one modal at a time).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery



[ 
https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656586#comment-16656586
 ] 

ASF GitHub Bot commented on METRON-1830:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226589952
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
 ---
@@ -259,19 +259,25 @@ export class TableViewComponent implements OnInit, 
OnChanges, OnDestroy {
   }
 
   deleteOneAlertFromMetaAlert($event, alert: Alert, metaAlertIndex: 
number) {
-this.metronDialogBox.showConfirmationMessage('Do you wish to remove 
the alert from the meta alert?').subscribe(response => {
-  if (response) {
+let confirmedSubscription = this.dialogService.confirm('Do you wish to 
remove the alert from the meta alert?').subscribe(r => {
--- End diff --

I would use const here (and all the other places) too.


> Re-implement Alerts dialog box without jQuery
> -
>
> Key: METRON-1830
> URL: https://issues.apache.org/jira/browse/METRON-1830
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Minor
>
> Currently, the dialog box class in both UIs directly manipulates the DOM with 
> jQuery. This is problematic when using a framework like Angular because it 
> causes a disconnect with how Angular handles and tracks DOM changes. We also 
> don't take advantage of Angular's rendering engine.
> The dialog box can and should be implemented as a component and a singleton 
> service (since we would never want to launch more than one modal at a time).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery



[ 
https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656587#comment-16656587
 ] 

ASF GitHub Bot commented on METRON-1830:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226597922
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts
 ---
@@ -421,10 +423,13 @@ export class TreeViewComponent extends 
TableViewComponent implements OnInit, OnC
 if (this.canCreateMetaAlert(group.total)) {
   let confirmationMsg = 'Do you wish to create a meta alert with ' +
 (group.total === 1 ? ' alert' : group.total + 
' selected alerts') + '?';
-  
this.metronDialogBox.showConfirmationMessage(confirmationMsg).subscribe((response)
 => {
-if (response) {
+  let confirmedSubscription = 
this.dialogService.confirm(confirmationMsg).subscribe(r => {
+if (r === 'Confirmed') {
--- End diff --

I suggest to use (r === ConfirmationType.Confirmed) instead of the string 
literal.


> Re-implement Alerts dialog box without jQuery
> -
>
> Key: METRON-1830
> URL: https://issues.apache.org/jira/browse/METRON-1830
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Minor
>
> Currently, the dialog box class in both UIs directly manipulates the DOM with 
> jQuery. This is problematic when using a framework like Angular because it 
> causes a disconnect with how Angular handles and tracks DOM changes. We also 
> don't take advantage of Angular's rendering engine.
> The dialog box can and should be implemented as a component and a singleton 
> service (since we would never want to launch more than one modal at a time).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery



[ 
https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656583#comment-16656583
 ] 

ASF GitHub Bot commented on METRON-1830:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226576832
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.ts
 ---
@@ -249,8 +249,8 @@ export class AlertDetailsComponent implements OnInit {
   commentText += ' \'' + 
this.alertCommentsWrapper[index].alertComment.comment + '\'';
 }
 
-
this.metronDialogBox.showConfirmationMessage(commentText).subscribe(response => 
{
-  if (response) {
+let confirmedSubscription = 
this.dialogService.confirm(commentText).subscribe(r => {
--- End diff --

Could we use const here instead of let?
It might be better to move the logic inside this subscription to a function 
like in saved-serches.component.ts #67 or in table-view.component.ts #428


> Re-implement Alerts dialog box without jQuery
> -
>
> Key: METRON-1830
> URL: https://issues.apache.org/jira/browse/METRON-1830
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Minor
>
> Currently, the dialog box class in both UIs directly manipulates the DOM with 
> jQuery. This is problematic when using a framework like Angular because it 
> causes a disconnect with how Angular handles and tracks DOM changes. We also 
> don't take advantage of Angular's rendering engine.
> The dialog box can and should be implemented as a component and a singleton 
> service (since we would never want to launch more than one modal at a time).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1830) Re-implement Alerts dialog box without jQuery



[ 
https://issues.apache.org/jira/browse/METRON-1830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16656584#comment-16656584
 ] 

ASF GitHub Bot commented on METRON-1830:


Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226589395
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
 ---
@@ -72,15 +72,15 @@ export class TableViewComponent implements OnInit, 
OnChanges, OnDestroy {
   @Output() onSelectedAlertsChange = new EventEmitter< Alert[]>();
 
   constructor(searchService: SearchService,
-  metronDialogBox: MetronDialogBox,
   updateService: UpdateService,
   metaAlertService: MetaAlertService,
-  globalConfigService: GlobalConfigService) {
+  globalConfigService: GlobalConfigService,
+  dialogService: DialogService) {
 this.searchService = searchService;
-this.metronDialogBox = metronDialogBox;
 this.updateService = updateService;
 this.metaAlertService = metaAlertService;
 this.globalConfigService = globalConfigService;
+this.dialogService = dialogService;
--- End diff --

In other places, you haven't assigning injected dialogService to a field 
but relying on Angular automagic. I see you tried to stay consistent with the 
original constructor impl, but might worth considering to building up a 
consistency on an application level.


> Re-implement Alerts dialog box without jQuery
> -
>
> Key: METRON-1830
> URL: https://issues.apache.org/jira/browse/METRON-1830
> Project: Metron
>  Issue Type: Bug
>Reporter: Shane Ardell
>Assignee: Shane Ardell
>Priority: Minor
>
> Currently, the dialog box class in both UIs directly manipulates the DOM with 
> jQuery. This is problematic when using a framework like Angular because it 
> causes a disconnect with how Angular handles and tracks DOM changes. We also 
> don't take advantage of Angular's rendering engine.
> The dialog box can and should be implemented as a component and a singleton 
> service (since we would never want to launch more than one modal at a time).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1240: METRON-1830: Re-implement Alerts dialog box witho...

Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226576832
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alert-details/alert-details.component.ts
 ---
@@ -249,8 +249,8 @@ export class AlertDetailsComponent implements OnInit {
   commentText += ' \'' + 
this.alertCommentsWrapper[index].alertComment.comment + '\'';
 }
 
-
this.metronDialogBox.showConfirmationMessage(commentText).subscribe(response => 
{
-  if (response) {
+let confirmedSubscription = 
this.dialogService.confirm(commentText).subscribe(r => {
--- End diff --

Could we use const here instead of let?
It might be better to move the logic inside this subscription to a function 
like in saved-serches.component.ts #67 or in table-view.component.ts #428


---

[GitHub] metron pull request #1240: METRON-1830: Re-implement Alerts dialog box witho...

Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226589395
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
 ---
@@ -72,15 +72,15 @@ export class TableViewComponent implements OnInit, 
OnChanges, OnDestroy {
   @Output() onSelectedAlertsChange = new EventEmitter< Alert[]>();
 
   constructor(searchService: SearchService,
-  metronDialogBox: MetronDialogBox,
   updateService: UpdateService,
   metaAlertService: MetaAlertService,
-  globalConfigService: GlobalConfigService) {
+  globalConfigService: GlobalConfigService,
+  dialogService: DialogService) {
 this.searchService = searchService;
-this.metronDialogBox = metronDialogBox;
 this.updateService = updateService;
 this.metaAlertService = metaAlertService;
 this.globalConfigService = globalConfigService;
+this.dialogService = dialogService;
--- End diff --

In other places, you haven't assigning injected dialogService to a field 
but relying on Angular automagic. I see you tried to stay consistent with the 
original constructor impl, but might worth considering to building up a 
consistency on an application level.


---

[GitHub] metron pull request #1240: METRON-1830: Re-implement Alerts dialog box witho...

Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226597922
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/tree-view/tree-view.component.ts
 ---
@@ -421,10 +423,13 @@ export class TreeViewComponent extends 
TableViewComponent implements OnInit, OnC
 if (this.canCreateMetaAlert(group.total)) {
   let confirmationMsg = 'Do you wish to create a meta alert with ' +
 (group.total === 1 ? ' alert' : group.total + 
' selected alerts') + '?';
-  
this.metronDialogBox.showConfirmationMessage(confirmationMsg).subscribe((response)
 => {
-if (response) {
+  let confirmedSubscription = 
this.dialogService.confirm(confirmationMsg).subscribe(r => {
+if (r === 'Confirmed') {
--- End diff --

I suggest to use (r === ConfirmationType.Confirmed) instead of the string 
literal.


---

[GitHub] metron pull request #1240: METRON-1830: Re-implement Alerts dialog box witho...

Github user tiborm commented on a diff in the pull request:

https://github.com/apache/metron/pull/1240#discussion_r226589952
  
--- Diff: 
metron-interface/metron-alerts/src/app/alerts/alerts-list/table-view/table-view.component.ts
 ---
@@ -259,19 +259,25 @@ export class TableViewComponent implements OnInit, 
OnChanges, OnDestroy {
   }
 
   deleteOneAlertFromMetaAlert($event, alert: Alert, metaAlertIndex: 
number) {
-this.metronDialogBox.showConfirmationMessage('Do you wish to remove 
the alert from the meta alert?').subscribe(response => {
-  if (response) {
+let confirmedSubscription = this.dialogService.confirm('Do you wish to 
remove the alert from the meta alert?').subscribe(r => {
--- End diff --

I would use const here (and all the other places) too.


---

[GitHub] metron pull request #1240: METRON-1830: Re-implement Alerts dialog box witho...