[jira] [Commented] (METRON-1831) Project Version Substitution Not Working
[ https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657891#comment-16657891 ] ASF GitHub Bot commented on METRON-1831: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1243 > Project Version Substitution Not Working > > > Key: METRON-1831 > URL: https://issues.apache.org/jira/browse/METRON-1831 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > Steps to Replicate: > # Spin-up development environment. > # Launch REPL. > # Notice the warning: > {code:java} > bin/stellar: line 36: ${project.version}: bad substitution{code} > > {code:java} > [root@node1 ~]# source /etc/default/metron > [root@node1 ~]# cd $METRON_HOME > [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER > bin/stellar: line 36: ${project.version}: bad substitution > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Stellar, Go! > Functions are loading lazily in the background and will be unavailable until > loaded fully. > {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, > parser.error.topic=indexing, update.hbase.table=metron_update, > update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, > es.document.id=, profiler.client.period.duration=15, > profiler.client.period.duration.units=MINUTES, > user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, > bootstrap.servers=node1:6667, source.type.field=source:type, > threat.triage.score.field=threat:triage:score, > enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, > profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, > geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code} > > This only seems to be a problem with the $METRON_HOME/bin/stellar script. > {code:java} > [root@node1 0.6.1]# grep project.version bin/* > bin/stellar:export > METRON_VERSION="${METRON_VERSION:-${project.version}}"{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1831) Project Version Substitution Not Working
[ https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657624#comment-16657624 ] ASF GitHub Bot commented on METRON-1831: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1243 Transient integration test failure? ``` Tests in error: ElasticsearchIndexingIntegrationTest>IndexingIntegrationTest.test:137 » Runtime ``` > Project Version Substitution Not Working > > > Key: METRON-1831 > URL: https://issues.apache.org/jira/browse/METRON-1831 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > Steps to Replicate: > # Spin-up development environment. > # Launch REPL. > # Notice the warning: > {code:java} > bin/stellar: line 36: ${project.version}: bad substitution{code} > > {code:java} > [root@node1 ~]# source /etc/default/metron > [root@node1 ~]# cd $METRON_HOME > [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER > bin/stellar: line 36: ${project.version}: bad substitution > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Stellar, Go! > Functions are loading lazily in the background and will be unavailable until > loaded fully. > {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, > parser.error.topic=indexing, update.hbase.table=metron_update, > update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, > es.document.id=, profiler.client.period.duration=15, > profiler.client.period.duration.units=MINUTES, > user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, > bootstrap.servers=node1:6667, source.type.field=source:type, > threat.triage.score.field=threat:triage:score, > enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, > profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, > geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code} > > This only seems to be a problem with the $METRON_HOME/bin/stellar script. > {code:java} > [root@node1 0.6.1]# grep project.version bin/* > bin/stellar:export > METRON_VERSION="${METRON_VERSION:-${project.version}}"{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1831) Project Version Substitution Not Working
[ https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657626#comment-16657626 ] ASF GitHub Bot commented on METRON-1831: GitHub user nickwallen reopened a pull request: https://github.com/apache/metron/pull/1243 METRON-1831 Project Version Substitution Not Working When running the stellar REPL script, a warning is reported. ``` bin/stellar: line 36: ${project.version}: bad substitution ``` For example... ``` [root@node1 ~]# source /etc/default/metron [root@node1 ~]# cd $METRON_HOME [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER bin/stellar: line 36: ${project.version}: bad substitution SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Stellar, Go! Functions are loading lazily in the background and will be unavailable until loaded fully. {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, parser.error.topic=indexing, update.hbase.table=metron_update, update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, es.document.id=, profiler.client.period.duration=15, profiler.client.period.duration.units=MINUTES, user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, bootstrap.servers=node1:6667, source.type.field=source:type, threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz} ``` ## Problem In the stellar REPL launch script, the ${project.version} is not getting substituted at build time, perhaps because it is wrapped inside another ${...}. This is probably why it is an issue in the stellar script and not the other scripts. ## Change * The stellar REPL launch script was changed in #1232 , but I did not see this warning occur when testing that change. * Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` automatically. There really is no need to set `METRON_HOME` and `METRON_VERSION` in the script any longer. It is cleaner to just remove this (which I almost did in #1232). * I use a `set -u` so the script will error if `METRON_HOME` is not defined by some odd happenstance. ## Testing 1. Stand-up a development environment. 1. Launch the Stellar REPL. Ensure there is no warning reported. ``` [vagrant@node1 ~]$ sudo su - [root@node1 ~]# cd /usr/metron/0.6.1/ [root@node1 0.6.1]# bin/stellar SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Stellar, Go! Functions are loading lazily in the background and will be unavailable until loaded fully. {} [Stellar]>>> ``` 1. Ensure the standard suite of Stellar functions are discovered. ``` [Stellar]>>> %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, BYTEARRAY_MATCHER, CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, ENRICHMENT_STELLAR_TRANSFORM_ADD, ENRICHMENT_STELLAR_TRANSFORM_PRINT, ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, GEOHASH_FROM_LATLONG, GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE, GET_LAST, GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, HDFS_READ_LINES, HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, INDEXING_SET_BATCH, INDEXING_SET_ENABLED, INDEXING_SET_INDEX, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_ENCODING, IS_INTEGER, IS_IP, IS_NAN, IS_URL, IT_ENTROPY, JOIN,
[jira] [Commented] (METRON-1831) Project Version Substitution Not Working
[ https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657625#comment-16657625 ] ASF GitHub Bot commented on METRON-1831: Github user nickwallen closed the pull request at: https://github.com/apache/metron/pull/1243 > Project Version Substitution Not Working > > > Key: METRON-1831 > URL: https://issues.apache.org/jira/browse/METRON-1831 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > Steps to Replicate: > # Spin-up development environment. > # Launch REPL. > # Notice the warning: > {code:java} > bin/stellar: line 36: ${project.version}: bad substitution{code} > > {code:java} > [root@node1 ~]# source /etc/default/metron > [root@node1 ~]# cd $METRON_HOME > [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER > bin/stellar: line 36: ${project.version}: bad substitution > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Stellar, Go! > Functions are loading lazily in the background and will be unavailable until > loaded fully. > {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, > parser.error.topic=indexing, update.hbase.table=metron_update, > update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, > es.document.id=, profiler.client.period.duration=15, > profiler.client.period.duration.units=MINUTES, > user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, > bootstrap.servers=node1:6667, source.type.field=source:type, > threat.triage.score.field=threat:triage:score, > enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, > profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, > geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code} > > This only seems to be a problem with the $METRON_HOME/bin/stellar script. > {code:java} > [root@node1 0.6.1]# grep project.version bin/* > bin/stellar:export > METRON_VERSION="${METRON_VERSION:-${project.version}}"{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1831) Project Version Substitution Not Working
[ https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657584#comment-16657584 ] ASF GitHub Bot commented on METRON-1831: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1243 +1 by inspection @nickwallen, thanks for the fix. > Project Version Substitution Not Working > > > Key: METRON-1831 > URL: https://issues.apache.org/jira/browse/METRON-1831 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > > Steps to Replicate: > # Spin-up development environment. > # Launch REPL. > # Notice the warning: > {code:java} > bin/stellar: line 36: ${project.version}: bad substitution{code} > > {code:java} > [root@node1 ~]# source /etc/default/metron > [root@node1 ~]# cd $METRON_HOME > [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER > bin/stellar: line 36: ${project.version}: bad substitution > SLF4J: Class path contains multiple SLF4J bindings. > SLF4J: Found binding in > [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: Found binding in > [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an > explanation. > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] > Stellar, Go! > Functions are loading lazily in the background and will be unavailable until > loaded fully. > {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, > parser.error.topic=indexing, update.hbase.table=metron_update, > update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, > es.document.id=, profiler.client.period.duration=15, > profiler.client.period.duration.units=MINUTES, > user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, > bootstrap.servers=node1:6667, source.type.field=source:type, > threat.triage.score.field=threat:triage:score, > enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, > profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, > geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz}{code} > > This only seems to be a problem with the $METRON_HOME/bin/stellar script. > {code:java} > [root@node1 0.6.1]# grep project.version bin/* > bin/stellar:export > METRON_VERSION="${METRON_VERSION:-${project.version}}"{code} -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1831) Project Version Substitution Not Working
[ https://issues.apache.org/jira/browse/METRON-1831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657516#comment-16657516 ] ASF GitHub Bot commented on METRON-1831: GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1243 METRON-1831 Project Version Substitution Not Working When running the stellar REPL script, a warning is reported. ``` bin/stellar: line 36: ${project.version}: bad substitution ``` For example... ``` [root@node1 ~]# source /etc/default/metron [root@node1 ~]# cd $METRON_HOME [root@node1 0.6.1]# bin/stellar -z $ZOOKEEPER bin/stellar: line 36: ${project.version}: bad substitution SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Stellar, Go! Functions are loading lazily in the background and will be unavailable until loaded fully. {es.clustername=metron, es.ip=node1:9300, es.date.format=.MM.dd.HH, parser.error.topic=indexing, update.hbase.table=metron_update, update.hbase.cf=t, es.client.settings={client.transport.ping_timeout=500s}, es.document.id=, profiler.client.period.duration=15, profiler.client.period.duration.units=MINUTES, user.settings.hbase.table=user_settings, user.settings.hbase.cf=cf, bootstrap.servers=node1:6667, source.type.field=source:type, threat.triage.score.field=threat:triage:score, enrichment.writer.batchSize=15, enrichment.writer.batchTimeout=0, profiler.writer.batchSize=15, profiler.writer.batchTimeout=0, geo.hdfs.file=/apps/metron/geo/default/GeoLite2-City.mmdb.gz} ``` ## Problem In the stellar REPL launch script, the ${project.version} is not getting substituted at build time, perhaps because it is wrapped inside another ${...}. This is probably why it is an issue in the stellar script and not the other scripts. ## Change * The stellar REPL launch script was changed in #1232 , but I did not see this warning occur when testing that change. * Since #1232, we source `/etc/default/metron` which defines `METRON_HOME` automatically. There really is no need to set `METRON_HOME` and `METRON_VERSION` in the script any longer. It is cleaner to just remove this (which I almost did in #1232). * I use a `set -u` so the script will error if `METRON_HOME` is not defined by some odd happenstance. ## Testing 1. Stand-up a development environment. 1. Launch the Stellar REPL. Ensure there is no warning reported. ``` [vagrant@node1 ~]$ sudo su - [root@node1 ~]# cd /usr/metron/0.6.1/ [root@node1 0.6.1]# bin/stellar SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/metron/0.6.1/lib/metron-profiler-repl-0.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.5.0-292/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] Stellar, Go! Functions are loading lazily in the background and will be unavailable until loaded fully. {} [Stellar]>>> ``` 1. Ensure the standard suite of Stellar functions are discovered. ``` [Stellar]>>> %functions ABS, APPEND_IF_MISSING, BIN, BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, BYTEARRAY_MATCHER, CEILING, CHOMP, CHOP, CONFIG_GET, CONFIG_PUT, COS, COUNT_MATCHES, DATE_FORMAT, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, DECODE, DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, DOMAIN_TYPOSQUAT, ENCODE, ENDS_WITH, ENRICHMENT_EXISTS, ENRICHMENT_GET, ENRICHMENT_STELLAR_TRANSFORM_ADD, ENRICHMENT_STELLAR_TRANSFORM_PRINT, ENRICHMENT_STELLAR_TRANSFORM_REMOVE, EXP, FILL_LEFT, FILL_RIGHT, FILTER, FLOOR, FORMAT, FUZZY_LANGS, FUZZY_SCORE, GEOHASH_CENTROID, GEOHASH_DIST, GEOHASH_FROM_LATLONG, GEOHASH_FROM_LOC, GEOHASH_MAX_DIST, GEOHASH_TO_LATLONG, GEO_GET, GET, GET_FIRST, GET_HASHES_AVAILABLE, GET_LAST, GET_SUPPORTED_ENCODINGS, GROK_EVAL, GROK_PREDICT, HASH, HDFS_LS, HDFS_READ, HDFS_READ_LINES, HDFS_RM, HDFS_WRITE, HLLP_ADD, HLLP_CARDINALITY, HLLP_INIT, HLLP_MERGE, INDEXING_SET_BATCH, INDEXING_SET_ENABLED, INDEXING_SET_INDEX, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_ENCODING, IS_INTEGER, IS_IP, IS_NAN, IS_URL, IT_ENTROPY, JOIN, KAFKA_FIND,