[jira] [Commented] (METRON-1003) ParserUtil parses dates incorrect
[
https://issues.apache.org/jira/browse/METRON-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087219#comment-16087219
]
ASF GitHub Bot commented on METRON-1003:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/623
> ParserUtil parses dates incorrect
> -
>
> Key: METRON-1003
> URL: https://issues.apache.org/jira/browse/METRON-1003
> Project: Metron
> Issue Type: Bug
>Reporter: Vladimir
>Priority: Minor
>
> ParserUtils class has method convertToEpoch that takes month, day and time
> (as strings), parses it and returns milliseconds since epoch.
> Month expected in "MMM" format (i.e. "Jun")
> Month is parsed and then it is tried to get int value as:
> {code}
> String month = String.valueOf(cal.get(Calendar.MONTH));
> {code}
> But according to documentation (see
> https://docs.oracle.com/javase/7/docs/api/java/util/Calendar.html#MONTH)
> months start from 0.
> So this method returns incorrect value.
> This method should be refactored, but would be great to fix it and write some
> tests before.
> This is minor bug as this method is used in FireEye parser only.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-838) Incorrect set of ts in FireEye parser
[ https://issues.apache.org/jira/browse/METRON-838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087222#comment-16087222 ] ASF GitHub Bot commented on METRON-838: --- Github user justinleet commented on the issue: https://github.com/apache/metron/pull/528 @bjigmp I just merged in https://github.com/apache/metron/pull/623, so you should be able to continue this now. Thanks again for the submissions! > Incorrect set of ts in FireEye parser > - > > Key: METRON-838 > URL: https://issues.apache.org/jira/browse/METRON-838 > Project: Metron > Issue Type: Bug >Reporter: Vladimir >Priority: Minor > > Although log line is parsed and day/month/year are extracted ts is not set to > correct value. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-633) Create better logging for HbaseEnrichmentWriter
[ https://issues.apache.org/jira/browse/METRON-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087224#comment-16087224 ] ASF GitHub Bot commented on METRON-633: --- Github user justinleet commented on the issue: https://github.com/apache/metron/pull/572 +1, thanks again for the contribution. > Create better logging for HbaseEnrichmentWriter > --- > > Key: METRON-633 > URL: https://issues.apache.org/jira/browse/METRON-633 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Tomas Zezula > Labels: newbie > > Right now our debug logging is nonexistent for this writer and it makes > tracking down issues almost impossible. This should be corrected. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-633) Create better logging for HbaseEnrichmentWriter
[ https://issues.apache.org/jira/browse/METRON-633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087225#comment-16087225 ] ASF GitHub Bot commented on METRON-633: --- Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/572 > Create better logging for HbaseEnrichmentWriter > --- > > Key: METRON-633 > URL: https://issues.apache.org/jira/browse/METRON-633 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Tomas Zezula > Labels: newbie > > Right now our debug logging is nonexistent for this writer and it makes > tracking down issues almost impossible. This should be corrected. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1008) Migrate Travis build to use Trusty
[
https://issues.apache.org/jira/browse/METRON-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087239#comment-16087239
]
ASF GitHub Bot commented on METRON-1008:
Github user justinleet commented on the issue:
https://github.com/apache/metron/pull/633
@ottobackwards Do you have any objections to the current state of the PR?
At this point, I think this is a pretty focused, incremental change. I'm +1 on
it, but I want to make sure you have a chance to chime in.
> Migrate Travis build to use Trusty
> --
>
> Key: METRON-1008
> URL: https://issues.apache.org/jira/browse/METRON-1008
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>
> [~zeo...@gmail.com] pinged someone from Travis while we were working on
> METRON-1004. In addition to confirming that we should be using a VM, it was
> also suggested we move to Trusty. This was briefly looked at in during
> METRON-1004, in particular because it (theoretically) has the correct version
> of compilers installed, so we could avoid the hit of setting that up. It
> blew up on `npm install`, possibly from my own ignorance.
> At minimum we should move to Trusty:
> {code}
> https://docs.travis-ci.com/user/trusty-ci-environment/
> {code}
> Ideally, we do some more thorough investigation of the compiler and see if we
> can cut out that step.
> See:
> https://docs.travis-ci.com/user/trusty-ci-environment/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1045) [Metron enrichment] Cannot restart service - enrichment table already exists
Rob Phipps created METRON-1045:
--
Summary: [Metron enrichment] Cannot restart service - enrichment
table already exists
Key: METRON-1045
URL: https://issues.apache.org/jira/browse/METRON-1045
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.0
Environment: CentOS 7
Metron 0.4.0
Ambari 2.5.1.10
Reporter: Rob Phipps
Hi,
This is a new install on Ambari, downloaded from the hortonworks repo if I
remember correctly. Although I have been able to start Metron once, for any
subsequent launches I get an error complaining the table 'enrichment' already
exists in the hdfs store. If I delete the table from the command line and try
again, then the startup process runs properly. Why is this script assuming that
the database is clean before it starts?
Although this is potentially unrelated, the Metron REST API module doesn't stay
running for very long after I start it, which means I am unable to log into the
management interface.
Any assistance would be much appreciated.
{quote}Execution of '/usr/metron/0.4.0/bin/start_enrichment_topology.sh
-s enrichment
-z metron:2181,hadoop-slave-1:2181,hadoop-master:2181' returned 1. Running:
/usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
-cp
/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar
org.apache.storm.daemon.ClientJarTransformerRunner
org.apache.storm.hack.StormShadeTransformer
/usr/metron/0.4.0/lib/metron-enrichment-0.4.0-uber.jar
/tmp/b70910e6688d11e78398000c2953f5b1.jar
Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
-Dstorm.conf.file= -cp
/usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/tmp/b70910e6688d11e78398000c2953f5b1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.1.0-129/storm/bin
-Dstorm.jar=/tmp/b70910e6688d11e78398000c2953f5b1.jar -Dstorm.dependency.jars=
-Dstorm.dependency.artifacts={} org.apache.storm.flux.Flux --remote
/usr/metron/0.4.0/flux/enrichment/remote.yaml --filter
/usr/metron/0.4.0/config/enrichment.properties{quote}
{quote}Exception in thread "main" java.lang.RuntimeException: Topology with
name `enrichment` already exists on cluster
at
org.apache.storm.StormSubmitter.submitTopologyAs(StormSubmitter.java:240)
at
org.apache.storm.StormSubmitter.submitTopology(StormSubmitter.java:390)
at org.apache.storm.flux.Flux.runCli(Flux.java:171)
at org.apache.storm.flux.Flux.main(Flux.java:98){quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1008) Migrate Travis build to use Trusty
[
https://issues.apache.org/jira/browse/METRON-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087257#comment-16087257
]
ASF GitHub Bot commented on METRON-1008:
Github user ottobackwards commented on the issue:
https://github.com/apache/metron/pull/633
+1
> Migrate Travis build to use Trusty
> --
>
> Key: METRON-1008
> URL: https://issues.apache.org/jira/browse/METRON-1008
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>
> [~zeo...@gmail.com] pinged someone from Travis while we were working on
> METRON-1004. In addition to confirming that we should be using a VM, it was
> also suggested we move to Trusty. This was briefly looked at in during
> METRON-1004, in particular because it (theoretically) has the correct version
> of compilers installed, so we could avoid the hit of setting that up. It
> blew up on `npm install`, possibly from my own ignorance.
> At minimum we should move to Trusty:
> {code}
> https://docs.travis-ci.com/user/trusty-ci-environment/
> {code}
> Ideally, we do some more thorough investigation of the compiler and see if we
> can cut out that step.
> See:
> https://docs.travis-ci.com/user/trusty-ci-environment/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1008) Migrate Travis build to use Trusty
[
https://issues.apache.org/jira/browse/METRON-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087265#comment-16087265
]
ASF GitHub Bot commented on METRON-1008:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/633
> Migrate Travis build to use Trusty
> --
>
> Key: METRON-1008
> URL: https://issues.apache.org/jira/browse/METRON-1008
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>
> [~zeo...@gmail.com] pinged someone from Travis while we were working on
> METRON-1004. In addition to confirming that we should be using a VM, it was
> also suggested we move to Trusty. This was briefly looked at in during
> METRON-1004, in particular because it (theoretically) has the correct version
> of compilers installed, so we could avoid the hit of setting that up. It
> blew up on `npm install`, possibly from my own ignorance.
> At minimum we should move to Trusty:
> {code}
> https://docs.travis-ci.com/user/trusty-ci-environment/
> {code}
> Ideally, we do some more thorough investigation of the compiler and see if we
> can cut out that step.
> See:
> https://docs.travis-ci.com/user/trusty-ci-environment/
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1046) STELLAR: Configurations should be able to reference Stellar File
Otto Fowler created METRON-1046: --- Summary: STELLAR: Configurations should be able to reference Stellar File Key: METRON-1046 URL: https://issues.apache.org/jira/browse/METRON-1046 Project: Metron Issue Type: Improvement Reporter: Otto Fowler Assignee: Otto Fowler As we get more and more capability in Stellar, we also get more complexity. Managing and testing complex stellar statements only by running things through the parsers and enrichment is not idea. An improvement would be to allow configurations ( enrichment and parser ) to be able to reference a Stellar 'file', which perhaps can be shared between parsers. This File should should be loaded into a class which presents a clean interface into multiple possible stellar executions, operating on the same message map, returning that map ( or modifying it ). These files, or what ever we call the abstraction should be testable on their own as well. The files and their 'updates' can be managed in the same way that the updates to their parent configurations are. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1045) [Metron enrichment] Cannot restart service - enrichment table already exists
[
https://issues.apache.org/jira/browse/METRON-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087368#comment-16087368
]
Nick Allen commented on METRON-1045:
> Although I have been able to start Metron once, for any subsequent launches I
> get an error complaining the table 'enrichment' already exists in the hdfs
> store
How exactly are you launching? What command are you running? What button are
you clicking?
> [Metron enrichment] Cannot restart service - enrichment table already exists
>
>
> Key: METRON-1045
> URL: https://issues.apache.org/jira/browse/METRON-1045
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.0
> Environment: CentOS 7
> Metron 0.4.0
> Ambari 2.5.1.10
>Reporter: Rob Phipps
> Labels: ambari, enrichment, hdfs
>
> Hi,
> This is a new install on Ambari, downloaded from the hortonworks repo if I
> remember correctly. Although I have been able to start Metron once, for any
> subsequent launches I get an error complaining the table 'enrichment' already
> exists in the hdfs store. If I delete the table from the command line and try
> again, then the startup process runs properly. Why is this script assuming
> that the database is clean before it starts?
> Although this is potentially unrelated, the Metron REST API module doesn't
> stay running for very long after I start it, which means I am unable to log
> into the management interface.
> Any assistance would be much appreciated.
> {quote}Execution of '/usr/metron/0.4.0/bin/start_enrichment_topology.sh
> -s enrichment
> -z metron:2181,hadoop-slave-1:2181,hadoop-master:2181' returned 1. Running:
> /usr/jdk64/jdk1.8.0_112/bin/java -server -Ddaemon.name= -Dstorm.options=
> -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
> -cp
> /usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar
> org.apache.storm.daemon.ClientJarTransformerRunner
> org.apache.storm.hack.StormShadeTransformer
> /usr/metron/0.4.0/lib/metron-enrichment-0.4.0-uber.jar
> /tmp/b70910e6688d11e78398000c2953f5b1.jar
> Running: /usr/jdk64/jdk1.8.0_112/bin/java -Ddaemon.name= -Dstorm.options=
> -Dstorm.home=/usr/hdp/2.6.1.0-129/storm -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib
> -Dstorm.conf.file= -cp
> /usr/hdp/2.6.1.0-129/storm/lib/asm-5.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-api-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-core-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.6.1.0-129/storm/lib/log4j-slf4j-impl-2.8.2.jar:/usr/hdp/2.6.1.0-129/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.6.1.0-129/storm/lib/objenesis-2.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.6.1.0-129/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.6.1.0-129/storm/lib/slf4j-api-1.7.21.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-core-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/storm-rename-hack-1.1.0.2.6.1.0-129.jar:/usr/hdp/2.6.1.0-129/storm/lib/zookeeper.jar:/usr/hdp/2.6.1.0-129/storm/lib/ambari-metrics-storm-sink.jar:/tmp/b70910e6688d11e78398000c2953f5b1.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/2.6.1.0-129/storm/bin
> -Dstorm.jar=/tmp/b70910e6688d11e78398000c2953f5b1.jar
> -Dstorm.dependency.jars= -Dstorm.dependency.artifacts={}
> org.apache.storm.flux.Flux --remote
> /usr/metron/0.4.0/flux/enrichment/remote.yaml --filter
> /usr/metron/0.4.0/config/enrichment.properties{quote}
>
> {quote}Exception in thread "main" java
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087373#comment-16087373
]
ASF GitHub Bot commented on METRON-1031:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127466486
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws
RestException {
protected int runCommand(String[] command) throws RestException {
ProcessBuilder pb = getProcessBuilder(command);
pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
try {
process = pb.start();
process.waitFor();
+
} catch (Exception e) {
throw new RestException(e);
}
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ",
command), exitValue);
+
+return exitValue;
}
protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
command[0] =
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
command[1] = "-k";
command[2] =
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
command[3] = "-z";
command[4] =
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
command[5] = "-s";
command[6] = name;
+command[7] = "-ksp";
+command[8] =
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
return command;
}
protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --
I don't think the -ksp flag is needed for the enrichment or indexing
topologies. This setting comes from enrichment.properties and
elasticsearch.properties.
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087387#comment-16087387
]
ASF GitHub Bot commented on METRON-1031:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127469430
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws
RestException {
protected int runCommand(String[] command) throws RestException {
ProcessBuilder pb = getProcessBuilder(command);
pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
try {
process = pb.start();
process.waitFor();
+
} catch (Exception e) {
throw new RestException(e);
}
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ",
command), exitValue);
+
+return exitValue;
}
protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
command[0] =
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
command[1] = "-k";
command[2] =
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
command[3] = "-z";
command[4] =
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
command[5] = "-s";
command[6] = name;
+command[7] = "-ksp";
+command[8] =
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
return command;
}
protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --
Makes sense now. I will remove -ksp from both Enrichment and Indexing
start commands. Thanks!
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087403#comment-16087403
]
ASF GitHub Bot commented on METRON-1031:
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127472746
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws
RestException {
protected int runCommand(String[] command) throws RestException {
ProcessBuilder pb = getProcessBuilder(command);
pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
try {
process = pb.start();
process.waitFor();
+
} catch (Exception e) {
throw new RestException(e);
}
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ",
command), exitValue);
+
+return exitValue;
}
protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
command[0] =
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
command[1] = "-k";
command[2] =
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
command[3] = "-z";
command[4] =
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
command[5] = "-s";
command[6] = name;
+command[7] = "-ksp";
+command[8] =
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
return command;
}
protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --
Has this changed since 0.4.0 RC1? Didn't work when I tried it until ksp
added in the start script.
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087406#comment-16087406
]
ASF GitHub Bot commented on METRON-1031:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127473184
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws
RestException {
protected int runCommand(String[] command) throws RestException {
ProcessBuilder pb = getProcessBuilder(command);
pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
try {
process = pb.start();
process.waitFor();
+
} catch (Exception e) {
throw new RestException(e);
}
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ",
command), exitValue);
+
+return exitValue;
}
protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
command[0] =
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
command[1] = "-k";
command[2] =
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
command[3] = "-z";
command[4] =
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
command[5] = "-s";
command[6] = name;
+command[7] = "-ksp";
+command[8] =
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
return command;
}
protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --
No it's always been like that. The parser topology start script is the
only one that supports the ksp flag. The enrichment and elasticsearch topology
start scripts don't expect any input parameters.
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087416#comment-16087416
]
ASF GitHub Bot commented on METRON-1031:
Github user simonellistonball commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127474849
--- Diff:
metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java
---
@@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws
RestException {
protected int runCommand(String[] command) throws RestException {
ProcessBuilder pb = getProcessBuilder(command);
pb.inheritIO();
-Process process = null;
+LOG.debug("Running command: cmd={}", String.join(" ", command));
+
+Process process;
try {
process = pb.start();
process.waitFor();
+
} catch (Exception e) {
throw new RestException(e);
}
-return process.exitValue();
+
+int exitValue = process.exitValue();
+LOG.debug("Command completed: cmd={}, exit={}", String.join(" ",
command), exitValue);
+
+return exitValue;
}
protected String[] getParserStartCommand(String name) {
-String[] command = new String[7];
+String[] command = new String[9];
command[0] =
environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY);
command[1] = "-k";
command[2] =
environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY);
command[3] = "-z";
command[4] =
environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY);
command[5] = "-s";
command[6] = name;
+command[7] = "-ksp";
+command[8] =
environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY);
return command;
}
protected String[] getEnrichmentStartCommand() {
-String[] command = new String[1];
+String[] command = new String[3];
--- End diff --
right, my bad, it was just parser script I had to change
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1018) Integration tests should reference flux yaml and property files deployed by Ambari
[
https://issues.apache.org/jira/browse/METRON-1018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087485#comment-16087485
]
ASF GitHub Bot commented on METRON-1018:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/635#discussion_r127484425
--- Diff:
metron-platform/metron-elasticsearch/src/test/java/org/apache/metron/elasticsearch/integration/ElasticsearchIndexingIntegrationTest.java
---
@@ -102,11 +102,16 @@ public void setAdditionalProperties(Properties
topologyProperties) {
topologyProperties.setProperty("es.clustername", "metron");
topologyProperties.setProperty("es.port", "9300");
topologyProperties.setProperty("es.ip", "localhost");
-topologyProperties.setProperty("indexing.writer.class.name",
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
+topologyProperties.setProperty("indexing_writer_class_name",
"org.apache.metron.elasticsearch.writer.ElasticsearchWriter");
}
@Override
public String cleanField(String field) {
return field;
}
--- End diff --
I do agree it does seem like a violation of separation of concerns, but to
be honest, I'm not terribly concerned about it in this case.
The copy-resources seems reasonable, if it's not too much work, but given
this particular problem, I'm mostly content with just fixing it.
> Integration tests should reference flux yaml and property files deployed by
> Ambari
> --
>
> Key: METRON-1018
> URL: https://issues.apache.org/jira/browse/METRON-1018
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> This is a follow-up to METRON-990. During the review of that Jira it was
> discovered that our integration tests are not referencing the actual flux
> yaml and property files that are used in a Metron installation. This means
> the integration tests must also be maintained separately with no automated
> protection against regression in cases where flux yaml or property files
> change.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1046) STELLAR: Configurations should be able to reference Stellar File
[ https://issues.apache.org/jira/browse/METRON-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087636#comment-16087636 ] Matt Foley commented on METRON-1046: This is a good idea. I see it as related to METRON-987, which was the first step in allowing sequences of Stellar statements (aka "programs" :-) ) instead of just unrelated groups of single statements. Your proposal lets us really work with programs as first-class entities. However, some concerns need to be resolved: 1. Syntax. Currently Stellar syntax and JSON fit neatly together. Where would be the cut line for file substitutions? Referencing METRON-987, would you only allow a file substitution where we currently allow square-bracketed Stellar string sequences? What about Profile config syntax, where several chunks of code are intimately related (hence want to be located in the same file), but don't all get executed at the same time? (This is not a showstopper question because Profile configs are usually simple and don't really need file substitution. The need is much greater in Enrichment.) 2. Config Updates. Currently Metron configuration is stored in ZK, but managed through Curator libraries. In return for considerable complexity, this gives instant update whenever a config changes, without effort in the BI part of the application. This differs sharply from file-based configuration, where updates in response to config changes require either a restart, an explicit reload command from the user, or frequent state-checking in the application. So currently people trying to develop a new enrichment can update the config, and immediately test the result, without restarting and without any explicit reload command. We probably want to not lose this. Rather than roll our own file pointer model, can we use JSON Pointers? Will they work with Curator? Both of those get into some fairly obscure features, that would need to be studied. It also actually relates to the syntax question presented above. > STELLAR: Configurations should be able to reference Stellar File > - > > Key: METRON-1046 > URL: https://issues.apache.org/jira/browse/METRON-1046 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Otto Fowler > > As we get more and more capability in Stellar, we also get more complexity. > Managing and testing complex stellar statements only by running things > through the parsers and enrichment is not idea. > An improvement would be to allow configurations ( enrichment and parser ) to > be able to reference a Stellar 'file', which perhaps can be shared between > parsers. > This File should should be loaded into a class which presents a clean > interface into multiple possible stellar executions, operating on the same > message map, returning that map ( or modifying it ). > These files, or what ever we call the abstraction should be testable on their > own as well. > The files and their 'updates' can be managed in the same way that the updates > to their parent configurations are. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087750#comment-16087750 ] ASF GitHub Bot commented on METRON-1005: Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/622 Your proposal has the advantage of making data in HBase self-identifying (if one has the key), which I always like. However, it's a large change and induces yet more complexity. There's an alternative I've been noodling occasionally, which I put forward here for consideration: Create a Profile Audit Log table in HBase. Every time a Profiler is configured, started, or stopped, make one entry in the audit log. The idea is to be able to answer exactly the kinds of questions posed in METRON-450, so the records should include things like the configuration, the first and last timestamps, and perhaps the key builder parameters. This would prevent historical profiles from being "lost" because the would-be querier doesn't have access to the exact config parameters used to write the profile. For the sake of housekeeping, one might do a scan, daily and/or at system restart, to assure that (a) the set Profiles with a "start" but not an "end" recorded in the audit log, and (b) the set of currently running Profiles, are actually consistent, and record "inferred end" entries in the audit log for orphans found. This solution is somewhat backward-applicable to existing Profile data; I think there are brute-force ways to scan the existing HBase tables and infer audit log entries, especially if historical configuration data is still available. We could write such a scanner. > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-450) Profiler Client - Add Queries About Set of Profiles
[ https://issues.apache.org/jira/browse/METRON-450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087763#comment-16087763 ] Matt Foley commented on METRON-450: --- [~nickwallen], it should be an explicit goal to prevent historical profiles from being "lost" because the would-be querier doesn't have access to the exact config parameters used to write the profile. I think being able to answer the above questions will indeed satisfy this goal, but I wanted to make it explicit. > Profiler Client - Add Queries About Set of Profiles > --- > > Key: METRON-450 > URL: https://issues.apache.org/jira/browse/METRON-450 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen > Labels: profiler > > Once I have used the Profiler for a while and have a collection of Profiles > created, there are some obvious questions that I am going to need answered. > * What profiles do I currently have data for? > * What entities do I have data for in profile X? > * How far back do I have data for profile X? > * How far back do I have data for profile X, entity Y? > * What is the period duration for profile X? -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1044) Disabled writers are not acking messages
[
https://issues.apache.org/jira/browse/METRON-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087795#comment-16087795
]
ASF GitHub Bot commented on METRON-1044:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/654#discussion_r127526102
--- Diff:
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch()
throws Exception {
}
@Test
+ public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws
Exception {
--- End diff --
It's not a huge deal, but is it possible to write a test where one writer
is enabled and one is disabled? I'm mostly concerned about making sure
everything is spelled out, to avoid any regressions if anything in here gets
refactored.
> Disabled writers are not acking messages
>
>
> Key: METRON-1044
> URL: https://issues.apache.org/jira/browse/METRON-1044
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a writer in the indexing topology is disabled, messages are not acked
> causing the spout to fail them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1044) Disabled writers are not acking messages
[
https://issues.apache.org/jira/browse/METRON-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087808#comment-16087808
]
ASF GitHub Bot commented on METRON-1044:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/654#discussion_r127527166
--- Diff:
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch()
throws Exception {
}
@Test
+ public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws
Exception {
--- End diff --
Do you mean write a test where a writer for one sensor is enabled and a
writer for another sensor is disabled? Or do you really want separate writers?
> Disabled writers are not acking messages
>
>
> Key: METRON-1044
> URL: https://issues.apache.org/jira/browse/METRON-1044
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a writer in the indexing topology is disabled, messages are not acked
> causing the spout to fail them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1044) Disabled writers are not acking messages
[
https://issues.apache.org/jira/browse/METRON-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087812#comment-16087812
]
ASF GitHub Bot commented on METRON-1044:
Github user justinleet commented on a diff in the pull request:
https://github.com/apache/metron/pull/654#discussion_r127527841
--- Diff:
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch()
throws Exception {
}
@Test
+ public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws
Exception {
--- End diff --
The first one (writer for one sensor is enabled and writer for another
sensor is disabled). Should have been more clear.
> Disabled writers are not acking messages
>
>
> Key: METRON-1044
> URL: https://issues.apache.org/jira/browse/METRON-1044
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a writer in the indexing topology is disabled, messages are not acked
> causing the spout to fail them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Created] (METRON-1047) REST should use core-site.xml for Hadoop configuration
Ryan Merriman created METRON-1047: - Summary: REST should use core-site.xml for Hadoop configuration Key: METRON-1047 URL: https://issues.apache.org/jira/browse/METRON-1047 Project: Metron Issue Type: Bug Reporter: Ryan Merriman Currently the REST application depends on a "hdfs.namenode.url" property for HDFS configuration. Instead, it should use core-site.xml for this property if possible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[ https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087858#comment-16087858 ] ASF GitHub Bot commented on METRON-1031: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/647 - I removed `-ksp` from Enrichment and Indexing. - I added `-e` also to the Parsers. This is also required to get them running. - Updated the test plan. I ran through a full test once successfully, but I changed a few minor things and just need to run through the manual test once more. > Management UI Cannot Start Topologies in Kerberized Environment > --- > > Key: METRON-1031 > URL: https://issues.apache.org/jira/browse/METRON-1031 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Assignee: Nick Allen > > The Metron Management UI cannot start topologies in a kerberized environment. > This includes the parser, indexing, and enrichment topologies. > It seems that Metron REST does not pass "-ksp" to the start topology > commands. As a result, topologies that are started with the Management UI > cannot start a producer against a kerberized Kafka. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1044) Disabled writers are not acking messages
[
https://issues.apache.org/jira/browse/METRON-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087857#comment-16087857
]
ASF GitHub Bot commented on METRON-1044:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/654#discussion_r127531933
--- Diff:
metron-platform/metron-writer/src/test/java/org/apache/metron/writer/BulkWriterComponentTest.java
---
@@ -118,6 +118,20 @@ public void writeShouldProperlyAckTuplesInBatch()
throws Exception {
}
@Test
+ public void writeShouldProperlyAckTuplesWhenWriterDisabled() throws
Exception {
--- End diff --
I combined the test for a disabled sensor into the previous test for
enabled sensors. Is this ok? Otherwise we'll have 2 tests that look almost
identical.
> Disabled writers are not acking messages
>
>
> Key: METRON-1044
> URL: https://issues.apache.org/jira/browse/METRON-1044
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>
> When a writer in the indexing topology is disabled, messages are not acked
> causing the spout to fail them.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087907#comment-16087907
]
ASF GitHub Bot commented on METRON-1031:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127534192
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
---
@@ -37,3 +37,5 @@ SECURITY_ENABLED={{security_enabled|lower}}
{% endif %}
{% if metron_keytab_path is defined
%}METRON_SERVICE_KEYTAB="{{metron_keytab_path}}"
{% endif %}
+KAFKA_SECURITY_PROTOCOL="{{kafka_security_protocol}}"
+PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config"
--- End diff --
I did not find any existing property that defines the
`PARSER_TOPOLOGY_OPTIONS` value. It is hard coded in one of the MPack Python
scripts (`parser_commands.py`) as `' -e ~' + self.__params.metron_user +
'/.storm/storm.config'`.
I thought that if I just put the exact value here, we are half way to
making it configurable in Ambari, if we choose to do so.
I also tried to use `~${METRON_HOME}/.storm/storm.config`, but the
`ProcessBuilder` does not do tilde expansion for user's home directories.
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-1044) Disabled writers are not acking messages
[ https://issues.apache.org/jira/browse/METRON-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087955#comment-16087955 ] ASF GitHub Bot commented on METRON-1044: Github user merrimanr closed the pull request at: https://github.com/apache/metron/pull/654 > Disabled writers are not acking messages > > > Key: METRON-1044 > URL: https://issues.apache.org/jira/browse/METRON-1044 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > When a writer in the indexing topology is disabled, messages are not acked > causing the spout to fail them. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1044) Disabled writers are not acking messages
[ https://issues.apache.org/jira/browse/METRON-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087956#comment-16087956 ] ASF GitHub Bot commented on METRON-1044: GitHub user merrimanr reopened a pull request: https://github.com/apache/metron/pull/654 METRON-1044: Disabled writers are not acking messages ## Contributor Comments The PR fixes a bug in the indexing topology where a disabled writer will cause messages to fail. This happens because the BulkWriterComponent simply returns (line 118) when it should ack the message before returning. This has been tested on full dev along with unit and integration tests. To verify, spin up full dev and disable the hdfs writer for bro. The indexing topology should now continue to function normally instead of the spout reporting failures. I also added a test case to ensure tuples are still acked when a writer is disabled. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/merrimanr/incubator-metron METRON-1044 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/654.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #654 commit 7fdf333bc5cbeddbc24ceda20df8922169ee418f Author: merrimanr Date: 2017-07-13T21:37:34Z initial commit commit 426b41501c01a91fac97e126d6122737e4b49124 Author: merrimanr Date: 2017-07-14T19:27:07Z combined enabled and disabled sensors into the same test > Disabled writers are not acking messages > > > Key: METRON-1044 > URL: https://issues.apache.org/jira/browse/METRON-1044 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman > > When a writer in the indexing topology is disabled, messages are not acked > causing the spout to fail them. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1031) Management UI Cannot Start Topologies in Kerberized Environment
[
https://issues.apache.org/jira/browse/METRON-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16088154#comment-16088154
]
ASF GitHub Bot commented on METRON-1031:
Github user merrimanr commented on a diff in the pull request:
https://github.com/apache/metron/pull/647#discussion_r127555429
--- Diff:
metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2
---
@@ -37,3 +37,5 @@ SECURITY_ENABLED={{security_enabled|lower}}
{% endif %}
{% if metron_keytab_path is defined
%}METRON_SERVICE_KEYTAB="{{metron_keytab_path}}"
{% endif %}
+KAFKA_SECURITY_PROTOCOL="{{kafka_security_protocol}}"
+PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config"
--- End diff --
I think that's fine. If you really wanted it to mirror the implementation
in the MPack, you could do:
`PARSER_TOPOLOGY_OPTIONS="/home/{{metron_user}}/.storm/storm.config"`
> Management UI Cannot Start Topologies in Kerberized Environment
> ---
>
> Key: METRON-1031
> URL: https://issues.apache.org/jira/browse/METRON-1031
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>
> The Metron Management UI cannot start topologies in a kerberized environment.
> This includes the parser, indexing, and enrichment topologies.
> It seems that Metron REST does not pass "-ksp" to the start topology
> commands. As a result, topologies that are started with the Management UI
> cannot start a producer against a kerberized Kafka.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
