[jira] [Assigned] (METRON-799) The MPack should function in a kerberized cluster
[ https://issues.apache.org/jira/browse/METRON-799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-799: -- Assignee: Justin Leet > The MPack should function in a kerberized cluster > - > > Key: METRON-799 > URL: https://issues.apache.org/jira/browse/METRON-799 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Justin Leet > Labels: kerberos > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-818) Ambari elasticsearch.properties template is missing topology.worker.childopts
Justin Leet created METRON-818:
--
Summary: Ambari elasticsearch.properties template is missing
topology.worker.childopts
Key: METRON-818
URL: https://issues.apache.org/jira/browse/METRON-818
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
Assignee: Justin Leet
Indexing fails on Ambari deploys, because indexing topology can't resolve a
property
{code}
Error: Could not find or load main class ${topology.worker.childopts}
{code}
"topology.worker.childopts=" needs to be added to metron-env.xml. An empty
value is fine (it's only set to nonempty for Kerberos)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-817) Customise output file path patterns for HDFS indexing
Justin Leet created METRON-817:
--
Summary: Customise output file path patterns for HDFS indexing
Key: METRON-817
URL: https://issues.apache.org/jira/browse/METRON-817
Project: Metron
Issue Type: Improvement
Reporter: Justin Leet
Assignee: Justin Leet
We need to be able to customize the filepaths for HDFS indexing, to allow
fields to be part of the naming. For example, if I have a 'tenant' field, I
should be able to direct it to a path
{code}
/apps/metron/.../{tenant}/{sensor}/{date}/filename-34324-432434.json
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Assigned] (METRON-617) Eliminate Javac warnings in the build
[
https://issues.apache.org/jira/browse/METRON-617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet reassigned METRON-617:
--
Assignee: (was: Justin Leet)
> Eliminate Javac warnings in the build
> -
>
> Key: METRON-617
> URL: https://issues.apache.org/jira/browse/METRON-617
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Priority: Minor
>
> We have a lot of Javac warnings (several hundred) in the build right now. The
> primary cause of these is unchecked types (for reasons including
> org.simple.json uses raw types under the hood). These should be labelled
> with @SuppressWarnings("unchecked") where appropriate. In addition, things
> like @SafeVarargs should be added where appropriate, etc.
> Given the number of changes in this ticket is significantly larger than even
> the Error Prone warnings (and may require tinkering with generics at
> appropriate points), I'm going to split this up by module (Analytics and
> Platform), and then split platform as necessary (parsers and enrichment in
> particular have a lot). Otherwise, anything resulting is likely to be too
> large and unmanageable to be properly tested and reviewed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-349) Switch ownership of topologies and files to metron user and update perms
[ https://issues.apache.org/jira/browse/METRON-349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-349: --- Description: Metron services are typically run under the storm user (e.g. spinning up topologies). The mpack deploy creates a Metron user and group. This install should be updated to be running and deploying as the metron user. In addition, many of the files are created or owned by users like the storm user (e.g. in HDFS). These files should also be owned by the metron user, and permissions restricted from 775. Notably, METRON-796 resulted from a partial fix to this. This ticket is the more complete solution to the ownership problem (796 is intended only to get things back in working order, and will actually revert ownership from metron:metron to metron:hadoop to allow storm user to write) was:Currently, Metron services are run under the root user- change this to run under a Metron user. Summary: Switch ownership of topologies and files to metron user and update perms (was: Switch Metron User from root to metron) Updated this ticket to be more complete per discussion on: https://github.com/apache/incubator-metron/pull/488 The original intent is still valid, but a bit outdated and incomplete, so this ticket title and description is updated appropriately. > Switch ownership of topologies and files to metron user and update perms > > > Key: METRON-349 > URL: https://issues.apache.org/jira/browse/METRON-349 > Project: Metron > Issue Type: Improvement >Reporter: David M. Lyle > Labels: deployment, platform > > Metron services are typically run under the storm user (e.g. spinning up > topologies). The mpack deploy creates a Metron user and group. This install > should be updated to be running and deploying as the metron user. > In addition, many of the files are created or owned by users like the storm > user (e.g. in HDFS). These files should also be owned by the metron user, > and permissions restricted from 775. > Notably, METRON-796 resulted from a partial fix to this. This ticket is the > more complete solution to the ownership problem (796 is intended only to get > things back in working order, and will actually revert ownership from > metron:metron to metron:hadoop to allow storm user to write) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-796) Mpack uses wrong group for owning HDFS directories
Justin Leet created METRON-796: -- Summary: Mpack uses wrong group for owning HDFS directories Key: METRON-796 URL: https://issues.apache.org/jira/browse/METRON-796 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet org.apache.hadoop.security.AccessControlException: Permission denied: user=storm, access=WRITE, inode="/apps/metron/indexing/indexed/snort/enrichment-null-0-0-1490305873514.json":metron:metron:drwxrwx The group got changed a bit ago from cluster_env.user_group (hadoop) to cluster_env.metron_group (metron). However, because everything right now runs as the storm user (which is in the hadoop group), it doesn't have perms to write anymore. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-791) Add links to website and downloads to top level POM
Justin Leet created METRON-791: -- Summary: Add links to website and downloads to top level POM Key: METRON-791 URL: https://issues.apache.org/jira/browse/METRON-791 Project: Metron Issue Type: Improvement Affects Versions: 0.3.1 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Right now our GitHub landing page doesn't link back to the Apache page. Both the main link, and the link to our releases section should be added. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (METRON-773) Intermittent unit test errors in the KafkaControllerIntegrationTest
[ https://issues.apache.org/jira/browse/METRON-773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15929984#comment-15929984 ] Justin Leet commented on METRON-773: I've also seen this in the fixes for the site build, intermittently. https://travis-ci.org/justinleet/incubator-metron/builds/209355250 > Intermittent unit test errors in the KafkaControllerIntegrationTest > --- > > Key: METRON-773 > URL: https://issues.apache.org/jira/browse/METRON-773 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella > > Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.514 sec <<< > FAILURE! - in org.apache.metron.rest.controller.KafkaControllerIntegrationTest > test(org.apache.metron.rest.controller.KafkaControllerIntegrationTest) Time > elapsed: 8.415 sec <<< FAILURE! > java.lang.AssertionError: Status expected:<200> but was:<404> > at > org.springframework.test.util.AssertionErrors.fail(AssertionErrors.java:54) > at > org.springframework.test.util.AssertionErrors.assertEquals(AssertionErrors.java:81) > at > org.springframework.test.web.servlet.result.StatusResultMatchers$10.match(StatusResultMatchers.java:664) > at > org.springframework.test.web.servlet.MockMvc$1.andExpect(MockMvc.java:171) > at > org.apache.metron.rest.controller.KafkaControllerIntegrationTest.test(KafkaControllerIntegrationTest.java:173) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:483) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at > org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86) > at > org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:252) > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:94) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at > org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61) > at > org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:191) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) > at > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-746) Build Custom Checkstyle and IDE formatting settings
[ https://issues.apache.org/jira/browse/METRON-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-746: --- Description: We need a custom checkstyle.xml based off the sun convention checkstyle. Based on a discussion thread, there are a few things that need to be setup * Two space indents * Line Length longer than 80 (pretty popular in the discussion, but not officially part of our code style) * Appropriate warn/error levels set so we don't immediately start failing every build. * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse if possible to avoid forcing a given dev environment. IntelliJ allows for importing a checkstyle.xml to use as the basis. We can export the resulting formatting settings for people. * Ensure Travis actually runs checkstyle during our builds was: We need a custom checkstyle.xml based off the sun convention checkstyle. Based on a discussion thread, there are a few things that need to be setup * Two space indents * Line Length longer than 80 (pretty popular in the discussion, but not officially part of our code style) * Appropriate warn/error levels set so we don't immediately start failing every build. * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse if possible to avoid forcing a given dev environment. IntelliJ allows for importing a checkstyle.xml to use as the basis. We can export the resulting formatting settings for people. > Build Custom Checkstyle and IDE formatting settings > --- > > Key: METRON-746 > URL: https://issues.apache.org/jira/browse/METRON-746 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Minor > > We need a custom checkstyle.xml based off the sun convention checkstyle. > Based on a discussion thread, there are a few things that need to be setup > * Two space indents > * Line Length longer than 80 (pretty popular in the discussion, but not > officially part of our code style) > * Appropriate warn/error levels set so we don't immediately start failing > every build. > * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse > if possible to avoid forcing a given dev environment. IntelliJ allows for > importing a checkstyle.xml to use as the basis. We can export the resulting > formatting settings for people. > * Ensure Travis actually runs checkstyle during our builds -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (METRON-746) Build Custom Checkstyle and IDE formatting settings
Justin Leet created METRON-746: -- Summary: Build Custom Checkstyle and IDE formatting settings Key: METRON-746 URL: https://issues.apache.org/jira/browse/METRON-746 Project: Metron Issue Type: Improvement Reporter: Justin Leet Priority: Minor We need a custom checkstyle.xml based off the sun convention checkstyle. Based on a discussion thread, there are a few things that need to be setup * Two space indents * Line Length longer than 80 (pretty popular in the discussion, but not officially part of our code style) * Appropriate warn/error levels set so we don't immediately start failing every build. * Importable IntelliJ code style at minimum, but I'd also like to see Eclipse if possible to avoid forcing a given dev environment. IntelliJ allows for importing a checkstyle.xml to use as the basis. We can export the resulting formatting settings for people. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (METRON-725) Javadoc is broken by the use of apiNote
[
https://issues.apache.org/jira/browse/METRON-725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-725:
---
Fix Version/s: 0.3.1
> Javadoc is broken by the use of apiNote
> ---
>
> Key: METRON-725
> URL: https://issues.apache.org/jira/browse/METRON-725
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
> Fix For: 0.3.1
>
>
> Error seen:
> {code}
> >mvn javadoc:javadoc
> ...
> [ERROR]
> /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java:127:
> error: unknown tag: apiNote
> ...
> {code}
> {{@apiNote}} doesn't work by default when generating Javadocs. Apparently,
> it's intended to be language level information rather than a widely adopted
> tag.
> This only shows up in ReaderSpliterator, in docs copied directly from the
> language construct. Given that all these methods are {{@Override}}, it seems
> reasonable to just drop the docs entirely, given that they inherit anyway.
> If desired, we could explicitly inherit the parent docs.
> Finally, we could enable the use of {{@apiNote}}, but given the intended
> usage and our copied use, I'm inclined to just change our code directly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-734) Builds failing because of MaxMind DB transitive dependency
[
https://issues.apache.org/jira/browse/METRON-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-734:
---
Fix Version/s: 0.3.1
> Builds failing because of MaxMind DB transitive dependency
> --
>
> Key: METRON-734
> URL: https://issues.apache.org/jira/browse/METRON-734
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.3.1
>
>
> Original Error
> {code}
> [ERROR] Failed to execute goal on project metron-enrichment: Could not
> resolve dependencies for project
> org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies
> at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1
> -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to
> read artifact descriptor for
> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to
> find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in
> the local repository, resolution will not be reattempted until the update
> interval of clojars.org has elapsed or updates are forced -> [Help 1]
> {code}
> Discussion on the lib
> https://github.com/maxmind/GeoIP2-java/issues/77
> The geoip2 lib from MaxMind has a transitive dependency on open ended version
> range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT,
> rather than a distinct version.
> The easy answer is to just exclude jackson from being pulled in by geoip2
> lib, and use our own. The catch is that the version geoip declares in its
> POM is higher than the version we use and I don't know if that causes any
> problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I
> haven't tested it).
> If there's a way to skip snapshots that might also contribute to solving our
> problem, but I still don't like the unrepeatability of the build if they
> depend on an open ended range.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (METRON-734) Builds failing because of MaxMind DB transitive dependency
[
https://issues.apache.org/jira/browse/METRON-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-734:
---
Description:
Original Error
{code}
[ERROR] Failed to execute goal on project metron-enrichment: Could not resolve
dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed
to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 ->
com.maxmind.db:maxmind-db:jar:1.2.1 ->
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read
artifact descriptor for
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to find
com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in the
local repository, resolution will not be reattempted until the update interval
of clojars.org has elapsed or updates are forced -> [Help 1]
{code}
Discussion on the lib
https://github.com/maxmind/GeoIP2-java/issues/77
The geoip2 lib from MaxMind has a transitive dependency on open ended version
range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, rather
than a distinct version.
The easy answer is to just exclude jackson from being pulled in by geoip2 lib,
and use our own. The catch is that the version geoip declares in its POM is
higher than the version we use and I don't know if that causes any problems
(I'm guessing not, since it's going from 2.7.x to 2.8.x, but I haven't tested
it).
If there's a way to skip snapshots that might also contribute to solving our
problem, but I still don't like the unrepeatability of the build if they depend
on an open ended range.
was:
Original Error
{code}
[ERROR] Failed to execute goal on project metron-enrichment: Could not resolve
dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed
to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 ->
com.maxmind.db:maxmind-db:jar:1.2.1 ->
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read
artifact descriptor for
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to find
com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in the
local repository, resolution will not be reattempted until the update interval
of clojars.org has elapsed or updates are forced -> [Help 1]
{code}
The geoip2 lib from MaxMind has a transitive dependency on open ended version
range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, rather
than a distinct version.
The easy answer is to just exclude jackson from being pulled in by geoip2 lib,
and use our own. The catch is that the version geoip declares in its POM is
higher than the version we use and I don't know if that causes any problems
(I'm guessing not, since it's going from 2.7.x to 2.8.x, but I haven't tested
it).
If there's a way to skip snapshots that might also contribute to solving our
problem, but I still don't like the unrepeatability of the build if they depend
on an open ended range.
> Builds failing because of MaxMind DB transitive dependency
> --
>
> Key: METRON-734
> URL: https://issues.apache.org/jira/browse/METRON-734
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> Original Error
> {code}
> [ERROR] Failed to execute goal on project metron-enrichment: Could not
> resolve dependencies for project
> org.apache.metron:metron-enrichment:jar:0.3.1: Failed to collect dependencies
> at com.maxmind.geoip2:geoip2:jar:2.8.0 -> com.maxmind.db:maxmind-db:jar:1.2.1
> -> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to
> read artifact descriptor for
> com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to
> find com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in
> the local repository, resolution will not be reattempted until the update
> interval of clojars.org has elapsed or updates are forced -> [Help 1]
> {code}
> Discussion on the lib
> https://github.com/maxmind/GeoIP2-java/issues/77
> The geoip2 lib from MaxMind has a transitive dependency on open ended version
> range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT,
> rather than a distinct version.
> The easy answer is to just exclude jackson from being pulled in by geoip2
> lib, and use our own. The catch is that the version geoip declares in its
> POM is higher than the version we use and I don't know if that causes any
> problems (I'm guessing not, since it's going from 2.7.x to 2.8.x, but I
> haven't tested it).
> If there's a way to skip snapshots that might also contribute to solving our
> problem, but I still don't like the unrepeatability of the build if they
> depend on an open ended range.
--
This message was sent by Atlassian JIRA
[jira] [Created] (METRON-734) Builds failing because of MaxMind DB transitive dependency
Justin Leet created METRON-734:
--
Summary: Builds failing because of MaxMind DB transitive dependency
Key: METRON-734
URL: https://issues.apache.org/jira/browse/METRON-734
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
Assignee: Justin Leet
Original Error
{code}
[ERROR] Failed to execute goal on project metron-enrichment: Could not resolve
dependencies for project org.apache.metron:metron-enrichment:jar:0.3.1: Failed
to collect dependencies at com.maxmind.geoip2:geoip2:jar:2.8.0 ->
com.maxmind.db:maxmind-db:jar:1.2.1 ->
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failed to read
artifact descriptor for
com.fasterxml.jackson.core:jackson-databind:jar:2.9.0-SNAPSHOT: Failure to find
com.fasterxml:oss-parent:pom:28 in http://clojars.org/repo was cached in the
local repository, resolution will not be reattempted until the update interval
of clojars.org has elapsed or updates are forced -> [Help 1]
{code}
The geoip2 lib from MaxMind has a transitive dependency on open ended version
range of jackson-databind. This causes it to try to use 2.9.0-SNAPSHOT, rather
than a distinct version.
The easy answer is to just exclude jackson from being pulled in by geoip2 lib,
and use our own. The catch is that the version geoip declares in its POM is
higher than the version we use and I don't know if that causes any problems
(I'm guessing not, since it's going from 2.7.x to 2.8.x, but I haven't tested
it).
If there's a way to skip snapshots that might also contribute to solving our
problem, but I still don't like the unrepeatability of the build if they depend
on an open ended range.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-733) Remove Geo database from ParserBolt
Justin Leet created METRON-733: -- Summary: Remove Geo database from ParserBolt Key: METRON-733 URL: https://issues.apache.org/jira/browse/METRON-733 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet The ParserBolt inits the Geo DB in its prepare() method. Parsers, unlike enrichments, do not use geo as a base capability. They should be able to run without the DB file existing in HDFS. This init causes issues if the file is missing, even if GEO_GET is unused in the parser definition. This change should preserve the ability of Parsers to employ Stellar's GEO_GET. The Stellar function already handles that init, so it shouldn't be an issue. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (METRON-728) ReaderSpliteratorTest fails randomly and extremely rarely
[
https://issues.apache.org/jira/browse/METRON-728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet reassigned METRON-728:
--
Assignee: Justin Leet
> ReaderSpliteratorTest fails randomly and extremely rarely
> -
>
> Key: METRON-728
> URL: https://issues.apache.org/jira/browse/METRON-728
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.1
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> See logs at
> https://travis-ci.org/justinleet/incubator-metron/builds/203298348
> I was able to reproduce this locally by calling
> {{testActuallyParallel_mediumBatch}} in a {{while(true)}} loop. It can also
> occur in {{testActuallyParallel_mediumBatchImplicitlyParallel()}}. I also
> had to add
> {{forkJoinPool.shutdownNow();}} to the end of the test, because otherwise OOM
> errors occur.
> My current assumption is that there's no guarantee you ever actually end up
> running in parallel, so in extremely rare cases you just end up running one
> thread.
> I've had it vary wildly when I hit it, from within a second or two to running
> for over a minute before an assertion failure occurs.
> We could just alter the assertion to be
> {code}Assert.assertTrue(threads.size() <= (int) Math.ceil(9.0 / 2) &&
> threads.size() >= 1);{code}
> This defeats the purpose of testing the parallelism a bit, but if there's no
> guarantee we actually get parallelism there's not a fantastic way to test it.
> Given the extreme rarity, we might want to just live with the fact that
> occasionally {{threads.size() >= 1}} hits the threads.size() == 1 case on the
> two tests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-728) ReaderSpliteratorTest fails randomly and extremely rarely
Justin Leet created METRON-728:
--
Summary: ReaderSpliteratorTest fails randomly and extremely rarely
Key: METRON-728
URL: https://issues.apache.org/jira/browse/METRON-728
Project: Metron
Issue Type: Bug
Affects Versions: 0.3.1
Reporter: Justin Leet
See logs at
https://travis-ci.org/justinleet/incubator-metron/builds/203298348
I was able to reproduce this locally by calling
{{testActuallyParallel_mediumBatch}} in a {{while(true)}} loop. It can also
occur in {{testActuallyParallel_mediumBatchImplicitlyParallel()}}. I also had
to add
{{forkJoinPool.shutdownNow();}} to the end of the test, because otherwise OOM
errors occur.
My current assumption is that there's no guarantee you ever actually end up
running in parallel, so in extremely rare cases you just end up running one
thread.
I've had it vary wildly when I hit it, from within a second or two to running
for over a minute before an assertion failure occurs.
We could just alter the assertion to be {code}Assert.assertTrue(threads.size()
<= (int) Math.ceil(9.0 / 2) && threads.size() >= 1);{code}
This defeats the purpose of testing the parallelism a bit, but if there's no
guarantee we actually get parallelism there's not a fantastic way to test it.
Given the extreme rarity, we might want to just live with the fact that
occasionally {{threads.size() >= 1}} hits the threads.size() == 1 case on the
two tests.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-726) Clean up mvn site generation
Justin Leet created METRON-726:
--
Summary: Clean up mvn site generation
Key: METRON-726
URL: https://issues.apache.org/jira/browse/METRON-726
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
Right now there's a couple issues with running mvn:site. The most obvious is
that EMMA appears to not work at all, but in attempting to fix that, several
other issues came to light.
Error seen:
{code}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-site-plugin:3.3:site (default-site) on project
metron-maas-common: failed to get report for
org.codehaus.mojo:emma-maven-plugin: Failed to execute goal
org.codehaus.mojo:emma-maven-plugin:1.0-alpha-3:instrument (report:emma) on
project metron-maas-common: Execution report:emma of goal
org.codehaus.mojo:emma-maven-plugin:1.0-alpha-3:instrument failed:
CONSTANT_info: invalid tag value [18] -> [Help 1]
{code}
After commenting out everything EMMA, there are still some issues seen:
{code}
[WARNING] Unable to process class
org/apache/metron/test/converters/BinaryConverters.class in JarAnalyzer File
/Users/jleet/.m2/repository/org/apache/metron/metron-test-utilities/0.3.1/metron-test-utilities-0.3.1.jar
org.apache.bcel.classfile.ClassFormatException: Invalid byte tag in constant
pool: 18
{code}
This seems to be a Java 8 issue, which means that EMMA likely is impossible to
make work. I'm unsure that's the root cause, but given the age of EMMA plus
(the outdated version of) BCEL thowing a very similar issue implies that the
Java version is related. This also implies that our {{mvn site}} hasn't worked
in a long time.
Cleaning this up should include at least
* Getting code coverage working again
* Consolidating reporting in our poms. A lot of it is repeated everywhere we
have Java.
* Ensure we can actually generate and look through the site.
* METRON-725 fixes Javadoc, so reporting will still have issues until that is
taken care of.
* Apparently checkstyle got dropped at some point. It's easy enough to add in,
and can be taken care of here.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-725) Javadoc is broken by the use of apiNote
Justin Leet created METRON-725:
--
Summary: Javadoc is broken by the use of apiNote
Key: METRON-725
URL: https://issues.apache.org/jira/browse/METRON-725
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
Error seen:
{code}
>mvn javadoc:javadoc
...
[ERROR]
/Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/src/main/java/org/apache/metron/common/utils/file/ReaderSpliterator.java:127:
error: unknown tag: apiNote
...
{code}
{{@apiNote}} doesn't work by default when generating Javadocs. Apparently,
it's intended to be language level information rather than a widely adopted tag.
This only shows up in ReaderSpliterator, in docs copied directly from the
language construct. Given that all these methods are {{@Override}}, it seems
reasonable to just drop the docs entirely, given that they inherit anyway.
If desired, we could explicitly inherit the parent docs.
Finally, we could enable the use of {{@apiNote}}, but given the intended usage
and our copied use, I'm inclined to just change our code directly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-711) StellarShell assigns variables even if an exception was thrown in the statement.
Justin Leet created METRON-711:
--
Summary: StellarShell assigns variables even if an exception was
thrown in the statement.
Key: METRON-711
URL: https://issues.apache.org/jira/browse/METRON-711
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
Discovered while reviewing https://github.com/apache/incubator-metron/pull/438.
If an exception is thrown during Stellar execution, the exception will be
logged, and null is returned. The assignment then goes through as normal,
leaving the assigned variable null.
Seen using THREAT_TRIAGE_REMOVE with a String arg, instead of a List. Resulted
in a null conf variable.
{code}
[Stellar]>>> conf := THREAT_TRIAGE_ADD(conf, [triage])
[Stellar]>>> conf := THREAT_TRIAGE_REMOVE(conf, 'Abnormal DNS Port')
[!] Unable to execute: java.lang.String cannot be cast to java.util.List
org.apache.metron.common.dsl.ParseException: Unable to execute:
java.lang.String cannot be cast to java.util.List
at
org.apache.metron.common.stellar.StellarCompiler.getResult(StellarCompiler.java:409)
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:127)
at
org.apache.metron.common.stellar.shell.StellarExecutor.execute(StellarExecutor.java:275)
at
org.apache.metron.common.stellar.shell.StellarShell.executeStellar(StellarShell.java:373)
at
org.apache.metron.common.stellar.shell.StellarShell.handleStellar(StellarShell.java:276)
at
org.apache.metron.common.stellar.shell.StellarShell.execute(StellarShell.java:412)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to
java.util.List
at
org.apache.metron.management.ThreatTriageFunctions$RemoveStellarTransformation.apply(ThreatTriageFunctions.java:232)
at
org.apache.metron.common.stellar.StellarCompiler.exitTransformationFunc(StellarCompiler.java:267)
at
org.apache.metron.common.stellar.generated.StellarParser$TransformationFuncContext.exitRule(StellarParser.java:1689)
at org.antlr.v4.runtime.Parser.triggerExitRuleEvent(Parser.java:422)
at org.antlr.v4.runtime.Parser.exitRule(Parser.java:632)
at
org.apache.metron.common.stellar.generated.StellarParser.functions(StellarParser.java:1712)
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_operands(StellarParser.java:1846)
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr_mul(StellarParser.java:1609)
at
org.apache.metron.common.stellar.generated.StellarParser.arithmetic_expr(StellarParser.java:1469)
at
org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:308)
at
org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:149)
at
org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:126)
... 8 more
[Stellar]>>> conf
[Stellar]>>> conf
[Stellar]>>> conf := THREAT_TRIAGE_REMOVE(conf, ['Abnormal DNS Port'])
[Stellar]>>> conf
{
"enrichment" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { }
},
"threatIntel" : {
"fieldMap" : { },
"fieldToTypeMap" : { },
"config" : { },
"triageConfig" : {
"riskLevelRules" : [ ],
"aggregator" : "MAX",
"aggregationConfig" : { }
}
},
"configuration" : { }
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (METRON-710) Rev MPack Version to 0.3.1.0
Justin Leet created METRON-710: -- Summary: Rev MPack Version to 0.3.1.0 Key: METRON-710 URL: https://issues.apache.org/jira/browse/METRON-710 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet Fix For: 0.3.1 For the release, need to rev mpack version. Dev list has so far agreed on 0.3.1.0 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city We actually use the field directly for the "Unique-Location(s)" visualization in Kibana. How we choose to populate the field changes the meaning of the count, which is where that question comes from. If we're fine counting only the city's geoname_id for that field, the switch to city is a one line change + test fix. The swapped data is in the same format as the current data, so the change to city is something I could have a PR for very quickly. Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title Justin Leet commented on METRON-680 Re: GeoLiteDatabase incorrectly using country geoname_id instead of city The part about including the country's code as the city isn't always true. See: 23.129.1.0/24,,6252001,,0,0 It's the U.S. (http://www.geonames.org/6252001/united-states.html), but doesn't attach any city information to it. Do we need to be doing a fallback of "Use city if available, else use country"? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-680) GeoLiteDatabase incorrectly using country geoname_id instead of city
Title: Message Title
Justin Leet created an issue
Metron / METRON-680
GeoLiteDatabase incorrectly using country geoname_id instead of city
Issue Type:
Bug
Assignee:
Justin Leet
Created:
31/Jan/17 14:12
Priority:
Major
Reporter:
Justin Leet
Due to misunderstanding exactly how things tied together with the updated database, the wrong field is used for the locId. Instead of using the city's geoname_id, we are using the country's.
This will effect Kibana dashboards and anything that depends on the locId, because it will be retrieved at the country level instead of the city level. The change will not break anything (anything not at the city level uses the country's code, e.g. if the IP is for Japan in general, the city code is 1861060, not empty or null). This example from the plaintext database can be seen in the second and third fields at:
1.112.0.0/15,1861060,1861060,,0,0,,35.6900,139.6900,500
The offending code is in `GeoLiteDatabase.java` and should be `geoInfo.put("locID", convertNullToEmptyString(country.getGeoNameId()));`
This should be updated to grab the city's geoname, and tests should be updated to reflect this (they didn't catch this error because of the misunderstood data change, not an error in coding).
Ideally, this field is renamed and better documented as part of METRON-679
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet commented on METRON-679 Re: Investigate including additional GeoIP fields James Sirota You're more familiar with what's potentially useful than I am, any insight here? Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] (METRON-679) Investigate including additional GeoIP fields
Title: Message Title Justin Leet created an issue Metron / METRON-679 Investigate including additional GeoIP fields Issue Type: Improvement Assignee: Unassigned Created: 31/Jan/17 13:56 Priority: Minor Reporter: Justin Leet The newer downloadable database we're using provides a wider array of fields than our old solution. We may also want to update some naming (we've been using metro code and calling it dma code, even in the old solution, for example). Data pulled from plaintext download at: http://dev.maxmind.com/geoip/geoip2/geolite2/ Currently used fields: geoname_id (renaming it LocId). city_name country_iso_code metro_code (but renaming it dma code) postal_code latitude longitude network (indirectly, it's the field used for pulling back data for IP and is
[jira] [Commented] (METRON-674) Archived Telemetry Filenames in HDFS Contain 'Null'
[
https://issues.apache.org/jira/browse/METRON-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15838562#comment-15838562
]
Justin Leet commented on METRON-674:
http://storm.apache.org/releases/1.0.1/storm-hdfs.html
Per the docs, the format is
{code}
{prefix}{componentId}-{taskId}-{rotationNum}-{timestamp}{extension}
{code}
Given a file like:
enrichment-null-0-0-1484752251563.json
The null field is componentId. I'd have to dig into why it's actually null
though.
> Archived Telemetry Filenames in HDFS Contain 'Null'
> ---
>
> Key: METRON-674
> URL: https://issues.apache.org/jira/browse/METRON-674
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Nick Allen
>Priority: Minor
>
> When running "Quick Dev", I have noticed that all of the archived telemetry
> files in HDFS contain null in the name.
> {code}
> [root@node1 0.3.0]# hdfs dfs -ls /apps/metron/indexing/indexed/*
> Found 2 items
> -rw-r--r-- 1 storm hadoop 644753 2017-01-19 17:47
> /apps/metron/indexing/indexed/bro/enrichment-null-0-0-1484847868551.json
> -rw-r--r-- 1 storm hadoop 14107767 2017-01-19 18:46
> /apps/metron/indexing/indexed/bro/enrichment-null-0-0-1484848728527.json
> Found 5 items
> -rwxrwxrwx 1 storm hadoop 205699 2017-01-16 21:57
> /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484603710250.json
> -rwxrwxrwx 1 storm hadoop5773871 2017-01-17 14:34
> /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484603925156.json
> -rwxrwxrwx 1 storm hadoop 253870 2017-01-17 13:43
> /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484660437793.json
> -rwxrwxrwx 1 storm hadoop 24023035 2017-01-17 19:45
> /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484660672723.json
> -rwxrwxrwx 1 storm hadoop2063857 2017-01-17 19:02
> /apps/metron/indexing/indexed/snort/enrichment-null-0-0-1484679265343.json
> Found 147 items
> -rwxrwxrwx 1 storm hadoop 18199681 2017-01-18 16:35
> /apps/metron/indexing/indexed/yaf/enrichment-null-0-0-1484752251563.json
> -rwxrwxrwx 1 storm hadoop 216895 2017-01-19 17:47
> /apps/metron/indexing/indexed/yaf/enrichment-null-0-0-1484846918122.json
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (METRON-283) Migrate Geo Enrichment outside of MySQL
[ https://issues.apache.org/jira/browse/METRON-283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-283: -- Assignee: Justin Leet > Migrate Geo Enrichment outside of MySQL > --- > > Key: METRON-283 > URL: https://issues.apache.org/jira/browse/METRON-283 > Project: Metron > Issue Type: Improvement >Reporter: James Sirota >Assignee: Justin Leet >Priority: Minor > > We need to migrate our enrichment SQL store from MySQL to Phoenix or some > other SQL on Hbase library. Or alternatively come up with a way to do this > without using SQL. This way we don't have a dependency on MySQL and there is > one less thing that we need to install on our platform -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-647) Parser unit test failures due to assumed year values
[ https://issues.apache.org/jira/browse/METRON-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-647: --- Fix Version/s: Next + 1 > Parser unit test failures due to assumed year values > > > Key: METRON-647 > URL: https://issues.apache.org/jira/browse/METRON-647 > Project: Metron > Issue Type: Bug >Reporter: Kyle Richardson >Assignee: Justin Leet >Priority: Blocker > Fix For: Next + 1 > > > Unit Tests Failing: > * BasicAsaParserTest.testIp6Addr:151 > * GrokWebSphereParserTest.testParseLoginLine:60 > * GrokWebSphereParserTest.testParseMalformedLoginLine:151 > * GrokWebSphereParserTest.tetsParseLogoutLine:84 > * GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175 > * GrokWebSphereParserTest.tetsParseMalformedOtherLine:220 > * GrokWebSphereParserTest.tetsParseMalformedRBMLine:198 > * GrokWebSphereParserTest.tetsParseOtherLine:129 > * GrokWebSphereParserTest.tetsParseRBMLine:107 > The issue appears to be the same for all. The original syslog message did not > include a year (fairly common); however, we have hard coded an assertion > based on the year the code was written (in our case 2016). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-647) Parser unit test failures due to assumed year values
[ https://issues.apache.org/jira/browse/METRON-647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-647: -- Assignee: Justin Leet > Parser unit test failures due to assumed year values > > > Key: METRON-647 > URL: https://issues.apache.org/jira/browse/METRON-647 > Project: Metron > Issue Type: Bug >Reporter: Kyle Richardson >Assignee: Justin Leet >Priority: Blocker > > Unit Tests Failing: > * BasicAsaParserTest.testIp6Addr:151 > * GrokWebSphereParserTest.testParseLoginLine:60 > * GrokWebSphereParserTest.testParseMalformedLoginLine:151 > * GrokWebSphereParserTest.tetsParseLogoutLine:84 > * GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175 > * GrokWebSphereParserTest.tetsParseMalformedOtherLine:220 > * GrokWebSphereParserTest.tetsParseMalformedRBMLine:198 > * GrokWebSphereParserTest.tetsParseOtherLine:129 > * GrokWebSphereParserTest.tetsParseRBMLine:107 > The issue appears to be the same for all. The original syslog message did not > include a year (fairly common); however, we have hard coded an assertion > based on the year the code was written (in our case 2016). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-618) Eliminate Javac Warnings in metron-analytics
[
https://issues.apache.org/jira/browse/METRON-618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-618:
---
Fix Version/s: Next + 1
> Eliminate Javac Warnings in metron-analytics
>
>
> Key: METRON-618
> URL: https://issues.apache.org/jira/browse/METRON-618
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
> Fix For: Next + 1
>
>
> Kill off the java compiler warnings in maas project, as much as possible.
> Most of these are related to missing @SuppressWarnings("unchecked") on code
> that either should have them, or be refactored so it's not necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-619) Eliminate Javac warnings in metron-parsers
[
https://issues.apache.org/jira/browse/METRON-619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-619:
---
Assignee: (was: Justin Leet)
> Eliminate Javac warnings in metron-parsers
> --
>
> Key: METRON-619
> URL: https://issues.apache.org/jira/browse/METRON-619
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Priority: Minor
>
> Metron-parsers has a lot of Javac warnings. Clean up the code as necessary
> to get rid of them. Probably mostly just missing
> @SuppressWarnings("unchecked") because things like our JSON library use raw
> types under the hood.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-621) Eliminate Javac warnings in metron-platform outside of parsers and enrichment
[
https://issues.apache.org/jira/browse/METRON-621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-621:
---
Assignee: (was: Justin Leet)
> Eliminate Javac warnings in metron-platform outside of parsers and enrichment
> -
>
> Key: METRON-621
> URL: https://issues.apache.org/jira/browse/METRON-621
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Priority: Minor
>
> metron-platform has a lot of Javac warnings. Clean up the code as necessary
> to get rid of them. Probably mostly just missing
> @SuppressWarnings("unchecked") because things like our JSON library use raw
> types under the hood.
> metron-parsers and metron-enrichment are covered by other tickets based on
> how many warnings they have.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-620) Eliminate Javac warnings in metron-enrichment
[
https://issues.apache.org/jira/browse/METRON-620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-620:
---
Assignee: (was: Justin Leet)
> Eliminate Javac warnings in metron-enrichment
> -
>
> Key: METRON-620
> URL: https://issues.apache.org/jira/browse/METRON-620
> Project: Metron
> Issue Type: Sub-task
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Priority: Minor
>
> metron-enrichment has a lot of Javac warnings. Clean up the code as
> necessary to get rid of them. Probably mostly just missing
> @SuppressWarnings("unchecked") because things like our JSON library use raw
> types under the hood.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-621) Eliminate Javac warnings in metron-platform outside of parsers and enrichment
Justin Leet created METRON-621:
--
Summary: Eliminate Javac warnings in metron-platform outside of
parsers and enrichment
Key: METRON-621
URL: https://issues.apache.org/jira/browse/METRON-621
Project: Metron
Issue Type: Sub-task
Affects Versions: 0.3.0
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
metron-platform has a lot of Javac warnings. Clean up the code as necessary to
get rid of them. Probably mostly just missing @SuppressWarnings("unchecked")
because things like our JSON library use raw types under the hood.
metron-parsers and metron-enrichment are covered by other tickets based on how
many warnings they have.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-620) Eliminate Javac warnings in metron-enrichment
Justin Leet created METRON-620:
--
Summary: Eliminate Javac warnings in metron-enrichment
Key: METRON-620
URL: https://issues.apache.org/jira/browse/METRON-620
Project: Metron
Issue Type: Sub-task
Affects Versions: 0.3.0
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
metron-enrichment has a lot of Javac warnings. Clean up the code as necessary
to get rid of them. Probably mostly just missing
@SuppressWarnings("unchecked") because things like our JSON library use raw
types under the hood.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-618) Eliminate Javac Warnings in metron-analytics
Justin Leet created METRON-618:
--
Summary: Eliminate Javac Warnings in metron-analytics
Key: METRON-618
URL: https://issues.apache.org/jira/browse/METRON-618
Project: Metron
Issue Type: Sub-task
Affects Versions: 0.3.0
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
Kill off the java compiler warnings in maas project, as much as possible. Most
of these are related to missing @SuppressWarnings("unchecked") on code that
either should have them, or be refactored so it's not necessary.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-617) Eliminate Javac warnings in the build
Justin Leet created METRON-617:
--
Summary: Eliminate Javac warnings in the build
Key: METRON-617
URL: https://issues.apache.org/jira/browse/METRON-617
Project: Metron
Issue Type: Improvement
Affects Versions: 0.3.0
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
We have a lot of Javac warnings (several hundred) in the build right now. The
primary cause of these is unchecked types (for reasons including
org.simple.json uses raw types under the hood). These should be labelled with
@SuppressWarnings("unchecked") where appropriate. In addition, things like
@SafeVarargs should be added where appropriate, etc.
Given the number of changes in this ticket is significantly larger than even
the Error Prone warnings (and may require tinkering with generics at
appropriate points), I'm going to split this up by module (Analytics and
Platform), and then split platform as necessary (parsers and enrichment in
particular have a lot). Otherwise, anything resulting is likely to be too
large and unmanageable to be properly tested and reviewed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-615) Prevent warnings in code from future contributions
[
https://issues.apache.org/jira/browse/METRON-615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15730011#comment-15730011
]
Justin Leet commented on METRON-615:
Yeah, I agree about new (and even current) contributors. Whatever we do,
documenting it for contributors is important, because I also don't want people
to get turned off because things blow up that they don't expect.
There should also be documentation around what tools we're using and how we
enable / disable things like this. These particular things aren't restrictive
while something is being built or improved, but I can see cases where you'd
want to disable/downgrade errors or warnings while you're working locally. To
the best of my knowledge, this doesn't exist in a fleshed out form and should
probably end up on the wiki somewhere.
> Prevent warnings in code from future contributions
> --
>
> Key: METRON-615
> URL: https://issues.apache.org/jira/browse/METRON-615
> Project: Metron
> Issue Type: Improvement
>Reporter: Justin Leet
>Priority: Minor
>
> After cleaning up most of the warnings in METRON-612, there's a question of
> locking out additional warnings in the code from Error Prone. This leads to
> the more general question of avoiding the addition of warnings in the future
> in general, not just from Error Prone but from javac or any other tooling we
> may add.
> The Hadoop project itself does something pretty extensive, e.g. [Hadoop
> QA|https://issues.apache.org/jira/browse/HADOOP-13827?focusedCommentId=15678418=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15678418].
> Settings up something like that might be pretty involved and overkill for
> where we are, but it might be a nice end goal.
> As an parallel measure, we can potentially turn Error Prone's warnings into
> errors for warnings we've stamped out. METRON-612's PR ([Github
> PR|https://github.com/apache/incubator-metron/pull/389]), assuming no
> additional warnings have come in on master, allows for 4 to be upgraded.
> [SynchronizeOnNonFinalField|http://errorprone.info/bugpattern/SynchronizeOnNonFinalField]
> [BoxedPrimitiveConstructor|http://errorprone.info/bugpattern/BoxedPrimitiveConstructor]
> [ClassCanBeStatic|http://errorprone.info/bugpattern/ClassCanBeStatic]
> [ClassNewInstance|http://errorprone.info/bugpattern/ClassNewInstance]
> Of these, I'm personally in favor of promoting SynchronizeOnNonFinalField,
> BoxedPrimitiveConstructor, and ClassNewInstance. ClassCanbeStatic I'm pretty
> neutral on.
> In particular, SynchronizeOnNonFinalField is a correctness issue.
> BoxedPrimitiveConstructor is a perf / code quality issue. ClassNewInstance
> avoids some issues around reflection. Both BoxedPrimitiveConstructor and
> ClassNewInstance are expected to have the originating problems become
> deprecated in Java 9 in favor of the more correct construction.
> [~dlyle]'s opinion per the PR:
> {quote}
> Totally For Erroring on:
> SynchronizeOnNonFinalField
> Not against it for:
> BoxedPrimitiveConstructor
> ClassCanBeStatic
> Okay with it for ClassNewInstance with additional input. I get why it's
> harmful, but is it more harmful than reflection in general? Or do you just
> want to not insert known future deprecated code?
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-612) Clean up Error Prone generated warnings
[ https://issues.apache.org/jira/browse/METRON-612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-612: --- Fix Version/s: Next + 1 > Clean up Error Prone generated warnings > --- > > Key: METRON-612 > URL: https://issues.apache.org/jira/browse/METRON-612 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: Next + 1 > > > As a result of METRON-593, we have a new set of warnings in our code that > should be addressed. 593 already addressed errors (as those would break the > build otherwise). > These can be seen in the mvn compile output of the various modules. It > includes things like missing Override annotations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-615) Prevent warnings in code from future contributions
Justin Leet created METRON-615:
--
Summary: Prevent warnings in code from future contributions
Key: METRON-615
URL: https://issues.apache.org/jira/browse/METRON-615
Project: Metron
Issue Type: Improvement
Reporter: Justin Leet
Priority: Minor
After cleaning up most of the warnings in METRON-612, there's a question of
locking out additional warnings in the code from Error Prone. This leads to
the more general question of avoiding the addition of warnings in the future in
general, not just from Error Prone but from javac or any other tooling we may
add.
The Hadoop project itself does something pretty extensive, e.g. [Hadoop
QA|https://issues.apache.org/jira/browse/HADOOP-13827?focusedCommentId=15678418=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15678418].
Settings up something like that might be pretty involved and overkill for
where we are, but it might be a nice end goal.
As an parallel measure, we can potentially turn Error Prone's warnings into
errors for warnings we've stamped out. METRON-612's PR ([Github
PR|https://github.com/apache/incubator-metron/pull/389]), assuming no
additional warnings have come in on master, allows for 4 to be upgraded.
[SynchronizeOnNonFinalField|http://errorprone.info/bugpattern/SynchronizeOnNonFinalField]
[BoxedPrimitiveConstructor|http://errorprone.info/bugpattern/BoxedPrimitiveConstructor]
[ClassCanBeStatic|http://errorprone.info/bugpattern/ClassCanBeStatic]
[ClassNewInstance|http://errorprone.info/bugpattern/ClassNewInstance]
Of these, I'm personally in favor of promoting SynchronizeOnNonFinalField,
BoxedPrimitiveConstructor, and ClassNewInstance. ClassCanbeStatic I'm pretty
neutral on.
In particular, SynchronizeOnNonFinalField is a correctness issue.
BoxedPrimitiveConstructor is a perf / code quality issue. ClassNewInstance
avoids some issues around reflection. Both BoxedPrimitiveConstructor and
ClassNewInstance are expected to have the originating problems become
deprecated in Java 9 in favor of the more correct construction.
[~dlyle]'s opinion per the PR:
{quote}
Totally For Erroring on:
SynchronizeOnNonFinalField
Not against it for:
BoxedPrimitiveConstructor
ClassCanBeStatic
Okay with it for ClassNewInstance with additional input. I get why it's
harmful, but is it more harmful than reflection in general? Or do you just want
to not insert known future deprecated code?
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-614) Eliminate use of the default Charset
[
https://issues.apache.org/jira/browse/METRON-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-614:
---
Description:
During work on METRON-612, I noticed a lot of our warnings are from implicit
use of the default Charset (e.g. when creating InputStreamReaders). Given that
this value is platform dependent, we should at least consider explicitly using
a particular Charset.
In addition, after fixing this, I would like to see it upgraded to a compile
error, because in my experience almost everyone tends to use methods that use
the implicit default Charset. This can be done by changing the compiler arg in
the pom to instead be
{code}
-Xlint:unchecked
-Xep:DefaultCharset:ERROR
{code}
My assumption is that we'll want UTF-8, but given a lack of expertise and that
usually there's a lot of opinions on character set issues, we may want to
consider other options.
http://errorprone.info/bugpattern/DefaultCharset
https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
standard Charsets and provides more details.
{code}
Charset Description
US-ASCIISeven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
of the Unicode character set
ISO-8859-1 ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
UTF-8 Eight-bit UCS Transformation Format
UTF-16BESixteen-bit UCS Transformation Format, big-endian byte order
UTF-16LESixteen-bit UCS Transformation Format, little-endian byte order
UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an
optional byte-order mark
{code}
was:
During work on Metron-612, I noticed a lot of our warnings are from implicit
use of the default Charset (e.g. when creating InputStreamReaders). Given that
this value is platform dependent, we should at least consider explicitly using
a particular Charset.
In addition, after fixing this, I would like to see it upgraded to a compile
error, because in my experience almost everyone tends to use methods that use
the implicit default Charset. This can be done by changing the compiler arg in
the pom to instead be
{code}
-Xlint:unchecked
-Xep:DefaultCharset:ERROR
{code}
My assumption is that we'll want UTF-8, but given a lack of expertise and that
usually there's a lot of opinions on character set issues, we may want to
consider other options.
http://errorprone.info/bugpattern/DefaultCharset
https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
standard Charsets and provides more details.
{code}
Charset Description
US-ASCIISeven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
of the Unicode character set
ISO-8859-1 ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
UTF-8 Eight-bit UCS Transformation Format
UTF-16BESixteen-bit UCS Transformation Format, big-endian byte order
UTF-16LESixteen-bit UCS Transformation Format, little-endian byte order
UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an
optional byte-order mark
{code}
> Eliminate use of the default Charset
>
>
> Key: METRON-614
> URL: https://issues.apache.org/jira/browse/METRON-614
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Priority: Minor
>
> During work on METRON-612, I noticed a lot of our warnings are from implicit
> use of the default Charset (e.g. when creating InputStreamReaders). Given
> that this value is platform dependent, we should at least consider explicitly
> using a particular Charset.
> In addition, after fixing this, I would like to see it upgraded to a compile
> error, because in my experience almost everyone tends to use methods that use
> the implicit default Charset. This can be done by changing the compiler arg
> in the pom to instead be
> {code}
>
> -Xlint:unchecked
> -Xep:DefaultCharset:ERROR
>
> {code}
> My assumption is that we'll want UTF-8, but given a lack of expertise and
> that usually there's a lot of opinions on character set issues, we may want
> to consider other options.
> http://errorprone.info/bugpattern/DefaultCharset
> https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
> standard Charsets and provides more details.
> {code}
> Charset Description
> US-ASCII Seven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
> of the Unicode character set
> ISO-8859-1ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
> UTF-8 Eight-bit UCS Transformation Format
> UTF-16BE Sixteen-bit UCS Transformation Format, big-endian byte order
> UTF-16LE Sixteen-bit UCS Transformation Format, little-endian byte order
> UTF-16Sixteen-bit UCS Transformation Format, byte order identified by
> an optional byte-order mark
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-614) Eliminate use of the default Charset
Justin Leet created METRON-614:
--
Summary: Eliminate use of the default Charset
Key: METRON-614
URL: https://issues.apache.org/jira/browse/METRON-614
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
Priority: Minor
During work on Metron-612, I noticed a lot of our warnings are from implicit
use of the default Charset (e.g. when creating InputStreamReaders). Given that
this value is platform dependent, we should at least consider explicitly using
a particular Charset.
In addition, after fixing this, I would like to see it upgraded to a compile
error, because in my experience almost everyone tends to use methods that use
the implicit default Charset. This can be done by changing the compiler arg in
the pom to instead be
{code}
-Xlint:unchecked
-Xep:DefaultCharset:ERROR
{code}
My assumption is that we'll want UTF-8, but given a lack of expertise and that
usually there's a lot of opinions on character set issues, we may want to
consider other options.
http://errorprone.info/bugpattern/DefaultCharset
https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
standard Charsets and provides more details.
{code}
Charset Description
US-ASCIISeven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
of the Unicode character set
ISO-8859-1 ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
UTF-8 Eight-bit UCS Transformation Format
UTF-16BESixteen-bit UCS Transformation Format, big-endian byte order
UTF-16LESixteen-bit UCS Transformation Format, little-endian byte order
UTF-16 Sixteen-bit UCS Transformation Format, byte order identified by an
optional byte-order mark
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-612) Clean up Error Prone generated warnings
[
https://issues.apache.org/jira/browse/METRON-612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15725463#comment-15725463
]
Justin Leet commented on METRON-612:
We have a (very) large number of warnings of the form
{code}
/home/travis/build/justinleet/incubator-metron/metron-analytics/metron-maas-common/src/main/java/org/apache/metron/maas/util/RESTUtil.java:64:
warning: [DefaultCharset] Implicit use of the platform default charset, which
can result in e.g. non-ASCII characters being silently replaced with '?' in
many environments
return new BufferedReader(new
InputStreamReader(response.getEntity().getContent()))
^
(see http://errorprone.info/bugpattern/DefaultCharset)
Did you mean 'return new BufferedReader(new
InputStreamReader(response.getEntity().getContent(), UTF_8))' or 'return new
BufferedReader(new InputStreamReader(response.getEntity().getContent(),
Charset.defaultCharset()))'?
{code}
See http://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html for
standard Charsets.
Given that the default Charset is platform dependent, we should at least
consider making a conscious choice on this. If we don't want to make a
decision, it is also possible to set this warning to ignore, but I'm not
generally a fan of doing that for warnings.
Does anybody with more knowledge / expertise have any input on this?
> Clean up Error Prone generated warnings
> ---
>
> Key: METRON-612
> URL: https://issues.apache.org/jira/browse/METRON-612
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
>
> As a result of METRON-593, we have a new set of warnings in our code that
> should be addressed. 593 already addressed errors (as those would break the
> build otherwise).
> These can be seen in the mvn compile output of the various modules. It
> includes things like missing Override annotations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-596) Eliminate Maven warnings from build
[
https://issues.apache.org/jira/browse/METRON-596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-596:
---
Fix Version/s: Next + 1
> Eliminate Maven warnings from build
> ---
>
> Key: METRON-596
> URL: https://issues.apache.org/jira/browse/METRON-596
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
> Fix For: Next + 1
>
>
> Right now our builds throw some warnings in Maven. Things like not specifying
> a version for plugins, etc. These should be cleaned up.
> Current Build warnings:
> {code}
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-common:jar:0.3.0
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.metron:metron-common:[unknown-version],
> /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/pom.xml,
> line 360, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-enrichment:jar:0.3.0
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.metron:metron-enrichment:[unknown-version],
> /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-enrichment/pom.xml,
> line 287, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-parsers:jar:0.3.0
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.metron:metron-parsers:[unknown-version],
> /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-parsers/pom.xml,
> line 250, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-indexing:jar:0.3.0
> [WARNING] 'build.plugins.plugin.version' for
> org.apache.maven.plugins:maven-jar-plugin is missing. @
> org.apache.metron:metron-indexing:[unknown-version],
> /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-indexing/pom.xml,
> line 183, column 21
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-management:jar:0.3.0
> [WARNING] The expression ${parent.version} is deprecated. Please use
> ${project.parent.version} instead.
> [WARNING] The expression ${parent.version} is deprecated. Please use
> ${project.parent.version} instead.
> [WARNING] The expression ${parent.version} is deprecated. Please use
> ${project.parent.version} instead.
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-hbase:jar:0.3.0
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.apache.hadoop:hadoop-hdfs:jar:tests -> duplicate declaration
> of version ${global_hadoop_version} @
> org.apache.metron:metron-hbase:[unknown-version],
> /Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-hbase/pom.xml,
> line 166, column 21
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-604) Mpack installs do not work on clean machines due to missing Elastic Curator repo
[
https://issues.apache.org/jira/browse/METRON-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-604:
---
Fix Version/s: Next + 1
> Mpack installs do not work on clean machines due to missing Elastic Curator
> repo
>
>
> Key: METRON-604
> URL: https://issues.apache.org/jira/browse/METRON-604
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: David M. Lyle
>Assignee: Justin Leet
> Fix For: Next + 1
>
>
> I cut fresh rpms/mpack for myself following [~justinleet]'s [readme|
> https://github.com/justinleet/metron-rpm]
> Add the mpack to Ambari and then build a fresh 10 node cluster, HDP 2.5.3 and
> Ambar 2.4.2 (note my last attempt was with HDP 2.5.0 and Ambari 2.4.1 iirc)
> Add the services, hit deploy, everything goes fine until it attempts to
> deploy kibana, the install fails because it cannot find the
> python-elasticsearch package within the repos we deploy using the mpack.
> If I manually add the following repo to the server where Ambari is attempting
> to install Kibana then I can retry and everything works fine.
> {noformat}
> [curator-4]
> name=CentOS/RHEL 7 repository for Elasticsearch Curator 4.x packages
> baseurl=http://packages.elastic.co/curator/4/centos/7
> gpgcheck=1
> gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
> enabled=1
> {noformat}
> I don’t know for sure this is the correct repository for us to use based on
> the version of ES/Kibana that we’re using, I just know it contains a package
> which satisfies the dependency requirements and allows the install to
> complete successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-604) Mpack installs do not work on clean machines due to missing Elastic Curator repo
[
https://issues.apache.org/jira/browse/METRON-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-604:
---
Summary: Mpack installs do not work on clean machines due to missing
Elastic Curator repo (was: Mpack installs do not work on clean machines due to
missing Elastic repo)
> Mpack installs do not work on clean machines due to missing Elastic Curator
> repo
>
>
> Key: METRON-604
> URL: https://issues.apache.org/jira/browse/METRON-604
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: David M. Lyle
>Assignee: Justin Leet
>
> I cut fresh rpms/mpack for myself following [~justinleet]'s [readme|
> https://github.com/justinleet/metron-rpm]
> Add the mpack to Ambari and then build a fresh 10 node cluster, HDP 2.5.3 and
> Ambar 2.4.2 (note my last attempt was with HDP 2.5.0 and Ambari 2.4.1 iirc)
> Add the services, hit deploy, everything goes fine until it attempts to
> deploy kibana, the install fails because it cannot find the
> python-elasticsearch package within the repos we deploy using the mpack.
> If I manually add the following repo to the server where Ambari is attempting
> to install Kibana then I can retry and everything works fine.
> {noformat}
> [curator-4]
> name=CentOS/RHEL 7 repository for Elasticsearch Curator 4.x packages
> baseurl=http://packages.elastic.co/curator/4/centos/7
> gpgcheck=1
> gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
> enabled=1
> {noformat}
> I don’t know for sure this is the correct repository for us to use based on
> the version of ES/Kibana that we’re using, I just know it contains a package
> which satisfies the dependency requirements and allows the install to
> complete successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-612) Clean up Error Prone generated warnings
Justin Leet created METRON-612: -- Summary: Clean up Error Prone generated warnings Key: METRON-612 URL: https://issues.apache.org/jira/browse/METRON-612 Project: Metron Issue Type: Bug Affects Versions: 0.3.0 Reporter: Justin Leet Assignee: Justin Leet Priority: Minor As a result of METRON-593, we have a new set of warnings in our code that should be addressed. 593 already addressed errors (as those would break the build otherwise). These can be seen in the mvn compile output of the various modules. It includes things like missing Override annotations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-604) Mpack installs do not work on clean machines due to missing Elastic repo
[
https://issues.apache.org/jira/browse/METRON-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet reassigned METRON-604:
--
Assignee: Justin Leet
> Mpack installs do not work on clean machines due to missing Elastic repo
>
>
> Key: METRON-604
> URL: https://issues.apache.org/jira/browse/METRON-604
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: David M. Lyle
>Assignee: Justin Leet
>
> I cut fresh rpms/mpack for myself following [~justinleet]'s [readme|
> https://github.com/justinleet/metron-rpm]
> Add the mpack to Ambari and then build a fresh 10 node cluster, HDP 2.5.3 and
> Ambar 2.4.2 (note my last attempt was with HDP 2.5.0 and Ambari 2.4.1 iirc)
> Add the services, hit deploy, everything goes fine until it attempts to
> deploy kibana, the install fails because it cannot find the
> python-elasticsearch package within the repos we deploy using the mpack.
> If I manually add the following repo to the server where Ambari is attempting
> to install Kibana then I can retry and everything works fine.
> {noformat}
> [curator-4]
> name=CentOS/RHEL 7 repository for Elasticsearch Curator 4.x packages
> baseurl=http://packages.elastic.co/curator/4/centos/7
> gpgcheck=1
> gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
> enabled=1
> {noformat}
> I don’t know for sure this is the correct repository for us to use based on
> the version of ES/Kibana that we’re using, I just know it contains a package
> which satisfies the dependency requirements and allows the install to
> complete successfully.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708818#comment-15708818
]
Justin Leet commented on METRON-597:
https://github.com/apache/incubator-metron/pull/378 has only one commit
difference with apache/master: METRON-588. It also just failed again on this
test
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-597) Sporadic Failures of Profiler Integration Tests
[
https://issues.apache.org/jira/browse/METRON-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15708659#comment-15708659
]
Justin Leet commented on METRON-597:
I'll try to keep an eye out for test failures in the other packages, too
[~nickwallen]. You're right that we could have larger integration test issues,
but given that those have had some refactoring lately, I'm not sure I want to
group the others into the same bucket yet.
> Sporadic Failures of Profiler Integration Tests
> ---
>
> Key: METRON-597
> URL: https://issues.apache.org/jira/browse/METRON-597
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> Seems to be some kind of timing issue.
> {code}
> ---
> T E S T S
> ---
> Running org.apache.metron.profiler.integration.ProfilerIntegrationTest
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 66.564 sec
> <<< FAILURE!
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest)
> Time elapsed: 16.759 sec <<< FAILURE!
> java.lang.AssertionError: expected:<20.0> but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:553)
> at org.junit.Assert.assertEquals(Assert.java:683)
> at
> org.apache.metron.profiler.integration.ProfilerIntegrationTest.testExample3(ProfilerIntegrationTest.java:203)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)
> Results :
> Failed tests:
> testExample3(org.apache.metron.profiler.integration.ProfilerIntegrationTest):
> expected:<20.0> but was:
> Tests run: 5, Failures: 1, Errors: 0, Skipped: 0
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-596) Eliminate Maven warnings from build
Justin Leet created METRON-596:
--
Summary: Eliminate Maven warnings from build
Key: METRON-596
URL: https://issues.apache.org/jira/browse/METRON-596
Project: Metron
Issue Type: Improvement
Affects Versions: 0.3.0
Reporter: Justin Leet
Assignee: Justin Leet
Priority: Minor
Right now our builds throw some warnings in Maven. Things like not specifying a
version for plugins, etc. These should be cleaned up.
Current Build warnings:
{code}
[INFO] Scanning for projects...
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-common:jar:0.3.0
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
org.apache.metron:metron-common:[unknown-version],
/Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-common/pom.xml,
line 360, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-enrichment:jar:0.3.0
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
org.apache.metron:metron-enrichment:[unknown-version],
/Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-enrichment/pom.xml,
line 287, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-parsers:jar:0.3.0
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
org.apache.metron:metron-parsers:[unknown-version],
/Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-parsers/pom.xml,
line 250, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-indexing:jar:0.3.0
[WARNING] 'build.plugins.plugin.version' for
org.apache.maven.plugins:maven-jar-plugin is missing. @
org.apache.metron:metron-indexing:[unknown-version],
/Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-indexing/pom.xml,
line 183, column 21
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-management:jar:0.3.0
[WARNING] The expression ${parent.version} is deprecated. Please use
${project.parent.version} instead.
[WARNING] The expression ${parent.version} is deprecated. Please use
${project.parent.version} instead.
[WARNING] The expression ${parent.version} is deprecated. Please use
${project.parent.version} instead.
[WARNING]
[WARNING] Some problems were encountered while building the effective model for
org.apache.metron:metron-hbase:jar:0.3.0
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
be unique: org.apache.hadoop:hadoop-hdfs:jar:tests -> duplicate declaration of
version ${global_hadoop_version} @
org.apache.metron:metron-hbase:[unknown-version],
/Users/jleet/Documents/workspace/incubator-metron/metron-platform/metron-hbase/pom.xml,
line 166, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten
the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support
building such malformed projects.
[WARNING]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-593) Enable an automated static analysis tool in the build
[ https://issues.apache.org/jira/browse/METRON-593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15705751#comment-15705751 ] Justin Leet commented on METRON-593: It's built in to a given version of error-prone. The actual bugpatterns are implemented at [Bug Patterns|https://github.com/google/error-prone/tree/master/core/src/main/java/com/google/errorprone/bugpatterns] in their repo. The webpage's docs are, as noted on the http://errorprone.info/bugpatterns, are autogenerated from that source. For example, MissingOverride.java results in http://errorprone.info/bugpattern/MissingOverride, which would be how the compilation knows what link to generate (I assume, but I haven't dug into their source that much). At that point, updating to latest ruleset is a matter of updating error-prone (which should be updating the version in the pom, and fixing any new errors that arise). As a sidenote, as we fix warnings, we can start promoting those to errors (or turn them off if we determine they're not something we're actually concerned bout) if we want as described in http://errorprone.info/docs/flags. > Enable an automated static analysis tool in the build > - > > Key: METRON-593 > URL: https://issues.apache.org/jira/browse/METRON-593 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > > From on a discussion thread I kicked off on the dev list. Some newer notes > follow > h4. Original Email > I wanted to kick off a discussion about integrating a static analysis tool > into our builds. > The main discussion points I wanted to start up (and feel encouraged to add > more): > 1) Most importantly, do we get enough value by adding a tool? > 2) What are we looking for out of a tool (Extensibility to add our own > checks, plugged into build cycle directly, ease of use, customizability, > etc.)? > 3) Are there any particular tools people have experience with? > 4) Assuming we want to roll something out, what's the best path? My current > assumption is that it's probably easiest to handle things on a pom by pom > basis, rather than trying to do everything at once, but there may be more > nuance people want to add. > The main one I've used FindBugs, but there's a been discussion lately about > issues with their community which led me to take a (very) brief glance at > into Google's errorprone. It seems like it's an alternative worth considering > from what I've seen. > Some links to errorprone info: > http://errorprone.info/ > https://github.com/google/error-prone > http://errorprone.info/bugpatterns > I played around with it for about 2 minutes, and was able to get it up and > running and happily complaining about metron-common during it's build cycle. > Haven't dug too much into the errors/warnings to get a sense of signal to > noise ratio. If anybody is interested in playing around with that setup for > metron-common, I have a branch at: > https://github.com/justinleet/incubator-metron/tree/errorprone > Just go to metron-platform/metron-common and run: > mvn compile > Gist with the error prone output. > https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 > h4. New notes > After playing around with error prone a bit more (I was able to get it > running on most of our modules and fix build breaking errors pretty quickly), > it provides significantly less check coverage than findbugs, but has the > benefit of being directly tied into the compile (meaning people can get > feedback and errors pretty quickly). Related to this, the errors that error > prone gave out were actual issues (although relatively minor in our codebase, > e.g. catching issues with format() calls). > It seems the benefits of error prone fall primarily on it coming earlier in > the lifecycle to give feedback quicker and being actively maintained and > improved. > The main benefit of findbugs is being a broader set of checks, but at the > expense of being later in the build cycle (because it operates on bytecode), > and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-593) Enable an automated static analysis tool in the build
Justin Leet created METRON-593: -- Summary: Enable an automated static analysis tool in the build Key: METRON-593 URL: https://issues.apache.org/jira/browse/METRON-593 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet >From on a discussion thread I kicked off on the dev list. Some newer notes >follow h4. Original Email I wanted to kick off a discussion about integrating a static analysis tool into our builds. The main discussion points I wanted to start up (and feel encouraged to add more): 1) Most importantly, do we get enough value by adding a tool? 2) What are we looking for out of a tool (Extensibility to add our own checks, plugged into build cycle directly, ease of use, customizability, etc.)? 3) Are there any particular tools people have experience with? 4) Assuming we want to roll something out, what's the best path? My current assumption is that it's probably easiest to handle things on a pom by pom basis, rather than trying to do everything at once, but there may be more nuance people want to add. The main one I've used FindBugs, but there's a been discussion lately about issues with their community which led me to take a (very) brief glance at into Google's errorprone. It seems like it's an alternative worth considering from what I've seen. Some links to errorprone info: http://errorprone.info/ https://github.com/google/error-prone http://errorprone.info/bugpatterns I played around with it for about 2 minutes, and was able to get it up and running and happily complaining about metron-common during it's build cycle. Haven't dug too much into the errors/warnings to get a sense of signal to noise ratio. If anybody is interested in playing around with that setup for metron-common, I have a branch at: https://github.com/justinleet/incubator-metron/tree/errorprone Just go to metron-platform/metron-common and run: mvn compile Gist with the error prone output. https://gist.github.com/justinleet/8d514727a0caeb5db2b4f76de0607214 h4. New notes After playing around with error prone a bit more (I was able to get it running on most of our modules and fix build breaking errors pretty quickly), it provides significantly less check coverage than findbugs, but has the benefit of being directly tied into the compile (meaning people can get feedback and errors pretty quickly). Related to this, the errors that error prone gave out were actual issues (although relatively minor in our codebase, e.g. catching issues with format() calls). It seems the benefits of error prone fall primarily on it coming earlier in the lifecycle to give feedback quicker and being actively maintained and improved. The main benefit of findbugs is being a broader set of checks, but at the expense of being later in the build cycle (because it operates on bytecode), and the community being in an odd place right now. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-522) Need to mandate Client installation on Metron Host
[ https://issues.apache.org/jira/browse/METRON-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-522: -- Assignee: Justin Leet > Need to mandate Client installation on Metron Host > -- > > Key: METRON-522 > URL: https://issues.apache.org/jira/browse/METRON-522 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Anand Subramanian >Assignee: Justin Leet > > This is while deploying Metron using Ambari mpacks on a existing HDP cluster > by adding Metron services subsequently. > While the Metron components colocation criteria is addressed by defect > [Metron-464|https://issues.apache.org/jira/browse/METRON-464], there needs to > be another check for ensuring that the Clients (ZK, HBASE etc.) must be > installed on the host where Metron components are installed. > This defect is created to track the enhancement for checking the presence of > client on the host selected for Metron components. > Note that one can use the Host->MetronHost->Add Clients option to install > client on the Metron host and subsequently "Add Service" again. However, it > would be good to check for this when the wizard is fired up. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-552) Ambari Mpack should be able to manage pcap topology
[
https://issues.apache.org/jira/browse/METRON-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15683694#comment-15683694
]
Justin Leet commented on METRON-552:
Addendum to the addendum. The metron user doesn't have a /user/metron created
on HDFS, so it's necessary to create and chown it appropriately. At that
point, the script should be run as the Metron user. This should all be handled
by the Ambari service.
{code}
su - hdfs
hadoop fs -mkdir /user/metron
hadoop fs -chown metron:hadoop /user/metron
{code}
> Ambari Mpack should be able to manage pcap topology
> ---
>
> Key: METRON-552
> URL: https://issues.apache.org/jira/browse/METRON-552
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.2.1BETA
>Reporter: Justin Leet
>
> Right now, the mpack installs the pcap RPM, but doesn't do any management of
> it. This should be handled by Ambari. With the current boundaries of the
> mpack (i.e. responsibility begins at kafka, not before), using pycapa to
> produce data to Kafka wouldn't be managed by Ambari.
> As a workaround (which will also have to happen in the Ambari service), pcap
> topology can be run on an Ambari managed Metron instance by
> 1) Updating /usr/metron/0.2.1BETA/config/pcap.properties by editing kafka.zk
> manually. Ambari should manage these configs itself, once the service is up.
> 2) Creating /apps/metron/pcap on HDFS, owning it to metron:hadoop, and
> updating permissions to 775 on the folder.
> At this point, /usr/metron/0.2.1BETA/bin/start_pcap_topology.sh should be
> able to be run normally.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (METRON-578) Missing error handling bolts for enrichment and threat intel
Justin Leet created METRON-578:
--
Summary: Missing error handling bolts for enrichment and threat
intel
Key: METRON-578
URL: https://issues.apache.org/jira/browse/METRON-578
Project: Metron
Issue Type: Improvement
Affects Versions: 0.2.1BETA
Reporter: Justin Leet
Assignee: Justin Leet
TL;DR - we need to add error handling to enrichments/threat intel
Metron has parsers, enrichment + threat intel, and indexing topologies
currently. Parsers and and enrichment have bolts that write to error topics in
Kafka
# indexing_error
# parser_error
# parser_invalid
The GenericEnrichmentBolt handles errors gracefully by passing along failed
enrichment tuples un-enriched and additionally emitting the tuple to an "error"
stream, however there is currently no plumbing to handle the error stream.
{code:java}
} catch (Exception e) {
LOG.error("[Metron] Unable to enrich message: " + rawMessage, e);
JSONObject error = ErrorUtils.generateErrorMessage("Enrichment problem: "
+ rawMessage, e);
if (key != null) {
collector.emit(enrichmentType, new Values(key, enrichedMessage,
subGroup));
}
collector.emit("error", new Values(error));
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (METRON-552) Ambari Mpack should be able to manage pcap topology
[ https://issues.apache.org/jira/browse/METRON-552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667358#comment-15667358 ] Justin Leet commented on METRON-552: As a addendum to this, before the topology can be started, it's also necessary to create the pcap kafka topic (doesn't need to be populated with data or anything, but it needs to actually exist to be read from in the topology). > Ambari Mpack should be able to manage pcap topology > --- > > Key: METRON-552 > URL: https://issues.apache.org/jira/browse/METRON-552 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.2.1BETA >Reporter: Justin Leet > > Right now, the mpack installs the pcap RPM, but doesn't do any management of > it. This should be handled by Ambari. With the current boundaries of the > mpack (i.e. responsibility begins at kafka, not before), using pycapa to > produce data to Kafka wouldn't be managed by Ambari. > As a workaround (which will also have to happen in the Ambari service), pcap > topology can be run on an Ambari managed Metron instance by > 1) Updating /usr/metron/0.2.1BETA/config/pcap.properties by editing kafka.zk > manually. Ambari should manage these configs itself, once the service is up. > 2) Creating /apps/metron/pcap on HDFS, owning it to metron:hadoop, and > updating permissions to 775 on the folder. > At this point, /usr/metron/0.2.1BETA/bin/start_pcap_topology.sh should be > able to be run normally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-564) Mpack shows UI error when changing configuration
[
https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-564:
---
Affects Version/s: (was: 0.3.0)
0.2.1BETA
Fix Version/s: 0.3.0
> Mpack shows UI error when changing configuration
>
>
> Key: METRON-564
> URL: https://issues.apache.org/jira/browse/METRON-564
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
>Reporter: Justin Leet
>Assignee: Justin Leet
> Fix For: 0.3.0
>
>
> When confirming configuration changes, the Ambari UI shows an error that it
> cannot validate configuration consistency.
> An error in the logs refers to the service advisor
> (getServiceConfigurationRecommendations), in particular a method that
> actually appears twice in the file, which is probably the root cause. This
> first instance of this method should be removed, and the error fixed.
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
> in main
> result = stackAdvisor.validateConfigurations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 536, in validateConfigurations
> validationItems = self.getConfigurationsValidationItems(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 612, in getConfigurationsValidationItems
> recommendations = self.recommendConfigurations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 766, in recommendConfigurations
> serviceAdvisor.getServiceConfigurationRecommendations(configurations,
> clusterSummary, services, hosts)
> File
> "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py",
> line 103, in getServiceConfigurationRecommendations
> stormUIServerPort =
> services["configurations"]["storm-site"]["properties"]["ui.port"]
> KeyError: 'storm-site'
> 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28]
> AbstractResourceProvider:90 - Error occurred during validation
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException:
> Stack Advisor reported an error: KeyError: 'storm-site'
> StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
[ https://issues.apache.org/jira/browse/METRON-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-563: --- Affects Version/s: (was: 0.2.1BETA) 0.3.0 > Ambari Metron service uses incorrect port for installing Elasticsearch > templates > > > Key: METRON-563 > URL: https://issues.apache.org/jira/browse/METRON-563 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 >Reporter: Justin Leet >Assignee: Justin Leet > > The port entered by the user at install time should be the binary port > (typically 9300), which is used by the indexing bolt. The installation of > templates expects the HTTP port for Elasticsearch (typically 9200). > There are (probably) two options for fixes here: > 1) Split the ES url and have something like ES Server, ES HTTP port, ES > Binary port > 2) Have the template loading go through the binary port. I assume this can > be done, but I don't know enough about it to say for sure. > This can be worked around by setting the ES url to :9200, installing > the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-564) Mpack shows UI error when changing configuration
[
https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-564:
---
Affects Version/s: (was: 0.2.1BETA)
0.3.0
> Mpack shows UI error when changing configuration
>
>
> Key: METRON-564
> URL: https://issues.apache.org/jira/browse/METRON-564
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> When confirming configuration changes, the Ambari UI shows an error that it
> cannot validate configuration consistency.
> An error in the logs refers to the service advisor
> (getServiceConfigurationRecommendations), in particular a method that
> actually appears twice in the file, which is probably the root cause. This
> first instance of this method should be removed, and the error fixed.
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
> in main
> result = stackAdvisor.validateConfigurations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 536, in validateConfigurations
> validationItems = self.getConfigurationsValidationItems(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 612, in getConfigurationsValidationItems
> recommendations = self.recommendConfigurations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 766, in recommendConfigurations
> serviceAdvisor.getServiceConfigurationRecommendations(configurations,
> clusterSummary, services, hosts)
> File
> "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py",
> line 103, in getServiceConfigurationRecommendations
> stormUIServerPort =
> services["configurations"]["storm-site"]["properties"]["ui.port"]
> KeyError: 'storm-site'
> 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28]
> AbstractResourceProvider:90 - Error occurred during validation
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException:
> Stack Advisor reported an error: KeyError: 'storm-site'
> StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
[ https://issues.apache.org/jira/browse/METRON-563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-563: -- Assignee: Justin Leet > Ambari Metron service uses incorrect port for installing Elasticsearch > templates > > > Key: METRON-563 > URL: https://issues.apache.org/jira/browse/METRON-563 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Justin Leet >Assignee: Justin Leet > > The port entered by the user at install time should be the binary port > (typically 9300), which is used by the indexing bolt. The installation of > templates expects the HTTP port for Elasticsearch (typically 9200). > There are (probably) two options for fixes here: > 1) Split the ES url and have something like ES Server, ES HTTP port, ES > Binary port > 2) Have the template loading go through the binary port. I assume this can > be done, but I don't know enough about it to say for sure. > This can be worked around by setting the ES url to :9200, installing > the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-564) Mpack shows UI error when changing configuration
[
https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15655158#comment-15655158
]
Justin Leet commented on METRON-564:
This does not appear to prevent config changes from actually happening.
> Mpack shows UI error when changing configuration
>
>
> Key: METRON-564
> URL: https://issues.apache.org/jira/browse/METRON-564
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> When confirming configuration changes, the Ambari UI shows an error that it
> cannot validate configuration consistency.
> An error in the logs refers to the service advisor
> (getServiceConfigurationRecommendations), in particular a method that
> actually appears twice in the file, which is probably the root cause. This
> first instance of this method should be removed, and the error fixed.
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
> in main
> result = stackAdvisor.validateConfigurations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 536, in validateConfigurations
> validationItems = self.getConfigurationsValidationItems(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 612, in getConfigurationsValidationItems
> recommendations = self.recommendConfigurations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 766, in recommendConfigurations
> serviceAdvisor.getServiceConfigurationRecommendations(configurations,
> clusterSummary, services, hosts)
> File
> "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py",
> line 103, in getServiceConfigurationRecommendations
> stormUIServerPort =
> services["configurations"]["storm-site"]["properties"]["ui.port"]
> KeyError: 'storm-site'
> 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28]
> AbstractResourceProvider:90 - Error occurred during validation
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException:
> Stack Advisor reported an error: KeyError: 'storm-site'
> StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (METRON-564) Mpack shows UI error when changing configuration
[
https://issues.apache.org/jira/browse/METRON-564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-564:
---
Description:
When confirming configuration changes, the Ambari UI shows an error that it
cannot validate configuration consistency.
An error in the logs refers to the service advisor
(getServiceConfigurationRecommendations), in particular a method that actually
appears twice in the file, which is probably the root cause. This first
instance of this method should be removed, and the error fixed.
{code}
Traceback (most recent call last):
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
in
main(sys.argv)
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
in main
result = stackAdvisor.validateConfigurations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 536, in validateConfigurations
validationItems = self.getConfigurationsValidationItems(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 612, in getConfigurationsValidationItems
recommendations = self.recommendConfigurations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 766, in recommendConfigurations
serviceAdvisor.getServiceConfigurationRecommendations(configurations,
clusterSummary, services, hosts)
File
"/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py",
line 103, in getServiceConfigurationRecommendations
stormUIServerPort =
services["configurations"]["storm-site"]["properties"]["ui.port"]
KeyError: 'storm-site'
10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28]
AbstractResourceProvider:90 - Error occurred during validation
org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack
Advisor reported an error: KeyError: 'storm-site'
StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out
{code}
was:
When confirming configuration changes, the Ambari UI shows an error that it
cannot validate configuration consistency.
An error in the logs refers to the service advisor
(getServiceConfigurationRecommendations), in particular a method that actually
appears twice in the file, which is probably the root cause. This first
instance of this method should be removed, and the error fixed.
Traceback (most recent call last):
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
in
main(sys.argv)
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
in main
result = stackAdvisor.validateConfigurations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 536, in validateConfigurations
validationItems = self.getConfigurationsValidationItems(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 612, in getConfigurationsValidationItems
recommendations = self.recommendConfigurations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 766, in recommendConfigurations
serviceAdvisor.getServiceConfigurationRecommendations(configurations,
clusterSummary, services, hosts)
File
"/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py",
line 103, in getServiceConfigurationRecommendations
stormUIServerPort =
services["configurations"]["storm-site"]["properties"]["ui.port"]
KeyError: 'storm-site'
10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28]
AbstractResourceProvider:90 - Error occurred during validation
org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack
Advisor reported an error: KeyError: 'storm-site'
StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out
> Mpack shows UI error when changing configuration
>
>
> Key: METRON-564
> URL: https://issues.apache.org/jira/browse/METRON-564
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.2.1BETA
>Reporter: Justin Leet
>Assignee: Justin Leet
>
> When confirming configuration changes, the Ambari UI shows an error that it
> cannot validate configuration consistency.
> An error in the logs refers to the service advisor
> (getServiceConfigurationRecommendations), in particular a method that
> actually appears twice in the file, which is probably the root cause. This
> first instance of this method should be removed, and the error fixed.
> {code}
> Traceback (most recent call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115,
[jira] [Created] (METRON-564) Mpack shows UI error when changing configuration
Justin Leet created METRON-564: -- Summary: Mpack shows UI error when changing configuration Key: METRON-564 URL: https://issues.apache.org/jira/browse/METRON-564 Project: Metron Issue Type: Bug Affects Versions: 0.2.1BETA Reporter: Justin Leet Assignee: Justin Leet When confirming configuration changes, the Ambari UI shows an error that it cannot validate configuration consistency. An error in the logs refers to the service advisor (getServiceConfigurationRecommendations), in particular a method that actually appears twice in the file, which is probably the root cause. This first instance of this method should be removed, and the error fixed. Traceback (most recent call last): File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 158, in main(sys.argv) File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 115, in main result = stackAdvisor.validateConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 536, in validateConfigurations validationItems = self.getConfigurationsValidationItems(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 612, in getConfigurationsValidationItems recommendations = self.recommendConfigurations(services, hosts) File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py", line 766, in recommendConfigurations serviceAdvisor.getServiceConfigurationRecommendations(configurations, clusterSummary, services, hosts) File "/var/lib/ambari-server/resources/common-services/METRON/0.3.0/service_advisor.py", line 103, in getServiceConfigurationRecommendations stormUIServerPort = services["configurations"]["storm-site"]["properties"]["ui.port"] KeyError: 'storm-site' 10 Nov 2016 15:29:58,186 WARN [ambari-client-thread-28] AbstractResourceProvider:90 - Error occurred during validation org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack Advisor reported an error: KeyError: 'storm-site' StdOut file: /var/run/ambari-server/stack-recommendations/18/stackadvisor.out -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-563) Ambari Metron service uses incorrect port for installing Elasticsearch templates
Justin Leet created METRON-563: -- Summary: Ambari Metron service uses incorrect port for installing Elasticsearch templates Key: METRON-563 URL: https://issues.apache.org/jira/browse/METRON-563 Project: Metron Issue Type: Bug Affects Versions: 0.2.1BETA Reporter: Justin Leet The port entered by the user at install time should be the binary port (typically 9300), which is used by the indexing bolt. The installation of templates expects the HTTP port for Elasticsearch (typically 9200). There are (probably) two options for fixes here: 1) Split the ES url and have something like ES Server, ES HTTP port, ES Binary port 2) Have the template loading go through the binary port. I assume this can be done, but I don't know enough about it to say for sure. This can be worked around by setting the ES url to :9200, installing the templates, and swapping back to :9300. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (METRON-525) Unable to start PCAP topology
[ https://issues.apache.org/jira/browse/METRON-525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644318#comment-15644318 ] Justin Leet commented on METRON-525: There's a couple things going on related to this: First, the PCAP topology has a different startup script (start_pcap_topology.sh). This is the core problem and [~james.sirota] updated some docs to make this more clear and accessible at: https://cwiki.apache.org/confluence/display/METRON/Launching+Metron+Topologies https://cwiki.apache.org/confluence/display/METRON/Bulk+Loading+Enrichments+and+Pruning+Data However, this doesn't apply to the Ambari Mpack, because until recently the PCAP RPM wasn't even installed and there is no service for managing it. I took a bit of time to spin it up and provide a manual workaround. It couple manual steps right now (assuming pycapa is up and running). Both need to be done before running start_pcap_topology.sh 1) /usr/metron/0.2.1BETA/config/pcap.properties needs to have kafka.zk set appropriately. 2) /apps/metron/pcap on HDFS needs to be created, chowned to metron:hadoop, and chmoded to 775 At this point, I was able to get both the topology and the pcap_query.sh running and producing data. To fix these as a service, 1) is just exposing the pcap.properties in Ambari and 2) is just creating the directory with appropriate perms (we do exactly the same thing for /apps/metron/enrichment), plus the surrounding build out of the Ambari definition. > Unable to start PCAP topology > - > > Key: METRON-525 > URL: https://issues.apache.org/jira/browse/METRON-525 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.2BETA >Reporter: Neha Sinha > > The following error is seen while starting PCAP topology :- > = > [root@metron-s-10 ~]# /usr/metron/0.2.1BETA/bin/start_parser_topology.sh -k > metron-s-10.openstacklocal:6667 -z metron-s-10.openstacklocal:2181 -s pcap > Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name= > -Dstorm.options= -Dstorm.home=/grid/0/hdp/2.4.3.0-227/storm > -Dstorm.log.dir=/grid/0/log/storm > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > -Dstorm.conf.file= -cp > /grid/0/hdp/2.4.3.0-227/storm/lib/log4j-api-2.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/cheshire-5.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/compojure-1.1.3.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/tools.logging-0.2.3.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/core.incubator-0.1.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/jline-0.9.94.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-core-1.1.5.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/java.classpath-0.2.2.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/slf4j-api-1.7.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/zookeeper.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/disruptor-2.10.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/log4j-core-2.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/jackson-core-2.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/tigris-0.1.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/reflectasm-1.07-shaded.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clj-stacktrace-0.2.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/commons-codec-1.6.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clojure-1.6.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-jetty-adapter-1.3.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-json-0.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/servlet-api-2.5.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/tools.namespace-0.2.4.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clj-time-0.8.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-devel-1.3.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/hadoop-auth-2.7.1.2.4.3.0-227.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/jackson-dataformat-smile-2.3.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/hiccup-0.3.6.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/asm-4.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/storm-core-0.10.0.2.4.3.0-227.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/clout-1.0.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ns-tracker-0.2.2.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/minlog-1.2.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/oncrpc-1.0.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/log4j-slf4j-impl-2.1.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/gmetric4j-1.0.7.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/ring-servlet-1.3.0.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/kryo-2.21.jar:/grid/0/hdp/2.4.3.0-227/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/metron/0.2.1BETA/lib/metron-parsers-0.2.1BETA-uber.jar:/usr/hdp/current/storm-supervisor/conf:/grid/0/hdp/2.4.3.0-227/storm/bin > -Dstorm.jar=/usr/metron/0.2.1BETA/lib/metron-parsers-0.2.1BETA-uber.jar > org.apache.metron.parsers.topology.ParserTopologyCLI -k > metron-s-10.openstacklocal:6667 -z metron-s-10.openstacklocal:2181 -s pcap > 05:59:01.065 [main] INFO
[jira] [Created] (METRON-553) Ambari Metron config should have "Test Connection" button for MySql and Elasticsearch
Justin Leet created METRON-553: -- Summary: Ambari Metron config should have "Test Connection" button for MySql and Elasticsearch Key: METRON-553 URL: https://issues.apache.org/jira/browse/METRON-553 Project: Metron Issue Type: Improvement Affects Versions: 0.2.1BETA Reporter: Justin Leet Right now, configs are entered while installing (or altering) Metron in Ambari. It would be nice to have a button to test connection for MySql and Elasticsearch so that you know the configs are correct before installing and spinning up everything. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-552) Ambari Mpack should be able to manage pcap topology
Justin Leet created METRON-552: -- Summary: Ambari Mpack should be able to manage pcap topology Key: METRON-552 URL: https://issues.apache.org/jira/browse/METRON-552 Project: Metron Issue Type: Improvement Affects Versions: 0.2.1BETA Reporter: Justin Leet Right now, the mpack installs the pcap RPM, but doesn't do any management of it. This should be handled by Ambari. With the current boundaries of the mpack (i.e. responsibility begins at kafka, not before), using pycapa to produce data to Kafka wouldn't be managed by Ambari. As a workaround (which will also have to happen in the Ambari service), pcap topology can be run on an Ambari managed Metron instance by 1) Updating /usr/metron/0.2.1BETA/config/pcap.properties by editing kafka.zk manually. Ambari should manage these configs itself, once the service is up. 2) Creating /apps/metron/pcap on HDFS, owning it to metron:hadoop, and updating permissions to 775 on the folder. At this point, /usr/metron/0.2.1BETA/bin/start_pcap_topology.sh should be able to be run normally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-520) /apps/metron/enrichment directory does not get created for Metron cluster deployed via Ambari
[ https://issues.apache.org/jira/browse/METRON-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-520: --- Fix Version/s: 0.2.2BETA > /apps/metron/enrichment directory does not get created for Metron cluster > deployed via Ambari > - > > Key: METRON-520 > URL: https://issues.apache.org/jira/browse/METRON-520 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Neha Sinha >Assignee: Justin Leet > Fix For: 0.2.2BETA > > > 1.Deploy Metron cluster via Ambari > 2. Replay Bro logs to generate bro elasticsearch indices > 3. The bro enriched and indexed data should be written to the HDFS at :- > /apps/metron/enrichment > The indexed data gets written to "/apps/metron/enrichment" for metron setups > that get deployed via Ansible, however, this path does not get created for > clusters deployed via Ambari. > Output for "hdfs dfs -ls" command for clusters deployed via Ansible > [hdfs@metron-ansible-3 ~]$ hdfs dfs -ls /apps/metron > Found 2 items > drwxrwxr-x - storm hadoop 0 2016-10-24 11:41 > /apps/metron/enrichment > drwxrwxr-x - hdfs hadoop 0 2016-10-24 11:03 /apps/metron/patterns > Output for "hdfs dfs -ls" command for clusters deployed via Ambari > [hdfs@metron-s-10 ~]$ hdfs dfs -ls /apps/metron > Found 7 items > -rwxr-xr-x 3 hdfs hdfs 13427 2016-10-25 10:02 /apps/metron/asa > -rwxr-xr-x 3 hdfs hdfs 5203 2016-10-25 10:02 /apps/metron/common > -rwxr-xr-x 3 hdfs hdfs524 2016-10-25 10:02 /apps/metron/fireeye > -rwxr-xr-x 3 hdfs hdfs 2552 2016-10-25 10:02 /apps/metron/sourcefire > -rwxr-xr-x 3 hdfs hdfs180 2016-10-25 10:02 /apps/metron/squid > -rwxr-xr-x 3 hdfs hdfs 2221 2016-10-25 10:02 /apps/metron/websphere > -rwxr-xr-x 3 hdfs hdfs879 2016-10-25 10:02 /apps/metron/yaf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-520) /apps/metron/enrichment directory does not get created for Metron cluster deployed via Ambari
[ https://issues.apache.org/jira/browse/METRON-520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-520: -- Assignee: Justin Leet > /apps/metron/enrichment directory does not get created for Metron cluster > deployed via Ambari > - > > Key: METRON-520 > URL: https://issues.apache.org/jira/browse/METRON-520 > Project: Metron > Issue Type: Bug >Affects Versions: 0.2.1BETA >Reporter: Neha Sinha >Assignee: Justin Leet > > 1.Deploy Metron cluster via Ambari > 2. Replay Bro logs to generate bro elasticsearch indices > 3. The bro enriched and indexed data should be written to the HDFS at :- > /apps/metron/enrichment > The indexed data gets written to "/apps/metron/enrichment" for metron setups > that get deployed via Ansible, however, this path does not get created for > clusters deployed via Ambari. > Output for "hdfs dfs -ls" command for clusters deployed via Ansible > [hdfs@metron-ansible-3 ~]$ hdfs dfs -ls /apps/metron > Found 2 items > drwxrwxr-x - storm hadoop 0 2016-10-24 11:41 > /apps/metron/enrichment > drwxrwxr-x - hdfs hadoop 0 2016-10-24 11:03 /apps/metron/patterns > Output for "hdfs dfs -ls" command for clusters deployed via Ambari > [hdfs@metron-s-10 ~]$ hdfs dfs -ls /apps/metron > Found 7 items > -rwxr-xr-x 3 hdfs hdfs 13427 2016-10-25 10:02 /apps/metron/asa > -rwxr-xr-x 3 hdfs hdfs 5203 2016-10-25 10:02 /apps/metron/common > -rwxr-xr-x 3 hdfs hdfs524 2016-10-25 10:02 /apps/metron/fireeye > -rwxr-xr-x 3 hdfs hdfs 2552 2016-10-25 10:02 /apps/metron/sourcefire > -rwxr-xr-x 3 hdfs hdfs180 2016-10-25 10:02 /apps/metron/squid > -rwxr-xr-x 3 hdfs hdfs 2221 2016-10-25 10:02 /apps/metron/websphere > -rwxr-xr-x 3 hdfs hdfs879 2016-10-25 10:02 /apps/metron/yaf -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-425) Stellar transformation fails to handle special characters
[
https://issues.apache.org/jira/browse/METRON-425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet reassigned METRON-425:
--
Assignee: Justin Leet
> Stellar transformation fails to handle special characters
> -
>
> Key: METRON-425
> URL: https://issues.apache.org/jira/browse/METRON-425
> Project: Metron
> Issue Type: Bug
>Reporter: Neha Sinha
>Assignee: Justin Leet
>
> I updated the snort parser file to have the following stellar transformation
> :-
> PARSER Config: snort
> {
> "parserClassName":"org.apache.metron.parsers.snort.BasicSnortParser",
> "sensorTopic":"snort",
> "parserConfig": {},
> "fieldTransformations" : [
> {
> "transformation" : "STELLAR"
> ,"output" : [ "is_alert","newStellarField","isAlert"]
> ,"config" :
> { "is_alert" : "false",
> "isAlert" : "false",
> "newStellarField" : "<>" }
> }
> ]
> }
> I get the following exception/error for the snort logs :-
> 2016-09-13 11:30:32.765 o.a.m.p.BasicParser [TRACE] [Metron] Message conforms
> to schema: {"msg":"\"'snort test
> alert'\"","sig_rev":"0","ip_dst_port":"80","ethsrc":"00:00:00:00:00:00","tcpseq":"0x5869E532","dgmlen":"40","icmpid":"","tcplen":"","tcpwindow":"0xFA02","icmpseq":"","tcpack":"0x3E05E218","protocol":"TCP","ip_dst_addr":"72.34.49.86","original_string":"09\/13-11:30:25.703857
> ,1,999158,0,\"'snort test
> alert'\",TCP,192.168.138.158,49204,72.34.49.86,80,00:00:00:00:00:00,00:00:00:00:00:00,0x3C,***A,0x5869E532,0x3E05E218,,0xFA02,128,0,2508,40,40960","icmpcode":"","tos":"0","id":"2508","ip_src_addr":"192.168.138.158","timestamp":1473766928857,"ethdst":"00:00:00:00:00:00","is_alert":"true","ttl":"128","ethlen":"0x3C","iplen":"40960","icmptype":"","ip_src_port":"49204","tcpflags":"***A","sig_id":"999158","sig_generator":"1"}
> 2016-09-13 11:30:32.766 b.s.d.executor [ERROR]
> org.apache.metron.common.dsl.ParseException: Syntax error @ 1:0 no viable
> alternative at input '<'
> at
> org.apache.metron.common.dsl.ErrorListener.syntaxError(ErrorListener.java:34)
> ~[stormjar.jar:?]
> at
> org.antlr.v4.runtime.ProxyErrorListener.syntaxError(ProxyErrorListener.java:65)
> ~[stormjar.jar:?]
> at org.antlr.v4.runtime.Parser.notifyErrorListeners(Parser.java:558)
> ~[stormjar.jar:?]
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.reportNoViableAlternative(DefaultErrorStrategy.java:310)
> ~[stormjar.jar:?]
> at
> org.antlr.v4.runtime.DefaultErrorStrategy.reportError(DefaultErrorStrategy.java:147)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.stellar.generated.StellarParser.transformation_expr(StellarParser.java:300)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.stellar.generated.StellarParser.transformation(StellarParser.java:146)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.stellar.BaseStellarProcessor.parse(BaseStellarProcessor.java:92)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.field.transformation.StellarTransformation.map(StellarTransformation.java:46)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.configuration.FieldTransformer.transform(FieldTransformer.java:111)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.configuration.FieldTransformer.transformAndUpdate(FieldTransformer.java:123)
> ~[stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:125)
> [stormjar.jar:?]
> at
> backtype.storm.daemon.executor$fn__5492$tuple_action_fn__5494.invoke(executor.clj:684)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at
> backtype.storm.daemon.executor$mk_task_receiver$fn__5415.invoke(executor.clj:431)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at
> backtype.storm.disruptor$clojure_handler$reify__4991.onEvent(disruptor.clj:58)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at
> backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at
> backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at
> backtype.storm.daemon.executor$fn__5492$fn__5505$fn__5556.invoke(executor.clj:813)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at backtype.storm.util$async_loop$fn__644.invoke(util.clj:479)
> [storm-core-0.10.0.2.4.2.0-258.jar:0.10.0.2.4.2.0-258]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.6.0.jar:?]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
> Caused by: org.antlr.v4.runtime.NoViableAltException
>
[jira] [Created] (METRON-482) Add logging to GrokParser to indicate supplied TimeZone
Justin Leet created METRON-482: -- Summary: Add logging to GrokParser to indicate supplied TimeZone Key: METRON-482 URL: https://issues.apache.org/jira/browse/METRON-482 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet TimeZone can be passed to the GrokParser and is used in timestamp conversions. However, it is not logged when this is used, making it difficult to tell why the conversion process resulted in the given values. Adding logging will make this easier to understand. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (METRON-410) mysql_server's MySQL install causes mutually assured destruction when installed on the same machine as the Ambari Hive MySQL
[ https://issues.apache.org/jira/browse/METRON-410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-410: --- Issue Type: Improvement (was: Sub-task) Parent: (was: METRON-427) > mysql_server's MySQL install causes mutually assured destruction when > installed on the same machine as the Ambari Hive MySQL > > > Key: METRON-410 > URL: https://issues.apache.org/jira/browse/METRON-410 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Priority: Minor > Original Estimate: 48h > Remaining Estimate: 48h > > Metron's mysql_server MySQL install causes mutually assured destruction when > installed on the same machine as the Ambari Hive MySQL. Here is the startup > error you get afterwards. > https://gist.github.com/JonZeolla/2ed4161c141ba32a3e8a0d6ce9718779 > In the short term, maybe add a check so they won't live on the same box. > Long term, allow them to coexist? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-427) Create Ambari Management Pack for Metron Installation
[ https://issues.apache.org/jira/browse/METRON-427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-427: -- Assignee: Justin Leet > Create Ambari Management Pack for Metron Installation > - > > Key: METRON-427 > URL: https://issues.apache.org/jira/browse/METRON-427 > Project: Metron > Issue Type: New Feature >Reporter: Justin Leet >Assignee: Justin Leet > > Right now, Metron depends on Ambari blueprints, in the Ansible scripts, to > deploy onto a cluster. > To ease installation, a full Ambari Management Pack > (https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs) can be > used to lay down topologies, etc. > The current expectation is that the boundaries of this would cover from Kafka > to the indexes. The dev list has additional discussion about whether or not > sensor install and what should exist beyond minimum viable product. > Additional follow-on tickets would be created based on the results of both > that discussion and any discussion on this ticket. > A minimum viable product for this would cover > * Laying down topologies (parsers, enrichment, and indexing) > * Starting and stopping topologies > * Setting up configuration > * Setting up bits (Using RPMs currently built locally) > * Set up infra dependencies (MySql and Elasticsearch) > At this point, the MVP could take data from Kafka, run it through the > topologies, and make it available in the output Elasticsearch Indexes. > A good deal of the ground work for this is already completed (several Service > Definitions, along with the RPM creation). Relevant Jira's are: > * METRON-383 (Create Ambari Service Definition for Metron Parsers) > * METRON-385 (Create Ambari Service Definition for Indexing) > * METRON-386 (Create Ambari Service Definition for Elasticsearch) > * METRON-357 (Create Ambari Service Definition for Kibana) > * METRON-214 (Build RPM Packages for Deployment) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-427) Create Ambari Management Pack for Metron Installation
Justin Leet created METRON-427: -- Summary: Create Ambari Management Pack for Metron Installation Key: METRON-427 URL: https://issues.apache.org/jira/browse/METRON-427 Project: Metron Issue Type: New Feature Reporter: Justin Leet Right now, Metron depends on Ambari blueprints, in the Ansible scripts, to deploy onto a cluster. To ease installation, a full Ambari Management Pack (https://cwiki.apache.org/confluence/display/AMBARI/Management+Packs) can be used to lay down topologies, etc. The current expectation is that the boundaries of this would cover from Kafka to the indexes. The dev list has additional discussion about whether or not sensor install and what should exist beyond minimum viable product. Additional follow-on tickets would be created based on the results of both that discussion and any discussion on this ticket. A minimum viable product for this would cover * Laying down topologies (parsers, enrichment, and indexing) * Starting and stopping topologies * Setting up configuration * Setting up bits (Using RPMs currently built locally) * Set up infra dependencies (MySql and Elasticsearch) At this point, the MVP could take data from Kafka, run it through the topologies, and make it available in the output Elasticsearch Indexes. A good deal of the ground work for this is already completed (several Service Definitions, along with the RPM creation). Relevant Jira's are: * METRON-383 (Create Ambari Service Definition for Metron Parsers) * METRON-385 (Create Ambari Service Definition for Indexing) * METRON-386 (Create Ambari Service Definition for Elasticsearch) * METRON-357 (Create Ambari Service Definition for Kibana) * METRON-214 (Build RPM Packages for Deployment) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-385) Create Ambari Service Definition for Indexing
Justin Leet created METRON-385: -- Summary: Create Ambari Service Definition for Indexing Key: METRON-385 URL: https://issues.apache.org/jira/browse/METRON-385 Project: Metron Issue Type: New Feature Reporter: Justin Leet Assignee: Justin Leet To pull everything into an easier install through Ambari, create a service definition to automatically install and handle the indexing topology appropriately. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (METRON-334) Travis CI cache Maven dependencies
Justin Leet created METRON-334: -- Summary: Travis CI cache Maven dependencies Key: METRON-334 URL: https://issues.apache.org/jira/browse/METRON-334 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet Priority: Minor Per https://docs.travis-ci.com/user/caching/ Dependencies can be cached to avoid having them be redownloaded every build. Specifically we want to cache $HOME/.m2 per the docs This should speed up Travis builds a little bit, although the majority of the time will still be on testing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-319) Automate RPM Builds
[ https://issues.apache.org/jira/browse/METRON-319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-319: -- Assignee: Justin Leet > Automate RPM Builds > --- > > Key: METRON-319 > URL: https://issues.apache.org/jira/browse/METRON-319 > Project: Metron > Issue Type: New Feature >Reporter: David M. Lyle >Assignee: Justin Leet > Fix For: 0.2.1BETA > > > Create automation that will build the RPMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (METRON-299) Build Parsers Deployment RPM
[ https://issues.apache.org/jira/browse/METRON-299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-299: -- Assignee: Justin Leet > Build Parsers Deployment RPM > > > Key: METRON-299 > URL: https://issues.apache.org/jira/browse/METRON-299 > Project: Metron > Issue Type: Sub-task >Reporter: David M. Lyle >Assignee: Justin Leet > > Relocate metron_streaming Parsers file copy and initial setup to RPM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
