[jira] [Created] (METRON-2324) verify_licenses.sh and list_dependencies.sh don't properly handle maven failures
Justin Leet created METRON-2324:
---
Summary: verify_licenses.sh and list_dependencies.sh don't
properly handle maven failures
Key: METRON-2324
URL: https://issues.apache.org/jira/browse/METRON-2324
Project: Metron
Issue Type: Bug
Reporter: Justin Leet
It produces different results for me on Travis than locally and implies I need
to readd a dep (because seems right: e.g. JUnit is a correct dependency of our
compiled source: specifically metron-integration-test).
The difference between local and Travis seems to be that our shell scripts are
faulty. They try to exit if an error occurs, but exit during a pipe chain
doesn't stop the chain, so it ends up passing along to builtins that exit with
0 (because for grep, sort, etc, nothing went wrong). Note that mvn
dependency:list should pretty much always fail midway because we don't have the
Metron artifacts built, so we can't run a build all the way through (because it
can't find intermodule deps). Locally, they tend to always be built for the
devs, so it succeeds.
To see this sort of thing in action:
{code:java}
{ echo "test" && exit 1 } | grep t
echo $?
{code}
This will still run the grep on test, and because the grep succeeds, the return
code is 0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (METRON-2311) Remove JUnit from all our uber jars
Justin Leet created METRON-2311: --- Summary: Remove JUnit from all our uber jars Key: METRON-2311 URL: https://issues.apache.org/jira/browse/METRON-2311 Project: Metron Issue Type: Bug Reporter: Justin Leet We're pulling in some JUnit (and probably other cruft) from some of our dependencies (e.g. json-simple), which ends up getting packaged into the uber jar. Should be cleaned up. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2310) Remove metron-integration-test from compile dependencies
[ https://issues.apache.org/jira/browse/METRON-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2310: Description: Discovered while debugging https://github.com/apache/metron/pull/1554. In several poms, we carry through metron-integration-test as a compile time dependency. This means it gets pulled into the uber jar, and suddenly we have a ton of mockito and other test time dependencies being carried through to what we deploy. was: Discovered while debugging https://github.com/apache/metron/pull/1554. In several poms, we carry through metron-integration-test as a compile time dependency. This means it gets pulled into the uber jar, and suddenly we have a ton of junit and other test time dependencies being carried through to what we deploy. > Remove metron-integration-test from compile dependencies > > > Key: METRON-2310 > URL: https://issues.apache.org/jira/browse/METRON-2310 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Discovered while debugging https://github.com/apache/metron/pull/1554. > In several poms, we carry through metron-integration-test as a compile time > dependency. This means it gets pulled into the uber jar, and suddenly we have > a ton of mockito and other test time dependencies being carried through to > what we deploy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2310) Remove metron-integration-test from compile dependencies
[ https://issues.apache.org/jira/browse/METRON-2310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2310: Summary: Remove metron-integration-test from compile dependencies (was: Remove metron-indexing-test from compile dependencies) > Remove metron-integration-test from compile dependencies > > > Key: METRON-2310 > URL: https://issues.apache.org/jira/browse/METRON-2310 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Priority: Major > > Discovered while debugging https://github.com/apache/metron/pull/1554. > In several poms, we carry through metron-integration-test as a compile time > dependency. This means it gets pulled into the uber jar, and suddenly we have > a ton of junit and other test time dependencies being carried through to what > we deploy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (METRON-2310) Remove metron-indexing-test from compile dependencies
Justin Leet created METRON-2310: --- Summary: Remove metron-indexing-test from compile dependencies Key: METRON-2310 URL: https://issues.apache.org/jira/browse/METRON-2310 Project: Metron Issue Type: Bug Reporter: Justin Leet Discovered while debugging https://github.com/apache/metron/pull/1554. In several poms, we carry through metron-integration-test as a compile time dependency. This means it gets pulled into the uber jar, and suddenly we have a ton of junit and other test time dependencies being carried through to what we deploy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (METRON-2307) Migrate to JUnit5
Justin Leet created METRON-2307:
---
Summary: Migrate to JUnit5
Key: METRON-2307
URL: https://issues.apache.org/jira/browse/METRON-2307
Project: Metron
Issue Type: Task
Reporter: Justin Leet
Opens up some possibilities in sharing our test infrastructure, and makes some
things easier/cleaner like properly testing exceptions. The catch is we have a
lot of JUnit and we occasionally mock static stuff and use {{@Rule}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (METRON-2263) Update Bro Plugin README with additional Docker testing requirements
Justin Leet created METRON-2263: --- Summary: Update Bro Plugin README with additional Docker testing requirements Key: METRON-2263 URL: https://issues.apache.org/jira/browse/METRON-2263 Project: Metron Issue Type: Improvement Reporter: Justin Leet Right now, there are (at least) 2 additional dependencies that need to be installed for running the Docker stuff with the Bro plugin on Mac. Both are available via homebrew. * wget - this gets used by the scripts. Alternatively, this could be converted to `curl`, which is available by default on Mac. * bash >= 4 - Associative arrays are used, but Mac only ships with bash 3.2, I believe for licensing issues. The scripts warn you, but there's no way to no without running them otherwise. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (METRON-2247) rpm-docker: Provide an option to bypass running rpmlint
[ https://issues.apache.org/jira/browse/METRON-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16924493#comment-16924493 ] Justin Leet commented on METRON-2247: - This runs through at least some common rpmlint issues that can show up: [https://fedoraproject.org/wiki/Common_Rpmlint_issues] I believe you can have runs that succeed in the dev environment, but that should still be corrected to meet expectations on an RPM, but could still potentially cause issues (permissions potentially spring to mind). This is purely speculative though, I barely qualify as an amateur in regards to RPMs. As an aside, we should try to brake the habit of calling it "full dev". There is no more quickdev/fulldev, and it's probably confusing for people new to Metron. It's just the development environment. > rpm-docker: Provide an option to bypass running rpmlint > --- > > Key: METRON-2247 > URL: https://issues.apache.org/jira/browse/METRON-2247 > Project: Metron > Issue Type: Improvement >Reporter: Dale Richardson >Assignee: Dale Richardson >Priority: Minor > > I notice that a large chunk of the time that an rpmbuild process takes is > spent running rpmlint. For people using the full dev process to spin up a > Metron cluster, running rpmlint is like running unit-testing - the assumption > is that the author of the code you are running has already done these tests. > I propose that running rpmlint should remain as part of a full dev build, but > that it would be useful to provide an argument that you pass to vagrant that > will cause the running of rpmlint to be skipped. This could speed up the > process of standing up a full dev cluster for those people who want it. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-1967) Create a base image for quick dev with HDP already installed and configured
[ https://issues.apache.org/jira/browse/METRON-1967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923413#comment-16923413 ] Justin Leet commented on METRON-1967: - I haven't looked at this in awhile, but I believe we'd probably just have a blueprint for the base image, then the Ambari REST APIs can be used to provision / configure / update a service. If it can all be done with blueprints, it probably makes it easier to move over, but I don't think the REST API calls are generally too different from the blueprint settings other than structurally. I just glanced through it, but [here's a walkthrough|https://markobigdata.com/2018/07/22/adding-service-to-hdp-using-rest-api/] I found on Google re: adding and configuring a service. It runs through Pig, but I can't imagine it'd be too different for Metron. > Create a base image for quick dev with HDP already installed and configured > --- > > Key: METRON-1967 > URL: https://issues.apache.org/jira/browse/METRON-1967 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Major > > What we should have is an image with HDP installed, such that quickdev only > deploys metron's packages. > > This should speed things up. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2242) Improve output of validate-jira-for-release script
[ https://issues.apache.org/jira/browse/METRON-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918954#comment-16918954 ] Justin Leet commented on METRON-2242: - Yeah, that's a pretty good summary. This doesn't get run very often, so I'd also say that it's not something that needs anything fancy. It just makes life a bit easier. Anything too complicated is just overkill. > Improve output of validate-jira-for-release script > -- > > Key: METRON-2242 > URL: https://issues.apache.org/jira/browse/METRON-2242 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Priority: Minor > > Right now, the validate-jira-for-release outputs the data in a whitespace > delimited format. Because values can have whitespace (e.g. "In Progress" or > "Justin Leet"), this means that easily sorting by contributor doesn't work. > This is important to contacting contributors to update Jira at release time. > Ideal output would be geared to the release manager being able to easily > contact the appropriate people to update their tickets. Speaking personally, > just making the output work with sort is sufficient, but it may be a better > default order. Additionally, if it's possible to pull out the contributor's > email (e.g. from the relevant commit), that would also make things easier. > If people want to rearrange the columns or something clever (e.g. have first > column be name, so people can easily find themselves in a general list > blast), that's another idea. > Finally, having a more obvious (or default) ability to just pass in version > "Next + 1" might make things a bit more clear, but it's not imperative. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2242) Improve output of validate-jira-for-release script
Justin Leet created METRON-2242: --- Summary: Improve output of validate-jira-for-release script Key: METRON-2242 URL: https://issues.apache.org/jira/browse/METRON-2242 Project: Metron Issue Type: Improvement Reporter: Justin Leet Right now, the validate-jira-for-release outputs the data in a whitespace delimited format. Because values can have whitespace (e.g. "In Progress" or "Justin Leet"), this means that easily sorting by contributor doesn't work. This is important to contacting contributors to update Jira at release time. Ideal output would be geared to the release manager being able to easily contact the appropriate people to update their tickets. Speaking personally, just making the output work with sort is sufficient, but it may be a better default order. Additionally, if it's possible to pull out the contributor's email (e.g. from the relevant commit), that would also make things easier. If people want to rearrange the columns or something clever (e.g. have first column be name, so people can easily find themselves in a general list blast), that's another idea. Finally, having a more obvious (or default) ability to just pass in version "Next + 1" might make things a bit more clear, but it's not imperative. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2111) Update public web site to point at 0.7.1 new release
[ https://issues.apache.org/jira/browse/METRON-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2111: Fix Version/s: Next + 1 > Update public web site to point at 0.7.1 new release > > > Key: METRON-2111 > URL: https://issues.apache.org/jira/browse/METRON-2111 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Update the website to reflect the release -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-682) Unify and Improve the Flat File Loader
[ https://issues.apache.org/jira/browse/METRON-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-682: --- Fix Version/s: 0.3.1 > Unify and Improve the Flat File Loader > -- > > Key: METRON-682 > URL: https://issues.apache.org/jira/browse/METRON-682 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.3.1 > > > Currently the flat file loader is deficient in a couple ways: > * It only supports importing local data despite there being a separate, > poorly named, application which supports importing enrichment via MapReduce > called threat_intel_loader.sh > * It does not support local imports from HDFS > * It does not support local imports from URLs > * It does not support importing zipped archives locally > * You cannot import more than one file at once > This JIRA will: > * Unify the MapReduce and local imports into one program and allow the user > to specify the import mode with a CLI flag > * Support local imports from HDFS and URLs > * Support local imports from zipped files > * Support importing more than one file at once -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-682) Unify and Improve the Flat File Loader
[ https://issues.apache.org/jira/browse/METRON-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918007#comment-16918007 ] Justin Leet commented on METRON-682: [~mmiklavcic] It's in the release changelog for 0.3.1, so I went ahead and updated and closed this out. > Unify and Improve the Flat File Loader > -- > > Key: METRON-682 > URL: https://issues.apache.org/jira/browse/METRON-682 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > Fix For: 0.3.1 > > > Currently the flat file loader is deficient in a couple ways: > * It only supports importing local data despite there being a separate, > poorly named, application which supports importing enrichment via MapReduce > called threat_intel_loader.sh > * It does not support local imports from HDFS > * It does not support local imports from URLs > * It does not support importing zipped archives locally > * You cannot import more than one file at once > This JIRA will: > * Unify the MapReduce and local imports into one program and allow the user > to specify the import mode with a CLI flag > * Support local imports from HDFS and URLs > * Support local imports from zipped files > * Support importing more than one file at once -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-682) Unify and Improve the Flat File Loader
[ https://issues.apache.org/jira/browse/METRON-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-682: -- Assignee: Casey Stella > Unify and Improve the Flat File Loader > -- > > Key: METRON-682 > URL: https://issues.apache.org/jira/browse/METRON-682 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella >Priority: Major > > Currently the flat file loader is deficient in a couple ways: > * It only supports importing local data despite there being a separate, > poorly named, application which supports importing enrichment via MapReduce > called threat_intel_loader.sh > * It does not support local imports from HDFS > * It does not support local imports from URLs > * It does not support importing zipped archives locally > * You cannot import more than one file at once > This JIRA will: > * Unify the MapReduce and local imports into one program and allow the user > to specify the import mode with a CLI flag > * Support local imports from HDFS and URLs > * Support local imports from zipped files > * Support importing more than one file at once -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2127) Update Maven repositories to https
Justin Leet created METRON-2127: --- Summary: Update Maven repositories to https Key: METRON-2127 URL: https://issues.apache.org/jira/browse/METRON-2127 Project: Metron Issue Type: Task Reporter: Justin Leet Assignee: Justin Leet A couple of our Maven repos can be updated to https. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2113) Update version to 0.7.2
[ https://issues.apache.org/jira/browse/METRON-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2113: Fix Version/s: Next + 1 > Update version to 0.7.2 > --- > > Key: METRON-2113 > URL: https://issues.apache.org/jira/browse/METRON-2113 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h > Remaining Estimate: 0h > > Need to update version to 0.7.2 following release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2111) Update public web site to point at 0.7.1 new release
Justin Leet created METRON-2111: --- Summary: Update public web site to point at 0.7.1 new release Key: METRON-2111 URL: https://issues.apache.org/jira/browse/METRON-2111 Project: Metron Issue Type: Task Reporter: Justin Leet Assignee: Justin Leet Update the website to reflect the release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2094) Create CentOS 7 Development Environment
[ https://issues.apache.org/jira/browse/METRON-2094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2094: Fix Version/s: 0.7.1 > Create CentOS 7 Development Environment > --- > > Key: METRON-2094 > URL: https://issues.apache.org/jira/browse/METRON-2094 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.1 > > Time Spent: 4h 50m > Remaining Estimate: 0h > > CentOS 6 RPMs are no longer distributed for HDP 3.1.0, only CentOS 7 RPMs. > Because of this we will likely need to transition Full Dev over to CentOS 7. > I don't see a downside to doing this since 6 is rather old and I assume that > most users run variants of 7 already anyways. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2018) Update prepare-commit to add Bro plugin tests
[ https://issues.apache.org/jira/browse/METRON-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2018: Fix Version/s: 0.7.1 > Update prepare-commit to add Bro plugin tests > - > > Key: METRON-2018 > URL: https://issues.apache.org/jira/browse/METRON-2018 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla >Assignee: Otto Fowler >Priority: Minor > Labels: patch-available > Fix For: 0.7.1 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > A significant amount of effort has been put into testing the bro plugin. We > should update the prepare-commit script in apache/metron to support it. > https://github.com/apache/metron/commit/42068d95192e7346bf0179399193a6afe01d9925#diff-1c903f7ce0ee24ca7d4e550f9dc31cc2R342 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2049) trunghuu
[ https://issues.apache.org/jira/browse/METRON-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2049: Affects Version/s: (was: 0.7.0) > trunghuu > > > Key: METRON-2049 > URL: https://issues.apache.org/jira/browse/METRON-2049 > Project: Metron > Issue Type: Bug >Reporter: Hà Trung Hữu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-2049) trunghuu
[ https://issues.apache.org/jira/browse/METRON-2049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831708#comment-16831708 ] Justin Leet commented on METRON-2049: - Closing per my comment at https://jira.apache.org/jira/browse/METRON-2047?focusedCommentId=16831705=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16831705 > trunghuu > > > Key: METRON-2049 > URL: https://issues.apache.org/jira/browse/METRON-2049 > Project: Metron > Issue Type: Bug >Affects Versions: 0.7.0 >Reporter: Hà Trung Hữu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-2048) trunghuu
[ https://issues.apache.org/jira/browse/METRON-2048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831707#comment-16831707 ] Justin Leet commented on METRON-2048: - Closing per my comment at https://jira.apache.org/jira/browse/METRON-2047?focusedCommentId=16831705=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16831705 > trunghuu > > > Key: METRON-2048 > URL: https://issues.apache.org/jira/browse/METRON-2048 > Project: Metron > Issue Type: Bug >Reporter: Hà Trung Hữu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-2047) hth
[ https://issues.apache.org/jira/browse/METRON-2047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16831705#comment-16831705 ] Justin Leet commented on METRON-2047: - [~trunghuu] It looks like there's a couple tickets that got created without any details? I'm going to close them, but if there's anything we can help with, let us know on the user list or flesh out a ticket. Thanks! > hth > --- > > Key: METRON-2047 > URL: https://issues.apache.org/jira/browse/METRON-2047 > Project: Metron > Issue Type: Bug >Reporter: Hà Trung Hữu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2093) Metron RC check script is outdated
[ https://issues.apache.org/jira/browse/METRON-2093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2093: Fix Version/s: 0.7.1 > Metron RC check script is outdated > -- > > Key: METRON-2093 > URL: https://issues.apache.org/jira/browse/METRON-2093 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 50m > Remaining Estimate: 0h > > There's a couple problems with the current script: > * It looks for the KEYS file in the current RC, which isn't where it lives > anymore. Now it's just top level of the actual dist. > * To be consistent between the repos, we're using '_' instead of '-' for the > separator between repo name and version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2093) Metron RC check script is outdated
Justin Leet created METRON-2093: --- Summary: Metron RC check script is outdated Key: METRON-2093 URL: https://issues.apache.org/jira/browse/METRON-2093 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet There's a couple problems with the current script: * It looks for the KEYS file in the current RC, which isn't where it lives anymore. Now it's just top level of the actual dist. * To be consistent between the repos, we're using '_' instead of '-' for the separator between repo name and version. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2090) Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency
[
https://issues.apache.org/jira/browse/METRON-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet reassigned METRON-2090:
---
Assignee: Ryan Merriman
> Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx
> dependency
> ---
>
> Key: METRON-2090
> URL: https://issues.apache.org/jira/browse/METRON-2090
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Full dev is failing during the build step with this error:
> {code:java}
> Failed to execute goal on project metron-writer-storm: Could not resolve
> dependencies for project org.apache.metron:metron-writer-storm:jar:0.7.1:
> Failure to find org.mortbay.jetty:jetty-util:jar:6.1.26.hwx in
> http://clojars.org/repo was cached in the local repository, resolution will
> not be reattempted until the update interval of clojars.org has elapsed or
> updates are forced{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-2090) Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency
[
https://issues.apache.org/jira/browse/METRON-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2090:
Fix Version/s: 0.7.1
> Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx
> dependency
> ---
>
> Key: METRON-2090
> URL: https://issues.apache.org/jira/browse/METRON-2090
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Full dev is failing during the build step with this error:
> {code:java}
> Failed to execute goal on project metron-writer-storm: Could not resolve
> dependencies for project org.apache.metron:metron-writer-storm:jar:0.7.1:
> Failure to find org.mortbay.jetty:jetty-util:jar:6.1.26.hwx in
> http://clojars.org/repo was cached in the local repository, resolution will
> not be reattempted until the update interval of clojars.org has elapsed or
> updates are forced{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-2091) SimpleHBaseEnrichmentWriterTest should be included in tests
[ https://issues.apache.org/jira/browse/METRON-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-2091: --- Assignee: Ryan Merriman > SimpleHBaseEnrichmentWriterTest should be included in tests > --- > > Key: METRON-2091 > URL: https://issues.apache.org/jira/browse/METRON-2091 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > The `SimpleHBaseEnrichmentWriterTest` test was accidentally left in the > metron-writer module instead of being moved to a new module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2091) SimpleHBaseEnrichmentWriterTest should be included in tests
[ https://issues.apache.org/jira/browse/METRON-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2091: Fix Version/s: 0.7.1 > SimpleHBaseEnrichmentWriterTest should be included in tests > --- > > Key: METRON-2091 > URL: https://issues.apache.org/jira/browse/METRON-2091 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The `SimpleHBaseEnrichmentWriterTest` test was accidentally left in the > metron-writer module instead of being moved to a new module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1963) Remove left over integration test from before refactoring
[ https://issues.apache.org/jira/browse/METRON-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1963: Fix Version/s: (was: 0.7.0) 0.7.1 > Remove left over integration test from before refactoring > - > > Key: METRON-1963 > URL: https://issues.apache.org/jira/browse/METRON-1963 > Project: Metron > Issue Type: Bug >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > https://github.com/apache/metron/tree/master/metron-platform/metron-parsers -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-589) Dist area should only contain latest release
[ https://issues.apache.org/jira/browse/METRON-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-589: --- Fix Version/s: (was: 0.7.1) 0.3.1 > Dist area should only contain latest release > > > Key: METRON-589 > URL: https://issues.apache.org/jira/browse/METRON-589 > Project: Metron > Issue Type: Bug >Reporter: John D. Ament >Assignee: Casey Stella >Priority: Major > Fix For: 0.3.1 > > > The dist area https://dist.apache.org/repos/dist/release/incubator/metron/ > should only contain the latest release. Older releases are found in the > archives. > Most projects don't directly link to this area, but instead provide the dyn > style links. See Geode for example: > http://geode.incubator.apache.org/releases/ - this works for older releases > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-589) Dist area should only contain latest release
[ https://issues.apache.org/jira/browse/METRON-589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825340#comment-16825340 ] Justin Leet commented on METRON-589: [~msmiklavcic] You're right, I'll figure it out. This is because I did a bulk search and apparently hit this ticket without catching it. > Dist area should only contain latest release > > > Key: METRON-589 > URL: https://issues.apache.org/jira/browse/METRON-589 > Project: Metron > Issue Type: Bug >Reporter: John D. Ament >Assignee: Casey Stella >Priority: Major > Fix For: 0.7.1 > > > The dist area https://dist.apache.org/repos/dist/release/incubator/metron/ > should only contain the latest release. Older releases are found in the > archives. > Most projects don't directly link to this area, but instead provide the dyn > style links. See Geode for example: > http://geode.incubator.apache.org/releases/ - this works for older releases > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-650) Remove Kraken Maven dependency on opensoc github repo
[
https://issues.apache.org/jira/browse/METRON-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16825330#comment-16825330
]
Justin Leet commented on METRON-650:
I don't think we have a typical answer. I'm fine with Done + no version.
> Remove Kraken Maven dependency on opensoc github repo
> -
>
> Key: METRON-650
> URL: https://issues.apache.org/jira/browse/METRON-650
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Priority: Major
> Fix For: 0.3.1
>
>
> Metron common's pom is referencing the opensoc repo to get Kraken:
>
>
> Metron-Kraken-Repo
> Metron Kraken Repository
> https://raw.github.com/opensoc/kraken/mvn-repo
>
>
> ...
>
> org.krakenapps
> kraken-pcap
> ${global_pcap_version}
>
>
> slf4j-api
> org.slf4j
>
>
> slf4j-simple
> org.slf4j
>
>
>
> This library does not currently reside in mvn central (searched by package
> and libname) and should instead be staged in Apache with Metron.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI
[ https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1434: Fix Version/s: (was: 0.4.3) Next + 1 > Ability to deploy Metron full dev as a single node (centos6) to AWS with > vagrant and AMI > > > Key: METRON-1434 > URL: https://issues.apache.org/jira/browse/METRON-1434 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.3 > Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 > Cores, 64 bit] based on metron-deployment/development/centos6/ > ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS > iii) AMI "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge >Reporter: ashah >Assignee: ashah >Priority: Minor > Labels: AWS, EC2, vagrant > Fix For: Next + 1 > > > The pull request has two potential solutions for deploying to Metron to AWS. > i) AWS Single Node Cluster Deployment Using Vagrant > ii) AWS Single Node Cluster Deployment Using an AMI > *i) AWS Single Node Cluster Deployment Using Vagrant:* > I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. > It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) > that was in git. I'm assuming the latest that was in git is 0.4.3 that is > progress. > _Instructions:_ > [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant] > > *ii) AWS Single Node Cluster Deployment Using an AMI:* > In addition to including instructions for deploying 0.4.3 using Vagrant on a > Mac I also included instructions for deploying to AWS using a preexisting AMI > (I made the following AMI public as a proof-of-concept: "GCR-Xetron > Demo"/"ami-93cb4ff7"). This might be the quicket option for deployment since > no preinstall requirments would be needed. > _Instructions:_ > [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami] > > PULL REQUEST CHECKLIST ITEMS > *"For documentation related changes:"* > Running cd site-book, mvn site produced "BUILD SUCCESS" > > *"For code changes:"* > Was not sure where to run "mvn -q clean integration-test install && > dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not > changed. For i) the following code were newly added/modified: > [Vagrant to work with aws] > [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile] > [changing eth1 to eth0] > [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/inventory/group_vars/all] > > > *"How the change may be verified: "* > You should be able to see Ambari after following instructions > (http://:8080) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-865) Additional Mpack bug fixes and improvements, that affect Ambari database
[
https://issues.apache.org/jira/browse/METRON-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-865:
---
Fix Version/s: (was: 0.3.1)
0.4.0
> Additional Mpack bug fixes and improvements, that affect Ambari database
>
>
> Key: METRON-865
> URL: https://issues.apache.org/jira/browse/METRON-865
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
> Environment: Centos7
>Reporter: Matt Foley
>Priority: Major
> Fix For: 0.4.0
>
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). Almost all items relate to Elasticsearch.
> About half the fixes did not impact the Ambari database, and were done in
> METRON-634 (PR#532).
> This jira provides the work items for changes that do impact the Ambari
> database, and should therefore be done with an Mpack version bump and
> database upgrade script. Implementation of these may be seen in the closed PR
> https://github.com/apache/metron/pull/425
> h2. Affects Ambari database, needs db upgrade script:
> * status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name (to distinguish ES cluster from Stack
> cluster)
> zen_discovery_ping_unicast_hosts -> es_cluster_hosts
> network_host -> network_bindings (these are in fact interface names, not
> host names)
> {code}
> * "data_dir" apparently should be eliminated (from elastic-sysconfig) in
> preference for "path_data" (in elastic-site.xml). The latter value ends up
> overriding the former anyway, but the existence of the former is confusing
> and unnecessary.
> * All four configuration parameters in elastic-env.xml should be moved to
> elastic-site.xml, because they are all reasonable to set in a "site.xml" file
> and do end up in the .yml file that ES uses instead of "site.xml", and do NOT
> end up in environment variables. The only parameters that end up in env vars
> are set in elastic-sysconfig, and the ES launch process in fact ignores the
> elastic-env.sh file that is templated in elastic-env.xml (which consists only
> of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and
> hence entirely remove elastic-env.xml, or we could choose to keep the small
> elastic-env.sh file and its template, just to remind people that it is
> necessary to have JAVA_HOME defined.
> * In METRON/0.3.0/configuration/metron-env.xml and
> METRON/0.3.0/package/scripts/params/params_linux.py, the value
> "metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is
> appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus
> it can be removed from metron-env.xml and set to
> "\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also
> eliminates a really unacceptable use of "double format".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-650) Remove Kraken Maven dependency on opensoc github repo
[
https://issues.apache.org/jira/browse/METRON-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824743#comment-16824743
]
Justin Leet commented on METRON-650:
[~msmiklavcic] Is this ticket still relevant? Can we get the fix version
cleaned up (either to nothing if it's not fixed, or to whatever version it's
fixed in)?
> Remove Kraken Maven dependency on opensoc github repo
> -
>
> Key: METRON-650
> URL: https://issues.apache.org/jira/browse/METRON-650
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Priority: Major
> Fix For: 0.3.1
>
>
> Metron common's pom is referencing the opensoc repo to get Kraken:
>
>
> Metron-Kraken-Repo
> Metron Kraken Repository
> https://raw.github.com/opensoc/kraken/mvn-repo
>
>
> ...
>
> org.krakenapps
> kraken-pcap
> ${global_pcap_version}
>
>
> slf4j-api
> org.slf4j
>
>
> slf4j-simple
> org.slf4j
>
>
>
> This library does not currently reside in mvn central (searched by package
> and libname) and should instead be staged in Apache with Metron.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-967) How to push Snort, Bro, YAF log to Metron
[ https://issues.apache.org/jira/browse/METRON-967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-967: --- Fix Version/s: (was: 0.3.0) > How to push Snort, Bro, YAF log to Metron > - > > Key: METRON-967 > URL: https://issues.apache.org/jira/browse/METRON-967 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: CentOS >Reporter: GS Peter >Priority: Major > Labels: features > Original Estimate: 96h > Remaining Estimate: 96h > > Hi everyone. > I configured Apache Metron full dev 0.3 but I don't push Snort, Bro, Yaf log > to Metron Server. Can you help me? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-737) Deploying additional model on an already running Maas model creates ghost cotainers
[ https://issues.apache.org/jira/browse/METRON-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-737: --- Fix Version/s: (was: 0.3.0) > Deploying additional model on an already running Maas model creates ghost > cotainers > --- > > Key: METRON-737 > URL: https://issues.apache.org/jira/browse/METRON-737 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 > Environment: centos 6,7 . > java version "1.8.0_60" >Reporter: sapinkumar amin >Priority: Major > Attachments: AppMaster.stderr.log, AppMaster.stderr.timeout.log, > container_successful_launch.log, mock_dga.zip > > > First deployment of Maas goes fine. If we ask for 1 container running model, > it does deploy total 2. One as AM and another as model itself. However adding > more number of containers of same model creates ghost containers in following > fashion > 1. $METRON_HOME/metron/bin/maas_deploy.sh -zq 'zookeeper_host:2181' -lmp > /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga1 -v 1.0 -ni 1 > Creates 2 container. 1 is AM, another is model (mock_dga) > (Running same as above command) > 2. $METRON_HOME/metron/bin/maas_deploy.sh -zq 'zookeeper_host:2181' -lmp > /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga1 -v 1.0 -ni 1 > Creates 4 containers. 1 is AM, two are models (mock_dga). 1 is ghost. > (Running same as above command) > 3. $METRON_HOME/metron/bin/maas_deploy.sh -zq 'zookeeper_host:2181' -lmp > /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga1 -v 1.0 -ni 1 > Creates 7 containers. 1 is AM, 3 are models, 3 are ghosts. > Please find AppMaster stderr logs attached with this JIRA. > 1. AppMaster.stderr.timeout.log has AM logs when after second attempts, one > of the ghost container times out. And total comes back to 3 which as expected. > 2. AppMaster.stderr.log has AM logs about running command 3 times and > launching 7 containers. > Please find the model (mock_dga.zip) and container launched log attached with > this jira. There are no logs for ghost container. > Please also note that if I issue REMOVE command, i.e. > $METRON_HOME/metron/bin/maas_deploy.sh -zq zookeeper_host:2181 -lmp > /root/mock_dga -hmp /user/root/models -mo REMOVE -m 512 -n dga1 -v 1.0 -ni 1, > It successfully removes the running model but not the ghost containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824739#comment-16824739 ] Justin Leet commented on METRON-1005: - [~nickwallen] This Jira can be closed and the fix version removed, right? > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1213) Flow diagram for Metron Kafka topics
[ https://issues.apache.org/jira/browse/METRON-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1213: Fix Version/s: (was: Next + 1) > Flow diagram for Metron Kafka topics > > > Key: METRON-1213 > URL: https://issues.apache.org/jira/browse/METRON-1213 > Project: Metron > Issue Type: New Feature >Affects Versions: 0.4.1 > Environment: Management UI >Reporter: Jasper Knulst >Priority: Major > > The insight that is really missing in Metron is an overview of Kafka topic > sizes and flow metrics (topic influx and outflux) > It is hard to answer questions like: > -Is the enrichments topic (lag) growing over time? > -What parser topics feed into enrichments, at what rate? > -Is the enrichment topology keeping up with the projected influx from all > parsers? > -Same questions, but then for indexing topic > When the capacity of the chained topologies is not aligned bad things can > happen, like a serious threat not reaching ES/Kibana fast enough to react > upon. Metron can only be a realtime alerter if there is no congestion along > the way. > This would take merging metrics from various Metron moving parts like Storm > API, kafka-offset-checker (state/snapshots and flow metrics units/sec) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-914) Build Metron failure
[ https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-914: --- Fix Version/s: (was: 0.4.0) > Build Metron failure > > > Key: METRON-914 > URL: https://issues.apache.org/jira/browse/METRON-914 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.0 > Environment: Ubuntu 16.04.1 >Reporter: GS Peter >Priority: Major > Attachments: build.log.tar.gz, build.test.log.gz, npm-debug.log > > Original Estimate: 504h > Remaining Estimate: 504h > > [INFO] metron-config .. FAILURE [04:08 > min] > [INFO] metron-rest-client . SKIPPED > [INFO] metron-rest SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 58:04 min > [INFO] Finished at: 2017-04-28T15:57:16+07:00 > [INFO] Final Memory: 141M/588M > [INFO] > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project > metron-config: Failed to run task: 'npm install' failed. (error code 1) -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :metron-config -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1331) ZKConfigurationsCacheIntegrationTest.validateBaseWrite Intermittent Test Failure
[
https://issues.apache.org/jira/browse/METRON-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1331:
Fix Version/s: (was: 0.4.1)
> ZKConfigurationsCacheIntegrationTest.validateBaseWrite Intermittent Test
> Failure
>
>
> Key: METRON-1331
> URL: https://issues.apache.org/jira/browse/METRON-1331
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Priority: Minor
>
> {code}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 40.267 sec
> <<< FAILURE! - in
> org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest
> validateBaseWrite(org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest)
> Time elapsed: 31.052 sec <<< FAILURE!
> java.lang.AssertionError: expected:<{hdfs={index=yaf, batchSize=5,
> enabled=true}, elasticsearch={index=yaf, batchSize=25, batchTimeout=7,
> enabled=true}, solr={index=yaf, batchSize=5, enabled=true}}> but was:<{}>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest.lambda$validateBaseWrite$14(ZKConfigurationsCacheIntegrationTest.java:265)
> at
> org.apache.metron.integration.utils.TestUtils.assertEventually(TestUtils.java:54)
> at
> org.apache.metron.integration.utils.TestUtils.assertEventually(TestUtils.java:39)
> at
> org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest.validateBaseWrite(ZKConfigurationsCacheIntegrationTest.java:265)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
>
> ZKConfigurationsCacheIntegrationTest.validateBaseWrite:265->lambda$validateBaseWrite$14:265
> expected:<{hdfs={index=yaf, batchSize=5, enabled=true},
> elasticsearch={index=yaf, batchSize=25, batchTimeout=7, enabled=true},
> solr={index=yaf, batchSize=5, enabled=true}}> but was:<{}>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1174) Metron Docker Document example not working
[
https://issues.apache.org/jira/browse/METRON-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1174:
Fix Version/s: (was: 0.4.0)
> Metron Docker Document example not working
> --
>
> Key: METRON-1174
> URL: https://issues.apache.org/jira/browse/METRON-1174
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.0
> Environment: CentOS 7
> Docker 1.12.6
> docker-compose version 1.16.1, build 6d1ac219
> docker-machine version 0.12.2, build 9371605
>Reporter: Choi BongJae
>Priority: Major
> Labels: docuentation, examples
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> This Section is not working with error
> {code:java}
> /usr/metron/0.4.0//bin/zk_load_configs.sh: line 35:
> /usr/hdp/current/hbase-client/bin/hbase: No such file or directory
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> {code}
> Upload configs to Zookeeper
> Parser configs and a global config configured for this Docker environment are
> included with the Kafka/Zookeeper image. Load them with:
> {code:java}
> $ docker-compose exec kafkazk bash
> # $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> $METRON_HOME/config/zookeeper
> # exit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-2081) Is Metron compatible with newer version of HDP?
[
https://issues.apache.org/jira/browse/METRON-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2081:
Fix Version/s: (was: 0.4.1)
> Is Metron compatible with newer version of HDP?
>
>
> Key: METRON-2081
> URL: https://issues.apache.org/jira/browse/METRON-2081
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Abhishek Sinha
>Priority: Major
>
> I am unable to deploy HDP 2.5.0 as it gives repo error, can anyone help me
> with the latest HDP version with which it is compatible with? The error that
> I receive is as follows:
>
> 2019-04-16 15:39:58,855 - Could not find stack selector for stack: HDP
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 155, in
> ApplicationTimelineServer().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 285, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 39, in install
> self.install_packages(env)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 576, in install_packages
> retry_count=agent_stack_retry_count)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 54, in action_install
> self.install_package(package_name, self.resource.use_repos,
> self.resource.skip_repos)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py",
> line 51, in install_package
> self.checked_call_with_retries(cmd, sudo=True,
> logoutput=self.get_logoutput())
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 86, in checked_call_with_retries
> return self._call_with_retries(cmd, is_checked=True, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 98, in _call_with_retries
> code, out = func(cmd, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_5_0_0_1245-yarn' returned 1.
> Error: Nothing to do
> *stdout:* */var/lib/ambari-agent/data/output-185.txt*
> 2019-04-16 15:38:35,068 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2019-04-16 15:38:35,070 - Group['hadoop'] {}
> 2019-04-16 15:38:35,071 - Group['users'] {}
> 2019-04-16 15:38:35,071 - Group['zeppelin'] {}
> 2019-04-16 15:38:35,071 - Group['metron'] {}
> 2019-04-16 15:38:35,072 - Group['kibana'] {}
> 2019-04-16 15:38:35,072 - Group['spark'] {}
> 2019-04-16 15:38:35,072 - Group['livy'] {}
> 2019-04-16 15:38:35,072 - Group['elasticsearch'] {}
> 2019-04-16 15:38:35,072 - User['hive'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,073 - User['zeppelin'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,074 - User['ambari-qa'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'users']}
> 2019-04-16 15:38:35,074 - User['flume'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,075 - User['metron'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['hdfs'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['storm'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,077 - User['spark'] \{'gid':
[jira] [Updated] (METRON-1285) Alerts UI Displays Error When Credentials Timeout
[ https://issues.apache.org/jira/browse/METRON-1285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1285: Fix Version/s: (was: 0.4.1) > Alerts UI Displays Error When Credentials Timeout > - > > Key: METRON-1285 > URL: https://issues.apache.org/jira/browse/METRON-1285 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > Labels: Alerts-UI > Attachments: Screen Shot 2017-10-27 at 1.06.48 PM.png > > > Steps to reproduce: > 1. Login to the Alarms UI. > 2. Turn off the auto-refresh option (not sure if this is required to > reproduce.) > 3. Remain on the main page, but wait until the credentials timeout. > 4. Attempt to perform an action in the stale browser window; like sort the > results. > 5. The browser will redirect the user to an ugly page showing a stack trace. > See screenshot. > I would expect the user to be redirected to the login page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2080) unable to access HDP 2.5.0 repo : http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0
[
https://issues.apache.org/jira/browse/METRON-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2080:
Fix Version/s: (was: 0.4.1)
> unable to access HDP 2.5.0 repo :
> http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0
> ---
>
> Key: METRON-2080
> URL: https://issues.apache.org/jira/browse/METRON-2080
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Abhishek Sinha
>Priority: Major
>
> HI,
> I am trying to deploy HDP 2.5.0 with Metron, getting following error :
>
> 2019-04-16 15:39:58,855 - Could not find stack selector for stack: HDP
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 155, in
> ApplicationTimelineServer().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 285, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 39, in install
> self.install_packages(env)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 576, in install_packages
> retry_count=agent_stack_retry_count)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 54, in action_install
> self.install_package(package_name, self.resource.use_repos,
> self.resource.skip_repos)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py",
> line 51, in install_package
> self.checked_call_with_retries(cmd, sudo=True,
> logoutput=self.get_logoutput())
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 86, in checked_call_with_retries
> return self._call_with_retries(cmd, is_checked=True, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 98, in _call_with_retries
> code, out = func(cmd, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_5_0_0_1245-yarn' returned 1.
> Error: Nothing to do
> *stdout:* */var/lib/ambari-agent/data/output-185.txt*
> 2019-04-16 15:38:35,068 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2019-04-16 15:38:35,070 - Group['hadoop'] {}
> 2019-04-16 15:38:35,071 - Group['users'] {}
> 2019-04-16 15:38:35,071 - Group['zeppelin'] {}
> 2019-04-16 15:38:35,071 - Group['metron'] {}
> 2019-04-16 15:38:35,072 - Group['kibana'] {}
> 2019-04-16 15:38:35,072 - Group['spark'] {}
> 2019-04-16 15:38:35,072 - Group['livy'] {}
> 2019-04-16 15:38:35,072 - Group['elasticsearch'] {}
> 2019-04-16 15:38:35,072 - User['hive'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,073 - User['zeppelin'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,074 - User['ambari-qa'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'users']}
> 2019-04-16 15:38:35,074 - User['flume'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,075 - User['metron'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['hdfs'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['storm'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,077 - User['spark'] \{'gid':
[jira] [Assigned] (METRON-2029) Configure Table should have filter
[ https://issues.apache.org/jira/browse/METRON-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-2029: --- Assignee: Shane Ardell > Configure Table should have filter > -- > > Key: METRON-2029 > URL: https://issues.apache.org/jira/browse/METRON-2029 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, when a user opens the configure table panel, they must scroll down > to see all their choices. This list can be really long, so it would be > beneficial to have an input for a user to type into and have the options > filtered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2029) Configure Table should have filter
[ https://issues.apache.org/jira/browse/METRON-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2029: Fix Version/s: 0.7.1 > Configure Table should have filter > -- > > Key: METRON-2029 > URL: https://issues.apache.org/jira/browse/METRON-2029 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, when a user opens the configure table panel, they must scroll down > to see all their choices. This list can be really long, so it would be > beneficial to have an input for a user to type into and have the options > filtered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1987) Upgrade Alert UI to stable Bootstrap 4
[ https://issues.apache.org/jira/browse/METRON-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1987: Fix Version/s: 0.7.1 > Upgrade Alert UI to stable Bootstrap 4 > -- > > Key: METRON-1987 > URL: https://issues.apache.org/jira/browse/METRON-1987 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Like we did for the Management UI, we should upgrade the version of Bootstrap > that we are using to the latest stable version. We are currently on an alpha > version which includes deprecated class names and values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1971) Short timeout value in Cypress may cause build failures
[ https://issues.apache.org/jira/browse/METRON-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1971: Fix Version/s: 0.7.1 > Short timeout value in Cypress may cause build failures > --- > > Key: METRON-1971 > URL: https://issues.apache.org/jira/browse/METRON-1971 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > According to the Cypress docs, it is unnecessary to add a wait command after > cy.visit. "Waiting for this is unnecessary because the cy.visit() resolves > once the page fires its load event. By that time all of your assets have been > loaded including javascript, stylesheets, and html." > However, the problem is with an Angular application, the page firing a load > event does not mean that the entire view has rendered yet. In systems where > memory is scarce, such as a Travis environment or a machine that has > memory-intensive applications running, this means a page load can take a few > seconds (or possibly even longer). > We currently use the default timeout in Cypress, which is only 4000ms. I > believe this short timeout can cause failures like this in Metron: > https://travis-ci.org/apache/metron/jobs/483945575 > We should increase the defaultCommandTimeout in cypress.json to something > like 3ms or greater to prevent failures like this in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1987) Upgrade Alert UI to stable Bootstrap 4
[ https://issues.apache.org/jira/browse/METRON-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1987: --- Assignee: Shane Ardell > Upgrade Alert UI to stable Bootstrap 4 > -- > > Key: METRON-1987 > URL: https://issues.apache.org/jira/browse/METRON-1987 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Like we did for the Management UI, we should upgrade the version of Bootstrap > that we are using to the latest stable version. We are currently on an alpha > version which includes deprecated class names and values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1971) Short timeout value in Cypress may cause build failures
[ https://issues.apache.org/jira/browse/METRON-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1971: --- Assignee: Shane Ardell > Short timeout value in Cypress may cause build failures > --- > > Key: METRON-1971 > URL: https://issues.apache.org/jira/browse/METRON-1971 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > According to the Cypress docs, it is unnecessary to add a wait command after > cy.visit. "Waiting for this is unnecessary because the cy.visit() resolves > once the page fires its load event. By that time all of your assets have been > loaded including javascript, stylesheets, and html." > However, the problem is with an Angular application, the page firing a load > event does not mean that the entire view has rendered yet. In systems where > memory is scarce, such as a Travis environment or a machine that has > memory-intensive applications running, this means a page load can take a few > seconds (or possibly even longer). > We currently use the default timeout in Cypress, which is only 4000ms. I > believe this short timeout can cause failures like this in Metron: > https://travis-ci.org/apache/metron/jobs/483945575 > We should increase the defaultCommandTimeout in cypress.json to something > like 3ms or greater to prevent failures like this in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1948) Dropped messages from REGEX_SELECT parser field transformation are not acked in Storm
[ https://issues.apache.org/jira/browse/METRON-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1948: Fix Version/s: 0.7.1 > Dropped messages from REGEX_SELECT parser field transformation are not acked > in Storm > - > > Key: METRON-1948 > URL: https://issues.apache.org/jira/browse/METRON-1948 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1795) General Purpose Regex Parser
[
https://issues.apache.org/jira/browse/METRON-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1795:
Fix Version/s: 0.7.1
> General Purpose Regex Parser
>
>
> Key: METRON-1795
> URL: https://issues.apache.org/jira/browse/METRON-1795
> Project: Metron
> Issue Type: New Feature
>Reporter: Jagdeep Singh
>Priority: Minor
> Fix For: 0.7.1
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> We have implemented a general purpose regex parser for Metron that we are
> interested in contributing back to the community.
>
> While the Metron Grok parser provides some regex based capability today, the
> intention of this general purpose regex parser is to:
> # Allow for more advanced parsing scenarios (specifically, dealing with
> multiple regex lines for devices that contain several log formats within them)
> # Give users and developers of Metron additional options for parsing
> # With the new parser chaining and regex routing feature available in
> Metron, this gives some additional flexibility to logically separate a flow
> by:
> # Regex routing to segregate logs at a device level and handle envelope
> unwrapping
> # This general purpose regex parser to parse an entire device type that
> contains multiple log formats within the single device (for example, RHEL
> logs)
> At the high-level control flow is like this:
> # Identify the record type if incoming raw message.
> # Find and apply the regular expression of corresponding record type to
> extract the fields (using named groups).
> # Apply the message header regex to extract the fields in the header part of
> the message (using named groups).
>
> The parser config uses the following structure:
>
> {code:java}
> "recordTypeRegex": "(?(?<=\\s)\\b(kernel|syslog)\\b(?=\\[|:))"
> "messageHeaderRegex":
> "(?(?<=^<)\\d{1,4}(?=>)).*?(?(?<=>)[A-Za-z]{3}\\s{1,2}\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2}(?=\\s)).*?(?(?<=\\s).*?(?=\\s))",
> "fields": [
> {
> "recordType": "kernel",
> "regex": ".*(?(?<=\\]|\\w\\:).*?(?=$))"
> },
> {
> "recordType": "syslog",
> "regex":
> ".*(?(?<=PID\\s=\\s).*?(?=\\sLine)).*(?(?<=64\\s)\/([A-Za-z0-9_-]+\/)+(?=\\w))(?.*?(?=\")).*(?(?<=\").*?(?=$))"
> }
> ]
> {code}
>
> Where:
> * *recordTypeRegex* is used to distinctly identify a record type. It inputs
> a valid regular expression and may also have named groups, which would be
> extracted into fields.
> * *messageHeaderRegex* is used to specify a regular expression to extract
> fields from a message part which is common across all the messages (i.e,
> syslog fields, standard headers)
> * *fields*: json list of objects containing recordType and regex. The
> expression that is evaluated is based on the output of the recordTypeRegex
> * Note: *recordTypeRegex* and *messageHeaderRegex* could be specified as
> lists also (as a JSON array), where the list will be evaluated in order until
> a matching regular expression is found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-2022) Metron rest creates large number of connections to ZK which causes subsequent connection to zk fail
[
https://issues.apache.org/jira/browse/METRON-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2022:
Fix Version/s: (was: Next + 1)
0.7.1
> Metron rest creates large number of connections to ZK which causes subsequent
> connection to zk fail
> ---
>
> Key: METRON-2022
> URL: https://issues.apache.org/jira/browse/METRON-2022
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> I have faced this issue while doing operations like changing status for
> metaalert
> I escalated a metaalert , then I dismissed , I tried open it again to see
> wether the status updated . I see that it failed to load the details and in
> the backend I see this error
>
> http://nat-r7-rrss-metron-1.openstacklocal:8082/api/v1/search/findOne
>
> {code:java}
> {"guid":"097c6a19-ccf0-4737-86f6-8ffd04f1e1c2","sensorType":"metaalert"}
> {
> "timestamp": "2019-02-28 13:36:17",
> "status": 500,
> "error": "Internal Server Error",
> "message": "java.util.concurrent.TimeoutException: Could not connect to
> ZooKeeper nat-r7-rrss-metron-1.openstacklocal:2181/solr within 1 ms",
> "path": "/api/v1/search/findOne"
> }
> {code}
>
> I see it's due to the number of connections to the ZK exceeded the allowed
> limit
> {code:java}
> 019-03-01 13:16:03,863 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too
> many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,158 -
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,752
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,865
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,397
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,667
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60
> {code}
> I stopped metron-rest service and I see the connections released on zk log
> {code:java}
> 2019-03-01 13:40:25,461 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:53099 which had sessionid
> 0x1693965ae4e0003 2019-03-01 13:40:25,462 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf002d, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,462 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47833 which had sessionid
> 0x169395872cf002d 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf001a, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,463 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47878 which had sessionid
> 0x169395872cf001a 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf0007, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,464 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47834 which had sessionid
> 0x169395872cf0007 2019-03-01 13:40:25,464 - WARN
>
[jira] [Updated] (METRON-2030) SensorParserGroupControllerIntegrationTest intermittent errors
[ https://issues.apache.org/jira/browse/METRON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2030: Fix Version/s: (was: Next + 1) 0.7.1 > SensorParserGroupControllerIntegrationTest intermittent errors > -- > > Key: METRON-2030 > URL: https://issues.apache.org/jira/browse/METRON-2030 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Recently I've seen some intermittent test failures in the recently added > `SensorParserGroupControllerIntegrationTest`. We likely need to add some > assertEventually statements to ensure REST state is consistent before testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2051) Improve stellar-zeppelin documentation
[ https://issues.apache.org/jira/browse/METRON-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2051: Fix Version/s: (was: Next + 1) 0.7.1 > Improve stellar-zeppelin documentation > -- > > Key: METRON-2051 > URL: https://issues.apache.org/jira/browse/METRON-2051 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add some notes and a link detailing the Zeppelin 3rd party interpreter > installation mechanism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1950) Site-book generation broken in master
[ https://issues.apache.org/jira/browse/METRON-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1950: Fix Version/s: (was: Next + 1) 0.7.1 > Site-book generation broken in master > - > > Key: METRON-1950 > URL: https://issues.apache.org/jira/browse/METRON-1950 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1951) Add site-book generation to Travis build
[ https://issues.apache.org/jira/browse/METRON-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1951: Fix Version/s: (was: Next + 1) 0.7.1 > Add site-book generation to Travis build > > > Key: METRON-1951 > URL: https://issues.apache.org/jira/browse/METRON-1951 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2041) RegularExpressionsParser in wrong source folder
[ https://issues.apache.org/jira/browse/METRON-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2041: Fix Version/s: (was: Next + 1) 0.7.1 > RegularExpressionsParser in wrong source folder > --- > > Key: METRON-2041 > URL: https://issues.apache.org/jira/browse/METRON-2041 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The RegularExpressionsParser was moved to the test directory, but should be > in src. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1371) Enhance Mpack for Ubuntu Deployments
[ https://issues.apache.org/jira/browse/METRON-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1371: Fix Version/s: (was: Next + 1) 0.7.1 > Enhance Mpack for Ubuntu Deployments > > > Key: METRON-1371 > URL: https://issues.apache.org/jira/browse/METRON-1371 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Labels: Ubuntu > Fix For: 0.7.1 > > > The current Mpack supports installation on CentOS and RPM-based environments > only. We should enhance the Mpack so that users can deploy on Ubuntu also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2016) Parser aggregate groups should be persisted and available through REST
[ https://issues.apache.org/jira/browse/METRON-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2016: Fix Version/s: (was: Next + 1) 0.7.1 > Parser aggregate groups should be persisted and available through REST > -- > > Key: METRON-2016 > URL: https://issues.apache.org/jira/browse/METRON-2016 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently a parser aggregate group is configured solely in Ambari as a > comma-separate list of sensors. These groups should be persisted so they can > be retrieved and modified by other applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-157) Create CEF Parser
[
https://issues.apache.org/jira/browse/METRON-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-157:
---
Fix Version/s: (was: Next + 1)
0.7.1
> Create CEF Parser
> -
>
> Key: METRON-157
> URL: https://issues.apache.org/jira/browse/METRON-157
> Project: Metron
> Issue Type: New Feature
>Reporter: Domenic Puzio
>Assignee: Kyle Richardson
>Priority: Minor
> Labels: platform
> Fix For: 0.7.1
>
>
> Create a parser for CEF (Common Event Format). CEF is a very common
> formatting for security data sources; it is used by FireEye, Adallom, Imperva
> WAF, CyberArk, and others. The parser should be flexible enough to work for
> any of these data sources. CEF uses shorthand field names, so field names
> should be changed to human-readable and Metron-friendly equivalents. CEF
> custom labels (cs1Label, flexString1Label, etc.) should be converted
> appropriately.
> Below are sample messages and their expected parsed output.
> Adallom CEF
> 2016-04-01T09:29:11.356-0400
> CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity
> policy 'User download/view file' was triggered by 'scolb...@gmail.com'
> suser=wander...@rock.com start=1459517280810 end=1459517280810
> audits=["AVPR-4oIPeFmuZ3CKKrg","AVPR-wx80cd9PUpAu2aj","AVPR-6XGPeFmuZ3CKKvx","AVPSALn_qE4Kgs_8_yK9","AVPSASW3gw_f3aEvgEmi"]
> services=["APPID_SXC"] users=["lva...@hotmail.com"]
> cs6=https://abcd-remote.console.arc.com/#/alerts/56fe779ee4b0459f4e9a484a
> cs6Label=consoleUrl
> ...
> {"source.type":"adallom","device_version":"1.0","severity":"0","device_product":"Adallom","services":"[\"APPID_SXC\"]","src_username":"wander...@rock.com","message":"Activity
> policy 'User download\/view file' was triggered by
> 'scolb...@gmail.com'","users":"[\"lva...@hotmail.com\"]","consoleUrl":"https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a","event_class_id":"56fe779ee4b0459f4e9a484a","original_string":"2016-04-01T09:29:11.356-0400
>
> CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity
> policy 'User download\/view file' was triggered by 'scolb...@gmail.com'
> suser=wander...@rock.com start=1459517280810 end=1459517280810
> audits=[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]
> services=[\"APPID_SXC\"] users=[\"lva...@hotmail.com\"]
> cs6=https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a
> cs6Label=consoleUrl","header":"2016-04-01T09:29:11.356-0400
> CEF:0","event_name":"ALERT_CABINET_EVENT_MATCH_AUDIT","startTime":"1459517280810","device_vendor":"Adallom","endTime":"1459517280810","audits":"[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]","timestamp":1459502951000}
> CyberArk CEF
> Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve
> password|5|act=Retrieve password suser=spilgrim fname=Root\ABC phobos3 - COMP
> dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason=
> cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=Security
> Vulnerability Mgmt cs3Label="Device Type" cs3= cs4Label="Database" cs4=
> cs5Label="Other info" cs5=101.198.70.93 cn1Label="Request Id" cn1=
> cn2Label="Ticket Id" cn2=Needed to verify config files being pulled
> msg=Needed to verify config files being pulled
> ...
> {"timestamp":1458569102000,"source.type":"cyberark","device_version":"7.20.0091","device_product":"Vault","fileName":"Root\\ABC
> phobos3 - COMP","src_username":"spilgrim","\"Other
> info\"":"101.198.70.93","\"Ticket Id\"":"Needed to verify config files being
> pulled
> ","deviceAddress":"120.99.70.3","severity":"5","deviceAction":"Retrieve
> password","message":"Needed to verify config files being
> pulled","event_class_id":"295","original_string":"Mar 21 14:05:02 HHHPVATN1
> CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password
> suser=spilgrim fname=Root\\ABC phobos3 - COMP dvc=120.99.70.3
> shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label=\"Affected
> User Name\" cs1= cs2Label=\"Safe Name\" cs2=Security Vulnerability Mgmt
> cs3Label=\"Device Type\" cs3= cs4Label=\"Database\" cs4= cs5Label=\"Other
> info\" cs5=101.198.70.93 cn1Label=\"Request Id\" cn1= cn2Label=\"Ticket Id\"
> cn2=Needed to verify config files being pulled msg=Needed to verify config
> files being pulled","\"Safe Name\"":"Security Vulnerability
> Mgmt","header":"Mar 21 14:05:02 HHHPVATN1 CEF:0","event_name":"Retrieve
> password","device_vendor":"Cyber-Ark","src_hostname":"10.44.134.78"}
> WAF CEF
> <14>CEF:0|Imperva
[jira] [Updated] (METRON-2065) Setting Parser Output Topic in Sensor Config is broken
[
https://issues.apache.org/jira/browse/METRON-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2065:
Fix Version/s: (was: Next + 1)
0.7.1
> Setting Parser Output Topic in Sensor Config is broken
> --
>
> Key: METRON-2065
> URL: https://issues.apache.org/jira/browse/METRON-2065
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Attachments: Screen Shot 2019-04-05 at 7.45.36 PM.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Login to management console
> Edit the parser config Advanced > Raw JSON !Screen Shot 2019-04-05 at
> 7.45.36 PM.png!
> Change the output topic for the 'snort' sensor.
> Verify that the changes taken effect using stellar shell
> {code:java}
> [Stellar]>>> conf := CONFIG_GET("PARSER","snort") { "parserClassName" :
> "org.apache.metron.parsers.snort.BasicSnortParser", "sensorTopic" : "snort",
> "outputTopic" : "new-topic", "readMetadata" : false, "mergeMetadata" : false,
> "spoutParallelism" : 1, "spoutNumTasks" : 1, "parserParallelism" : 1,
> "parserNumTasks" : 1, "errorWriterParallelism" : 1, "errorWriterNumTasks" :
> 1, "spoutConfig" : { }, "stormConfig" : { }, "parserConfig" : { },
> "fieldTransformations" : [ ], "cacheConfig" : { }, "rawMessageStrategy" :
> "DEFAULT", "rawMessageStrategyConfig" : { } }
> {code}
> publish the message to 'snort' topic
> I use the console consumer to validate output is being piped into "new_topic"
> and verified that no messages were sent to the topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic new-topic
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper]. [2019-04-05 14:08:08,796] WARN
> SASL configuration failed: javax.security.auth.login.LoginException: No JAAS
> configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn) [2019-04-05 14:08:09,005]
> WARN SASL configuration failed: javax.security.auth.login.LoginException: No
> JAAS configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {code}
> where as I see that the messages were sent to "enrichments" topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic enrichments
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper].
> [2019-04-05 14:10:18,930] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> [2019-04-05 14:10:19,095] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {"msg":"snort test
> alert","sig_rev":"0","ip_dst_port":"80","ethsrc":"00:00:00:00:00:00","tcpseq":"0xF017C4DA","dgmlen":"40","icmpid":"","tcplen":"","tcpwindow":"0xF6C9","icmpseq":"","tcpack":"0xABDB8426","protocol":"TCP","ip_dst_addr":"62.75.195.236","original_string":"09\/09\/16-09:09:09.844676
> ,1,999158,0,\"snort test
>
[jira] [Updated] (METRON-2066) Documentation and logging corrections
[ https://issues.apache.org/jira/browse/METRON-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2066: Fix Version/s: (was: Next + 1) 0.7.1 > Documentation and logging corrections > - > > Key: METRON-2066 > URL: https://issues.apache.org/jira/browse/METRON-2066 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Grab-bag of issues and/or omissions I found with documentation while working > through a parser use case with REGEX_SELECT. Also need to tweak the logging > for GrokParser so it no longer outputs nothing for the grok expressions > during init. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2064) Metron REST API overwriting global.json values
[ https://issues.apache.org/jira/browse/METRON-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2064: Fix Version/s: (was: Next + 1) 0.7.1 > Metron REST API overwriting global.json values > -- > > Key: METRON-2064 > URL: https://issues.apache.org/jira/browse/METRON-2064 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We added the capability for Ambari to modify global config properties without > overwriting non-conflicting externally-provided properties in this PR - > [https://github.com/apache/metron/pull/760]. The REST API is currently > overwriting the global config when it's restarted, thus losing user changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2060) Improving Alerts table config pane
[ https://issues.apache.org/jira/browse/METRON-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2060: Fix Version/s: (was: Next + 1) 0.7.1 > Improving Alerts table config pane > -- > > Key: METRON-2060 > URL: https://issues.apache.org/jira/browse/METRON-2060 > Project: Metron > Issue Type: Improvement >Reporter: Tibor Meller >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2067) Maven pom file duplicate dependency fixes
[
https://issues.apache.org/jira/browse/METRON-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2067:
Fix Version/s: (was: Next + 1)
0.7.1
> Maven pom file duplicate dependency fixes
> -
>
> Key: METRON-2067
> URL: https://issues.apache.org/jira/browse/METRON-2067
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Fix a couple duplicate dependencies in pom files. Address error/warning
> output like the following:
> {code:java}
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-profiler-storm:jar:0.7.1
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.apache.metron:metron-integration-test:jar -> duplicate
> declaration of version ${project.parent.version} @
> org.apache.metron:metron-profiler-storm:[unknown-version],
> metron/metron-analytics/metron-profiler-storm/pom.xml, line 314, column 21
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-writer:jar:0.7.1
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.apache.httpcomponents:httpclient:jar -> duplicate declaration
> of version ${global_httpclient_version} @
> org.apache.metron:metron-writer:[unknown-version],
> metron/metron-platform/metron-writer/pom.xml, line 248, column 21
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1998) Only one sensor is flushed by tick tuple
[ https://issues.apache.org/jira/browse/METRON-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1998: Fix Version/s: (was: Next + 1) 0.7.1 > Only one sensor is flushed by tick tuple > > > Key: METRON-1998 > URL: https://issues.apache.org/jira/browse/METRON-1998 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 3.5h > Remaining Estimate: 0h > > The BulkWriterComponent maintains an internal map of sensors to message > batches. The flushTimeouts method incorrectly returns after one of the > sensor's batch timeout is reached and flushed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2078) Remove Storm dependency from metron-writer
[ https://issues.apache.org/jira/browse/METRON-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2078: Fix Version/s: (was: Next + 1) 0.7.1 > Remove Storm dependency from metron-writer > -- > > Key: METRON-2078 > URL: https://issues.apache.org/jira/browse/METRON-2078 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-writer and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1654) findOne request after an alert patch returns with the original state of the alert item
[ https://issues.apache.org/jira/browse/METRON-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1654: Fix Version/s: (was: Next + 1) 0.7.1 > findOne request after an alert patch returns with the original state of the > alert item > -- > > Key: METRON-1654 > URL: https://issues.apache.org/jira/browse/METRON-1654 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > /findOne request after an alert patch returns with the original state of the > alert item. > The UI stays inconsistent until the next search request. > This leads to the following bugs: > - Alert status change on details panel haven't reflected on the list view. > - Adding an element to a meta-alert doesn't change the items counter beside > the name of the meta-alert > - Removing an element from a meta-alert doesn't change the items counter > beside the name of the meta-alert > - Clicking on remove/unlink button in a meta-alert group doesn't remove item > immediately > This behavior only validated on the full dev single node VM and might not > exist in production. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2014) Add architectural documentation for metron-writer
[ https://issues.apache.org/jira/browse/METRON-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2014: Fix Version/s: (was: Next + 1) 0.7.1 > Add architectural documentation for metron-writer > - > > Key: METRON-2014 > URL: https://issues.apache.org/jira/browse/METRON-2014 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This task is based on the discussion here: > [https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E] > Follow on architectural documentation is needed for the writer module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1968) Messages are lost when a parser produces multiple messages and batch size is greater than 1
[ https://issues.apache.org/jira/browse/METRON-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1968: Fix Version/s: (was: Next + 1) 0.7.1 > Messages are lost when a parser produces multiple messages and batch size is > greater than 1 > --- > > Key: METRON-1968 > URL: https://issues.apache.org/jira/browse/METRON-1968 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 15h 20m > Remaining Estimate: 0h > > A bug was discovered where messages are lost when a parser produces multiple > messages. This happens anytime the batch size for that sensor is set to > greater than 1. For example, consider a parser that produces 30 messages > from a single input message. Assume the batch size for this sensor/parser is > set to 10. The batch is currently flushed only after 10 tuples are received > and only 10 messages are written out. I think the correct behavior would be > for 3 batches of 10 messages to be flushed for every tuple and a total of 300 > messages written for every 10 tuples. > This is happening because the various writer classes/interfaces > (BulkWriterComponent, BulkMessageWriter, KafkaWriter, etc) assume a 1 to 1 > relationship between messages and tuples. The root cause of this specific > issue is > [here|https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2036) Maven builds fail locally in HDFSWriterTest
[
https://issues.apache.org/jira/browse/METRON-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2036:
Fix Version/s: (was: Next + 1)
0.7.1
> Maven builds fail locally in HDFSWriterTest
> ---
>
> Key: METRON-2036
> URL: https://issues.apache.org/jira/browse/METRON-2036
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.7.1
>
> Attachments: METRON-2036-ide-fail.txt
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I'm getting the following when running mvn clean install from master locally.
> {code:java}
> testSingleFileIfNoStreamClosed(org.apache.metron.writer.hdfs.HdfsWriterTest)
> Time elapsed: 0.087 sec <<< ERROR!
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at java.nio.file.Files.readAllLines(Files.java:3205)
> at java.nio.file.Files.readAllLines(Files.java:3242)
> at
> org.apache.metron.writer.hdfs.HdfsWriterTest.testSingleFileIfNoStreamClosed(HdfsWriterTest.java:444)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> ...
> Results :
> Tests in error:
> HdfsWriterTest.testSingleFileIfNoStreamClosed:444 » ClosedByInterrupt
> HdfsWriterTest.testWriteMultipleFiles:356 » ClosedByInterrupt
> HdfsWriterTest.testWriteNoOutputFunction:271 » ClosedByInterrupt
> HdfsWriterTest.testWriteSingleFile:314 » ClosedByInterrupt
> HdfsWriterTest.testWriteSingleFileWithNull:407 » ClosedByInterrupt
> Tests run: 55, Failures: 0, Errors: 5, Skipped: 0
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Metron . SUCCESS [ 8.093 s]
> [INFO] metron-stellar . SUCCESS [ 0.241 s]
> [INFO]
[jira] [Updated] (METRON-2026) Remove Storm dependency from metron-common
[ https://issues.apache.org/jira/browse/METRON-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2026: Fix Version/s: (was: Next + 1) 0.7.1 > Remove Storm dependency from metron-common > -- > > Key: METRON-2026 > URL: https://issues.apache.org/jira/browse/METRON-2026 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-common and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1469) Kafka Plugin for Bro - Configurable JSON Timestamps
[ https://issues.apache.org/jira/browse/METRON-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1469: Fix Version/s: (was: Next + 1) 0.7.1 > Kafka Plugin for Bro - Configurable JSON Timestamps > --- > > Key: METRON-1469 > URL: https://issues.apache.org/jira/browse/METRON-1469 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.1 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.1 > > > The Kafka Plugin for Bro allows the user to output JSON to a Kafka topic. > Allow the user to define the format of timestamps that get embedded in the > JSON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1878) Add Metron as a Knox service
[ https://issues.apache.org/jira/browse/METRON-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1878: Fix Version/s: (was: Next + 1) 0.7.1 > Add Metron as a Knox service > > > Key: METRON-1878 > URL: https://issues.apache.org/jira/browse/METRON-1878 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > This feature would add Apache Knox as an option for access to Metron REST and > the Metron UIs. LDAP authentication would be required for Metron REST in > this scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2054) Update Metron wiki Streaming Enrichment instructions
[ https://issues.apache.org/jira/browse/METRON-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2054: Fix Version/s: (was: Next + 1) 0.7.1 > Update Metron wiki Streaming Enrichment instructions > > > Key: METRON-2054 > URL: https://issues.apache.org/jira/browse/METRON-2054 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > > Classpath for the SimpleHbaseEnrichmentWriter will change on this wiki page - > [https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1945) Metron MPack support for Knox SSO setup
[ https://issues.apache.org/jira/browse/METRON-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1945: Fix Version/s: (was: Next + 1) 0.7.1 > Metron MPack support for Knox SSO setup > --- > > Key: METRON-1945 > URL: https://issues.apache.org/jira/browse/METRON-1945 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Knox SSO support for Metron is being added in > [https://github.com/apache/metron/pull/1275.] This PR depends on that work > and enables Ambari to automate most of the setup work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2039) Time range queries do not work with Solr
[ https://issues.apache.org/jira/browse/METRON-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2039: Fix Version/s: (was: Next + 1) 0.7.1 > Time range queries do not work with Solr > > > Key: METRON-2039 > URL: https://issues.apache.org/jira/browse/METRON-2039 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > A bug was discovered recently where a query generated from the time widget in > the alerts UI only works on Elasticsearch. The syntax should be updated to > work with both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1965) Knox should work on a multi-node installation
[ https://issues.apache.org/jira/browse/METRON-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1965: Fix Version/s: (was: Next + 1) 0.7.1 > Knox should work on a multi-node installation > - > > Key: METRON-1965 > URL: https://issues.apache.org/jira/browse/METRON-1965 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We discovered a few issues when deploying Metron in a multi-node environment > and enabling Knox: > * Metron scripts and resources need to be installed by a Metron Client > component > * The flag used to determine if Metron Knox assets are installed does not > work correctly if Knox is installed after a Metron Client > * The Knox host needs to be added to the SSO whitelist -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2074) Script to handle TGT renewal with Storm and Kerberos enabled
[ https://issues.apache.org/jira/browse/METRON-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2074: Fix Version/s: (was: Next + 1) 0.7.1 > Script to handle TGT renewal with Storm and Kerberos enabled > > > Key: METRON-2074 > URL: https://issues.apache.org/jira/browse/METRON-2074 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Storm doesn't automatically manage TGT renewal. This means it's up to the > topology deployer to handle providing a renewed TGT to their topologies. We > should provide a script that users can schedule with cron or similar > scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-589) Dist area should only contain latest release
[ https://issues.apache.org/jira/browse/METRON-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-589: --- Fix Version/s: (was: Next + 1) 0.7.1 > Dist area should only contain latest release > > > Key: METRON-589 > URL: https://issues.apache.org/jira/browse/METRON-589 > Project: Metron > Issue Type: Bug >Reporter: John D. Ament >Assignee: Casey Stella >Priority: Major > Fix For: 0.7.1 > > > The dist area https://dist.apache.org/repos/dist/release/incubator/metron/ > should only contain the latest release. Older releases are found in the > archives. > Most projects don't directly link to this area, but instead provide the dyn > style links. See Geode for example: > http://geode.incubator.apache.org/releases/ - this works for older releases > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2007) Management UI not loading grok statements correctly
[
https://issues.apache.org/jira/browse/METRON-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2007:
Fix Version/s: (was: Next + 1)
0.7.1
> Management UI not loading grok statements correctly
> ---
>
> Key: METRON-2007
> URL: https://issues.apache.org/jira/browse/METRON-2007
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When I open a Grok sensor in the management UI (like squid) I get an error:
> {code:java}
> Could not find grok statement in HDFS or classpath at
> /apps/metron/patterns/squid{code}
> I verified that file exists in HDFS. It also happens when I open a sensor
> that gets a grok statement from the classpath (yaf for example):
> {code:java}
> Could not find grok statement in HDFS or classpath at /patterns/yaf{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1993) Stellar REST_GET should handle responses when content length is less than zero
[ https://issues.apache.org/jira/browse/METRON-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1993: Fix Version/s: (was: Next + 1) 0.7.1 > Stellar REST_GET should handle responses when content length is less than zero > -- > > Key: METRON-1993 > URL: https://issues.apache.org/jira/browse/METRON-1993 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > There are cases where a HTTP response content length header is missing but > content exists. The Stellar REST_GET function should still parse the content > in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2009) Address Javadoc checkstyle issues in metron-common
[ https://issues.apache.org/jira/browse/METRON-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2009: Fix Version/s: 0.7.1 > Address Javadoc checkstyle issues in metron-common > -- > > Key: METRON-2009 > URL: https://issues.apache.org/jira/browse/METRON-2009 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: 0.7.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We have a lot of missing Javadoc or broken style in various modules, > including metron-common. We should fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2006) Reenable JacoCo code coverage
[ https://issues.apache.org/jira/browse/METRON-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2006: Fix Version/s: 0.7.1 > Reenable JacoCo code coverage > - > > Key: METRON-2006 > URL: https://issues.apache.org/jira/browse/METRON-2006 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > It got disabled here: > https://github.com/apache/metron/pull/840/commits/54c2486e4e618a6d445667ed2040faacf165ca8c > We should get it reenabled and working with the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1939) Update version to 0.7.1
[ https://issues.apache.org/jira/browse/METRON-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1939: Fix Version/s: 0.7.1 > Update version to 0.7.1 > --- > > Key: METRON-1939 > URL: https://issues.apache.org/jira/browse/METRON-1939 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Need to update the version for ongoing development post 0.7.0 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1937) Update public web site to point at 0.7.0 new release
[ https://issues.apache.org/jira/browse/METRON-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1937: Fix Version/s: 0.7.1 > Update public web site to point at 0.7.0 new release > > > Key: METRON-1937 > URL: https://issues.apache.org/jira/browse/METRON-1937 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Need to update site to reflect the release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1929) Build GET_ASN Stellar function
[ https://issues.apache.org/jira/browse/METRON-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1929: Fix Version/s: 0.7.1 > Build GET_ASN Stellar function > -- > > Key: METRON-1929 > URL: https://issues.apache.org/jira/browse/METRON-1929 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 7.5h > Remaining Estimate: 0h > > MaxMind has a database for ASN information, very similar to the Geo databases > we use. We should be able to have a GET_ASN function similar to that, and > reuse (and refactor) the existing infrastructure from GEO_GET to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1815) Separate metron-parsers into metron-parsers-common and metron-parsers-storm
[ https://issues.apache.org/jira/browse/METRON-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1815: Fix Version/s: 0.7.1 > Separate metron-parsers into metron-parsers-common and metron-parsers-storm > --- > > Key: METRON-1815 > URL: https://issues.apache.org/jira/browse/METRON-1815 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h > Remaining Estimate: 0h > > In order to expose our parsers to 3rd party components (e.g. the discussions > on NiFi and potentially other platforms like Spark), we should > separate the storm-bits into its own project. The metron-parsers-common > project should contain only parser-oriented code, whereas the > metron-parsers-storm project should contain the storm specific code > (e.g. the parser bolt). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1582) after installation of metron in ubuntu my system is hang
[ https://issues.apache.org/jira/browse/METRON-1582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1582: Fix Version/s: (was: 0.4.2) > after installation of metron in ubuntu my system is hang > > > Key: METRON-1582 > URL: https://issues.apache.org/jira/browse/METRON-1582 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.2 >Reporter: manisha tank >Priority: Major > Labels: issue, metron, system-browser > Original Estimate: 36h > Remaining Estimate: 36h > > I installed single node Apache metron in Ubuntu Operating system. Apache > metron system require VM assigned 20 GB RAM, 100GB disk, and 2 CPUs. we > already assign 32GB RAM in our system. I used all prerequisite require of > metron. once installation complete I ingested sample squid logs inside apache > metron. Once i ingested squid logs my sytem taked much more burden and loads. > My full system is hang. I can't do any work in my system. > > Anybody have idea about this issue -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1583:
Fix Version/s: (was: 0.4.2)
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Attachments: cisco_log_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1597) EC2 10 Node Deployment - New function to check os and update *.cfg control_path
[ https://issues.apache.org/jira/browse/METRON-1597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1597: Fix Version/s: (was: 0.5.0) > EC2 10 Node Deployment - New function to check os and update *.cfg > control_path > --- > > Key: METRON-1597 > URL: https://issues.apache.org/jira/browse/METRON-1597 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.5.0 > Environment: ClearOS release 7.4.0, CentOS 6.9, Ubuntu 18.04, Mac OS > X 10.13.3 >Reporter: ashah >Assignee: ashah >Priority: Minor > > This PR updates run.sh to include a function (changeAnsibleControlPath) that > will check for the OS that the run.sh file is run on. It will update the > control_path in ansible.cfg accordingly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2009) Address Javadoc checkstyle issues in metron-common
Justin Leet created METRON-2009: --- Summary: Address Javadoc checkstyle issues in metron-common Key: METRON-2009 URL: https://issues.apache.org/jira/browse/METRON-2009 Project: Metron Issue Type: Improvement Reporter: Justin Leet Assignee: Justin Leet We have a lot of missing Javadoc or broken style in various modules, including metron-common. We should fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2006) Reenable JacoCo code coverage
Justin Leet created METRON-2006: --- Summary: Reenable JacoCo code coverage Key: METRON-2006 URL: https://issues.apache.org/jira/browse/METRON-2006 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet It got disabled here: https://github.com/apache/metron/pull/840/commits/54c2486e4e618a6d445667ed2040faacf165ca8c We should get it reenabled and working with the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-2005) Batch Writer writes 0-byte files to HDFS on rotation
Justin Leet created METRON-2005: --- Summary: Batch Writer writes 0-byte files to HDFS on rotation Key: METRON-2005 URL: https://issues.apache.org/jira/browse/METRON-2005 Project: Metron Issue Type: Bug Reporter: Justin Leet Assignee: Justin Leet This results from https://github.com/apache/metron/pull/505 That PR breaks the standard convention of just choose a file name and rotate the file repeatedly, because now any message could get routed to a different file based on a Stellar statement. This break was noted in the PR, because we didn't care about the rotation number anyway. This works fine for the 0th rotation (a new file is opened, data is written, file is closed), but on the first rotation we signal to the HdfsWriter that the file has been closed in order to limit the maximum number of open files, but still create a new file with rotation 1. This file never receives any data (because we no longer maintain an open file reference to it), and the SourceHandler for it stays open with the Timer still attempting further (pointless rotations). Note that no data is lost, any data that would go into this file just instead goes into a new 0 rotation file. This becomes more obvious the longer the cluster is running or the shorter the timeout on a file is. As each open file attempts rotations, eventually large numbers of 0-byte files are created. An easy fix for this is to remove the creation of new files during rotations (but still perform RotationActions). This means that every file will have a 0 rotation (which we don't actually use for anything anyway). More complicated things could be done (e.g. evict oldest file from a cache), but it seems heavy handed for maintaining a rotation count we don't care about anyway. Additionally, the Timer should be cancelled when the reference is removed from HdfsWriter. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1942) Travis failure during caching of integration tests
Justin Leet created METRON-1942: --- Summary: Travis failure during caching of integration tests Key: METRON-1942 URL: https://issues.apache.org/jira/browse/METRON-1942 Project: Metron Issue Type: Bug Reporter: Justin Leet Travis had a failure during the caching phase after running the integration tests. It took too long to finish up and got killed from the build limit. See https://travis-ci.org/apache/metron/builds/469535418 Seems unrelated to the PR itself, which was for this PR: https://github.com/apache/metron/pull/1249 The follow-on build after kicking Travis succeeded with several minutes to spare. If it happens more frequently, it'll start to become higher priority. I'm labelling this with test-error, although technically it's not a test, but the framework running the tests. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1939) Update version to 0.7.1
Justin Leet created METRON-1939: --- Summary: Update version to 0.7.1 Key: METRON-1939 URL: https://issues.apache.org/jira/browse/METRON-1939 Project: Metron Issue Type: Task Reporter: Justin Leet Assignee: Justin Leet Need to update the version for ongoing development post 0.7.0 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
