[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065879#comment-16065879 ] ASF GitHub Bot commented on METRON-777: --- Github user justinleet commented on the issue: https://github.com/apache/metron/pull/530 To confirm @mmiklavc about maven, I did hit the exact same issue using 3.5.0, although I didn't validate with 3.3.9. > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[ https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065877#comment-16065877 ] ASF GitHub Bot commented on METRON-1004: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/624 I added a fix to actually clear out the correct directory of Maven artifacts before caching. In a separate, experimental branch, there's an attempt to cache the artifacts resulting from npm. See: https://github.com/justinleet/metron/tree/caching and https://travis-ci.org/justinleet/metron. This required a run without the integration tests on in order to make it to the populating the cache successfully, then reenabling them next commit. At this point we do have intermittent successful builds on my Travis, although I'm doubt it's consistent. > Travis CI - Job Exceeded Maximum Time Limit > --- > > Key: METRON-1004 > URL: https://issues.apache.org/jira/browse/METRON-1004 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > Fix For: Next + 1 > > > Most Travis CI jobs are failing with the following error message. > bq. The job exceeded the maximum time limit for jobs, and has been terminated. > Travis apparently has a hard 50 minute limit that will cause jobs to timeout. > > https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts > There is some additional time for setting up the container that is counted in > this hard 50 minute limit that is not reported in the UI. So even if a build > says it took 44 minutes with a timeout, it likely in fact did hit the 50 > minute limit. > Here are some examples of these failures. > * https://travis-ci.org/apache/metron/builds/244574987 > * https://travis-ci.org/apache/metron/builds/244552768 > * https://travis-ci.org/apache/metron/builds/241249480 > This is a separate issue from the "Storm slots didn't shut down" issue that > is tracked in a separate PR. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065568#comment-16065568 ] ASF GitHub Bot commented on METRON-777: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Start of the deployment readme, parser deployment to follow. > NOTE: METRON-942 has some changes in this area, since there where fixes when actually writing the 3rd party extension installer > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-508) Expand Elasticsearch templates to support the standard bro logs
[ https://issues.apache.org/jira/browse/METRON-508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065303#comment-16065303 ] ASF GitHub Bot commented on METRON-508: --- Github user JonZeolla closed the pull request at: https://github.com/apache/metron/pull/586 > Expand Elasticsearch templates to support the standard bro logs > --- > > Key: METRON-508 > URL: https://issues.apache.org/jira/browse/METRON-508 > Project: Metron > Issue Type: Sub-task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > The current elasticsearch templates do not support any logs other than Conn, > HTTP, and DNS. We should provide additional templates so that an > out-of-the-box bro install can send all of its logs into Metron and they will > get probably indexed in elasticsearch. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (METRON-1006) Remove Incubator DISCLAIMER file and fix Release Process doc
[ https://issues.apache.org/jira/browse/METRON-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley reassigned METRON-1006: -- Assignee: Matt Foley > Remove Incubator DISCLAIMER file and fix Release Process doc > > > Key: METRON-1006 > URL: https://issues.apache.org/jira/browse/METRON-1006 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley >Assignee: Matt Foley > Fix For: 0.4.0 > > > Incubator projects have to have a DISCLAIMER that says: > ``` > Apache Metron is an effort undergoing incubation at The Apache Software > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > required > of all newly accepted projects until a further review indicates that the > infrastructure, communications, and decision making process have stabilized in > a manner consistent with other successful ASF projects. While incubation > status > is not necessarily a reflection of the completeness or stability of the code, > it does indicate that the project has yet to be fully endorsed by the ASF. > ``` > This is not appropriate for a graduated TLP, and there is no evident similar > disclaimer for TLPs as shown in other projects. The usual disclaimers remain > in the LICENSE file, of course. > Removing the DISCLAIMER file from our project root source directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-1006) Remove Incubator DISCLAIMER file and fix Release Process doc
[ https://issues.apache.org/jira/browse/METRON-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated METRON-1006: --- Fix Version/s: 0.4.0 > Remove Incubator DISCLAIMER file and fix Release Process doc > > > Key: METRON-1006 > URL: https://issues.apache.org/jira/browse/METRON-1006 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley > Fix For: 0.4.0 > > > Incubator projects have to have a DISCLAIMER that says: > ``` > Apache Metron is an effort undergoing incubation at The Apache Software > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > required > of all newly accepted projects until a further review indicates that the > infrastructure, communications, and decision making process have stabilized in > a manner consistent with other successful ASF projects. While incubation > status > is not necessarily a reflection of the completeness or stability of the code, > it does indicate that the project has yet to be fully endorsed by the ASF. > ``` > This is not appropriate for a graduated TLP, and there is no evident similar > disclaimer for TLPs as shown in other projects. The usual disclaimers remain > in the LICENSE file, of course. > Removing the DISCLAIMER file from our project root source directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1006) Remove Incubator DISCLAIMER file and fix Release Process doc
[ https://issues.apache.org/jira/browse/METRON-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065160#comment-16065160 ] ASF GitHub Bot commented on METRON-1006: Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/625 > Remove Incubator DISCLAIMER file and fix Release Process doc > > > Key: METRON-1006 > URL: https://issues.apache.org/jira/browse/METRON-1006 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley > > Incubator projects have to have a DISCLAIMER that says: > ``` > Apache Metron is an effort undergoing incubation at The Apache Software > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > required > of all newly accepted projects until a further review indicates that the > infrastructure, communications, and decision making process have stabilized in > a manner consistent with other successful ASF projects. While incubation > status > is not necessarily a reflection of the completeness or stability of the code, > it does indicate that the project has yet to be fully endorsed by the ASF. > ``` > This is not appropriate for a graduated TLP, and there is no evident similar > disclaimer for TLPs as shown in other projects. The usual disclaimers remain > in the LICENSE file, of course. > Removing the DISCLAIMER file from our project root source directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1006) Remove Incubator DISCLAIMER file and fix Release Process doc
[ https://issues.apache.org/jira/browse/METRON-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065153#comment-16065153 ] ASF GitHub Bot commented on METRON-1006: Github user mattf-horton commented on the issue: https://github.com/apache/metron/pull/625 This has been +1'ed in email by the following reviewers: @nickwallen : +1 Yes, definitely cruft. Good find. @cestella : +1 to removing it. Other top level projects do not have a disclaimer (see, for example, hbase: http://www-eu.apache.org/dist/hbase/stable) @ottobackwards : +1 Thanks, all. > Remove Incubator DISCLAIMER file and fix Release Process doc > > > Key: METRON-1006 > URL: https://issues.apache.org/jira/browse/METRON-1006 > Project: Metron > Issue Type: Bug >Reporter: Matt Foley > > Incubator projects have to have a DISCLAIMER that says: > ``` > Apache Metron is an effort undergoing incubation at The Apache Software > Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is > required > of all newly accepted projects until a further review indicates that the > infrastructure, communications, and decision making process have stabilized in > a manner consistent with other successful ASF projects. While incubation > status > is not necessarily a reflection of the completeness or stability of the code, > it does indicate that the project has yet to be fully endorsed by the ASF. > ``` > This is not appropriate for a graduated TLP, and there is no evident similar > disclaimer for TLPs as shown in other projects. The usual disclaimers remain > in the LICENSE file, of course. > Removing the DISCLAIMER file from our project root source directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (METRON-1006) Remove Incubator DISCLAIMER file and fix Release Process doc
Matt Foley created METRON-1006: -- Summary: Remove Incubator DISCLAIMER file and fix Release Process doc Key: METRON-1006 URL: https://issues.apache.org/jira/browse/METRON-1006 Project: Metron Issue Type: Bug Reporter: Matt Foley Incubator projects have to have a DISCLAIMER that says: ``` Apache Metron is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator PMC. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. ``` This is not appropriate for a graduated TLP, and there is no evident similar disclaimer for TLPs as shown in other projects. The usual disclaimers remain in the LICENSE file, of course. Removing the DISCLAIMER file from our project root source directory. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065046#comment-16065046 ] ASF GitHub Bot commented on METRON-777: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 OK, IWOMM. I'm on irc if you want to jump on > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065027#comment-16065027 ] ASF GitHub Bot commented on METRON-777: --- Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/530 Looking into it now, but building locally I see this: ``` $ mvn clean install -DskipTests -T 2C ... [INFO] elasticsearch-shaded ... SKIPPED [INFO] metron-elasticsearch ... SKIPPED [INFO] metron-maven-parser-extension-archetype FAILURE [ 0.995 s] [INFO] metron-maven-archetypes SKIPPED [INFO] metron-deployment .. SKIPPED [INFO] Metron Ambari Management Pack .. SKIPPED [INFO] metron-docker .. SKIPPED [INFO] metron-interface ... SKIPPED [INFO] metron-config .. SKIPPED [INFO] metron-rest-client . SKIPPED [INFO] metron-rest SKIPPED [INFO] site-book .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 2.138 s (Wall Clock) [INFO] Finished at: 2017-06-27T09:51:59-06:00 [INFO] Final Memory: 38M/418M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-archetype-plugin:3.0.0:integration-test (default-integration-test) on project metron-maven-parser-extension-archetype: [ERROR] Archetype IT 'basic' failed: org.codehaus.plexus.util.xml.pull.XmlPullParserException: end tag not allowed in epilog but got / (position: END_TAG seen ...\n\n\n [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :metron-maven-parser-extension-archetype ``` > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[ https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065022#comment-16065022 ] ASF GitHub Bot commented on METRON-1004: Github user justinleet commented on a diff in the pull request: https://github.com/apache/metron/pull/624#discussion_r124317746 --- Diff: .travis.yml --- @@ -17,7 +17,7 @@ before_install: - export PATH=$M2_HOME/bin:$PATH script: - | --- End diff -- Sorry, my bad. Usually I tend to consider the commits less important because it's usually a full feature, and it's just minor changes / fixes afterwards. I'll try to make sure the messages are easier to follow, since this is pretty ongoing until it's consistent. > Travis CI - Job Exceeded Maximum Time Limit > --- > > Key: METRON-1004 > URL: https://issues.apache.org/jira/browse/METRON-1004 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > Fix For: Next + 1 > > > Most Travis CI jobs are failing with the following error message. > bq. The job exceeded the maximum time limit for jobs, and has been terminated. > Travis apparently has a hard 50 minute limit that will cause jobs to timeout. > > https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts > There is some additional time for setting up the container that is counted in > this hard 50 minute limit that is not reported in the UI. So even if a build > says it took 44 minutes with a timeout, it likely in fact did hit the 50 > minute limit. > Here are some examples of these failures. > * https://travis-ci.org/apache/metron/builds/244574987 > * https://travis-ci.org/apache/metron/builds/244552768 > * https://travis-ci.org/apache/metron/builds/241249480 > This is a separate issue from the "Storm slots didn't shut down" issue that > is tracked in a separate PR. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[ https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065017#comment-16065017 ] ASF GitHub Bot commented on METRON-1004: Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/624#discussion_r124314669 --- Diff: .travis.yml --- @@ -17,7 +17,7 @@ before_install: - export PATH=$M2_HOME/bin:$PATH script: - | --- End diff -- because it makes it slow right? Can we document with the commits, as you go, the rationale behind the changes, so we can look back and understand a little bit? "why did we get rid of FOO?" Let me check the commit log > " Remove foo. It is seen to cause an increase of X in Y and do z. it is also pretty snarky and fresh" "Oh, that makes sense" > Travis CI - Job Exceeded Maximum Time Limit > --- > > Key: METRON-1004 > URL: https://issues.apache.org/jira/browse/METRON-1004 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > Fix For: Next + 1 > > > Most Travis CI jobs are failing with the following error message. > bq. The job exceeded the maximum time limit for jobs, and has been terminated. > Travis apparently has a hard 50 minute limit that will cause jobs to timeout. > > https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts > There is some additional time for setting up the container that is counted in > this hard 50 minute limit that is not reported in the UI. So even if a build > says it took 44 minutes with a timeout, it likely in fact did hit the 50 > minute limit. > Here are some examples of these failures. > * https://travis-ci.org/apache/metron/builds/244574987 > * https://travis-ci.org/apache/metron/builds/244552768 > * https://travis-ci.org/apache/metron/builds/241249480 > This is a separate issue from the "Storm slots didn't shut down" issue that > is tracked in a separate PR. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[
https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065015#comment-16065015
]
ASF GitHub Bot commented on METRON-1004:
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/metron/pull/624#discussion_r124314003
--- Diff:
metron-platform/metron-pcap-backend/src/test/java/org/apache/metron/pcap/integration/PcapTopologyIntegrationTest.java
---
@@ -90,23 +90,6 @@ public boolean accept(File dir, String name) {
}
--- End diff --
I think until this is completely deprecated, we should keep the test, but
disable it.
> Travis CI - Job Exceeded Maximum Time Limit
> ---
>
> Key: METRON-1004
> URL: https://issues.apache.org/jira/browse/METRON-1004
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
> Fix For: Next + 1
>
>
> Most Travis CI jobs are failing with the following error message.
> bq. The job exceeded the maximum time limit for jobs, and has been terminated.
> Travis apparently has a hard 50 minute limit that will cause jobs to timeout.
>
> https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts
> There is some additional time for setting up the container that is counted in
> this hard 50 minute limit that is not reported in the UI. So even if a build
> says it took 44 minutes with a timeout, it likely in fact did hit the 50
> minute limit.
> Here are some examples of these failures.
> * https://travis-ci.org/apache/metron/builds/244574987
> * https://travis-ci.org/apache/metron/builds/244552768
> * https://travis-ci.org/apache/metron/builds/241249480
> This is a separate issue from the "Storm slots didn't shut down" issue that
> is tracked in a separate PR.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065014#comment-16065014 ] ASF GitHub Bot commented on METRON-777: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 I have added more documentation and I'm working on documenting where things are deployed > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-895) Ambari "Metron Enrichment Start" Fails - The TGT found is not renewable
[
https://issues.apache.org/jira/browse/METRON-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064980#comment-16064980
]
Anand Subramanian commented on METRON-895:
--
Marking bug as resolved, since the above comment from Nick works, and it is
already in place.
> Ambari "Metron Enrichment Start" Fails - The TGT found is not renewable
> ---
>
> Key: METRON-895
> URL: https://issues.apache.org/jira/browse/METRON-895
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>
> After Kerberizing a cluster in Ambari with Metron already installed, I am
> unable to launch the enrichment topology. It complains that the "The TGT
> found is not renewable".
> I am running on CentOS 7 and is likely specific to this version of CentOS.
> {code}
> [root@y113 ~]# cat /etc/centos-release
> CentOS Linux release 7.2.1511 (Core)
> {code}
> {code}
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_master.py",
> line 113, in
> Enrichment().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 280, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_master.py",
> line 74, in start
> commands.start_enrichment_topology()
> File
> "/var/lib/ambari-agent/cache/common-services/METRON/0.4.0/package/scripts/enrichment_commands.py",
> line 146, in start_enrichment_topology
> user=self.__params.metron_user)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/metron/0.4.0/bin/start_enrichment_topology.sh
> -s enrichment -z
> y113.l42scl.hortonworks.com:2181,y114.l42scl.hortonworks.com:2181,y115.l42scl.hortonworks.com:2181'
> returned 1. Running: /usr/jdk64/jdk1.8.0_77/bin/java -server -Ddaemon.name=
> -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.3.0-37/storm
> -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
> -cp
> /usr/hdp/2.5.3.0-37/storm/lib/clojure-1.7.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/disruptor-3.3.2.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-rename-hack-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ring-cors-0.1.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/asm-5.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.5.3.0-37/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.5.3.0-37/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.5.3.0-37/storm/lib/zookeeper.jar:/usr/hdp/2.5.3.0-37/storm/lib/minlog-1.3.0.jar:/usr/hdp/2.5.3.0-37/storm/lib/kryo-3.0.3.jar:/usr/hdp/2.5.3.0-37/storm/lib/storm-core-1.0.1.2.5.3.0-37.jar:/usr/hdp/2.5.3.0-37/storm/lib/reflectasm-1.10.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/objenesis-2.1.jar:/usr/hdp/2.5.3.0-37/storm/lib/ambari-metrics-storm-sink.jar
> org.apache.storm.daemon.ClientJarTransformerRunner
> org.apache.storm.hack.StormShadeTransformer
> /usr/metron/0.4.0/lib/metron-enrichment-0.4.0-uber.jar
> /tmp/0bea323c2aba11e781e900351a9eb24a.jar
> Running: /usr/jdk64/jdk1.8.0_77/bin/java -client -Ddaemon.name=
> -Dstorm.options= -Dstorm.home=/usr/hdp/2.5.3.0-37/storm
> -Dstorm.log.dir=/var/log/storm
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -Dstorm.conf.file=
> -cp
>
[jira] [Commented] (METRON-939) Upgrade ElasticSearch and Kibana
[ https://issues.apache.org/jira/browse/METRON-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064875#comment-16064875 ] ASF GitHub Bot commented on METRON-939: --- Github user wardbekker commented on the issue: https://github.com/apache/metron/pull/619 hey @cestella, @simonellistonball, see updated contributor notes. It's not ready for a official pull request, but this gives a good idea on the impact on the code for a working ES5.x implementation. > Upgrade ElasticSearch and Kibana > > > Key: METRON-939 > URL: https://issues.apache.org/jira/browse/METRON-939 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla > > Upgrade ElasticSearch and Kibana (latest is 5.4 as of writing this). Among > other benefits, this allows us to use periods in field names > (https://github.com/elastic/elasticsearch/pull/19937/files), which has been > available as of 5.0 and 2.4, and the ability to index an IPv6 address > properly > (https://www.elastic.co/blog/indexing-ipv6-addresses-in-elasticsearch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[ https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064820#comment-16064820 ] ASF GitHub Bot commented on METRON-1004: Github user justinleet commented on the issue: https://github.com/apache/metron/pull/624 As a note, what I have is currently the first steps towards reusing infra. It's not perfect, and it's not reused across classes. There was an attempt to use the build matrix to split fast and slow tests, but it resulted in inconsistent failures. Seems like Maven gets tangled up between the builds. Could merit further investigation. it'll increase processing time (because both unit and integration tests have to actually build), but should avoid having either portion of the build timeout. > Travis CI - Job Exceeded Maximum Time Limit > --- > > Key: METRON-1004 > URL: https://issues.apache.org/jira/browse/METRON-1004 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > Fix For: Next + 1 > > > Most Travis CI jobs are failing with the following error message. > bq. The job exceeded the maximum time limit for jobs, and has been terminated. > Travis apparently has a hard 50 minute limit that will cause jobs to timeout. > > https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts > There is some additional time for setting up the container that is counted in > this hard 50 minute limit that is not reported in the UI. So even if a build > says it took 44 minutes with a timeout, it likely in fact did hit the 50 > minute limit. > Here are some examples of these failures. > * https://travis-ci.org/apache/metron/builds/244574987 > * https://travis-ci.org/apache/metron/builds/244552768 > * https://travis-ci.org/apache/metron/builds/241249480 > This is a separate issue from the "Storm slots didn't shut down" issue that > is tracked in a separate PR. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[ https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064798#comment-16064798 ] ASF GitHub Bot commented on METRON-1004: Github user cestella commented on the issue: https://github.com/apache/metron/pull/624 I submitted PRs against this branch to incorporate the suggested changes above for: * Selective shading for non-leaf projects to cut the build times dramatically. * `TaxiiIntegrationTest` * `PcapIntegrationTest` I submit them without credit. > Travis CI - Job Exceeded Maximum Time Limit > --- > > Key: METRON-1004 > URL: https://issues.apache.org/jira/browse/METRON-1004 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen > Fix For: Next + 1 > > > Most Travis CI jobs are failing with the following error message. > bq. The job exceeded the maximum time limit for jobs, and has been terminated. > Travis apparently has a hard 50 minute limit that will cause jobs to timeout. > > https://docs.travis-ci.com/user/customizing-the-build#Build-Timeouts > There is some additional time for setting up the container that is counted in > this hard 50 minute limit that is not reported in the UI. So even if a build > says it took 44 minutes with a timeout, it likely in fact did hit the 50 > minute limit. > Here are some examples of these failures. > * https://travis-ci.org/apache/metron/builds/244574987 > * https://travis-ci.org/apache/metron/builds/244552768 > * https://travis-ci.org/apache/metron/builds/241249480 > This is a separate issue from the "Storm slots didn't shut down" issue that > is tracked in a separate PR. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-777) Create a plugin system for Metron based on 'NAR'
[ https://issues.apache.org/jira/browse/METRON-777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064716#comment-16064716 ] ASF GitHub Bot commented on METRON-777: --- Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/530 Thank you @mattf-horton, I was able to fix the links. So, @mmiklavc et. al., can we frame our documentation discussion around filling out and improving these documents? > Create a plugin system for Metron based on 'NAR' > > > Key: METRON-777 > URL: https://issues.apache.org/jira/browse/METRON-777 > Project: Metron > Issue Type: New Feature >Reporter: Otto Fowler >Assignee: Otto Fowler > > The success of the Metron project will be greatly dependent on community > participation, and with that the ability to adapt and extend Metron without > having to maintain a fork of the project. > As organizations and individuals look to extend the Metron system with custom > parsers, enrichments, and stellar functions that may be proprietary in > nature, the ability to develop and deploy these extensions outside the Metron > code base is critically important. > To that end, and after community discussion and proposal we create or > formalize the 'plugin' development story in Metron. > The proposal is to adapt the Apache Nifi NAR system for use in Metron. This > will provide the system with: > * archetype(s) for developer projects and independent development > * defined packaging and metadata for 'plugin' products > * loading and instantiation with classloader isolation capabilities > * removing the necessity for shading plugin jars > These capabilities will also enable other features, such as plugin lifecycle, > plugin configuration+redeployment, and other things. > The plugin archetypes and their installation will be a followon -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (METRON-1004) Travis CI - Job Exceeded Maximum Time Limit
[ https://issues.apache.org/jira/browse/METRON-1004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064627#comment-16064627 ] ASF GitHub Bot commented on METRON-1004: Github user cestella commented on the issue: https://github.com/apache/metron/pull/624 This does look good. A couple of observations in no particular order of importance; just wanted to get this out there for discussion. # Considering the overhead I want to consider the overhead not in our tests for a moment. In the last run, I count the following timings: * build - 5:41 * unit tests - 2:59 * integration tests - 14:44 * metron-config - 2:17 * verify licenses - 0:16 That's 25:57 out of a total run from Travis of 31:53, which is 5:56 overhead. We should factor that in. # Where to Focus ## Build Time The natural conclusion is to focus on the long pole, those integration tests, but we may be served to also consider the build time. Our build takes a long time and we depend upon parallelization to make the build return in a sensible time (the user time for the build is 26 minutes!). Furthermore, our build is extremely IO heavy due to the shading that we (necessarily) do. While we are on a shared system with the rest of the apache projects, I think reducing the IO burden of our build. While I think that shading is important, we have a very ham-fisted way of doing it. We shade for two reasons: * Relocation of dependencies that conflict * Creating uber jars for Storm One issue is that if we consider the tree of projects induced by their dependent nature, is that we shade non-leaf projects for purpose of relocation. I propose we stop doing that. Let's take, for instance, `metron-common`. We shade that project to relocate guava and beanutils. The consequences of relocating 2 packages is 47M of dependencies. Those 47M of dependencies also gets bundled again into all of the leaf projects (e.g. `metron-parsers`, etc.), thus shading twice. I propose fixing this one of two ways: * aggressively exclude ALL dependencies other than `org.apache.metron` and the relocated dependencies in any project that needs shading purely for relocation * Extract the shaded/relocated dependencies across the project into a separate project and make all of our non-leaf dependencies non-shaded I think the first may be the easiest to achieve and most surgical. Ultimately, it may even be advantageous to have a single jar created as the deployable output of our process (or maybe a small handful representing the independent subcomponents: `metron`, `MaaS` and `stellar`). ## Integration Tests Obviously the integration tests are the long pole in the tent. A couple of thoughts on these: ### `TaxiiIntegrationTest` My impression was that it was slow because parsing taxii via the mitre library was downright arduous. It costs us ~2:30 as of the working build above. We are passing a relatively large blob of taxii in and should consider trimming the taxii example data down to something more manageable and see if that will drop the timing down. ### `PcapIntegrationTest` We currently test two modes for the PcapIntegrationTest, pulling the timestamp from the key and pulling the timestamp from the message itself. We know that in production, we only want to support pulling the timestamp from the key. We might cut this test time in half by only testing the supported approach (it's 81 seconds as of last count). ### `Parser Integration Tests` We might want to reconsider what we integration test here. We currently have an integration test for every parser and we may get the same coverage by mocking out the `ParserWriterBolt` and constructing a shim to pass data in, execute against the real parser bolt, capture data written and evaluate the output. This would drop the overhead for each parser test dramatically (no storm or kafka) and would keep the semantics of the tests. Admittedly this may not be a focus in terms of bang-for-buck because total parser cost is around 86 seconds. # Reuse Integration Test Infrastructure This seems to be the persistent conversation whenever our tests start to push us over the edge. We incur quite a bit of overhead because we spin up and down integration test infrastructure in our `InMemoryComponent`s. We could consider correcting this in a couple of ways: * Reusing the infrastructure * Either use the in memory components or spin up light weight versions of the infrastructure and then run the tests against that (i.e. docker or separate-process versions of the in-memory components). * We'd need to refactor each integration test to clean up after itself so other tests are not splashed *
[jira] [Updated] (METRON-907) Zeppelin Dashboard to execute and download pcap queries
[ https://issues.apache.org/jira/browse/METRON-907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-907: Fix Version/s: 0.4.0 > Zeppelin Dashboard to execute and download pcap queries > --- > > Key: METRON-907 > URL: https://issues.apache.org/jira/browse/METRON-907 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.4.0 > > > We should have a zeppelin dashboard which allowed users to specify stellar > queries and date ranges and be able to download the resulting pcap files. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-955) Make the default sync policy for HDFS Writer be based on the batch size
[ https://issues.apache.org/jira/browse/METRON-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-955: Fix Version/s: 0.4.0 > Make the default sync policy for HDFS Writer be based on the batch size > --- > > Key: METRON-955 > URL: https://issues.apache.org/jira/browse/METRON-955 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.4.0 > > > Right now, we do a sync per record written in HDFS. This has performance > penalties as it requires a Namenode contact. We get around this by letting > the user define the sync policy in Flux, but that's less convenient and more > sensible defaults could be created. We should use the batch size as a > sensible default for the sync policy (i.e. sync every batch). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-995) Temporary variables in stellar enrichments which are maps do not function as expected
[
https://issues.apache.org/jira/browse/METRON-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Casey Stella updated METRON-995:
Fix Version/s: 0.4.0
> Temporary variables in stellar enrichments which are maps do not function as
> expected
> -
>
> Key: METRON-995
> URL: https://issues.apache.org/jira/browse/METRON-995
> Project: Metron
> Issue Type: Bug
>Reporter: Casey Stella
>Assignee: Casey Stella
> Fix For: 0.4.0
>
>
> If you construct a variable which returns a map in a stellar enrichment, it
> is not able to be used in statements afterwards due to the fact that we
> explode the map.
> For instance:
> {code}
> foo := { 'foo' : 7}
> foo_num := MAP_GET('foo', foo)
> foo := null
> {code}
> would yield the following fields:
> * {code}foo.foo == 7 {code}
> * {code}foo_num := null {code}
> What we would prefer is just:
> * {code}foo_num == 7{code}
> As a policy, we should only explode the fields which are maps that are around
> at the end of the enrichments. Also, the map itself should be available to
> be used in downstream enrichments.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (METRON-965) In the case where we specify the syncpolicy in the HDFS Writer, we do not properly clone and end up syncing for every record
[ https://issues.apache.org/jira/browse/METRON-965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-965: Fix Version/s: 0.4.0 > In the case where we specify the syncpolicy in the HDFS Writer, we do not > properly clone and end up syncing for every record > > > Key: METRON-965 > URL: https://issues.apache.org/jira/browse/METRON-965 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.4.0 > > > Right now the SyncPolicy works as follows: > * If you specify the sync policy in the flux file, it will use that policy to > determine when to sync the HDFS Writer. > * If you do not, then it will create a count sync policy based on the batch > size. > In the first case, we are not creating the sync policy properly and end up > sync'ing per record, which has performance impact and is incorrect. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-980) Short circuit operations for Stellar
[ https://issues.apache.org/jira/browse/METRON-980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-980: Fix Version/s: 0.4.0 > Short circuit operations for Stellar > > > Key: METRON-980 > URL: https://issues.apache.org/jira/browse/METRON-980 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.4.0 > > > Stellar does not currently contain short circuit operations. In most > languages, this is an important optimization, but for Stellar on Metron, this > is a requirement due to the fact that some variables may be null legitimately > and we cannot create multi-line conditionals or temporary variables at the > moment. > The short circuit operations supported: > * short circuited `or` (e.g. true or FUNC(...) would never execute FUNC) > * short circuited `and` (e.g. false and FUNC(...) would never execute FUNC) > * short circuited if/then/else (e.g. if true then FUNC(...) else FUNC2(...) > will never call FUNC2) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (METRON-966) Pcap topology does not commit offsets
[ https://issues.apache.org/jira/browse/METRON-966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Casey Stella updated METRON-966: Fix Version/s: 0.4.0 > Pcap topology does not commit offsets > - > > Key: METRON-966 > URL: https://issues.apache.org/jira/browse/METRON-966 > Project: Metron > Issue Type: Bug >Reporter: Casey Stella >Assignee: Casey Stella > Fix For: 0.4.0 > > > Due to a bug in the storm-kafka-client in 1.1.0, offsets are not committed in > the pcap topology. This can be worked around by ensuring that ack is called > after nextTuple() rather than during nextTuple(). This is ONLY a problem in > spout-only topologies (which self-ack). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
