[jira] [Updated] (METRON-2032) Create summary table having list of stellar functions in README
[ https://issues.apache.org/jira/browse/METRON-2032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-2032: -- Fix Version/s: 0.7.1 > Create summary table having list of stellar functions in README > --- > > Key: METRON-2032 > URL: https://issues.apache.org/jira/browse/METRON-2032 > Project: Metron > Issue Type: Task >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Create a summary table with hyperlinks to stellar functions under: > https://github.com/apache/metron/blob/master/metron-platform/metron-management/README.md#functions -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2046) IS_EMPTY stellar functions fails on empty map
[
https://issues.apache.org/jira/browse/METRON-2046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anand Subramanian updated METRON-2046:
--
Fix Version/s: 0.7.1
> IS_EMPTY stellar functions fails on empty map
> -
>
> Key: METRON-2046
> URL: https://issues.apache.org/jira/browse/METRON-2046
> Project: Metron
> Issue Type: Bug
>Reporter: Anand Subramanian
>Assignee: Anand Subramanian
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The following case is not working as expected:
> {code:java}
> [Stellar]>>> val := GEO_GET("10.1.1.1")
> {}
> [Stellar]>>> IS_EMPTY( val)
> false{code}
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1962) Make entering JDBC details in REST config to be optional
[ https://issues.apache.org/jira/browse/METRON-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1962: -- Fix Version/s: 0.7.1 > Make entering JDBC details in REST config to be optional > > > Key: METRON-1962 > URL: https://issues.apache.org/jira/browse/METRON-1962 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > In the case where user chooses to have LDAP enabled for REST authentication, > the REST config still asks for entering the following parameters as mandatory: > * Metron JDBC URL > * Metron JDBC Driver > * Metron JDBC username > * Metron JDBC password > * Metron JDBC platform > These should be optional so user can choose not to enter values for them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1955) Update metron SPEC file to include syslog 3164 parser
[ https://issues.apache.org/jira/browse/METRON-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anand Subramanian updated METRON-1955: -- Fix Version/s: 0.7.1 > Update metron SPEC file to include syslog 3164 parser > - > > Key: METRON-1955 > URL: https://issues.apache.org/jira/browse/METRON-1955 > Project: Metron > Issue Type: Bug >Reporter: Anand Subramanian >Assignee: Anand Subramanian >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h > Remaining Estimate: 0h > > Update metron.spec file to include syslog 3164 parser -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1933) Improve build-utils helper scripts
[ https://issues.apache.org/jira/browse/METRON-1933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Zeolla updated METRON-1933: --- Fix Version/s: 0.7.1 > Improve build-utils helper scripts > -- > > Key: METRON-1933 > URL: https://issues.apache.org/jira/browse/METRON-1933 > Project: Metron > Issue Type: Bug >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently generate_license.py doesn't handle errors properly, and the README > doesn't mention you should have run `mvn clean install -DskipTests` > beforehand. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1932) Update ES and Kibana to 5.6.14
[ https://issues.apache.org/jira/browse/METRON-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Zeolla updated METRON-1932: --- Fix Version/s: 0.7.1 > Update ES and Kibana to 5.6.14 > -- > > Key: METRON-1932 > URL: https://issues.apache.org/jira/browse/METRON-1932 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Fix For: 0.7.1 > > Time Spent: 4.5h > Remaining Estimate: 0h > > I'm seeing some issues that appear to have been fixed in ES 5.6.4, however we > should update to the latest ES/Kibana patch, currently 5.6.13. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1434) Ability to deploy Metron full dev as a single node (centos6) to AWS with vagrant and AMI
[ https://issues.apache.org/jira/browse/METRON-1434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1434: Fix Version/s: (was: 0.4.3) Next + 1 > Ability to deploy Metron full dev as a single node (centos6) to AWS with > vagrant and AMI > > > Key: METRON-1434 > URL: https://issues.apache.org/jira/browse/METRON-1434 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.4.3 > Environment: i) AWS Image: Centos6 - AWS t2.t2xlarge [32 GB Memory, 8 > Cores, 64 bit] based on metron-deployment/development/centos6/ > ii) AWS Single Node Cluster Deployment Using Vagrant: MacOS > iii) AMI "GCR-Xetron Demo"/"ami-93cb4ff7" - AWS t2.t2xlarge >Reporter: ashah >Assignee: ashah >Priority: Minor > Labels: AWS, EC2, vagrant > Fix For: Next + 1 > > > The pull request has two potential solutions for deploying to Metron to AWS. > i) AWS Single Node Cluster Deployment Using Vagrant > ii) AWS Single Node Cluster Deployment Using an AMI > *i) AWS Single Node Cluster Deployment Using Vagrant:* > I was able to deploy Metron as a single node to AWS using Vagrant on MacOS. > It was tested using the latest Metron code (as of 2018-01-30 - 8:00am UTC) > that was in git. I'm assuming the latest that was in git is 0.4.3 that is > progress. > _Instructions:_ > [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-vagrant] > > *ii) AWS Single Node Cluster Deployment Using an AMI:* > In addition to including instructions for deploying 0.4.3 using Vagrant on a > Mac I also included instructions for deploying to AWS using a preexisting AMI > (I made the following AMI public as a proof-of-concept: "GCR-Xetron > Demo"/"ami-93cb4ff7"). This might be the quicket option for deployment since > no preinstall requirments would be needed. > _Instructions:_ > [https://github.com/as22323/metron/blob/master/metron-deployment/README.md#aws-single-node-cluster-deployment-using-an-ami] > > PULL REQUEST CHECKLIST ITEMS > *"For documentation related changes:"* > Running cd site-book, mvn site produced "BUILD SUCCESS" > > *"For code changes:"* > Was not sure where to run "mvn -q clean integration-test install && > dev-utilities/build-utils/verify_licenses.sh ". Core Metron Code was not > changed. For i) the following code were newly added/modified: > [Vagrant to work with aws] > [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/Vagrantfile] > [changing eth1 to eth0] > [https://github.com/as22323/metron/blob/master/metron-deployment/development/aws-centos6/inventory/group_vars/all] > > > *"How the change may be verified: "* > You should be able to see Ambari after following instructions > (http://:8080) > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-865) Additional Mpack bug fixes and improvements, that affect Ambari database
[
https://issues.apache.org/jira/browse/METRON-865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-865:
---
Fix Version/s: (was: 0.3.1)
0.4.0
> Additional Mpack bug fixes and improvements, that affect Ambari database
>
>
> Key: METRON-865
> URL: https://issues.apache.org/jira/browse/METRON-865
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.3.0
> Environment: Centos7
>Reporter: Matt Foley
>Priority: Major
> Fix For: 0.4.0
>
>
> Multiple bug fixes and recommended improvements were found in the course of
> implementing METRON-608 that are unrelated to METRON-609 (singlenode
> install). Almost all items relate to Elasticsearch.
> About half the fixes did not impact the Ambari database, and were done in
> METRON-634 (PR#532).
> This jira provides the work items for changes that do impact the Ambari
> database, and should therefore be done with an Mpack version bump and
> database upgrade script. Implementation of these may be seen in the closed PR
> https://github.com/apache/metron/pull/425
> h2. Affects Ambari database, needs db upgrade script:
> * status_params.py, which redundantly defines pid_dir as a python variable,
> is unnecessary and unused by the ES portion of the Mpack. It can be removed.
> * pid_dir SHOULD be specified in elastic-sysconfig.xml, rather than
> elastic-env.xml, as it is a parameter that must be provided to ES at
> launch-time, but is not something there's any reason for the admin to change
> in usual circumstances.
> * conf_dir SHOULD be specified in elastic-env.xml or elastic-site.xml, not in
> elastic-sysconfig.xml. While it too is a parameter that must be provided to
> ES at launch-time, it is typically left to the installing admin where to put
> the config files.
> * The configuration parameter names in elastic-site.xml should be improved in
> several instances to make the semantics more obvious to the human reader (who
> may not be real familiar with Elasticsearch configuration). Mouse-over
> documentation will continue to provide the ES config parameter equivalents.
> In particular, suggest:
> {code}
> cluster_name -> es_cluster_name (to distinguish ES cluster from Stack
> cluster)
> zen_discovery_ping_unicast_hosts -> es_cluster_hosts
> network_host -> network_bindings (these are in fact interface names, not
> host names)
> {code}
> * "data_dir" apparently should be eliminated (from elastic-sysconfig) in
> preference for "path_data" (in elastic-site.xml). The latter value ends up
> overriding the former anyway, but the existence of the former is confusing
> and unnecessary.
> * All four configuration parameters in elastic-env.xml should be moved to
> elastic-site.xml, because they are all reasonable to set in a "site.xml" file
> and do end up in the .yml file that ES uses instead of "site.xml", and do NOT
> end up in environment variables. The only parameters that end up in env vars
> are set in elastic-sysconfig, and the ES launch process in fact ignores the
> elastic-env.sh file that is templated in elastic-env.xml (which consists only
> of JAVA_HOME and PATH). Therefore we could also eliminate elastic-env.sh and
> hence entirely remove elastic-env.xml, or we could choose to keep the small
> elastic-env.sh file and its template, just to remind people that it is
> necessary to have JAVA_HOME defined.
> * In METRON/0.3.0/configuration/metron-env.xml and
> METRON/0.3.0/package/scripts/params/params_linux.py, the value
> "metron_apps_indexed_hdfs_dir" does not need to be settable by admin; it is
> appropriate to require it to be subordinate to "metron_apps_hdfs_dir". Thus
> it can be removed from metron-env.xml and set to
> "\{metron_apps_hdfs_dir\}/indexing/indexed" in params_linux.py. This also
> eliminates a really unacceptable use of "double format".
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-650) Remove Kraken Maven dependency on opensoc github repo
[
https://issues.apache.org/jira/browse/METRON-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824743#comment-16824743
]
Justin Leet commented on METRON-650:
[~msmiklavcic] Is this ticket still relevant? Can we get the fix version
cleaned up (either to nothing if it's not fixed, or to whatever version it's
fixed in)?
> Remove Kraken Maven dependency on opensoc github repo
> -
>
> Key: METRON-650
> URL: https://issues.apache.org/jira/browse/METRON-650
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Priority: Major
> Fix For: 0.3.1
>
>
> Metron common's pom is referencing the opensoc repo to get Kraken:
>
>
> Metron-Kraken-Repo
> Metron Kraken Repository
> https://raw.github.com/opensoc/kraken/mvn-repo
>
>
> ...
>
> org.krakenapps
> kraken-pcap
> ${global_pcap_version}
>
>
> slf4j-api
> org.slf4j
>
>
> slf4j-simple
> org.slf4j
>
>
>
> This library does not currently reside in mvn central (searched by package
> and libname) and should instead be staged in Apache with Metron.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-967) How to push Snort, Bro, YAF log to Metron
[ https://issues.apache.org/jira/browse/METRON-967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-967: --- Fix Version/s: (was: 0.3.0) > How to push Snort, Bro, YAF log to Metron > - > > Key: METRON-967 > URL: https://issues.apache.org/jira/browse/METRON-967 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 > Environment: CentOS >Reporter: GS Peter >Priority: Major > Labels: features > Original Estimate: 96h > Remaining Estimate: 96h > > Hi everyone. > I configured Apache Metron full dev 0.3 but I don't push Snort, Bro, Yaf log > to Metron Server. Can you help me? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-737) Deploying additional model on an already running Maas model creates ghost cotainers
[ https://issues.apache.org/jira/browse/METRON-737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-737: --- Fix Version/s: (was: 0.3.0) > Deploying additional model on an already running Maas model creates ghost > cotainers > --- > > Key: METRON-737 > URL: https://issues.apache.org/jira/browse/METRON-737 > Project: Metron > Issue Type: Bug >Affects Versions: 0.3.0 > Environment: centos 6,7 . > java version "1.8.0_60" >Reporter: sapinkumar amin >Priority: Major > Attachments: AppMaster.stderr.log, AppMaster.stderr.timeout.log, > container_successful_launch.log, mock_dga.zip > > > First deployment of Maas goes fine. If we ask for 1 container running model, > it does deploy total 2. One as AM and another as model itself. However adding > more number of containers of same model creates ghost containers in following > fashion > 1. $METRON_HOME/metron/bin/maas_deploy.sh -zq 'zookeeper_host:2181' -lmp > /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga1 -v 1.0 -ni 1 > Creates 2 container. 1 is AM, another is model (mock_dga) > (Running same as above command) > 2. $METRON_HOME/metron/bin/maas_deploy.sh -zq 'zookeeper_host:2181' -lmp > /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga1 -v 1.0 -ni 1 > Creates 4 containers. 1 is AM, two are models (mock_dga). 1 is ghost. > (Running same as above command) > 3. $METRON_HOME/metron/bin/maas_deploy.sh -zq 'zookeeper_host:2181' -lmp > /root/mock_dga -hmp /user/root/models -mo ADD -m 512 -n dga1 -v 1.0 -ni 1 > Creates 7 containers. 1 is AM, 3 are models, 3 are ghosts. > Please find AppMaster stderr logs attached with this JIRA. > 1. AppMaster.stderr.timeout.log has AM logs when after second attempts, one > of the ghost container times out. And total comes back to 3 which as expected. > 2. AppMaster.stderr.log has AM logs about running command 3 times and > launching 7 containers. > Please find the model (mock_dga.zip) and container launched log attached with > this jira. There are no logs for ghost container. > Please also note that if I issue REMOVE command, i.e. > $METRON_HOME/metron/bin/maas_deploy.sh -zq zookeeper_host:2181 -lmp > /root/mock_dga -hmp /user/root/models -mo REMOVE -m 512 -n dga1 -v 1.0 -ni 1, > It successfully removes the running model but not the ghost containers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1005) Create Decodable Row Key for Profiler
[ https://issues.apache.org/jira/browse/METRON-1005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824739#comment-16824739 ] Justin Leet commented on METRON-1005: - [~nickwallen] This Jira can be closed and the fix version removed, right? > Create Decodable Row Key for Profiler > - > > Key: METRON-1005 > URL: https://issues.apache.org/jira/browse/METRON-1005 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.3.0 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > > To be able to answer the types of questions that I outlined in METRON-450, we > need a row key that is decodable. Right now there is no logic to decode a > row key, nor is the existing row key easily decodable. > Once the row keys can be decoded, you could scan all of the row keys in the > Profiler's HBase table, decode each of them and extract things like, the > names of all your profiles, the names of entities within a profile, the > period duration of a given profile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1213) Flow diagram for Metron Kafka topics
[ https://issues.apache.org/jira/browse/METRON-1213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1213: Fix Version/s: (was: Next + 1) > Flow diagram for Metron Kafka topics > > > Key: METRON-1213 > URL: https://issues.apache.org/jira/browse/METRON-1213 > Project: Metron > Issue Type: New Feature >Affects Versions: 0.4.1 > Environment: Management UI >Reporter: Jasper Knulst >Priority: Major > > The insight that is really missing in Metron is an overview of Kafka topic > sizes and flow metrics (topic influx and outflux) > It is hard to answer questions like: > -Is the enrichments topic (lag) growing over time? > -What parser topics feed into enrichments, at what rate? > -Is the enrichment topology keeping up with the projected influx from all > parsers? > -Same questions, but then for indexing topic > When the capacity of the chained topologies is not aligned bad things can > happen, like a serious threat not reaching ES/Kibana fast enough to react > upon. Metron can only be a realtime alerter if there is no congestion along > the way. > This would take merging metrics from various Metron moving parts like Storm > API, kafka-offset-checker (state/snapshots and flow metrics units/sec) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-914) Build Metron failure
[ https://issues.apache.org/jira/browse/METRON-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-914: --- Fix Version/s: (was: 0.4.0) > Build Metron failure > > > Key: METRON-914 > URL: https://issues.apache.org/jira/browse/METRON-914 > Project: Metron > Issue Type: Bug >Affects Versions: 0.4.0 > Environment: Ubuntu 16.04.1 >Reporter: GS Peter >Priority: Major > Attachments: build.log.tar.gz, build.test.log.gz, npm-debug.log > > Original Estimate: 504h > Remaining Estimate: 504h > > [INFO] metron-config .. FAILURE [04:08 > min] > [INFO] metron-rest-client . SKIPPED > [INFO] metron-rest SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 58:04 min > [INFO] Finished at: 2017-04-28T15:57:16+07:00 > [INFO] Final Memory: 141M/588M > [INFO] > > [ERROR] Failed to execute goal > com.github.eirslett:frontend-maven-plugin:1.3:npm (npm install) on project > metron-config: Failed to run task: 'npm install' failed. (error code 1) -> > [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :metron-config -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1331) ZKConfigurationsCacheIntegrationTest.validateBaseWrite Intermittent Test Failure
[
https://issues.apache.org/jira/browse/METRON-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1331:
Fix Version/s: (was: 0.4.1)
> ZKConfigurationsCacheIntegrationTest.validateBaseWrite Intermittent Test
> Failure
>
>
> Key: METRON-1331
> URL: https://issues.apache.org/jira/browse/METRON-1331
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Priority: Minor
>
> {code}
> Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 40.267 sec
> <<< FAILURE! - in
> org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest
> validateBaseWrite(org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest)
> Time elapsed: 31.052 sec <<< FAILURE!
> java.lang.AssertionError: expected:<{hdfs={index=yaf, batchSize=5,
> enabled=true}, elasticsearch={index=yaf, batchSize=25, batchTimeout=7,
> enabled=true}, solr={index=yaf, batchSize=5, enabled=true}}> but was:<{}>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest.lambda$validateBaseWrite$14(ZKConfigurationsCacheIntegrationTest.java:265)
> at
> org.apache.metron.integration.utils.TestUtils.assertEventually(TestUtils.java:54)
> at
> org.apache.metron.integration.utils.TestUtils.assertEventually(TestUtils.java:39)
> at
> org.apache.metron.common.zookeeper.ZKConfigurationsCacheIntegrationTest.validateBaseWrite(ZKConfigurationsCacheIntegrationTest.java:265)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Results :
> Failed tests:
>
> ZKConfigurationsCacheIntegrationTest.validateBaseWrite:265->lambda$validateBaseWrite$14:265
> expected:<{hdfs={index=yaf, batchSize=5, enabled=true},
> elasticsearch={index=yaf, batchSize=25, batchTimeout=7, enabled=true},
> solr={index=yaf, batchSize=5, enabled=true}}> but was:<{}>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1174) Metron Docker Document example not working
[
https://issues.apache.org/jira/browse/METRON-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1174:
Fix Version/s: (was: 0.4.0)
> Metron Docker Document example not working
> --
>
> Key: METRON-1174
> URL: https://issues.apache.org/jira/browse/METRON-1174
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.0
> Environment: CentOS 7
> Docker 1.12.6
> docker-compose version 1.16.1, build 6d1ac219
> docker-machine version 0.12.2, build 9371605
>Reporter: Choi BongJae
>Priority: Major
> Labels: docuentation, examples
> Original Estimate: 6h
> Remaining Estimate: 6h
>
> This Section is not working with error
> {code:java}
> /usr/metron/0.4.0//bin/zk_load_configs.sh: line 35:
> /usr/hdp/current/hbase-client/bin/hbase: No such file or directory
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> {code}
> Upload configs to Zookeeper
> Parser configs and a global config configured for this Docker environment are
> included with the Kafka/Zookeeper image. Load them with:
> {code:java}
> $ docker-compose exec kafkazk bash
> # $METRON_HOME/bin/zk_load_configs.sh -z localhost:2181 -m PUSH -i
> $METRON_HOME/config/zookeeper
> # exit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-2081) Is Metron compatible with newer version of HDP?
[
https://issues.apache.org/jira/browse/METRON-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2081:
Fix Version/s: (was: 0.4.1)
> Is Metron compatible with newer version of HDP?
>
>
> Key: METRON-2081
> URL: https://issues.apache.org/jira/browse/METRON-2081
> Project: Metron
> Issue Type: Improvement
>Affects Versions: 0.4.1
>Reporter: Abhishek Sinha
>Priority: Major
>
> I am unable to deploy HDP 2.5.0 as it gives repo error, can anyone help me
> with the latest HDP version with which it is compatible with? The error that
> I receive is as follows:
>
> 2019-04-16 15:39:58,855 - Could not find stack selector for stack: HDP
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 155, in
> ApplicationTimelineServer().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 285, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 39, in install
> self.install_packages(env)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 576, in install_packages
> retry_count=agent_stack_retry_count)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 54, in action_install
> self.install_package(package_name, self.resource.use_repos,
> self.resource.skip_repos)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py",
> line 51, in install_package
> self.checked_call_with_retries(cmd, sudo=True,
> logoutput=self.get_logoutput())
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 86, in checked_call_with_retries
> return self._call_with_retries(cmd, is_checked=True, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 98, in _call_with_retries
> code, out = func(cmd, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_5_0_0_1245-yarn' returned 1.
> Error: Nothing to do
> *stdout:* */var/lib/ambari-agent/data/output-185.txt*
> 2019-04-16 15:38:35,068 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2019-04-16 15:38:35,070 - Group['hadoop'] {}
> 2019-04-16 15:38:35,071 - Group['users'] {}
> 2019-04-16 15:38:35,071 - Group['zeppelin'] {}
> 2019-04-16 15:38:35,071 - Group['metron'] {}
> 2019-04-16 15:38:35,072 - Group['kibana'] {}
> 2019-04-16 15:38:35,072 - Group['spark'] {}
> 2019-04-16 15:38:35,072 - Group['livy'] {}
> 2019-04-16 15:38:35,072 - Group['elasticsearch'] {}
> 2019-04-16 15:38:35,072 - User['hive'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,073 - User['zeppelin'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,074 - User['ambari-qa'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'users']}
> 2019-04-16 15:38:35,074 - User['flume'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,075 - User['metron'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['hdfs'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['storm'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,077 - User['spark'] \{'gid':
[jira] [Updated] (METRON-1285) Alerts UI Displays Error When Credentials Timeout
[ https://issues.apache.org/jira/browse/METRON-1285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1285: Fix Version/s: (was: 0.4.1) > Alerts UI Displays Error When Credentials Timeout > - > > Key: METRON-1285 > URL: https://issues.apache.org/jira/browse/METRON-1285 > Project: Metron > Issue Type: Bug >Reporter: Nick Allen >Priority: Major > Labels: Alerts-UI > Attachments: Screen Shot 2017-10-27 at 1.06.48 PM.png > > > Steps to reproduce: > 1. Login to the Alarms UI. > 2. Turn off the auto-refresh option (not sure if this is required to > reproduce.) > 3. Remain on the main page, but wait until the credentials timeout. > 4. Attempt to perform an action in the stale browser window; like sort the > results. > 5. The browser will redirect the user to an ugly page showing a stack trace. > See screenshot. > I would expect the user to be redirected to the login page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2080) unable to access HDP 2.5.0 repo : http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0
[
https://issues.apache.org/jira/browse/METRON-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2080:
Fix Version/s: (was: 0.4.1)
> unable to access HDP 2.5.0 repo :
> http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.5.0.0
> ---
>
> Key: METRON-2080
> URL: https://issues.apache.org/jira/browse/METRON-2080
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.1
>Reporter: Abhishek Sinha
>Priority: Major
>
> HI,
> I am trying to deploy HDP 2.5.0 with Metron, getting following error :
>
> 2019-04-16 15:39:58,855 - Could not find stack selector for stack: HDP
> Traceback (most recent call last):
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 155, in
> ApplicationTimelineServer().execute()
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 285, in execute
> method(env)
> File
> "/var/lib/ambari-agent/cache/common-services/YARN/2.1.0.2.0/package/scripts/application_timeline_server.py",
> line 39, in install
> self.install_packages(env)
> File
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 576, in install_packages
> retry_count=agent_stack_retry_count)
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 54, in action_install
> self.install_package(package_name, self.resource.use_repos,
> self.resource.skip_repos)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py",
> line 51, in install_package
> self.checked_call_with_retries(cmd, sudo=True,
> logoutput=self.get_logoutput())
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 86, in checked_call_with_retries
> return self._call_with_retries(cmd, is_checked=True, **kwargs)
> File
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py",
> line 98, in _call_with_retries
> code, out = func(cmd, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of
> '/usr/bin/yum -d 0 -e 0 -y install hadoop_2_5_0_0_1245-yarn' returned 1.
> Error: Nothing to do
> *stdout:* */var/lib/ambari-agent/data/output-185.txt*
> 2019-04-16 15:38:35,068 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2019-04-16 15:38:35,070 - Group['hadoop'] {}
> 2019-04-16 15:38:35,071 - Group['users'] {}
> 2019-04-16 15:38:35,071 - Group['zeppelin'] {}
> 2019-04-16 15:38:35,071 - Group['metron'] {}
> 2019-04-16 15:38:35,072 - Group['kibana'] {}
> 2019-04-16 15:38:35,072 - Group['spark'] {}
> 2019-04-16 15:38:35,072 - Group['livy'] {}
> 2019-04-16 15:38:35,072 - Group['elasticsearch'] {}
> 2019-04-16 15:38:35,072 - User['hive'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,073 - User['zeppelin'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,074 - User['ambari-qa'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'users']}
> 2019-04-16 15:38:35,074 - User['flume'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,075 - User['metron'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['hdfs'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,076 - User['storm'] \{'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': [u'hadoop']}
> 2019-04-16 15:38:35,077 - User['spark'] \{'gid':
[jira] [Assigned] (METRON-2029) Configure Table should have filter
[ https://issues.apache.org/jira/browse/METRON-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-2029: --- Assignee: Shane Ardell > Configure Table should have filter > -- > > Key: METRON-2029 > URL: https://issues.apache.org/jira/browse/METRON-2029 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, when a user opens the configure table panel, they must scroll down > to see all their choices. This list can be really long, so it would be > beneficial to have an input for a user to type into and have the options > filtered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2029) Configure Table should have filter
[ https://issues.apache.org/jira/browse/METRON-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2029: Fix Version/s: 0.7.1 > Configure Table should have filter > -- > > Key: METRON-2029 > URL: https://issues.apache.org/jira/browse/METRON-2029 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently, when a user opens the configure table panel, they must scroll down > to see all their choices. This list can be really long, so it would be > beneficial to have an input for a user to type into and have the options > filtered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1987) Upgrade Alert UI to stable Bootstrap 4
[ https://issues.apache.org/jira/browse/METRON-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1987: Fix Version/s: 0.7.1 > Upgrade Alert UI to stable Bootstrap 4 > -- > > Key: METRON-1987 > URL: https://issues.apache.org/jira/browse/METRON-1987 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Like we did for the Management UI, we should upgrade the version of Bootstrap > that we are using to the latest stable version. We are currently on an alpha > version which includes deprecated class names and values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1971) Short timeout value in Cypress may cause build failures
[ https://issues.apache.org/jira/browse/METRON-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1971: Fix Version/s: 0.7.1 > Short timeout value in Cypress may cause build failures > --- > > Key: METRON-1971 > URL: https://issues.apache.org/jira/browse/METRON-1971 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > According to the Cypress docs, it is unnecessary to add a wait command after > cy.visit. "Waiting for this is unnecessary because the cy.visit() resolves > once the page fires its load event. By that time all of your assets have been > loaded including javascript, stylesheets, and html." > However, the problem is with an Angular application, the page firing a load > event does not mean that the entire view has rendered yet. In systems where > memory is scarce, such as a Travis environment or a machine that has > memory-intensive applications running, this means a page load can take a few > seconds (or possibly even longer). > We currently use the default timeout in Cypress, which is only 4000ms. I > believe this short timeout can cause failures like this in Metron: > https://travis-ci.org/apache/metron/jobs/483945575 > We should increase the defaultCommandTimeout in cypress.json to something > like 3ms or greater to prevent failures like this in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1987) Upgrade Alert UI to stable Bootstrap 4
[ https://issues.apache.org/jira/browse/METRON-1987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1987: --- Assignee: Shane Ardell > Upgrade Alert UI to stable Bootstrap 4 > -- > > Key: METRON-1987 > URL: https://issues.apache.org/jira/browse/METRON-1987 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Like we did for the Management UI, we should upgrade the version of Bootstrap > that we are using to the latest stable version. We are currently on an alpha > version which includes deprecated class names and values. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1971) Short timeout value in Cypress may cause build failures
[ https://issues.apache.org/jira/browse/METRON-1971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet reassigned METRON-1971: --- Assignee: Shane Ardell > Short timeout value in Cypress may cause build failures > --- > > Key: METRON-1971 > URL: https://issues.apache.org/jira/browse/METRON-1971 > Project: Metron > Issue Type: Bug >Reporter: Shane Ardell >Assignee: Shane Ardell >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > According to the Cypress docs, it is unnecessary to add a wait command after > cy.visit. "Waiting for this is unnecessary because the cy.visit() resolves > once the page fires its load event. By that time all of your assets have been > loaded including javascript, stylesheets, and html." > However, the problem is with an Angular application, the page firing a load > event does not mean that the entire view has rendered yet. In systems where > memory is scarce, such as a Travis environment or a machine that has > memory-intensive applications running, this means a page load can take a few > seconds (or possibly even longer). > We currently use the default timeout in Cypress, which is only 4000ms. I > believe this short timeout can cause failures like this in Metron: > https://travis-ci.org/apache/metron/jobs/483945575 > We should increase the defaultCommandTimeout in cypress.json to something > like 3ms or greater to prevent failures like this in the future. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1948) Dropped messages from REGEX_SELECT parser field transformation are not acked in Storm
[ https://issues.apache.org/jira/browse/METRON-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1948: Fix Version/s: 0.7.1 > Dropped messages from REGEX_SELECT parser field transformation are not acked > in Storm > - > > Key: METRON-1948 > URL: https://issues.apache.org/jira/browse/METRON-1948 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1795) General Purpose Regex Parser
[
https://issues.apache.org/jira/browse/METRON-1795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-1795:
Fix Version/s: 0.7.1
> General Purpose Regex Parser
>
>
> Key: METRON-1795
> URL: https://issues.apache.org/jira/browse/METRON-1795
> Project: Metron
> Issue Type: New Feature
>Reporter: Jagdeep Singh
>Priority: Minor
> Fix For: 0.7.1
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> We have implemented a general purpose regex parser for Metron that we are
> interested in contributing back to the community.
>
> While the Metron Grok parser provides some regex based capability today, the
> intention of this general purpose regex parser is to:
> # Allow for more advanced parsing scenarios (specifically, dealing with
> multiple regex lines for devices that contain several log formats within them)
> # Give users and developers of Metron additional options for parsing
> # With the new parser chaining and regex routing feature available in
> Metron, this gives some additional flexibility to logically separate a flow
> by:
> # Regex routing to segregate logs at a device level and handle envelope
> unwrapping
> # This general purpose regex parser to parse an entire device type that
> contains multiple log formats within the single device (for example, RHEL
> logs)
> At the high-level control flow is like this:
> # Identify the record type if incoming raw message.
> # Find and apply the regular expression of corresponding record type to
> extract the fields (using named groups).
> # Apply the message header regex to extract the fields in the header part of
> the message (using named groups).
>
> The parser config uses the following structure:
>
> {code:java}
> "recordTypeRegex": "(?(?<=\\s)\\b(kernel|syslog)\\b(?=\\[|:))"
> "messageHeaderRegex":
> "(?(?<=^<)\\d{1,4}(?=>)).*?(?(?<=>)[A-Za-z]{3}\\s{1,2}\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2}(?=\\s)).*?(?(?<=\\s).*?(?=\\s))",
> "fields": [
> {
> "recordType": "kernel",
> "regex": ".*(?(?<=\\]|\\w\\:).*?(?=$))"
> },
> {
> "recordType": "syslog",
> "regex":
> ".*(?(?<=PID\\s=\\s).*?(?=\\sLine)).*(?(?<=64\\s)\/([A-Za-z0-9_-]+\/)+(?=\\w))(?.*?(?=\")).*(?(?<=\").*?(?=$))"
> }
> ]
> {code}
>
> Where:
> * *recordTypeRegex* is used to distinctly identify a record type. It inputs
> a valid regular expression and may also have named groups, which would be
> extracted into fields.
> * *messageHeaderRegex* is used to specify a regular expression to extract
> fields from a message part which is common across all the messages (i.e,
> syslog fields, standard headers)
> * *fields*: json list of objects containing recordType and regex. The
> expression that is evaluated is based on the output of the recordTypeRegex
> * Note: *recordTypeRegex* and *messageHeaderRegex* could be specified as
> lists also (as a JSON array), where the list will be evaluated in order until
> a matching regular expression is found.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-2022) Metron rest creates large number of connections to ZK which causes subsequent connection to zk fail
[
https://issues.apache.org/jira/browse/METRON-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2022:
Fix Version/s: (was: Next + 1)
0.7.1
> Metron rest creates large number of connections to ZK which causes subsequent
> connection to zk fail
> ---
>
> Key: METRON-2022
> URL: https://issues.apache.org/jira/browse/METRON-2022
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> I have faced this issue while doing operations like changing status for
> metaalert
> I escalated a metaalert , then I dismissed , I tried open it again to see
> wether the status updated . I see that it failed to load the details and in
> the backend I see this error
>
> http://nat-r7-rrss-metron-1.openstacklocal:8082/api/v1/search/findOne
>
> {code:java}
> {"guid":"097c6a19-ccf0-4737-86f6-8ffd04f1e1c2","sensorType":"metaalert"}
> {
> "timestamp": "2019-02-28 13:36:17",
> "status": 500,
> "error": "Internal Server Error",
> "message": "java.util.concurrent.TimeoutException: Could not connect to
> ZooKeeper nat-r7-rrss-metron-1.openstacklocal:2181/solr within 1 ms",
> "path": "/api/v1/search/findOne"
> }
> {code}
>
> I see it's due to the number of connections to the ZK exceeded the allowed
> limit
> {code:java}
> 019-03-01 13:16:03,863 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too
> many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,158 -
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,752
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,865
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,397
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,667
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60
> {code}
> I stopped metron-rest service and I see the connections released on zk log
> {code:java}
> 2019-03-01 13:40:25,461 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:53099 which had sessionid
> 0x1693965ae4e0003 2019-03-01 13:40:25,462 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf002d, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,462 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47833 which had sessionid
> 0x169395872cf002d 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf001a, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,463 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47878 which had sessionid
> 0x169395872cf001a 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf0007, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,464 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47834 which had sessionid
> 0x169395872cf0007 2019-03-01 13:40:25,464 - WARN
>
[jira] [Updated] (METRON-2030) SensorParserGroupControllerIntegrationTest intermittent errors
[ https://issues.apache.org/jira/browse/METRON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2030: Fix Version/s: (was: Next + 1) 0.7.1 > SensorParserGroupControllerIntegrationTest intermittent errors > -- > > Key: METRON-2030 > URL: https://issues.apache.org/jira/browse/METRON-2030 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Recently I've seen some intermittent test failures in the recently added > `SensorParserGroupControllerIntegrationTest`. We likely need to add some > assertEventually statements to ensure REST state is consistent before testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2051) Improve stellar-zeppelin documentation
[ https://issues.apache.org/jira/browse/METRON-2051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2051: Fix Version/s: (was: Next + 1) 0.7.1 > Improve stellar-zeppelin documentation > -- > > Key: METRON-2051 > URL: https://issues.apache.org/jira/browse/METRON-2051 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Add some notes and a link detailing the Zeppelin 3rd party interpreter > installation mechanism. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1950) Site-book generation broken in master
[ https://issues.apache.org/jira/browse/METRON-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1950: Fix Version/s: (was: Next + 1) 0.7.1 > Site-book generation broken in master > - > > Key: METRON-1950 > URL: https://issues.apache.org/jira/browse/METRON-1950 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1951) Add site-book generation to Travis build
[ https://issues.apache.org/jira/browse/METRON-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1951: Fix Version/s: (was: Next + 1) 0.7.1 > Add site-book generation to Travis build > > > Key: METRON-1951 > URL: https://issues.apache.org/jira/browse/METRON-1951 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2041) RegularExpressionsParser in wrong source folder
[ https://issues.apache.org/jira/browse/METRON-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2041: Fix Version/s: (was: Next + 1) 0.7.1 > RegularExpressionsParser in wrong source folder > --- > > Key: METRON-2041 > URL: https://issues.apache.org/jira/browse/METRON-2041 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > The RegularExpressionsParser was moved to the test directory, but should be > in src. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1371) Enhance Mpack for Ubuntu Deployments
[ https://issues.apache.org/jira/browse/METRON-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1371: Fix Version/s: (was: Next + 1) 0.7.1 > Enhance Mpack for Ubuntu Deployments > > > Key: METRON-1371 > URL: https://issues.apache.org/jira/browse/METRON-1371 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Labels: Ubuntu > Fix For: 0.7.1 > > > The current Mpack supports installation on CentOS and RPM-based environments > only. We should enhance the Mpack so that users can deploy on Ubuntu also. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2016) Parser aggregate groups should be persisted and available through REST
[ https://issues.apache.org/jira/browse/METRON-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2016: Fix Version/s: (was: Next + 1) 0.7.1 > Parser aggregate groups should be persisted and available through REST > -- > > Key: METRON-2016 > URL: https://issues.apache.org/jira/browse/METRON-2016 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently a parser aggregate group is configured solely in Ambari as a > comma-separate list of sensors. These groups should be persisted so they can > be retrieved and modified by other applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-157) Create CEF Parser
[
https://issues.apache.org/jira/browse/METRON-157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-157:
---
Fix Version/s: (was: Next + 1)
0.7.1
> Create CEF Parser
> -
>
> Key: METRON-157
> URL: https://issues.apache.org/jira/browse/METRON-157
> Project: Metron
> Issue Type: New Feature
>Reporter: Domenic Puzio
>Assignee: Kyle Richardson
>Priority: Minor
> Labels: platform
> Fix For: 0.7.1
>
>
> Create a parser for CEF (Common Event Format). CEF is a very common
> formatting for security data sources; it is used by FireEye, Adallom, Imperva
> WAF, CyberArk, and others. The parser should be flexible enough to work for
> any of these data sources. CEF uses shorthand field names, so field names
> should be changed to human-readable and Metron-friendly equivalents. CEF
> custom labels (cs1Label, flexString1Label, etc.) should be converted
> appropriately.
> Below are sample messages and their expected parsed output.
> Adallom CEF
> 2016-04-01T09:29:11.356-0400
> CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity
> policy 'User download/view file' was triggered by 'scolb...@gmail.com'
> suser=wander...@rock.com start=1459517280810 end=1459517280810
> audits=["AVPR-4oIPeFmuZ3CKKrg","AVPR-wx80cd9PUpAu2aj","AVPR-6XGPeFmuZ3CKKvx","AVPSALn_qE4Kgs_8_yK9","AVPSASW3gw_f3aEvgEmi"]
> services=["APPID_SXC"] users=["lva...@hotmail.com"]
> cs6=https://abcd-remote.console.arc.com/#/alerts/56fe779ee4b0459f4e9a484a
> cs6Label=consoleUrl
> ...
> {"source.type":"adallom","device_version":"1.0","severity":"0","device_product":"Adallom","services":"[\"APPID_SXC\"]","src_username":"wander...@rock.com","message":"Activity
> policy 'User download\/view file' was triggered by
> 'scolb...@gmail.com'","users":"[\"lva...@hotmail.com\"]","consoleUrl":"https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a","event_class_id":"56fe779ee4b0459f4e9a484a","original_string":"2016-04-01T09:29:11.356-0400
>
> CEF:0|Adallom|Adallom|1.0|56fe779ee4b0459f4e9a484a|ALERT_CABINET_EVENT_MATCH_AUDIT|0|msg=Activity
> policy 'User download\/view file' was triggered by 'scolb...@gmail.com'
> suser=wander...@rock.com start=1459517280810 end=1459517280810
> audits=[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]
> services=[\"APPID_SXC\"] users=[\"lva...@hotmail.com\"]
> cs6=https:\/\/abcd-remote.console.arc.com\/#\/alerts\/56fe779ee4b0459f4e9a484a
> cs6Label=consoleUrl","header":"2016-04-01T09:29:11.356-0400
> CEF:0","event_name":"ALERT_CABINET_EVENT_MATCH_AUDIT","startTime":"1459517280810","device_vendor":"Adallom","endTime":"1459517280810","audits":"[\"AVPR-4oIPeFmuZ3CKKrg\",\"AVPR-wx80cd9PUpAu2aj\",\"AVPR-6XGPeFmuZ3CKKvx\",\"AVPSALn_qE4Kgs_8_yK9\",\"AVPSASW3gw_f3aEvgEmi\"]","timestamp":1459502951000}
> CyberArk CEF
> Mar 21 14:05:02 HHHPVATN1 CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve
> password|5|act=Retrieve password suser=spilgrim fname=Root\ABC phobos3 - COMP
> dvc=120.99.70.3 shost=10.44.134.78 dhost= duser= externalId= app= reason=
> cs1Label="Affected User Name" cs1= cs2Label="Safe Name" cs2=Security
> Vulnerability Mgmt cs3Label="Device Type" cs3= cs4Label="Database" cs4=
> cs5Label="Other info" cs5=101.198.70.93 cn1Label="Request Id" cn1=
> cn2Label="Ticket Id" cn2=Needed to verify config files being pulled
> msg=Needed to verify config files being pulled
> ...
> {"timestamp":1458569102000,"source.type":"cyberark","device_version":"7.20.0091","device_product":"Vault","fileName":"Root\\ABC
> phobos3 - COMP","src_username":"spilgrim","\"Other
> info\"":"101.198.70.93","\"Ticket Id\"":"Needed to verify config files being
> pulled
> ","deviceAddress":"120.99.70.3","severity":"5","deviceAction":"Retrieve
> password","message":"Needed to verify config files being
> pulled","event_class_id":"295","original_string":"Mar 21 14:05:02 HHHPVATN1
> CEF:0|Cyber-Ark|Vault|7.20.0091|295|Retrieve password|5|act=Retrieve password
> suser=spilgrim fname=Root\\ABC phobos3 - COMP dvc=120.99.70.3
> shost=10.44.134.78 dhost= duser= externalId= app= reason= cs1Label=\"Affected
> User Name\" cs1= cs2Label=\"Safe Name\" cs2=Security Vulnerability Mgmt
> cs3Label=\"Device Type\" cs3= cs4Label=\"Database\" cs4= cs5Label=\"Other
> info\" cs5=101.198.70.93 cn1Label=\"Request Id\" cn1= cn2Label=\"Ticket Id\"
> cn2=Needed to verify config files being pulled msg=Needed to verify config
> files being pulled","\"Safe Name\"":"Security Vulnerability
> Mgmt","header":"Mar 21 14:05:02 HHHPVATN1 CEF:0","event_name":"Retrieve
> password","device_vendor":"Cyber-Ark","src_hostname":"10.44.134.78"}
> WAF CEF
> <14>CEF:0|Imperva
[jira] [Updated] (METRON-2065) Setting Parser Output Topic in Sensor Config is broken
[
https://issues.apache.org/jira/browse/METRON-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2065:
Fix Version/s: (was: Next + 1)
0.7.1
> Setting Parser Output Topic in Sensor Config is broken
> --
>
> Key: METRON-2065
> URL: https://issues.apache.org/jira/browse/METRON-2065
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Attachments: Screen Shot 2019-04-05 at 7.45.36 PM.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Login to management console
> Edit the parser config Advanced > Raw JSON !Screen Shot 2019-04-05 at
> 7.45.36 PM.png!
> Change the output topic for the 'snort' sensor.
> Verify that the changes taken effect using stellar shell
> {code:java}
> [Stellar]>>> conf := CONFIG_GET("PARSER","snort") { "parserClassName" :
> "org.apache.metron.parsers.snort.BasicSnortParser", "sensorTopic" : "snort",
> "outputTopic" : "new-topic", "readMetadata" : false, "mergeMetadata" : false,
> "spoutParallelism" : 1, "spoutNumTasks" : 1, "parserParallelism" : 1,
> "parserNumTasks" : 1, "errorWriterParallelism" : 1, "errorWriterNumTasks" :
> 1, "spoutConfig" : { }, "stormConfig" : { }, "parserConfig" : { },
> "fieldTransformations" : [ ], "cacheConfig" : { }, "rawMessageStrategy" :
> "DEFAULT", "rawMessageStrategyConfig" : { } }
> {code}
> publish the message to 'snort' topic
> I use the console consumer to validate output is being piped into "new_topic"
> and verified that no messages were sent to the topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic new-topic
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper]. [2019-04-05 14:08:08,796] WARN
> SASL configuration failed: javax.security.auth.login.LoginException: No JAAS
> configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn) [2019-04-05 14:08:09,005]
> WARN SASL configuration failed: javax.security.auth.login.LoginException: No
> JAAS configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {code}
> where as I see that the messages were sent to "enrichments" topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic enrichments
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper].
> [2019-04-05 14:10:18,930] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> [2019-04-05 14:10:19,095] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {"msg":"snort test
> alert","sig_rev":"0","ip_dst_port":"80","ethsrc":"00:00:00:00:00:00","tcpseq":"0xF017C4DA","dgmlen":"40","icmpid":"","tcplen":"","tcpwindow":"0xF6C9","icmpseq":"","tcpack":"0xABDB8426","protocol":"TCP","ip_dst_addr":"62.75.195.236","original_string":"09\/09\/16-09:09:09.844676
> ,1,999158,0,\"snort test
>
[jira] [Updated] (METRON-2066) Documentation and logging corrections
[ https://issues.apache.org/jira/browse/METRON-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2066: Fix Version/s: (was: Next + 1) 0.7.1 > Documentation and logging corrections > - > > Key: METRON-2066 > URL: https://issues.apache.org/jira/browse/METRON-2066 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 20m > Remaining Estimate: 0h > > Grab-bag of issues and/or omissions I found with documentation while working > through a parser use case with REGEX_SELECT. Also need to tweak the logging > for GrokParser so it no longer outputs nothing for the grok expressions > during init. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2064) Metron REST API overwriting global.json values
[ https://issues.apache.org/jira/browse/METRON-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2064: Fix Version/s: (was: Next + 1) 0.7.1 > Metron REST API overwriting global.json values > -- > > Key: METRON-2064 > URL: https://issues.apache.org/jira/browse/METRON-2064 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We added the capability for Ambari to modify global config properties without > overwriting non-conflicting externally-provided properties in this PR - > [https://github.com/apache/metron/pull/760]. The REST API is currently > overwriting the global config when it's restarted, thus losing user changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2060) Improving Alerts table config pane
[ https://issues.apache.org/jira/browse/METRON-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2060: Fix Version/s: (was: Next + 1) 0.7.1 > Improving Alerts table config pane > -- > > Key: METRON-2060 > URL: https://issues.apache.org/jira/browse/METRON-2060 > Project: Metron > Issue Type: Improvement >Reporter: Tibor Meller >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2067) Maven pom file duplicate dependency fixes
[
https://issues.apache.org/jira/browse/METRON-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2067:
Fix Version/s: (was: Next + 1)
0.7.1
> Maven pom file duplicate dependency fixes
> -
>
> Key: METRON-2067
> URL: https://issues.apache.org/jira/browse/METRON-2067
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Fix a couple duplicate dependencies in pom files. Address error/warning
> output like the following:
> {code:java}
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-profiler-storm:jar:0.7.1
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.apache.metron:metron-integration-test:jar -> duplicate
> declaration of version ${project.parent.version} @
> org.apache.metron:metron-profiler-storm:[unknown-version],
> metron/metron-analytics/metron-profiler-storm/pom.xml, line 314, column 21
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.metron:metron-writer:jar:0.7.1
> [WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must
> be unique: org.apache.httpcomponents:httpclient:jar -> duplicate declaration
> of version ${global_httpclient_version} @
> org.apache.metron:metron-writer:[unknown-version],
> metron/metron-platform/metron-writer/pom.xml, line 248, column 21
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1998) Only one sensor is flushed by tick tuple
[ https://issues.apache.org/jira/browse/METRON-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1998: Fix Version/s: (was: Next + 1) 0.7.1 > Only one sensor is flushed by tick tuple > > > Key: METRON-1998 > URL: https://issues.apache.org/jira/browse/METRON-1998 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 3.5h > Remaining Estimate: 0h > > The BulkWriterComponent maintains an internal map of sensors to message > batches. The flushTimeouts method incorrectly returns after one of the > sensor's batch timeout is reached and flushed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2078) Remove Storm dependency from metron-writer
[ https://issues.apache.org/jira/browse/METRON-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2078: Fix Version/s: (was: Next + 1) 0.7.1 > Remove Storm dependency from metron-writer > -- > > Key: METRON-2078 > URL: https://issues.apache.org/jira/browse/METRON-2078 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-writer and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1654) findOne request after an alert patch returns with the original state of the alert item
[ https://issues.apache.org/jira/browse/METRON-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1654: Fix Version/s: (was: Next + 1) 0.7.1 > findOne request after an alert patch returns with the original state of the > alert item > -- > > Key: METRON-1654 > URL: https://issues.apache.org/jira/browse/METRON-1654 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > /findOne request after an alert patch returns with the original state of the > alert item. > The UI stays inconsistent until the next search request. > This leads to the following bugs: > - Alert status change on details panel haven't reflected on the list view. > - Adding an element to a meta-alert doesn't change the items counter beside > the name of the meta-alert > - Removing an element from a meta-alert doesn't change the items counter > beside the name of the meta-alert > - Clicking on remove/unlink button in a meta-alert group doesn't remove item > immediately > This behavior only validated on the full dev single node VM and might not > exist in production. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2014) Add architectural documentation for metron-writer
[ https://issues.apache.org/jira/browse/METRON-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2014: Fix Version/s: (was: Next + 1) 0.7.1 > Add architectural documentation for metron-writer > - > > Key: METRON-2014 > URL: https://issues.apache.org/jira/browse/METRON-2014 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 50m > Remaining Estimate: 0h > > This task is based on the discussion here: > [https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E] > Follow on architectural documentation is needed for the writer module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1968) Messages are lost when a parser produces multiple messages and batch size is greater than 1
[ https://issues.apache.org/jira/browse/METRON-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1968: Fix Version/s: (was: Next + 1) 0.7.1 > Messages are lost when a parser produces multiple messages and batch size is > greater than 1 > --- > > Key: METRON-1968 > URL: https://issues.apache.org/jira/browse/METRON-1968 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 15h 20m > Remaining Estimate: 0h > > A bug was discovered where messages are lost when a parser produces multiple > messages. This happens anytime the batch size for that sensor is set to > greater than 1. For example, consider a parser that produces 30 messages > from a single input message. Assume the batch size for this sensor/parser is > set to 10. The batch is currently flushed only after 10 tuples are received > and only 10 messages are written out. I think the correct behavior would be > for 3 batches of 10 messages to be flushed for every tuple and a total of 300 > messages written for every 10 tuples. > This is happening because the various writer classes/interfaces > (BulkWriterComponent, BulkMessageWriter, KafkaWriter, etc) assume a 1 to 1 > relationship between messages and tuples. The root cause of this specific > issue is > [here|https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2036) Maven builds fail locally in HDFSWriterTest
[
https://issues.apache.org/jira/browse/METRON-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2036:
Fix Version/s: (was: Next + 1)
0.7.1
> Maven builds fail locally in HDFSWriterTest
> ---
>
> Key: METRON-2036
> URL: https://issues.apache.org/jira/browse/METRON-2036
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Fix For: 0.7.1
>
> Attachments: METRON-2036-ide-fail.txt
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> I'm getting the following when running mvn clean install from master locally.
> {code:java}
> testSingleFileIfNoStreamClosed(org.apache.metron.writer.hdfs.HdfsWriterTest)
> Time elapsed: 0.087 sec <<< ERROR!
> java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:164)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
> at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
> at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
> at java.io.InputStreamReader.read(InputStreamReader.java:184)
> at java.io.BufferedReader.fill(BufferedReader.java:161)
> at java.io.BufferedReader.readLine(BufferedReader.java:324)
> at java.io.BufferedReader.readLine(BufferedReader.java:389)
> at java.nio.file.Files.readAllLines(Files.java:3205)
> at java.nio.file.Files.readAllLines(Files.java:3242)
> at
> org.apache.metron.writer.hdfs.HdfsWriterTest.testSingleFileIfNoStreamClosed(HdfsWriterTest.java:444)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> ...
> Results :
> Tests in error:
> HdfsWriterTest.testSingleFileIfNoStreamClosed:444 » ClosedByInterrupt
> HdfsWriterTest.testWriteMultipleFiles:356 » ClosedByInterrupt
> HdfsWriterTest.testWriteNoOutputFunction:271 » ClosedByInterrupt
> HdfsWriterTest.testWriteSingleFile:314 » ClosedByInterrupt
> HdfsWriterTest.testWriteSingleFileWithNull:407 » ClosedByInterrupt
> Tests run: 55, Failures: 0, Errors: 5, Skipped: 0
> [INFO]
>
> [INFO] Reactor Summary:
> [INFO]
> [INFO] Metron . SUCCESS [ 8.093 s]
> [INFO] metron-stellar . SUCCESS [ 0.241 s]
> [INFO]
[jira] [Updated] (METRON-2026) Remove Storm dependency from metron-common
[ https://issues.apache.org/jira/browse/METRON-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2026: Fix Version/s: (was: Next + 1) 0.7.1 > Remove Storm dependency from metron-common > -- > > Key: METRON-2026 > URL: https://issues.apache.org/jira/browse/METRON-2026 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-common and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1469) Kafka Plugin for Bro - Configurable JSON Timestamps
[ https://issues.apache.org/jira/browse/METRON-1469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1469: Fix Version/s: (was: Next + 1) 0.7.1 > Kafka Plugin for Bro - Configurable JSON Timestamps > --- > > Key: METRON-1469 > URL: https://issues.apache.org/jira/browse/METRON-1469 > Project: Metron > Issue Type: Improvement >Affects Versions: 0.1 >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: 0.7.1 > > > The Kafka Plugin for Bro allows the user to output JSON to a Kafka topic. > Allow the user to define the format of timestamps that get embedded in the > JSON. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1878) Add Metron as a Knox service
[ https://issues.apache.org/jira/browse/METRON-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1878: Fix Version/s: (was: Next + 1) 0.7.1 > Add Metron as a Knox service > > > Key: METRON-1878 > URL: https://issues.apache.org/jira/browse/METRON-1878 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > This feature would add Apache Knox as an option for access to Metron REST and > the Metron UIs. LDAP authentication would be required for Metron REST in > this scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2054) Update Metron wiki Streaming Enrichment instructions
[ https://issues.apache.org/jira/browse/METRON-2054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2054: Fix Version/s: (was: Next + 1) 0.7.1 > Update Metron wiki Streaming Enrichment instructions > > > Key: METRON-2054 > URL: https://issues.apache.org/jira/browse/METRON-2054 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > > Classpath for the SimpleHbaseEnrichmentWriter will change on this wiki page - > [https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1945) Metron MPack support for Knox SSO setup
[ https://issues.apache.org/jira/browse/METRON-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1945: Fix Version/s: (was: Next + 1) 0.7.1 > Metron MPack support for Knox SSO setup > --- > > Key: METRON-1945 > URL: https://issues.apache.org/jira/browse/METRON-1945 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Knox SSO support for Metron is being added in > [https://github.com/apache/metron/pull/1275.] This PR depends on that work > and enables Ambari to automate most of the setup work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2039) Time range queries do not work with Solr
[ https://issues.apache.org/jira/browse/METRON-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2039: Fix Version/s: (was: Next + 1) 0.7.1 > Time range queries do not work with Solr > > > Key: METRON-2039 > URL: https://issues.apache.org/jira/browse/METRON-2039 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > A bug was discovered recently where a query generated from the time widget in > the alerts UI only works on Elasticsearch. The syntax should be updated to > work with both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1965) Knox should work on a multi-node installation
[ https://issues.apache.org/jira/browse/METRON-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1965: Fix Version/s: (was: Next + 1) 0.7.1 > Knox should work on a multi-node installation > - > > Key: METRON-1965 > URL: https://issues.apache.org/jira/browse/METRON-1965 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We discovered a few issues when deploying Metron in a multi-node environment > and enabling Knox: > * Metron scripts and resources need to be installed by a Metron Client > component > * The flag used to determine if Metron Knox assets are installed does not > work correctly if Knox is installed after a Metron Client > * The Knox host needs to be added to the SSO whitelist -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2074) Script to handle TGT renewal with Storm and Kerberos enabled
[ https://issues.apache.org/jira/browse/METRON-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2074: Fix Version/s: (was: Next + 1) 0.7.1 > Script to handle TGT renewal with Storm and Kerberos enabled > > > Key: METRON-2074 > URL: https://issues.apache.org/jira/browse/METRON-2074 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: 0.7.1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Storm doesn't automatically manage TGT renewal. This means it's up to the > topology deployer to handle providing a renewed TGT to their topologies. We > should provide a script that users can schedule with cron or similar > scheduler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-589) Dist area should only contain latest release
[ https://issues.apache.org/jira/browse/METRON-589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-589: --- Fix Version/s: (was: Next + 1) 0.7.1 > Dist area should only contain latest release > > > Key: METRON-589 > URL: https://issues.apache.org/jira/browse/METRON-589 > Project: Metron > Issue Type: Bug >Reporter: John D. Ament >Assignee: Casey Stella >Priority: Major > Fix For: 0.7.1 > > > The dist area https://dist.apache.org/repos/dist/release/incubator/metron/ > should only contain the latest release. Older releases are found in the > archives. > Most projects don't directly link to this area, but instead provide the dyn > style links. See Geode for example: > http://geode.incubator.apache.org/releases/ - this works for older releases > as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2007) Management UI not loading grok statements correctly
[
https://issues.apache.org/jira/browse/METRON-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Leet updated METRON-2007:
Fix Version/s: (was: Next + 1)
0.7.1
> Management UI not loading grok statements correctly
> ---
>
> Key: METRON-2007
> URL: https://issues.apache.org/jira/browse/METRON-2007
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: 0.7.1
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When I open a Grok sensor in the management UI (like squid) I get an error:
> {code:java}
> Could not find grok statement in HDFS or classpath at
> /apps/metron/patterns/squid{code}
> I verified that file exists in HDFS. It also happens when I open a sensor
> that gets a grok statement from the classpath (yaf for example):
> {code:java}
> Could not find grok statement in HDFS or classpath at /patterns/yaf{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1993) Stellar REST_GET should handle responses when content length is less than zero
[ https://issues.apache.org/jira/browse/METRON-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1993: Fix Version/s: (was: Next + 1) 0.7.1 > Stellar REST_GET should handle responses when content length is less than zero > -- > > Key: METRON-1993 > URL: https://issues.apache.org/jira/browse/METRON-1993 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > There are cases where a HTTP response content length header is missing but > content exists. The Stellar REST_GET function should still parse the content > in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2009) Address Javadoc checkstyle issues in metron-common
[ https://issues.apache.org/jira/browse/METRON-2009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2009: Fix Version/s: 0.7.1 > Address Javadoc checkstyle issues in metron-common > -- > > Key: METRON-2009 > URL: https://issues.apache.org/jira/browse/METRON-2009 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Minor > Fix For: 0.7.1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > We have a lot of missing Javadoc or broken style in various modules, > including metron-common. We should fix them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2006) Reenable JacoCo code coverage
[ https://issues.apache.org/jira/browse/METRON-2006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-2006: Fix Version/s: 0.7.1 > Reenable JacoCo code coverage > - > > Key: METRON-2006 > URL: https://issues.apache.org/jira/browse/METRON-2006 > Project: Metron > Issue Type: Bug >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > It got disabled here: > https://github.com/apache/metron/pull/840/commits/54c2486e4e618a6d445667ed2040faacf165ca8c > We should get it reenabled and working with the build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1939) Update version to 0.7.1
[ https://issues.apache.org/jira/browse/METRON-1939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1939: Fix Version/s: 0.7.1 > Update version to 0.7.1 > --- > > Key: METRON-1939 > URL: https://issues.apache.org/jira/browse/METRON-1939 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Need to update the version for ongoing development post 0.7.0 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1937) Update public web site to point at 0.7.0 new release
[ https://issues.apache.org/jira/browse/METRON-1937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1937: Fix Version/s: 0.7.1 > Update public web site to point at 0.7.0 new release > > > Key: METRON-1937 > URL: https://issues.apache.org/jira/browse/METRON-1937 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 40m > Remaining Estimate: 0h > > Need to update site to reflect the release. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1929) Build GET_ASN Stellar function
[ https://issues.apache.org/jira/browse/METRON-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1929: Fix Version/s: 0.7.1 > Build GET_ASN Stellar function > -- > > Key: METRON-1929 > URL: https://issues.apache.org/jira/browse/METRON-1929 > Project: Metron > Issue Type: Task >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 7.5h > Remaining Estimate: 0h > > MaxMind has a database for ASN information, very similar to the Geo databases > we use. We should be able to have a GET_ASN function similar to that, and > reuse (and refactor) the existing infrastructure from GEO_GET to support it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1815) Separate metron-parsers into metron-parsers-common and metron-parsers-storm
[ https://issues.apache.org/jira/browse/METRON-1815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Justin Leet updated METRON-1815: Fix Version/s: 0.7.1 > Separate metron-parsers into metron-parsers-common and metron-parsers-storm > --- > > Key: METRON-1815 > URL: https://issues.apache.org/jira/browse/METRON-1815 > Project: Metron > Issue Type: Improvement >Reporter: Justin Leet >Assignee: Justin Leet >Priority: Major > Fix For: 0.7.1 > > Time Spent: 2h > Remaining Estimate: 0h > > In order to expose our parsers to 3rd party components (e.g. the discussions > on NiFi and potentially other platforms like Spark), we should > separate the storm-bits into its own project. The metron-parsers-common > project should contain only parser-oriented code, whereas the > metron-parsers-storm project should contain the storm specific code > (e.g. the parser bolt). > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1965) Knox should work on a multi-node installation
[ https://issues.apache.org/jira/browse/METRON-1965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1965: -- Fix Version/s: Next + 1 > Knox should work on a multi-node installation > - > > Key: METRON-1965 > URL: https://issues.apache.org/jira/browse/METRON-1965 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > We discovered a few issues when deploying Metron in a multi-node environment > and enabling Knox: > * Metron scripts and resources need to be installed by a Metron Client > component > * The flag used to determine if Metron Knox assets are installed does not > work correctly if Knox is installed after a Metron Client > * The Knox host needs to be added to the SSO whitelist -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1945) Metron MPack support for Knox SSO setup
[ https://issues.apache.org/jira/browse/METRON-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1945: -- Fix Version/s: Next + 1 > Metron MPack support for Knox SSO setup > --- > > Key: METRON-1945 > URL: https://issues.apache.org/jira/browse/METRON-1945 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > Knox SSO support for Metron is being added in > [https://github.com/apache/metron/pull/1275.] This PR depends on that work > and enables Ambari to automate most of the setup work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1945) Metron MPack support for Knox SSO setup
[ https://issues.apache.org/jira/browse/METRON-1945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1945: - Assignee: Ryan Merriman > Metron MPack support for Knox SSO setup > --- > > Key: METRON-1945 > URL: https://issues.apache.org/jira/browse/METRON-1945 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > Knox SSO support for Metron is being added in > [https://github.com/apache/metron/pull/1275.] This PR depends on that work > and enables Ambari to automate most of the setup work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1878) Add Metron as a Knox service
[ https://issues.apache.org/jira/browse/METRON-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1878: -- Fix Version/s: Next + 1 > Add Metron as a Knox service > > > Key: METRON-1878 > URL: https://issues.apache.org/jira/browse/METRON-1878 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 6h 10m > Remaining Estimate: 0h > > This feature would add Apache Knox as an option for access to Metron REST and > the Metron UIs. LDAP authentication would be required for Metron REST in > this scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1878) Add Metron as a Knox service
[ https://issues.apache.org/jira/browse/METRON-1878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1878: - Assignee: Ryan Merriman > Add Metron as a Knox service > > > Key: METRON-1878 > URL: https://issues.apache.org/jira/browse/METRON-1878 > Project: Metron > Issue Type: New Feature >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > This feature would add Apache Knox as an option for access to Metron REST and > the Metron UIs. LDAP authentication would be required for Metron REST in > this scenario. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2007) Management UI not loading grok statements correctly
[
https://issues.apache.org/jira/browse/METRON-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Merriman updated METRON-2007:
--
Fix Version/s: Next + 1
> Management UI not loading grok statements correctly
> ---
>
> Key: METRON-2007
> URL: https://issues.apache.org/jira/browse/METRON-2007
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When I open a Grok sensor in the management UI (like squid) I get an error:
> {code:java}
> Could not find grok statement in HDFS or classpath at
> /apps/metron/patterns/squid{code}
> I verified that file exists in HDFS. It also happens when I open a sensor
> that gets a grok statement from the classpath (yaf for example):
> {code:java}
> Could not find grok statement in HDFS or classpath at /patterns/yaf{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (METRON-1993) Stellar REST_GET should handle responses when content length is less than zero
[ https://issues.apache.org/jira/browse/METRON-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1993: - Assignee: Ryan Merriman > Stellar REST_GET should handle responses when content length is less than zero > -- > > Key: METRON-1993 > URL: https://issues.apache.org/jira/browse/METRON-1993 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 2h 20m > Remaining Estimate: 0h > > There are cases where a HTTP response content length header is missing but > content exists. The Stellar REST_GET function should still parse the content > in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2007) Management UI not loading grok statements correctly
[
https://issues.apache.org/jira/browse/METRON-2007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Merriman reassigned METRON-2007:
-
Assignee: Ryan Merriman
> Management UI not loading grok statements correctly
> ---
>
> Key: METRON-2007
> URL: https://issues.apache.org/jira/browse/METRON-2007
> Project: Metron
> Issue Type: Bug
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When I open a Grok sensor in the management UI (like squid) I get an error:
> {code:java}
> Could not find grok statement in HDFS or classpath at
> /apps/metron/patterns/squid{code}
> I verified that file exists in HDFS. It also happens when I open a sensor
> that gets a grok statement from the classpath (yaf for example):
> {code:java}
> Could not find grok statement in HDFS or classpath at /patterns/yaf{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1993) Stellar REST_GET should handle responses when content length is less than zero
[ https://issues.apache.org/jira/browse/METRON-1993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1993: -- Fix Version/s: Next + 1 > Stellar REST_GET should handle responses when content length is less than zero > -- > > Key: METRON-1993 > URL: https://issues.apache.org/jira/browse/METRON-1993 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > There are cases where a HTTP response content length header is missing but > content exists. The Stellar REST_GET function should still parse the content > in this case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1998) Only one sensor is flushed by tick tuple
[ https://issues.apache.org/jira/browse/METRON-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1998: -- Fix Version/s: Next + 1 > Only one sensor is flushed by tick tuple > > > Key: METRON-1998 > URL: https://issues.apache.org/jira/browse/METRON-1998 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 3.5h > Remaining Estimate: 0h > > The BulkWriterComponent maintains an internal map of sensors to message > batches. The flushTimeouts method incorrectly returns after one of the > sensor's batch timeout is reached and flushed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1998) Only one sensor is flushed by tick tuple
[ https://issues.apache.org/jira/browse/METRON-1998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-1998: - Assignee: Ryan Merriman > Only one sensor is flushed by tick tuple > > > Key: METRON-1998 > URL: https://issues.apache.org/jira/browse/METRON-1998 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 3.5h > Remaining Estimate: 0h > > The BulkWriterComponent maintains an internal map of sensors to message > batches. The flushTimeouts method incorrectly returns after one of the > sensor's batch timeout is reached and flushed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1968) Messages are lost when a parser produces multiple messages and batch size is greater than 1
[ https://issues.apache.org/jira/browse/METRON-1968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1968: -- Fix Version/s: Next + 1 > Messages are lost when a parser produces multiple messages and batch size is > greater than 1 > --- > > Key: METRON-1968 > URL: https://issues.apache.org/jira/browse/METRON-1968 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 15h 20m > Remaining Estimate: 0h > > A bug was discovered where messages are lost when a parser produces multiple > messages. This happens anytime the batch size for that sensor is set to > greater than 1. For example, consider a parser that produces 30 messages > from a single input message. Assume the batch size for this sensor/parser is > set to 10. The batch is currently flushed only after 10 tuples are received > and only 10 messages are written out. I think the correct behavior would be > for 3 batches of 10 messages to be flushed for every tuple and a total of 300 > messages written for every 10 tuples. > This is happening because the various writer classes/interfaces > (BulkWriterComponent, BulkMessageWriter, KafkaWriter, etc) assume a 1 to 1 > relationship between messages and tuples. The root cause of this specific > issue is > [here|https://github.com/apache/metron/blob/master/metron-platform/metron-writer/src/main/java/org/apache/metron/writer/kafka/KafkaWriter.java#L236]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2016) Parser aggregate groups should be persisted and available through REST
[ https://issues.apache.org/jira/browse/METRON-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-2016: - Assignee: Ryan Merriman > Parser aggregate groups should be persisted and available through REST > -- > > Key: METRON-2016 > URL: https://issues.apache.org/jira/browse/METRON-2016 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently a parser aggregate group is configured solely in Ambari as a > comma-separate list of sensors. These groups should be persisted so they can > be retrieved and modified by other applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2016) Parser aggregate groups should be persisted and available through REST
[ https://issues.apache.org/jira/browse/METRON-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2016: -- Fix Version/s: Next + 1 > Parser aggregate groups should be persisted and available through REST > -- > > Key: METRON-2016 > URL: https://issues.apache.org/jira/browse/METRON-2016 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently a parser aggregate group is configured solely in Ambari as a > comma-separate list of sensors. These groups should be persisted so they can > be retrieved and modified by other applications. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2019) Improve Metron REST Logging
[ https://issues.apache.org/jira/browse/METRON-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman reassigned METRON-2019: - Assignee: Ryan Merriman > Improve Metron REST Logging > --- > > Key: METRON-2019 > URL: https://issues.apache.org/jira/browse/METRON-2019 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > When a bad request is submitted to the REST API, often the only indication of > an error is the response code that you get back. This makes it difficult to > troubleshoot problems and add features. This can make life for the UI team > and others more difficult than it needs to be. > I think the following enhancements would be very helpful. > 1. Every request received by the REST API along with the headers and body > should be logged. > 2. Every response returned by the REST API should be logged. > 3. Instructions should be provided to turn on debug logging when the REST API > is deployed. For example, how do I turn on debug logging in Full Dev? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2019) Improve Metron REST Logging
[ https://issues.apache.org/jira/browse/METRON-2019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2019: -- Fix Version/s: Next + 1 > Improve Metron REST Logging > --- > > Key: METRON-2019 > URL: https://issues.apache.org/jira/browse/METRON-2019 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > When a bad request is submitted to the REST API, often the only indication of > an error is the response code that you get back. This makes it difficult to > troubleshoot problems and add features. This can make life for the UI team > and others more difficult than it needs to be. > I think the following enhancements would be very helpful. > 1. Every request received by the REST API along with the headers and body > should be logged. > 2. Every response returned by the REST API should be logged. > 3. Instructions should be provided to turn on debug logging when the REST API > is deployed. For example, how do I turn on debug logging in Full Dev? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1654) findOne request after an alert patch returns with the original state of the alert item
[ https://issues.apache.org/jira/browse/METRON-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-1654: -- Fix Version/s: Next + 1 > findOne request after an alert patch returns with the original state of the > alert item > -- > > Key: METRON-1654 > URL: https://issues.apache.org/jira/browse/METRON-1654 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > /findOne request after an alert patch returns with the original state of the > alert item. > The UI stays inconsistent until the next search request. > This leads to the following bugs: > - Alert status change on details panel haven't reflected on the list view. > - Adding an element to a meta-alert doesn't change the items counter beside > the name of the meta-alert > - Removing an element from a meta-alert doesn't change the items counter > beside the name of the meta-alert > - Clicking on remove/unlink button in a meta-alert group doesn't remove item > immediately > This behavior only validated on the full dev single node VM and might not > exist in production. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2078) Remove Storm dependency from metron-writer
[ https://issues.apache.org/jira/browse/METRON-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2078: -- Fix Version/s: Next + 1 > Remove Storm dependency from metron-writer > -- > > Key: METRON-2078 > URL: https://issues.apache.org/jira/browse/METRON-2078 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-writer and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2026) Remove Storm dependency from metron-common
[ https://issues.apache.org/jira/browse/METRON-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2026: -- Fix Version/s: Next + 1 > Remove Storm dependency from metron-common > -- > > Key: METRON-2026 > URL: https://issues.apache.org/jira/browse/METRON-2026 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-common and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2030) SensorParserGroupControllerIntegrationTest intermittent errors
[ https://issues.apache.org/jira/browse/METRON-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2030: -- Fix Version/s: Next + 1 > SensorParserGroupControllerIntegrationTest intermittent errors > -- > > Key: METRON-2030 > URL: https://issues.apache.org/jira/browse/METRON-2030 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Recently I've seen some intermittent test failures in the recently added > `SensorParserGroupControllerIntegrationTest`. We likely need to add some > assertEventually statements to ensure REST state is consistent before testing. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2039) Time range queries do not work with Solr
[ https://issues.apache.org/jira/browse/METRON-2039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2039: -- Fix Version/s: Next + 1 > Time range queries do not work with Solr > > > Key: METRON-2039 > URL: https://issues.apache.org/jira/browse/METRON-2039 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > A bug was discovered recently where a query generated from the time widget in > the alerts UI only works on Elasticsearch. The syntax should be updated to > work with both. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2022) Metron rest creates large number of connections to ZK which causes subsequent connection to zk fail
[
https://issues.apache.org/jira/browse/METRON-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Merriman updated METRON-2022:
--
Fix Version/s: Next + 1
> Metron rest creates large number of connections to ZK which causes subsequent
> connection to zk fail
> ---
>
> Key: METRON-2022
> URL: https://issues.apache.org/jira/browse/METRON-2022
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> I have faced this issue while doing operations like changing status for
> metaalert
> I escalated a metaalert , then I dismissed , I tried open it again to see
> wether the status updated . I see that it failed to load the details and in
> the backend I see this error
>
> http://nat-r7-rrss-metron-1.openstacklocal:8082/api/v1/search/findOne
>
> {code:java}
> {"guid":"097c6a19-ccf0-4737-86f6-8ffd04f1e1c2","sensorType":"metaalert"}
> {
> "timestamp": "2019-02-28 13:36:17",
> "status": 500,
> "error": "Internal Server Error",
> "message": "java.util.concurrent.TimeoutException: Could not connect to
> ZooKeeper nat-r7-rrss-metron-1.openstacklocal:2181/solr within 1 ms",
> "path": "/api/v1/search/findOne"
> }
> {code}
>
> I see it's due to the number of connections to the ZK exceeded the allowed
> limit
> {code:java}
> 019-03-01 13:16:03,863 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too
> many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,158 -
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,752
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,865
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,397
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,667
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60
> {code}
> I stopped metron-rest service and I see the connections released on zk log
> {code:java}
> 2019-03-01 13:40:25,461 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:53099 which had sessionid
> 0x1693965ae4e0003 2019-03-01 13:40:25,462 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf002d, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,462 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47833 which had sessionid
> 0x169395872cf002d 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf001a, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,463 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47878 which had sessionid
> 0x169395872cf001a 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf0007, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,464 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47834 which had sessionid
> 0x169395872cf0007 2019-03-01 13:40:25,464 - WARN
>
[jira] [Assigned] (METRON-2022) Metron rest creates large number of connections to ZK which causes subsequent connection to zk fail
[
https://issues.apache.org/jira/browse/METRON-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Merriman reassigned METRON-2022:
-
Assignee: Ryan Merriman
> Metron rest creates large number of connections to ZK which causes subsequent
> connection to zk fail
> ---
>
> Key: METRON-2022
> URL: https://issues.apache.org/jira/browse/METRON-2022
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> I have faced this issue while doing operations like changing status for
> metaalert
> I escalated a metaalert , then I dismissed , I tried open it again to see
> wether the status updated . I see that it failed to load the details and in
> the backend I see this error
>
> http://nat-r7-rrss-metron-1.openstacklocal:8082/api/v1/search/findOne
>
> {code:java}
> {"guid":"097c6a19-ccf0-4737-86f6-8ffd04f1e1c2","sensorType":"metaalert"}
> {
> "timestamp": "2019-02-28 13:36:17",
> "status": 500,
> "error": "Internal Server Error",
> "message": "java.util.concurrent.TimeoutException: Could not connect to
> ZooKeeper nat-r7-rrss-metron-1.openstacklocal:2181/solr within 1 ms",
> "path": "/api/v1/search/findOne"
> }
> {code}
>
> I see it's due to the number of connections to the ZK exceeded the allowed
> limit
> {code:java}
> 019-03-01 13:16:03,863 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] - Too
> many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,158 -
> WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,752
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:04,865
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,397
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60 2019-03-01 13:16:05,667
> - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxnFactory@193] -
> Too many connections from /172.22.107.254 - max is 60
> {code}
> I stopped metron-rest service and I see the connections released on zk log
> {code:java}
> 2019-03-01 13:40:25,461 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:53099 which had sessionid
> 0x1693965ae4e0003 2019-03-01 13:40:25,462 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf002d, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,462 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47833 which had sessionid
> 0x169395872cf002d 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf001a, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,463 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47878 which had sessionid
> 0x169395872cf001a 2019-03-01 13:40:25,463 - WARN
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@357] - caught end of
> stream exception EndOfStreamException: Unable to read additional data from
> client sessionid 0x169395872cf0007, likely client has closed socket at
> org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:230) at
> org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
> at java.lang.Thread.run(Thread.java:748) 2019-03-01 13:40:25,464 - INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1033] - Closed
> socket connection for client /172.22.107.254:47834 which had sessionid
> 0x169395872cf0007 2019-03-01 13:40:25,464 - WARN
>
[jira] [Updated] (METRON-2064) Metron REST API overwriting global.json values
[ https://issues.apache.org/jira/browse/METRON-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Merriman updated METRON-2064: -- Fix Version/s: Next + 1 > Metron REST API overwriting global.json values > -- > > Key: METRON-2064 > URL: https://issues.apache.org/jira/browse/METRON-2064 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > We added the capability for Ambari to modify global config properties without > overwriting non-conflicting externally-provided properties in this PR - > [https://github.com/apache/metron/pull/760]. The REST API is currently > overwriting the global config when it's restarted, thus losing user changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-2065) Setting Parser Output Topic in Sensor Config is broken
[
https://issues.apache.org/jira/browse/METRON-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Merriman updated METRON-2065:
--
Fix Version/s: Next + 1
> Setting Parser Output Topic in Sensor Config is broken
> --
>
> Key: METRON-2065
> URL: https://issues.apache.org/jira/browse/METRON-2065
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: Next + 1
>
> Attachments: Screen Shot 2019-04-05 at 7.45.36 PM.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Login to management console
> Edit the parser config Advanced > Raw JSON !Screen Shot 2019-04-05 at
> 7.45.36 PM.png!
> Change the output topic for the 'snort' sensor.
> Verify that the changes taken effect using stellar shell
> {code:java}
> [Stellar]>>> conf := CONFIG_GET("PARSER","snort") { "parserClassName" :
> "org.apache.metron.parsers.snort.BasicSnortParser", "sensorTopic" : "snort",
> "outputTopic" : "new-topic", "readMetadata" : false, "mergeMetadata" : false,
> "spoutParallelism" : 1, "spoutNumTasks" : 1, "parserParallelism" : 1,
> "parserNumTasks" : 1, "errorWriterParallelism" : 1, "errorWriterNumTasks" :
> 1, "spoutConfig" : { }, "stormConfig" : { }, "parserConfig" : { },
> "fieldTransformations" : [ ], "cacheConfig" : { }, "rawMessageStrategy" :
> "DEFAULT", "rawMessageStrategyConfig" : { } }
> {code}
> publish the message to 'snort' topic
> I use the console consumer to validate output is being piped into "new_topic"
> and verified that no messages were sent to the topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic new-topic
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper]. [2019-04-05 14:08:08,796] WARN
> SASL configuration failed: javax.security.auth.login.LoginException: No JAAS
> configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn) [2019-04-05 14:08:09,005]
> WARN SASL configuration failed: javax.security.auth.login.LoginException: No
> JAAS configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {code}
> where as I see that the messages were sent to "enrichments" topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic enrichments
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper].
> [2019-04-05 14:10:18,930] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> [2019-04-05 14:10:19,095] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {"msg":"snort test
> alert","sig_rev":"0","ip_dst_port":"80","ethsrc":"00:00:00:00:00:00","tcpseq":"0xF017C4DA","dgmlen":"40","icmpid":"","tcplen":"","tcpwindow":"0xF6C9","icmpseq":"","tcpack":"0xABDB8426","protocol":"TCP","ip_dst_addr":"62.75.195.236","original_string":"09\/09\/16-09:09:09.844676
> ,1,999158,0,\"snort test
>
[GitHub] [metron] merrimanr opened a new pull request #1391: METRON-2090: Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency
merrimanr opened a new pull request #1391: METRON-2090: Full dev is failing
with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency
URL: https://github.com/apache/metron/pull/1391
## Contributor Comments
I was able to track this down to a special version of `org.mortbay.jetty`
dependencies being pulled in by the `org.apache.storm:storm-hdfs` dependency:
https://community.hortonworks.com/questions/6256/jetty-util-6126hwxjar.html
The `org.mortbay.jetty` dependencies are included transitively and the
recent metron-writer pom.xml refactor in
https://github.com/apache/metron/pull/1386 caused this special version to now
be included. This PR fixes this by simply excluding the special version.
### Testing
I was able to successfully build full dev and verified data was flowing
through to HDFS and ES.
## Pull Request Checklist
Thank you for submitting a contribution to Apache Metron.
Please refer to our [Development
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
for the complete guide to follow for contributions.
Please refer also to our [Build Verification
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
for complete smoke testing guides.
In order to streamline the review of the contribution we ask you follow
these guidelines and ask you to double check the following:
### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to be
created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [x] Has your PR been rebased against the latest commit within the target
branch (typically master)?
### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
```
mvn -q clean integration-test install &&
dev-utilities/build-utils/verify_licenses.sh
```
- [x] Have you written or updated unit tests and or integration tests to
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [x] Have you verified the basic functionality of the build by building and
running locally with Vagrant full-dev environment or the equivalent?
### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in which
it is rendered by building and verifying the site-book? If not then run the
following commands and the verify changes via
`site-book/target/site/index.html`:
```
cd site-book
mvn site
```
Note:
Please ensure that once the PR is submitted, you check travis-ci for build
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up for
your personal repository such that your branches are built there before
submitting a pull request.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [metron] merrimanr closed pull request #1391: METRON-2090: Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency
merrimanr closed pull request #1391: METRON-2090: Full dev is failing with missing org.mortbay.jetty:jetty-util:jar:6.1.26.hwx dependency URL: https://github.com/apache/metron/pull/1391 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-1950) Site-book generation broken in master
[ https://issues.apache.org/jira/browse/METRON-1950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-1950: -- Fix Version/s: Next + 1 > Site-book generation broken in master > - > > Key: METRON-1950 > URL: https://issues.apache.org/jira/browse/METRON-1950 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2026) Remove Storm dependency from metron-common
[ https://issues.apache.org/jira/browse/METRON-2026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-2026: -- Assignee: Ryan Merriman > Remove Storm dependency from metron-common > -- > > Key: METRON-2026 > URL: https://issues.apache.org/jira/browse/METRON-2026 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 2h 40m > Remaining Estimate: 0h > > Migrating to other streaming engine will require common modules that don't > depend on Storm. We need to split Storm-dependent classes out of > metron-common and into a separate module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-1654) findOne request after an alert patch returns with the original state of the alert item
[ https://issues.apache.org/jira/browse/METRON-1654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-1654: -- Assignee: Ryan Merriman > findOne request after an alert patch returns with the original state of the > alert item > -- > > Key: METRON-1654 > URL: https://issues.apache.org/jira/browse/METRON-1654 > Project: Metron > Issue Type: Bug >Reporter: Tibor Meller >Assignee: Ryan Merriman >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > /findOne request after an alert patch returns with the original state of the > alert item. > The UI stays inconsistent until the next search request. > This leads to the following bugs: > - Alert status change on details panel haven't reflected on the list view. > - Adding an element to a meta-alert doesn't change the items counter beside > the name of the meta-alert > - Removing an element from a meta-alert doesn't change the items counter > beside the name of the meta-alert > - Clicking on remove/unlink button in a meta-alert group doesn't remove item > immediately > This behavior only validated on the full dev single node VM and might not > exist in production. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1914) Upgrade the bro sensor in full-dev to latest
[ https://issues.apache.org/jira/browse/METRON-1914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16824482#comment-16824482 ] Jon Zeolla commented on METRON-1914: Since there is talk about moving full-dev to centos7, which greatly simplifies what we need to do to upgrade bro, I am adding a dependency to this task. > Upgrade the bro sensor in full-dev to latest > > > Key: METRON-1914 > URL: https://issues.apache.org/jira/browse/METRON-1914 > Project: Metron > Issue Type: Improvement >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Minor > > Bro 2.6 and 2.6.1 have been released, we should support it by default. We > should maintain backwards compatability at least to 2.5. > Note regarding metron-bro-plugin-kafka: > In docker/in_docker_scripts/configure_bro_plugin.sh:41, this will error with > bro 2.6 and newer, and generally isn't a very clean appraoch. In 2.6, the > DHCP framework was reworked and known-devices-and-hostnames.bro no longer > exists (zeek/zeek@e76b56c#diff-cdafd4a1100bf5b58e63223c5d25a906). This > command was initially made as a hack to test Metron ES support for bro 2.5.5 > logs. I suggest we figure out what the right approach here is - don't have > anything off the top of my head right now. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (METRON-1951) Add site-book generation to Travis build
[ https://issues.apache.org/jira/browse/METRON-1951?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-1951: -- Fix Version/s: Next + 1 > Add site-book generation to Travis build > > > Key: METRON-1951 > URL: https://issues.apache.org/jira/browse/METRON-1951 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2014) Add architectural documentation for metron-writer
[ https://issues.apache.org/jira/browse/METRON-2014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-2014: -- Assignee: Ryan Merriman > Add architectural documentation for metron-writer > - > > Key: METRON-2014 > URL: https://issues.apache.org/jira/browse/METRON-2014 > Project: Metron > Issue Type: Improvement >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 50m > Remaining Estimate: 0h > > This task is based on the discussion here: > [https://lists.apache.org/thread.html/13bd0ed5606ad4f3427f24a8e759d6bcb61ace76d4afcc9f48310a00@%3Cdev.metron.apache.org%3E] > Follow on architectural documentation is needed for the writer module. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2064) Metron REST API overwriting global.json values
[ https://issues.apache.org/jira/browse/METRON-2064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Allen reassigned METRON-2064: -- Assignee: Ryan Merriman > Metron REST API overwriting global.json values > -- > > Key: METRON-2064 > URL: https://issues.apache.org/jira/browse/METRON-2064 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Ryan Merriman >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We added the capability for Ambari to modify global config properties without > overwriting non-conflicting externally-provided properties in this PR - > [https://github.com/apache/metron/pull/760]. The REST API is currently > overwriting the global config when it's restarted, thus losing user changes. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (METRON-2065) Setting Parser Output Topic in Sensor Config is broken
[
https://issues.apache.org/jira/browse/METRON-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-2065:
--
Assignee: Ryan Merriman
> Setting Parser Output Topic in Sensor Config is broken
> --
>
> Key: METRON-2065
> URL: https://issues.apache.org/jira/browse/METRON-2065
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Ryan Merriman
>Priority: Major
> Attachments: Screen Shot 2019-04-05 at 7.45.36 PM.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Login to management console
> Edit the parser config Advanced > Raw JSON !Screen Shot 2019-04-05 at
> 7.45.36 PM.png!
> Change the output topic for the 'snort' sensor.
> Verify that the changes taken effect using stellar shell
> {code:java}
> [Stellar]>>> conf := CONFIG_GET("PARSER","snort") { "parserClassName" :
> "org.apache.metron.parsers.snort.BasicSnortParser", "sensorTopic" : "snort",
> "outputTopic" : "new-topic", "readMetadata" : false, "mergeMetadata" : false,
> "spoutParallelism" : 1, "spoutNumTasks" : 1, "parserParallelism" : 1,
> "parserNumTasks" : 1, "errorWriterParallelism" : 1, "errorWriterNumTasks" :
> 1, "spoutConfig" : { }, "stormConfig" : { }, "parserConfig" : { },
> "fieldTransformations" : [ ], "cacheConfig" : { }, "rawMessageStrategy" :
> "DEFAULT", "rawMessageStrategyConfig" : { } }
> {code}
> publish the message to 'snort' topic
> I use the console consumer to validate output is being piped into "new_topic"
> and verified that no messages were sent to the topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic new-topic
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper]. [2019-04-05 14:08:08,796] WARN
> SASL configuration failed: javax.security.auth.login.LoginException: No JAAS
> configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn) [2019-04-05 14:08:09,005]
> WARN SASL configuration failed: javax.security.auth.login.LoginException: No
> JAAS configuration section named 'Client' was found in specified JAAS
> configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {code}
> where as I see that the messages were sent to "enrichments" topic
> {code:java}
> [metron@nat-r7-udos-metron-1 bin]$ ./kafka-console-consumer.sh --zookeeper
> $ZOOKEEPER --security-protocol PLAINTEXTSASL --topic enrichments
> Using the ConsoleConsumer with old consumer is deprecated and will be removed
> in a future major release. Consider using the new consumer by passing
> [bootstrap-server] instead of [zookeeper].
> [2019-04-05 14:10:18,930] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> [2019-04-05 14:10:19,095] WARN SASL configuration failed:
> javax.security.auth.login.LoginException: No JAAS configuration section named
> 'Client' was found in specified JAAS configuration file:
> '/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf'. Will continue
> connection to Zookeeper server without SASL authentication, if Zookeeper
> server allows it. (org.apache.zookeeper.ClientCnxn)
> {"msg":"snort test
> alert","sig_rev":"0","ip_dst_port":"80","ethsrc":"00:00:00:00:00:00","tcpseq":"0xF017C4DA","dgmlen":"40","icmpid":"","tcplen":"","tcpwindow":"0xF6C9","icmpseq":"","tcpack":"0xABDB8426","protocol":"TCP","ip_dst_addr":"62.75.195.236","original_string":"09\/09\/16-09:09:09.844676
> ,1,999158,0,\"snort test
>
[jira] [Updated] (METRON-1817) Remove bro package dependency on bro source
[ https://issues.apache.org/jira/browse/METRON-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon Zeolla updated METRON-1817: --- Fix Version/s: Next + 1 > Remove bro package dependency on bro source > --- > > Key: METRON-1817 > URL: https://issues.apache.org/jira/browse/METRON-1817 > Project: Metron > Issue Type: Task >Reporter: Jon Zeolla >Assignee: Jon Zeolla >Priority: Major > Fix For: Next + 1 > > Time Spent: 20m > Remaining Estimate: 0h > > With Bro 2.6 we no longer need to instruct users to point to the bro source > to compile. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
