[jira] [Updated] (METRON-2341) [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest
[ https://issues.apache.org/jira/browse/METRON-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2341: -- Summary: [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest (was: Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest) > [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in > /metron-interface/metron-rest > - > > Key: METRON-2341 > URL: https://issues.apache.org/jira/browse/METRON-2341 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Priority: Major > > This came automatically from dependabot on Github. > https://github.com/apache/metron/pull/1552. Need a Jira to associate with the > PR for review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2341) [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest
[ https://issues.apache.org/jira/browse/METRON-2341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2341: - Assignee: Michael Miklavcic > [dependabot] Bump nimbus-jose-jwt from 4.41.2 to 7.9 in > /metron-interface/metron-rest > - > > Key: METRON-2341 > URL: https://issues.apache.org/jira/browse/METRON-2341 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > > This came automatically from dependabot on Github. > https://github.com/apache/metron/pull/1552. Need a Jira to associate with the > PR for review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (METRON-2341) Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest
Michael Miklavcic created METRON-2341: - Summary: Bump nimbus-jose-jwt from 4.41.2 to 7.9 in /metron-interface/metron-rest Key: METRON-2341 URL: https://issues.apache.org/jira/browse/METRON-2341 Project: Metron Issue Type: Task Reporter: Michael Miklavcic This came automatically from dependabot on Github. https://github.com/apache/metron/pull/1552. Need a Jira to associate with the PR for review. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (METRON-2336) Stack advisor provides some components multiple times
[ https://issues.apache.org/jira/browse/METRON-2336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993622#comment-16993622 ] Michael Miklavcic commented on METRON-2336: --- Hi [~sziszo], thanks for reaching out - I see you work on Ambari. I'm hoping we can collaborate on figuring out a solution for this problem. There is a ticket already open for this against Ambari here - https://issues.apache.org/jira/browse/AMBARI-25375. Can you provide any detail as to why you believe this is a client issue and not an issue with how Ambari is handling the host component map? There has been quite a bit of discussion about the service_advisor parent code misbehaving and some attempted workarounds here: # https://github.com/apache/metron/pull/1505 # https://github.com/apache/metron/pull/1512 > Stack advisor provides some components multiple times > - > > Key: METRON-2336 > URL: https://issues.apache.org/jira/browse/METRON-2336 > Project: Metron > Issue Type: Bug >Affects Versions: 0.7.1 >Reporter: Szilard Antal >Priority: Major > Attachments: Screenshot 2019-11-26 at 12.42.59 PM.png > > > Customer is in Ambari 2.7.4.0-118 and metron mpack version > hcp-ambari-mpack-2.0.0.0-4 > Currently when try to install metron its showing Metron PCAP ,Metron Profiler > ,Metron Enrichment twice in ambari. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2322) Add Ambari connection check to upgrade_helper script
[ https://issues.apache.org/jira/browse/METRON-2322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2322: -- Fix Version/s: Next + 1 > Add Ambari connection check to upgrade_helper script > > > Key: METRON-2322 > URL: https://issues.apache.org/jira/browse/METRON-2322 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 2h > Remaining Estimate: 0h > > The current script does not output any error information if the > username/password or cluster name for Ambari is incorrect. This adds a return > code status check to the output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2326) Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
[
https://issues.apache.org/jira/browse/METRON-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2326:
--
Fix Version/s: Next + 1
> Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
> --
>
> Key: METRON-2326
> URL: https://issues.apache.org/jira/browse/METRON-2326
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> A Threat Triage Rule's "reason" field can contain executable Stellar to
> provide an operator context as to why a rule fired during Threat Triage. I
> am unable to call any function that requires a StellarContext during
> initialization, from the 'Reason' field of a Threat Triage Rule. For
> example, I cannot call `ENRICHMENT_GET`.
> h3. Steps to Replicate
> 1. Create a simple file called `user.csv`.
> {code:java}
> [root@node1 ~]# cat user.csv
> jdoe,192.168.138.2
> jane,192.168.66.1
> ciana,192.168.138.158
> danixa,95.163.121.204
> jim,192.168.66.121
> {code}
> 2 . Create a file called `user-extractor.json`.
> {code:java}
> {
> "config": {
> "columns": {
> "user": 0,
> "ip": 1
> },
> "indicator_column": "ip",
> "separator": ",",
> "type": "user"
> },
> "extractor": "CSV"
> }
> {code}
> 3. Import the enrichment data.
> {code:java}
> source /etc/default/metron
> $METRON_HOME/bin/flatfile_loader.sh -i ./user.csv -t enrichment -c t -e
> ./user-extractor.json
> {code}
> 4. Validate that the enrichment loaded successfully.
> {code:java}
> [root@node1 0.7.2]# source /etc/default/metron
> [root@node1 0.7.2]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
>
> [Stellar]>>> ip_dst_addr := "192.168.138.2"
> 192.168.138.2
>
> [Stellar]>>> ENRICHMENT_GET('user', ip_dst_addr, 'enrichment', 't')
> \{ip=192.168.138.2, user=jdoe}
> {code}
> 5. Create a threat triage rule that attempts an ENRICHMENT_GET.
> {code}
> [Stellar]>>> conf := SHELL_EDIT()
> {
> "enrichment": {
> "fieldMap": {
> "stellar": {
> "config": {
> "is_alert": "true"
> }
> }
> },
> "fieldToTypeMap": {},
> "config": {}
> },
> "threatIntel": {
> "fieldMap": {},
> "fieldToTypeMap": {},
> "config": {},
> "triageConfig": {
> "riskLevelRules": [
> {
> "name": "Rule",
> "comment": "This rule does not work when executing the 'reason' field.",
> "rule": "true",
> "reason": "FORMAT('Call to ENRICHMENT_GET=%s', ENRICHMENT_GET('user',
> ip_dst_addr, 'enrichment', 't'))",
> "score": "100"
> }
> ],
> "aggregator": "MAX",
> "aggregationConfig": {}
> }
> },
> "configuration": {}
> }
>
> [Stellar]>>> CONFIG_PUT("ENRICHMENT", conf, "snort")
> {code}
>
> 6. The Storm worker logs for Enrichment show the following error.
> {code:java}
> 2019-11-21 03:54:34.370 o.a.c.f.r.c.TreeCache Curator-TreeCache-4 [ERROR]
> org.apache.metron.jackson.databind.JsonMappingException: Unable to find
> capability GLOBAL_CONFIG; it may not be available in your context.
> at [Source: java.io.ByteArrayInputStream@1f55bdda; line: 24, column: 11]
> (through reference chain:
> org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig["threatIntel"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatIntelConfig["triageConfig"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig["riskLevelRules"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:262)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:537)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:518)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> ~[stormjar.jar:?]
> at
>
[jira] [Updated] (METRON-2285) Batch Profiler Cannot Persist Data Sketches
[
https://issues.apache.org/jira/browse/METRON-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2285:
--
Fix Version/s: Next + 1
> Batch Profiler Cannot Persist Data Sketches
> ---
>
> Key: METRON-2285
> URL: https://issues.apache.org/jira/browse/METRON-2285
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: Maxim Dashenko
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class
> org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file
> /usr/hcp/current/metron/config/batch-profiler.properties
> ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config
> /usr/hcp/current/metron/config/batch-profiler.properties --profiles
> ~/profiler.json
> {code}
> cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>
> cat ~/profiler.json
> {code}
> {
>"profiles":[
> {
> "profile":"batchteststat",
> "onlyif":"source.type == 'testsource' and devicehostname ==
> 'windows9.something.com'",
> "foreach":"devicehostname",
> "update":{
> "s":"STATS_ADD(s, devicehostname)"
> },
> "result":{
> "profile":"s"
> }
> }
>],
>"timestampField":"timestamp"
> }
> {code}
> cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000,
> "source.type": "testsource"}
> {code}
> The command raises an exception:
> {code}
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 68 in stage 8.0 failed 1 times, most recent failure:
> Lost task 68.0 in stage 8.0 (TID 274, localhost, executor driver):
> com.esotericsoftware.kryo.KryoException: Unable to find class:
> org.apache.metron.statistics.OnlineStatisticsProvider
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.metron.common.utils.SerDeUtils.fromBytes(SerDeUtils.java:262)
> at
> org.apache.metron.profiler.spark.ProfileMeasurementAdapter.toProfileMeasurement(ProfileMeasurementAdapter.java:85)
> at
> org.apache.metron.profiler.spark.function.HBaseWriterFunction.call(HBaseWriterFunction.java:124)
> at org.apache.spark.sql.Dataset$$anonfun$48.apply(Dataset.scala:2266)
> at org.apache.spark.sql.Dataset$$anonfun$48.apply(Dataset.scala:2266)
> at
> org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:196)
> at
> org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:193)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:108)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.metron.statistics.OnlineStatisticsProvider
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at
[jira] [Created] (METRON-2322) Add Ambari connection check to upgrade_helper script
Michael Miklavcic created METRON-2322: - Summary: Add Ambari connection check to upgrade_helper script Key: METRON-2322 URL: https://issues.apache.org/jira/browse/METRON-2322 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic The current script does not output any error information if the username/password or cluster name for Ambari is incorrect. This adds a return code status check to the output. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2239) Metron Automated backup and restore
[ https://issues.apache.org/jira/browse/METRON-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2239: -- Fix Version/s: Next + 1 > Metron Automated backup and restore > --- > > Key: METRON-2239 > URL: https://issues.apache.org/jira/browse/METRON-2239 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Michael Miklavcic >Priority: Blocker > Fix For: Next + 1 > > Time Spent: 3h > Remaining Estimate: 0h > > Metron, for upgrading to HDP 3.1 should have the ability to backup and > restore metron specific configurations. > For many, this upgrade will involve OS upgrade and Hadoop/HDP upgrade/Amabari > etc. > Such a tool would: > Backup to file metron configurations from different locations, along with > enough meta data on the files to restore them. The directory/backup location > would be structured. > * Disk on nodes > * HDFS > * Zookeeper > * ?? > This backup will then be archived by the tool, and could then be saved by the > user. > The tool ( or a companion tool ) would be able to take these archives and > restore them back to the cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2293) Fix some inaccuracies in the MaaS README example
[ https://issues.apache.org/jira/browse/METRON-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2293: -- Fix Version/s: Next + 1 > Fix some inaccuracies in the MaaS README example > > > Key: METRON-2293 > URL: https://issues.apache.org/jira/browse/METRON-2293 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2239) Metron Automated backup and restore
[ https://issues.apache.org/jira/browse/METRON-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2239: - Assignee: Michael Miklavcic > Metron Automated backup and restore > --- > > Key: METRON-2239 > URL: https://issues.apache.org/jira/browse/METRON-2239 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Michael Miklavcic >Priority: Blocker > > Metron, for upgrading to HDP 3.1 should have the ability to backup and > restore metron specific configurations. > For many, this upgrade will involve OS upgrade and Hadoop/HDP upgrade/Amabari > etc. > Such a tool would: > Backup to file metron configurations from different locations, along with > enough meta data on the files to restore them. The directory/backup location > would be structured. > * Disk on nodes > * HDFS > * Zookeeper > * ?? > This backup will then be archived by the tool, and could then be saved by the > user. > The tool ( or a companion tool ) would be able to take these archives and > restore them back to the cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2300) Fix Brad Kolarov's Apache ID
[ https://issues.apache.org/jira/browse/METRON-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2300: - Assignee: Billie Rinaldi > Fix Brad Kolarov's Apache ID > > > Key: METRON-2300 > URL: https://issues.apache.org/jira/browse/METRON-2300 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Billie Rinaldi >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (METRON-2300) Fix Brad Kolarov's Apache ID
Michael Miklavcic created METRON-2300: - Summary: Fix Brad Kolarov's Apache ID Key: METRON-2300 URL: https://issues.apache.org/jira/browse/METRON-2300 Project: Metron Issue Type: Task Reporter: Michael Miklavcic -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2280) PCAP queries no longer work
[
https://issues.apache.org/jira/browse/METRON-2280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2280:
-
Assignee: Michael Miklavcic
> PCAP queries no longer work
> ---
>
> Key: METRON-2280
> URL: https://issues.apache.org/jira/browse/METRON-2280
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Run a PCAP query and get the following exception
> {code:java}
> $METRON_HOME/bin/pcap_query.sh fixed -st 20191009 -df "MMdd" -p 6 -rpf 500
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/metron/stellar/common/utils/ConversionUtils
> at
> org.apache.metron.common.configuration.ConfigOption.get(ConfigOption.java:81)
> at org.apache.metron.pcap.mr.PcapJob.submit(PcapJob.java:219)
> at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:107)
> at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:55)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.metron.stellar.common.utils.ConversionUtils
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 10 more
> {code}
> This appears to have been introduced in
> https://github.com/apache/metron/pull/1436 and missed in testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (METRON-2284) Metron Profiler for Spark doesn't work as expected
[
https://issues.apache.org/jira/browse/METRON-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954000#comment-16954000
]
Michael Miklavcic commented on METRON-2284:
---
Can we have you try the following exercise from the REPL using your profile?
https://github.com/apache/metron/blob/master/metron-analytics/metron-profiler-repl/README.md#getting-started
{code:java}
[Stellar]>>> val := SET_INIT()
[]
[Stellar]>>> devicehostname := 'windows9.something.com'
windows9.something.com
[Stellar]>>> val := SET_ADD(val, IS_EMPTY(devicehostname))
[false]
[Stellar]>>> conf := SHELL_EDIT()
# add the following profile contents in the vi editor that comes up:
{
"profiles":[
{
"profile":"batchtest5",
"onlyif":"devicehostname == 'windows9.something.com'",
"foreach":"devicehostname",
"init":{
"val":"SET_INIT()"
},
"update":{
"val":"SET_ADD(val, IS_EMPTY(devicehostname))"
},
"result":{
"profile":"val"
}
}
],
"timestampField":"timestamp"
}
[Stellar]>>> profiler := PROFILER_INIT(conf)
Profiler{1 profile(s), 0 messages(s), 0 route(s)}
[Stellar]>>> msg := SHELL_EDIT()
# add this record
{"devicehostname": "windows9.something.com", "timestamp": 1567241981000}
[Stellar]>>> PROFILER_APPLY(msg, profiler)
Profiler{1 profile(s), 1 messages(s), 1 route(s)}
[Stellar]>>> values := PROFILER_FLUSH(profiler)
[{period={duration=90, period=1741379, start=156724110,
end=156724200}, profile=batchtest5, groups=[], value=[false],
entity=windows9.something.com}]
{code}
I'm seeing "value" set to false, as expected, at least from the REPL. Let's see
if we can verify that part of the functionality matches up as expected and go
from there.
> Metron Profiler for Spark doesn't work as expected
> --
>
> Key: METRON-2284
> URL: https://issues.apache.org/jira/browse/METRON-2284
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: Maxim Dashenko
>Priority: Major
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class
> org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file
> /usr/hcp/current/metron/config/batch-profiler.properties
> ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config
> /usr/hcp/current/metron/config/batch-profiler.properties --profiles
> ~/profiler.json
> {code}
> cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>
> cat ~/profiler.json
> {code}
> {
> "profiles":[
>{
> "profile":"batchtest5",
> "onlyif":"source.type == 'testsource' and devicehostname ==
> 'windows9.something.com'",
> "foreach":"devicehostname",
> "init":{
>"val":"SET_INIT()"
> },
> "update":{
>"val":"SET_ADD(val, IS_EMPTY(devicehostname))"
> },
> "result":{
>"profile":"val"
> }
>}
> ],
> "timestampField":"timestamp"
> }
> {code}
> cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000,
> "source.type": "testsource"}
> {code}
> Stellar statement
> {code}
> PROFILE_GET('batchtest5', 'windows9.something.com', PROFILE_FIXED(100,
> 'DAYS'))
> {code}
> Returns:
> {code}
> [[true]]
> {code}
> Expected result:
> {code}
> [[false]]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (METRON-2293) Fix some inaccuracies in the MaaS README example
Michael Miklavcic created METRON-2293: - Summary: Fix some inaccuracies in the MaaS README example Key: METRON-2293 URL: https://issues.apache.org/jira/browse/METRON-2293 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2264) Upgrade metron-hbase-client to HBase 2.0.2
[ https://issues.apache.org/jira/browse/METRON-2264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2264: -- Fix Version/s: Next + 1 > Upgrade metron-hbase-client to HBase 2.0.2 > -- > > Key: METRON-2264 > URL: https://issues.apache.org/jira/browse/METRON-2264 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > The metron-hbase-client has some special handling for pulling in a shaded > HBase client library. This pulls in a different version (1.1.2) than the > global HBase version to work around a known HBase bug. After the upgrade to > 2.0.2, this work around is no longer needed and can be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2170) Deprecate custom Kafka module
[ https://issues.apache.org/jira/browse/METRON-2170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2170: -- Parent: (was: METRON-2088) Issue Type: Task (was: Sub-task) > Deprecate custom Kafka module > - > > Key: METRON-2170 > URL: https://issues.apache.org/jira/browse/METRON-2170 > Project: Metron > Issue Type: Task >Reporter: Ryan Merriman >Priority: Major > > Previously we had to add a "metron-storm-kafka-override" module to include > fixes for the Storm Kafka spout included in the version of Storm we were > using. This should be removed now that we have upgraded. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2171) Deprecate Kafka authentication handling
[ https://issues.apache.org/jira/browse/METRON-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2171: -- Parent: (was: METRON-2088) Issue Type: Task (was: Sub-task) > Deprecate Kafka authentication handling > --- > > Key: METRON-2171 > URL: https://issues.apache.org/jira/browse/METRON-2171 > Project: Metron > Issue Type: Task >Reporter: Ryan Merriman >Priority: Major > > Previously we needed to detect and properly set the Kafka authentication > setting, depending on the version of Kafka deployed. This should not be > necessary after upgrading to Kafka 2.0.0 and should be removed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2265) Update Kerberos settings
[ https://issues.apache.org/jira/browse/METRON-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2265: - Assignee: Ryan Merriman > Update Kerberos settings > > > Key: METRON-2265 > URL: https://issues.apache.org/jira/browse/METRON-2265 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > With various components being upgraded, there will be some necessary > adjustments for the system to continue working when Kerberos is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2265) Update Kerberos settings
[ https://issues.apache.org/jira/browse/METRON-2265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2265: -- Fix Version/s: Next + 1 > Update Kerberos settings > > > Key: METRON-2265 > URL: https://issues.apache.org/jira/browse/METRON-2265 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 4h 10m > Remaining Estimate: 0h > > With various components being upgraded, there will be some necessary > adjustments for the system to continue working when Kerberos is enabled. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2262) Upgrade to Curator 4.2.0
[ https://issues.apache.org/jira/browse/METRON-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2262: -- Parent: (was: METRON-2088) Issue Type: Task (was: Sub-task) > Upgrade to Curator 4.2.0 > > > Key: METRON-2262 > URL: https://issues.apache.org/jira/browse/METRON-2262 > Project: Metron > Issue Type: Task >Reporter: Nick Allen >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Upgrade to Curator 4.2.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (METRON-2262) Upgrade to Curator 4.2.0
[ https://issues.apache.org/jira/browse/METRON-2262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16952171#comment-16952171 ] Michael Miklavcic commented on METRON-2262: --- This should become a separate Jira for a future effort. This is no longer necessary for the upgrade. > Upgrade to Curator 4.2.0 > > > Key: METRON-2262 > URL: https://issues.apache.org/jira/browse/METRON-2262 > Project: Metron > Issue Type: Sub-task >Reporter: Nick Allen >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Upgrade to Curator 4.2.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (METRON-2286) Create a classpath-isolating classloader for the Metron topologies
Michael Miklavcic created METRON-2286: - Summary: Create a classpath-isolating classloader for the Metron topologies Key: METRON-2286 URL: https://issues.apache.org/jira/browse/METRON-2286 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic This would be an improvement based on existing working started in https://issues.apache.org/jira/browse/METRON-744. https://accumulo.apache.org/blog/2014/05/03/accumulo-classloader.html Storm provides no classpath isolation in its topology runtime, so it is a very common occurrence to run into version compatibility problems from libraries such as: Guava, Jackson, SLF4J, and Log4j between kafka, hadoop, hbase, and Storm. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (METRON-2280) PCAP queries no longer work
Michael Miklavcic created METRON-2280:
-
Summary: PCAP queries no longer work
Key: METRON-2280
URL: https://issues.apache.org/jira/browse/METRON-2280
Project: Metron
Issue Type: Bug
Reporter: Michael Miklavcic
Run a PCAP query and get the following exception
{code:java}
$METRON_HOME/bin/pcap_query.sh fixed -st 20191009 -df "MMdd" -p 6 -rpf 500
Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/metron/stellar/common/utils/ConversionUtils
at
org.apache.metron.common.configuration.ConfigOption.get(ConfigOption.java:81)
at org.apache.metron.pcap.mr.PcapJob.submit(PcapJob.java:219)
at org.apache.metron.pcap.query.PcapCli.run(PcapCli.java:107)
at org.apache.metron.pcap.query.PcapCli.main(PcapCli.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:318)
at org.apache.hadoop.util.RunJar.main(RunJar.java:232)
Caused by: java.lang.ClassNotFoundException:
org.apache.metron.stellar.common.utils.ConversionUtils
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 10 more
{code}
This appears to have been introduced in
https://github.com/apache/metron/pull/1436 and missed in testing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (METRON-2274) Flatfile loader and summarizer mapreduce mode broken
Michael Miklavcic created METRON-2274: - Summary: Flatfile loader and summarizer mapreduce mode broken Key: METRON-2274 URL: https://issues.apache.org/jira/browse/METRON-2274 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic Assignee: Michael Miklavcic Fix mapreduce mode for the loaders. This was broken in METRON-2149 in 2 places: [https://github.com/apache/metron/pull/1436/files#diff-1a76680a25447c0c0b3a245ce767b3f8R39] and [https://github.com/apache/metron/pull/1436/files#diff-8c698ee5b00cdf64f09bcd48847eae33R39] HADOOP_CLASSPATH only sets the classpath for the local client MR application, not for the JVM's running the map and reduce tasks. When Stellar was extracted from our uber jars, the metron-data-management jar no longer contained requisite classes from Stellar common. The correct way to handle adding dependencies for mapreduce jobs is via the "-libjars" option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2235) Full dev deployment server startup timeout
[ https://issues.apache.org/jira/browse/METRON-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2235: - Assignee: Dale Richardson > Full dev deployment server startup timeout > -- > > Key: METRON-2235 > URL: https://issues.apache.org/jira/browse/METRON-2235 > Project: Metron > Issue Type: Bug >Reporter: Dale Richardson >Assignee: Dale Richardson >Priority: Minor > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > When building full dev on slower virtual systems, it may take longer then 50 > seconds for Ambari to fully startup. The startup script will kill the Ambari > server process if it takes longer then 50 seconds. > The amber config item 'server.startup.web.timeout' Can override this timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (METRON-2235) Full dev deployment server startup timeout
[ https://issues.apache.org/jira/browse/METRON-2235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2235: -- Fix Version/s: Next + 1 > Full dev deployment server startup timeout > -- > > Key: METRON-2235 > URL: https://issues.apache.org/jira/browse/METRON-2235 > Project: Metron > Issue Type: Bug >Reporter: Dale Richardson >Priority: Minor > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > When building full dev on slower virtual systems, it may take longer then 50 > seconds for Ambari to fully startup. The startup script will kill the Ambari > server process if it takes longer then 50 seconds. > The amber config item 'server.startup.web.timeout' Can override this timeout. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (METRON-2232) Upgrade to Hadoop 3.1.1
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934753#comment-16934753 ] Michael Miklavcic commented on METRON-2232: --- Reverting Curator back to 2.7.1 allows the MaaS tests to run again. I'm unsure of the original impetus for revving to 2.12. > Upgrade to Hadoop 3.1.1 > --- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (METRON-2250) Missing services in HDP 3.1 metron mpack and installer stuck
[
https://issues.apache.org/jira/browse/METRON-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2250:
-
Assignee: Mohan (was: Michael Miklavcic)
> Missing services in HDP 3.1 metron mpack and installer stuck
>
>
> Key: METRON-2250
> URL: https://issues.apache.org/jira/browse/METRON-2250
> Project: Metron
> Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Mohan
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> We missed this during our full dev deployment because we use blueprints,
> which seems to bypass the service_advisor mechanism in Ambari.
>
> If you uninstall Metron and attempt to reinstall the service (or do any fresh
> install for that matter), the service recommendations will fail due to
> missing hostnames in the services list. The following stack traces appear in
> the Ambari server logs.
>
> {code:java}
> Error occured in stack advisor.
> Error details: list index out of range
> 2019-09-03 20:42:10,911 INFO [ambari-client-thread-34] StackAdvisorRunner:167
> - Advisor script stderr: Traceback (most recent call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 190,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 122,
> in main
> result = stackAdvisor.validateComponentLayout(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 1072, in validateComponentLayout
> validationItems = self.getComponentLayoutValidations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 1095, in getComponentLayoutValidations
> items.extend(serviceAdvisor.getServiceComponentLayoutValidations(services,
> hosts))
> File
> "/var/lib/ambari-server/resources/common-services/METRON/0.7.2/service_advisor.py",
> line 46, in getServiceComponentLayoutValidations
> metronParsersHost = self.getHosts(componentsList, "METRON_PARSERS")[0]
> IndexError: list index out of range
> 2019-09-03 20:42:10,912 WARN [ambari-client-thread-34]
> ValidationResourceProvider:132 - Error occurred during validation
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException:
> Stack Advisor reported an error. Exit Code: 2. Error: IndexError: list index
> out of range
> StdOut file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.out
> StdErr file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.err
> at
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.processLogs(StackAdvisorRunner.java:149)
> at
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.runScript(StackAdvisorRunner.java:89)
> at
> org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:314)
> at
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.validate(StackAdvisorHelper.java:94)
> at
> org.apache.ambari.server.controller.internal.ValidationResourceProvider.createResources(ValidationResourceProvider.java:127)
> at
> org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:296)
> at
> org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
> at
> org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:50)
> at
> org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:68)
> at
> org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
> at
> org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:163)
> at
> org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:127)
> at
> org.apache.ambari.server.api.services.ValidationService.getValidation(ValidationService.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
>
[jira] [Commented] (METRON-2232) Upgrade to Hadoop 3.1.1
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16931753#comment-16931753 ] Michael Miklavcic commented on METRON-2232: --- The exception shown in the integration test looks like a bit of a red herring. I'm seeing the ADD operation work just fine [https://github.com/apache/metron/blob/master/metron-analytics/metron-maas-service/src/test/java/org/apache/metron/maas/service/MaasIntegrationTest.java#L214] and the REMOVE operation never seem to remove the endpoint [https://github.com/apache/metron/blob/master/metron-analytics/metron-maas-service/src/test/java/org/apache/metron/maas/service/MaasIntegrationTest.java#L252], so the whole thing loops indefinitely here until the test times out [https://github.com/apache/metron/blob/master/metron-analytics/metron-maas-service/src/test/java/org/apache/metron/maas/service/MaasIntegrationTest.java#L261] > Upgrade to Hadoop 3.1.1 > --- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2232) Upgrade to Hadoop 3.1.1
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16929399#comment-16929399 ] Michael Miklavcic commented on METRON-2232: --- Current exceptions in the MaasIntegrationTest [https://gist.github.com/mmiklavc/429b2207c52dfc59e80ea6e621a60c4f] > Upgrade to Hadoop 3.1.1 > --- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2232) Upgrade to Hadoop 3.1.1
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2232: - Assignee: Michael Miklavcic > Upgrade to Hadoop 3.1.1 > --- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2255) Move npm install from RPM build to Maven build
Michael Miklavcic created METRON-2255: - Summary: Move npm install from RPM build to Maven build Key: METRON-2255 URL: https://issues.apache.org/jira/browse/METRON-2255 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Packaging should remain in the purview of Maven with RPM simply bundling those results. [https://github.com/apache/metron/blob/master/metron-deployment/packaging/docker/rpm-docker/SPECS/metron.spec#L121] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2222) Deprecate Storm 1.0.x
[ https://issues.apache.org/jira/browse/METRON-?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-: -- Summary: Deprecate Storm 1.0.x (was: Deprecate Storm 1.x) > Deprecate Storm 1.0.x > - > > Key: METRON- > URL: https://issues.apache.org/jira/browse/METRON- > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Priority: Major > > There isn't much to this task other than calling out the major Storm version > deprecation once this upgrade makes its way to master. We should double-check > that Upgrading.md reflects the current state of changes from this feature > branch. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2250) Missing services in HDP 3.1 metron mpack and installer stuck
[
https://issues.apache.org/jira/browse/METRON-2250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926031#comment-16926031
]
Michael Miklavcic commented on METRON-2250:
---
I created a ticket against Ambari and have linked it back here for reference.
> Missing services in HDP 3.1 metron mpack and installer stuck
>
>
> Key: METRON-2250
> URL: https://issues.apache.org/jira/browse/METRON-2250
> Project: Metron
> Issue Type: Sub-task
>Reporter: Michael Miklavcic
>Assignee: Michael Miklavcic
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> We missed this during our full dev deployment because we use blueprints,
> which seems to bypass the service_advisor mechanism in Ambari.
>
> If you uninstall Metron and attempt to reinstall the service (or do any fresh
> install for that matter), the service recommendations will fail due to
> missing hostnames in the services list. The following stack traces appear in
> the Ambari server logs.
>
> {code:java}
> Error occured in stack advisor.
> Error details: list index out of range
> 2019-09-03 20:42:10,911 INFO [ambari-client-thread-34] StackAdvisorRunner:167
> - Advisor script stderr: Traceback (most recent call last):
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 190,
> in
> main(sys.argv)
> File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 122,
> in main
> result = stackAdvisor.validateComponentLayout(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 1072, in validateComponentLayout
> validationItems = self.getComponentLayoutValidations(services, hosts)
> File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
> line 1095, in getComponentLayoutValidations
> items.extend(serviceAdvisor.getServiceComponentLayoutValidations(services,
> hosts))
> File
> "/var/lib/ambari-server/resources/common-services/METRON/0.7.2/service_advisor.py",
> line 46, in getServiceComponentLayoutValidations
> metronParsersHost = self.getHosts(componentsList, "METRON_PARSERS")[0]
> IndexError: list index out of range
> 2019-09-03 20:42:10,912 WARN [ambari-client-thread-34]
> ValidationResourceProvider:132 - Error occurred during validation
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException:
> Stack Advisor reported an error. Exit Code: 2. Error: IndexError: list index
> out of range
> StdOut file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.out
> StdErr file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.err
> at
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.processLogs(StackAdvisorRunner.java:149)
> at
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.runScript(StackAdvisorRunner.java:89)
> at
> org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:314)
> at
> org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.validate(StackAdvisorHelper.java:94)
> at
> org.apache.ambari.server.controller.internal.ValidationResourceProvider.createResources(ValidationResourceProvider.java:127)
> at
> org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:296)
> at
> org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
> at
> org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:50)
> at
> org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:68)
> at
> org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
> at
> org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:163)
> at
> org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:127)
> at
> org.apache.ambari.server.api.services.ValidationService.getValidation(ValidationService.java:59)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> at
>
[jira] [Created] (METRON-2250) Missing services in HDP 3.1 metron mpack and installer stuck
Michael Miklavcic created METRON-2250:
-
Summary: Missing services in HDP 3.1 metron mpack and installer
stuck
Key: METRON-2250
URL: https://issues.apache.org/jira/browse/METRON-2250
Project: Metron
Issue Type: Sub-task
Reporter: Michael Miklavcic
Assignee: Michael Miklavcic
We missed this during our full dev deployment because we use blueprints, which
seems to bypass the service_advisor mechanism in Ambari.
If you uninstall Metron and attempt to reinstall the service (or do any fresh
install for that matter), the service recommendations will fail due to missing
hostnames in the services list. The following stack traces appear in the Ambari
server logs.
{code:java}
Error occured in stack advisor.
Error details: list index out of range
2019-09-03 20:42:10,911 INFO [ambari-client-thread-34] StackAdvisorRunner:167 -
Advisor script stderr: Traceback (most recent call last):
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 190, in
main(sys.argv)
File "/var/lib/ambari-server/resources/scripts/stack_advisor.py", line 122, in
main
result = stackAdvisor.validateComponentLayout(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 1072, in validateComponentLayout
validationItems = self.getComponentLayoutValidations(services, hosts)
File "/var/lib/ambari-server/resources/scripts/../stacks/stack_advisor.py",
line 1095, in getComponentLayoutValidations
items.extend(serviceAdvisor.getServiceComponentLayoutValidations(services,
hosts))
File
"/var/lib/ambari-server/resources/common-services/METRON/0.7.2/service_advisor.py",
line 46, in getServiceComponentLayoutValidations
metronParsersHost = self.getHosts(componentsList, "METRON_PARSERS")[0]
IndexError: list index out of range
2019-09-03 20:42:10,912 WARN [ambari-client-thread-34]
ValidationResourceProvider:132 - Error occurred during validation
org.apache.ambari.server.api.services.stackadvisor.StackAdvisorException: Stack
Advisor reported an error. Exit Code: 2. Error: IndexError: list index out of
range
StdOut file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.out
StdErr file: /var/run/ambari-server/stack-recommendations/7/stackadvisor.err
at
org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.processLogs(StackAdvisorRunner.java:149)
at
org.apache.ambari.server.api.services.stackadvisor.StackAdvisorRunner.runScript(StackAdvisorRunner.java:89)
at
org.apache.ambari.server.api.services.stackadvisor.commands.StackAdvisorCommand.invoke(StackAdvisorCommand.java:314)
at
org.apache.ambari.server.api.services.stackadvisor.StackAdvisorHelper.validate(StackAdvisorHelper.java:94)
at
org.apache.ambari.server.controller.internal.ValidationResourceProvider.createResources(ValidationResourceProvider.java:127)
at
org.apache.ambari.server.controller.internal.ClusterControllerImpl.createResources(ClusterControllerImpl.java:296)
at
org.apache.ambari.server.api.services.persistence.PersistenceManagerImpl.create(PersistenceManagerImpl.java:97)
at
org.apache.ambari.server.api.handlers.CreateHandler.persist(CreateHandler.java:50)
at
org.apache.ambari.server.api.handlers.BaseManagementHandler.handleRequest(BaseManagementHandler.java:68)
at
org.apache.ambari.server.api.services.BaseRequest.process(BaseRequest.java:144)
at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:163)
at
org.apache.ambari.server.api.services.BaseService.handleRequest(BaseService.java:127)
at
org.apache.ambari.server.api.services.ValidationService.getValidation(ValidationService.java:59)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at
com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at
com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at
com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
at
com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at
com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at
com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at
com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
at
[jira] [Commented] (METRON-2247) rpm-docker: Provide an option to bypass running rpmlint
[ https://issues.apache.org/jira/browse/METRON-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923579#comment-16923579 ] Michael Miklavcic commented on METRON-2247: --- Can you verify all of what rpmlint checks for us? It's possible that we might not need it by default at all when you consider a normal fulldev workflow, i.e. spin up full dev to verify your changes haven't broken anything with the RPMs. If successfully spinning up full dev with data flowing e2e effectively confirms that the RPMs are in spec, then that step is redundant. I would still want the option as a spot check, as needed. > rpm-docker: Provide an option to bypass running rpmlint > --- > > Key: METRON-2247 > URL: https://issues.apache.org/jira/browse/METRON-2247 > Project: Metron > Issue Type: Improvement >Reporter: Dale Richardson >Assignee: Dale Richardson >Priority: Minor > > I notice that a large chunk of the time that an rpmbuild process takes is > spent running rpmlint. For people using the full dev process to spin up a > Metron cluster, running rpmlint is like running unit-testing - the assumption > is that the author of the code you are running has already done these tests. > I propose that running rpmlint should remain as part of a full dev build, but > that it would be useful to provide an argument that you pass to vagrant that > will cause the running of rpmlint to be skipped. This could speed up the > process of standing up a full dev cluster for those people who want it. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2239) Metron Automated backup and restore
[ https://issues.apache.org/jira/browse/METRON-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922898#comment-16922898 ] Michael Miklavcic commented on METRON-2239: --- [~otto] - this is marked as "Blocker", but I'm not sure I understand the scope of that. Is this assumed to be blocker for a 1.0 release only? For the feature branch? > Metron Automated backup and restore > --- > > Key: METRON-2239 > URL: https://issues.apache.org/jira/browse/METRON-2239 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Blocker > > Metron, for upgrading to HDP 3.1 should have the ability to backup and > restore metron specific configurations. > For many, this upgrade will involve OS upgrade and Hadoop/HDP upgrade/Amabari > etc. > Such a tool would: > Backup to file metron configurations from different locations, along with > enough meta data on the files to restore them. The directory/backup location > would be structured. > * Disk on nodes > * HDFS > * Zookeeper > * ?? > This backup will then be archived by the tool, and could then be saved by the > user. > The tool ( or a companion tool ) would be able to take these archives and > restore them back to the cluster. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2217) Migrate current HBase client from HTableInterface to Table
[ https://issues.apache.org/jira/browse/METRON-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2217: -- Fix Version/s: Next + 1 > Migrate current HBase client from HTableInterface to Table > -- > > Key: METRON-2217 > URL: https://issues.apache.org/jira/browse/METRON-2217 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 5h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2201) The description for the IS_IP method default behavior needs to corrected as per implementation
[ https://issues.apache.org/jira/browse/METRON-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2201: -- Fix Version/s: Next + 1 > The description for the IS_IP method default behavior needs to corrected as > per implementation > --- > > Key: METRON-2201 > URL: https://issues.apache.org/jira/browse/METRON-2201 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Michael Miklavcic >Priority: Minor > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > The description for the IS_IP method behavior needs to corrected as per > implementation , > The description say , >Description: Determine if an string is an IP or not. > Input: > ip - An object which we wish to test is an ip > type (optional) - Object of string or collection type (e.g. list) one of IPV4 > or IPV6 or both. The default is IPV4. > The description say 'The default is IPV4' where as the implementation is it > validates the given input is valid ip (ipv4/ipv6) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2201) The description for the IS_IP method default behavior needs to corrected as per implementation
[ https://issues.apache.org/jira/browse/METRON-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2201: - Assignee: Michael Miklavcic (was: Mohan) > The description for the IS_IP method default behavior needs to corrected as > per implementation > --- > > Key: METRON-2201 > URL: https://issues.apache.org/jira/browse/METRON-2201 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Michael Miklavcic >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > The description for the IS_IP method behavior needs to corrected as per > implementation , > The description say , >Description: Determine if an string is an IP or not. > Input: > ip - An object which we wish to test is an ip > type (optional) - Object of string or collection type (e.g. list) one of IPV4 > or IPV6 or both. The default is IPV4. > The description say 'The default is IPV4' where as the implementation is it > validates the given input is valid ip (ipv4/ipv6) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2201) The description for the IS_IP method default behavior needs to corrected as per implementation
[ https://issues.apache.org/jira/browse/METRON-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2201: - Assignee: Mohan (was: Michael Miklavcic) > The description for the IS_IP method default behavior needs to corrected as > per implementation > --- > > Key: METRON-2201 > URL: https://issues.apache.org/jira/browse/METRON-2201 > Project: Metron > Issue Type: Bug >Reporter: Mohan >Assignee: Mohan >Priority: Minor > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > > The description for the IS_IP method behavior needs to corrected as per > implementation , > The description say , >Description: Determine if an string is an IP or not. > Input: > ip - An object which we wish to test is an ip > type (optional) - Object of string or collection type (e.g. list) one of IPV4 > or IPV6 or both. The default is IPV4. > The description say 'The default is IPV4' where as the implementation is it > validates the given input is valid ip (ipv4/ipv6) -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2227) Kafka test harness has a hardcoded 5 second timeout
[ https://issues.apache.org/jira/browse/METRON-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2227: - Assignee: Dale Richardson (was: Michael Miklavcic) > Kafka test harness has a hardcoded 5 second timeout > --- > > Key: METRON-2227 > URL: https://issues.apache.org/jira/browse/METRON-2227 > Project: Metron > Issue Type: Bug >Reporter: Dale Richardson >Assignee: Dale Richardson >Priority: Minor > Fix For: Next + 1 > > Time Spent: 1h > Remaining Estimate: 0h > > metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java > has a hard coded timeout in the Kafka test harness method: > waitUntilMetadataIsPropagated(String topic, int numPartitions). > On machines that are busy or just a bit slower this surfaces as occasional > test failures. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2227) Kafka test harness has a hardcoded 5 second timeout
[ https://issues.apache.org/jira/browse/METRON-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2227: - Assignee: Michael Miklavcic > Kafka test harness has a hardcoded 5 second timeout > --- > > Key: METRON-2227 > URL: https://issues.apache.org/jira/browse/METRON-2227 > Project: Metron > Issue Type: Bug >Reporter: Dale Richardson >Assignee: Michael Miklavcic >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java > has a hard coded timeout in the Kafka test harness method: > waitUntilMetadataIsPropagated(String topic, int numPartitions). > On machines that are busy or just a bit slower this surfaces as occasional > test failures. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2227) Kafka test harness has a hardcoded 5 second timeout
[ https://issues.apache.org/jira/browse/METRON-2227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2227: -- Fix Version/s: Next + 1 > Kafka test harness has a hardcoded 5 second timeout > --- > > Key: METRON-2227 > URL: https://issues.apache.org/jira/browse/METRON-2227 > Project: Metron > Issue Type: Bug >Reporter: Dale Richardson >Assignee: Michael Miklavcic >Priority: Minor > Fix For: Next + 1 > > Time Spent: 1h > Remaining Estimate: 0h > > metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/KafkaComponent.java > has a hard coded timeout in the Kafka test harness method: > waitUntilMetadataIsPropagated(String topic, int numPartitions). > On machines that are busy or just a bit slower this surfaces as occasional > test failures. > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2229) Stellar shell should not output debugging detail by default
[
https://issues.apache.org/jira/browse/METRON-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16920532#comment-16920532
]
Michael Miklavcic commented on METRON-2229:
---
Good catch - I believe this happens for many of the scripts. I may have copied
the wrong output, but I think this applies to both the zk utils as well as
stellar reply. I'll double check and adjust jiras accordingly.
> Stellar shell should not output debugging detail by default
> ---
>
> Key: METRON-2229
> URL: https://issues.apache.org/jira/browse/METRON-2229
> Project: Metron
> Issue Type: Improvement
>Reporter: Michael Miklavcic
>Priority: Major
>
> When starting up the REPL the user is greeted with many lines of classpath
> and other useful debugging detail. By default, we should not be printing all
> of this to the CLI. This should be a command line option, similar to Maven's
> "-X" command line switch.
> {code:java}
> [root@node1(127.0.0.1 192.168.66.121): ~]
> # $METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z $ZOOKEEPER -i
> $METRON_HOME/config/zookeeper/
> 2019-08-22 20:25:50,625 INFO [main] imps.CuratorFrameworkImpl: Starting
> 2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
> environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
> 2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
> environment:host.name=node1
> 2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.version=1.8.0_222
> 2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.vendor=Oracle Corporation
> 2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre
> 2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
> environment:java.class.path=/usr/metron/0.7.2/lib/metron-parsers-common-0.7.2-uber.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/conf:/usr/jdk64/jdk1.8.0_112/lib/tools.jar:/usr/hdp/current/hbase-client:/usr/hdp/current/hbase-client/lib/activation-1.1.jar:/usr/hdp/current/hbase-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-client/lib/asm-3.1.jar:/usr/hdp/current/hbase-client/lib/avro-1.7.6.jar:/usr/hdp/current/hbase-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hbase-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/hbase-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hbase-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hbase-client/lib/azure-storage-5.4.0.jar:/usr/hdp/current/hbase-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-client/lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-client/lib/commons-compress-1.4.1.jar:/usr/hdp/cur...
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (METRON-2238) Streaming enrichments regression
[
https://issues.apache.org/jira/browse/METRON-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2238:
--
Fix Version/s: Next + 1
> Streaming enrichments regression
>
>
> Key: METRON-2238
> URL: https://issues.apache.org/jira/browse/METRON-2238
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I spun up testing the streaming enrichments use case () as part of testing
> another unrelated PR. The user parser fails with the following stacktrace:
> {code:java}
> 2019-08-27 20:05:59.858 o.a.s.d.executor Thread-12-parserBolt-executor[5 5]
> [ERROR]
> java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1126)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1331)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1439)
> ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1042)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter.write(SimpleHbaseEnrichmentWriter.java:345)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:123)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.applyShouldFlush(BulkWriterComponent.java:179)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flushAll(BulkWriterComponent.java:152)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.WriterHandler.flush(WriterHandler.java:98)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.handleTickTuple(ParserBolt.java:313)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:242)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10252$tuple_action_fn__10254.invoke(executor.clj:735)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:469)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.daemon.executor$fn__10252$fn__10265$fn__10320.invoke(executor.clj:855)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
[jira] [Assigned] (METRON-2149) Shaded jar classifier is not consistent
[ https://issues.apache.org/jira/browse/METRON-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2149: - Assignee: Ryan Merriman > Shaded jar classifier is not consistent > --- > > Key: METRON-2149 > URL: https://issues.apache.org/jira/browse/METRON-2149 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > The Maven Shade plugin is used in several modules for packaging up code > assets into a single, executable jar. There is an option to use a classifier > but it is used the inconsistently: about half of the modules use it while > others don't. This causes inconsistent mvn dependency resolution and it > might be worth investigating a consistent strategy. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2238) Streaming enrichments regression
[
https://issues.apache.org/jira/browse/METRON-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2238:
-
Assignee: Ryan Merriman
> Streaming enrichments regression
>
>
> Key: METRON-2238
> URL: https://issues.apache.org/jira/browse/METRON-2238
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Assignee: Ryan Merriman
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> I spun up testing the streaming enrichments use case () as part of testing
> another unrelated PR. The user parser fails with the following stacktrace:
> {code:java}
> 2019-08-27 20:05:59.858 o.a.s.d.executor Thread-12-parserBolt-executor[5 5]
> [ERROR]
> java.lang.IllegalAccessError: tried to access method
> com.google.common.base.Stopwatch.()V from class
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1126)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1331)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
> ~[stormjar.jar:?]
> at
> org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
> ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1439)
> ~[stormjar.jar:?]
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1042)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter.write(SimpleHbaseEnrichmentWriter.java:345)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:123)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.applyShouldFlush(BulkWriterComponent.java:179)
> [stormjar.jar:?]
> at
> org.apache.metron.writer.BulkWriterComponent.flushAll(BulkWriterComponent.java:152)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.WriterHandler.flush(WriterHandler.java:98)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.handleTickTuple(ParserBolt.java:313)
> [stormjar.jar:?]
> at
> org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:242)
> [stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10252$tuple_action_fn__10254.invoke(executor.clj:735)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:469)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
> at
> org.apache.storm.daemon.executor$fn__10252$fn__10265$fn__10320.invoke(executor.clj:855)
> [storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
>
[jira] [Updated] (METRON-2149) Shaded jar classifier is not consistent
[ https://issues.apache.org/jira/browse/METRON-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2149: -- Fix Version/s: Next + 1 > Shaded jar classifier is not consistent > --- > > Key: METRON-2149 > URL: https://issues.apache.org/jira/browse/METRON-2149 > Project: Metron > Issue Type: Bug >Reporter: Ryan Merriman >Priority: Major > Fix For: Next + 1 > > Time Spent: 4h 20m > Remaining Estimate: 0h > > The Maven Shade plugin is used in several modules for packaging up code > assets into a single, executable jar. There is an option to use a classifier > but it is used the inconsistently: about half of the modules use it while > others don't. This causes inconsistent mvn dependency resolution and it > might be worth investigating a consistent strategy. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2239) Metron Automated backup and restore
[ https://issues.apache.org/jira/browse/METRON-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16918040#comment-16918040 ] Michael Miklavcic commented on METRON-2239: --- We probably want to distinguish between a full platform DR replication type solution and a metron-specific backup. The Hadoop components themselves have steps and tooling for managing those upgrades and/or migrations. I think we should make note of them, and provide some links to the appropriate documentation, but I don't think the Metron project should own those integration points. Here's one such example from when we upgraded Elasticsearch - [https://github.com/apache/metron/pull/840/files#diff-325aea0d364d12c8637eef347ebcfca6R64] > Metron Automated backup and restore > --- > > Key: METRON-2239 > URL: https://issues.apache.org/jira/browse/METRON-2239 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Blocker > > Metron, for upgrading to HDP 3.1 should have the ability to backup and > restore metron specific configurations. > For many, this upgrade will involve OS upgrade and Hadoop/HDP upgrade/Amabari > etc. > Such a tool would: > Backup to file metron configurations from different locations, along with > enough meta data on the files to restore them. The directory/backup location > would be structured. > * Disk on nodes > * HDFS > * Zookeeper > * ?? > This backup will then be archived by the tool, and could then be saved by the > user. > The tool ( or a companion tool ) would be able to take these archives and > restore them back to the cluster. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2240) Missing documentation for streaming enrichment
Michael Miklavcic created METRON-2240: - Summary: Missing documentation for streaming enrichment Key: METRON-2240 URL: https://issues.apache.org/jira/browse/METRON-2240 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic Metron provides a streaming enrichment capability, e.g. [https://cwiki.apache.org/confluence/display/METRON/2016/06/16/Metron+Tutorial+-+Fundamentals+Part+6%3A+Streaming+Enrichment]. This is a core use case for the platform, however it does not appear to have made it into the project READMEs. This feature crosses 3 major project boundaries: # writers (in particular hbase) # enrichment # parsers As such, there are a few places this documentation should have a presence: # In the metron parser module [https://github.com/apache/metron/tree/master/metron-platform/metron-parsing]. Streaming enrichment is managed by a specialized parser topology. # In the metron enrichment module [https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment]. There is no code tied to enrichments, but for the sake of making it more intuitive to find, there should at least be a link to the main documentation in parsers. # In the use cases directory [https://github.com/apache/metron/tree/master/use-cases]. This last case can should probably be handled as part of a bigger effort for migrating the blog use cases to source control. Jira still pending, but it should be linked to this topic. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-614) Eliminate use of the default Charset
[
https://issues.apache.org/jira/browse/METRON-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-614:
Assignee: Michael Miklavcic
> Eliminate use of the default Charset
>
>
> Key: METRON-614
> URL: https://issues.apache.org/jira/browse/METRON-614
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Michael Miklavcic
>Priority: Minor
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> During work on METRON-612, I noticed a lot of our warnings are from implicit
> use of the default Charset (e.g. when creating InputStreamReaders). Given
> that this value is platform dependent, we should at least consider explicitly
> using a particular Charset.
> In addition, after fixing this, I would like to see it upgraded to a compile
> error, because in my experience almost everyone tends to use methods that use
> the implicit default Charset. This can be done by changing the compiler arg
> in the pom to instead be
> {code}
>
> -Xlint:unchecked
> -Xep:DefaultCharset:ERROR
>
> {code}
> My assumption is that we'll want UTF-8, but given a lack of expertise and
> that usually there's a lot of opinions on character set issues, we may want
> to consider other options.
> http://errorprone.info/bugpattern/DefaultCharset
> https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
> standard Charsets and provides more details.
> {code}
> Charset Description
> US-ASCII Seven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
> of the Unicode character set
> ISO-8859-1ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
> UTF-8 Eight-bit UCS Transformation Format
> UTF-16BE Sixteen-bit UCS Transformation Format, big-endian byte order
> UTF-16LE Sixteen-bit UCS Transformation Format, little-endian byte order
> UTF-16Sixteen-bit UCS Transformation Format, byte order identified by
> an optional byte-order mark
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (METRON-614) Eliminate use of the default Charset
[
https://issues.apache.org/jira/browse/METRON-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-614:
Assignee: Justin Leet (was: Michael Miklavcic)
> Eliminate use of the default Charset
>
>
> Key: METRON-614
> URL: https://issues.apache.org/jira/browse/METRON-614
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Justin Leet
>Priority: Minor
> Fix For: Next + 1
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> During work on METRON-612, I noticed a lot of our warnings are from implicit
> use of the default Charset (e.g. when creating InputStreamReaders). Given
> that this value is platform dependent, we should at least consider explicitly
> using a particular Charset.
> In addition, after fixing this, I would like to see it upgraded to a compile
> error, because in my experience almost everyone tends to use methods that use
> the implicit default Charset. This can be done by changing the compiler arg
> in the pom to instead be
> {code}
>
> -Xlint:unchecked
> -Xep:DefaultCharset:ERROR
>
> {code}
> My assumption is that we'll want UTF-8, but given a lack of expertise and
> that usually there's a lot of opinions on character set issues, we may want
> to consider other options.
> http://errorprone.info/bugpattern/DefaultCharset
> https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
> standard Charsets and provides more details.
> {code}
> Charset Description
> US-ASCII Seven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
> of the Unicode character set
> ISO-8859-1ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
> UTF-8 Eight-bit UCS Transformation Format
> UTF-16BE Sixteen-bit UCS Transformation Format, big-endian byte order
> UTF-16LE Sixteen-bit UCS Transformation Format, little-endian byte order
> UTF-16Sixteen-bit UCS Transformation Format, byte order identified by
> an optional byte-order mark
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Updated] (METRON-614) Eliminate use of the default Charset
[
https://issues.apache.org/jira/browse/METRON-614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-614:
-
Fix Version/s: Next + 1
> Eliminate use of the default Charset
>
>
> Key: METRON-614
> URL: https://issues.apache.org/jira/browse/METRON-614
> Project: Metron
> Issue Type: Bug
>Reporter: Justin Leet
>Assignee: Michael Miklavcic
>Priority: Minor
> Fix For: Next + 1
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> During work on METRON-612, I noticed a lot of our warnings are from implicit
> use of the default Charset (e.g. when creating InputStreamReaders). Given
> that this value is platform dependent, we should at least consider explicitly
> using a particular Charset.
> In addition, after fixing this, I would like to see it upgraded to a compile
> error, because in my experience almost everyone tends to use methods that use
> the implicit default Charset. This can be done by changing the compiler arg
> in the pom to instead be
> {code}
>
> -Xlint:unchecked
> -Xep:DefaultCharset:ERROR
>
> {code}
> My assumption is that we'll want UTF-8, but given a lack of expertise and
> that usually there's a lot of opinions on character set issues, we may want
> to consider other options.
> http://errorprone.info/bugpattern/DefaultCharset
> https://docs.oracle.com/javase/8/docs/api/java/nio/charset/Charset.html lists
> standard Charsets and provides more details.
> {code}
> Charset Description
> US-ASCII Seven-bit ASCII, a.k.a. ISO646-US, a.k.a. the Basic Latin block
> of the Unicode character set
> ISO-8859-1ISO Latin Alphabet No. 1, a.k.a. ISO-LATIN-1
> UTF-8 Eight-bit UCS Transformation Format
> UTF-16BE Sixteen-bit UCS Transformation Format, big-endian byte order
> UTF-16LE Sixteen-bit UCS Transformation Format, little-endian byte order
> UTF-16Sixteen-bit UCS Transformation Format, byte order identified by
> an optional byte-order mark
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (METRON-682) Unify and Improve the Flat File Loader
[ https://issues.apache.org/jira/browse/METRON-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917985#comment-16917985 ] Michael Miklavcic commented on METRON-682: -- This appears to have been completed in Feb of 2017. Did we miss closing this out and tagging it in a release? [~cestella] [~justinleet] [~otto] > Unify and Improve the Flat File Loader > -- > > Key: METRON-682 > URL: https://issues.apache.org/jira/browse/METRON-682 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > Currently the flat file loader is deficient in a couple ways: > * It only supports importing local data despite there being a separate, > poorly named, application which supports importing enrichment via MapReduce > called threat_intel_loader.sh > * It does not support local imports from HDFS > * It does not support local imports from URLs > * It does not support importing zipped archives locally > * You cannot import more than one file at once > This JIRA will: > * Unify the MapReduce and local imports into one program and allow the user > to specify the import mode with a CLI flag > * Support local imports from HDFS and URLs > * Support local imports from zipped files > * Support importing more than one file at once -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2239) Metron Automated backup and restore
[ https://issues.apache.org/jira/browse/METRON-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16917304#comment-16917304 ] Michael Miklavcic commented on METRON-2239: --- "Disk on nodes" - can you elaborate on this? Outside of METRON_HOME on the Metron node and some of the Ambari configs, I can't think of much else to backup. > Metron Automated backup and restore > --- > > Key: METRON-2239 > URL: https://issues.apache.org/jira/browse/METRON-2239 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Priority: Blocker > > Metron, for upgrading to HDP 3.1 should have the ability to backup and > restore metron specific configurations. > For many, this upgrade will involve OS upgrade and Hadoop/HDP upgrade/Amabari > etc. > Such a tool would: > Backup to file metron configurations from different locations, along with > enough meta data on the files to restore them. The directory/backup location > would be structured. > * Disk on nodes > * HDFS > * Zookeeper > * ?? > This backup will then be archived by the tool, and could then be saved by the > user. > The tool ( or a companion tool ) would be able to take these archives and > restore them back to the cluster. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2238) Streaming enrichments regression
Michael Miklavcic created METRON-2238:
-
Summary: Streaming enrichments regression
Key: METRON-2238
URL: https://issues.apache.org/jira/browse/METRON-2238
Project: Metron
Issue Type: Bug
Reporter: Michael Miklavcic
I spun up testing the streaming enrichments use case () as part of testing
another unrelated PR. The user parser fails with the following stacktrace:
{code:java}
2019-08-27 20:05:59.858 o.a.s.d.executor Thread-12-parserBolt-executor[5 5]
[ERROR]
java.lang.IllegalAccessError: tried to access method
com.google.common.base.Stopwatch.()V from class
org.apache.hadoop.hbase.zookeeper.MetaTableLocator
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:596)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:580)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.zookeeper.MetaTableLocator.blockUntilAvailable(MetaTableLocator.java:559)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.ZooKeeperRegistry.getMetaRegionLocation(ZooKeeperRegistry.java:61)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateMeta(ConnectionManager.java:1185)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1152)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.relocateRegion(ConnectionManager.java:1126)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1331)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1155)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:370)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:321)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:206)
~[stormjar.jar:?]
at
org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:183)
~[stormjar.jar:?]
at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1439)
~[stormjar.jar:?]
at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1042)
~[stormjar.jar:?]
at
org.apache.metron.writer.hbase.SimpleHbaseEnrichmentWriter.write(SimpleHbaseEnrichmentWriter.java:345)
~[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.flush(BulkWriterComponent.java:123)
[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.applyShouldFlush(BulkWriterComponent.java:179)
[stormjar.jar:?]
at
org.apache.metron.writer.BulkWriterComponent.flushAll(BulkWriterComponent.java:152)
[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.WriterHandler.flush(WriterHandler.java:98)
[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.handleTickTuple(ParserBolt.java:313)
[stormjar.jar:?]
at
org.apache.metron.parsers.bolt.ParserBolt.execute(ParserBolt.java:242)
[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__10252$tuple_action_fn__10254.invoke(executor.clj:735)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:469)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at
org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at
org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at
org.apache.storm.daemon.executor$fn__10252$fn__10265$fn__10320.invoke(executor.clj:855)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484)
[storm-core-1.1.0.2.6.5.1175-1.jar:1.1.0.2.6.5.1175-1]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (METRON-2234) Metron full dev deployment can persist bad state preventing builds
[
https://issues.apache.org/jira/browse/METRON-2234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916214#comment-16916214
]
Michael Miklavcic commented on METRON-2234:
---
Looks like this is related to the following from the spec file.
{code:java}
# allows node dependencies to be packaged in the RPMs
npm install --prefix="%\{buildroot}%\{metron_home}/web/expressjs"
--only=production
{code}
It probably makes sense to keep this sort of packaging detail external to the
RPM build and as a responsibility of the respective Maven modules.
> Metron full dev deployment can persist bad state preventing builds
> --
>
> Key: METRON-2234
> URL: https://issues.apache.org/jira/browse/METRON-2234
> Project: Metron
> Issue Type: Bug
>Reporter: Dale Richardson
>Priority: Minor
>
> In some cases when a deployment is cancelled or halts due to an error, rpm
> build state is left behind in the rpm-docker directory. I've especially
> noticed this when building from a Centos7 box.
> This causes in subsequent maven commands (even a maven clean command) to
> fail, due to (I am assuming) changing permissions of the files when they get
> mapped as Docker RPM/FPM input files.
> The directory of metron-deployment/packaging/docker/rpm-docker/.npm seems to
> be a common culprit.
> Manually deleting the directory before re-running a build seems to fix the
> problem.
> Below is a typical error being reported in this case:
>
> TASK [metron-builder : Clean Metron]
> ***
> changed: [node1 -> localhost] => (item=mvn clean -P HDP-2.5.0.0)
> changed: [node1 -> localhost] => (item=mvn clean -P mpack)
> failed: [node1 -> localhost] (item=mvn clean -P build-rpms) => {"changed":
> true, "cmd": "mvn clean -P build-rpms", "delta": "0:00:04.986465", "end":
> "2019-08-26 20:18:36.774539", "item": "mvn clean -P build-rpms", "msg":
> "non-zero return code", "rc": 1, "start": "2019-08-26 20:18:31.788074",
> "stderr": "", "stderr_lines": [], "stdout": "[INFO] Scanning for
> projects...\n[INFO]
> \n[INFO]
> Reactor Build Order:\n[INFO] \n[INFO] Metron
> [pom]\n[INFO] metron-stellar
> [pom]\n[INFO] stellar-common
> [jar]\n[INFO] metron-analytics
> [pom]\n[INFO] metron-maas-common
> [jar]\n[INFO] metron-platform
> [pom]\n[INFO] metron-zookeeper
> [jar]\n[INFO]
> metron-test-utilities
> [jar]\n[INFO] metron-integration-test
> [jar]\n[INFO] metron-maas-service
> [jar]\n[INFO] metron-common
> [jar]\n[INFO] metron-statistics
> [jar]\n[INFO] metron-common-streaming
> [pom]\n[INFO] metron-common-storm
> [jar]\n[INFO] metron-hbase
> [pom]\n[INFO] metron-hbase-common
> [jar]\n[INFO] metron-enrichment
> [pom]\n[INFO] metron-enrichment-common
> [jar]\n[INFO] metron-writer
> [pom]\n[INFO] metron-writer-common
> [jar]\n[INFO] metron-writer-storm
> [jar]\n[INFO]
> metron-storm-kafka-override
> [jar]\n[INFO] metron-storm-kafka
> [jar]\n[INFO] metron-profiler-common
> [jar]\n[INFO] metron-profiler-client
> [jar]\n[INFO] metron-profiler-storm
> [jar]\n[INFO] metron-profiler-spark
> [jar]\n[INFO] metron-profiler-repl
> [jar]\n[INFO] metron-hbase-client
> [jar]\n[INFO] metron-enrichment-storm
> [jar]\n[INFO] metron-indexing
> [pom]\n[INFO]
[jira] [Created] (METRON-2236) KAFKA_SEEK and KAFKA_FIND missing documentation
Michael Miklavcic created METRON-2236: - Summary: KAFKA_SEEK and KAFKA_FIND missing documentation Key: METRON-2236 URL: https://issues.apache.org/jira/browse/METRON-2236 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic Neither function has documentation in the Stellar README. [https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#stellar-core-functions] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (METRON-2232) Upgrade Hadoop
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915874#comment-16915874 ] Michael Miklavcic edited comment on METRON-2232 at 8/26/19 3:32 PM: Should include tests for: # MaaS # PCAP # Data management - flatfile loader, flatfile summarizer, threatintel taxi loader # Batch profiler load - a new version of Spark should come in at this point was (Author: mmiklavcic): Should include tests for: # MaaS # PCAP # Data management - flatfile loader, flatfile summarizer, threatintel taxi loader > Upgrade Hadoop > -- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Comment Edited] (METRON-2232) Upgrade Hadoop
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915874#comment-16915874 ] Michael Miklavcic edited comment on METRON-2232 at 8/26/19 3:21 PM: Should include tests for: # MaaS # PCAP # Data management - flatfile loader, flatfile summarizer, threatintel taxi loader was (Author: mmiklavcic): Should include tests for MaaS > Upgrade Hadoop > -- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2232) Upgrade Hadoop
[ https://issues.apache.org/jira/browse/METRON-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915874#comment-16915874 ] Michael Miklavcic commented on METRON-2232: --- Should include tests for MaaS > Upgrade Hadoop > -- > > Key: METRON-2232 > URL: https://issues.apache.org/jira/browse/METRON-2232 > Project: Metron > Issue Type: Sub-task >Reporter: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2233) Deprecate Centos 6
Michael Miklavcic created METRON-2233: - Summary: Deprecate Centos 6 Key: METRON-2233 URL: https://issues.apache.org/jira/browse/METRON-2233 Project: Metron Issue Type: Sub-task Reporter: Michael Miklavcic Centos 6 will no longer work for with HDP 3.1. We have a Centos 7 setup that should replace it entirely. We will need to migrate and update any related documentation. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2232) Upgrade Hadoop
Michael Miklavcic created METRON-2232: - Summary: Upgrade Hadoop Key: METRON-2232 URL: https://issues.apache.org/jira/browse/METRON-2232 Project: Metron Issue Type: Sub-task Reporter: Michael Miklavcic -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2231) Revert METRON-2175, METRON-2176, METRON-2177 in HDP 3.1 upgrade feature branch
Michael Miklavcic created METRON-2231: - Summary: Revert METRON-2175, METRON-2176, METRON-2177 in HDP 3.1 upgrade feature branch Key: METRON-2231 URL: https://issues.apache.org/jira/browse/METRON-2231 Project: Metron Issue Type: Sub-task Reporter: Michael Miklavcic -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2169) Upgrade Kafka/Storm
[ https://issues.apache.org/jira/browse/METRON-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2169: - Assignee: Michael Miklavcic (was: Ryan Merriman) > Upgrade Kafka/Storm > --- > > Key: METRON-2169 > URL: https://issues.apache.org/jira/browse/METRON-2169 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Michael Miklavcic >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > To support HDP 3.1, we need to upgrade Kafka to version 2.0.0 and Storm to > version 1.2.1. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2169) Upgrade Kafka/Storm
[ https://issues.apache.org/jira/browse/METRON-2169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2169: - Assignee: Ryan Merriman (was: Michael Miklavcic) > Upgrade Kafka/Storm > --- > > Key: METRON-2169 > URL: https://issues.apache.org/jira/browse/METRON-2169 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > To support HDP 3.1, we need to upgrade Kafka to version 2.0.0 and Storm to > version 1.2.1. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2223) Reconcile Maven versions
[ https://issues.apache.org/jira/browse/METRON-2223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914340#comment-16914340 ] Michael Miklavcic commented on METRON-2223: --- This is a gating task for finishing out the FB > Reconcile Maven versions > > > Key: METRON-2223 > URL: https://issues.apache.org/jira/browse/METRON-2223 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Priority: Major > > It may be useful to upgrade a version independently of other components in > HDP 3.1. We will need to reconcile these versions before the feature branch > is merged into master. This task ensures all Maven profile dependency > versions match the correct HDP version. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2224) Upgrade Zeppelin
[ https://issues.apache.org/jira/browse/METRON-2224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2224: - Assignee: Ryan Merriman (was: Michael Miklavcic) > Upgrade Zeppelin > > > Key: METRON-2224 > URL: https://issues.apache.org/jira/browse/METRON-2224 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > We need to upgrade the Zeppelin dependency from 0.7.3 to 0.8.0. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2225) Upgrade Solr
[ https://issues.apache.org/jira/browse/METRON-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2225: - Assignee: Ryan Merriman > Upgrade Solr > > > Key: METRON-2225 > URL: https://issues.apache.org/jira/browse/METRON-2225 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We need to upgrade Solr from 6.6.2 to 7.4.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2225) Upgrade Solr
[ https://issues.apache.org/jira/browse/METRON-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2225: - Assignee: Michael Miklavcic (was: Ryan Merriman) > Upgrade Solr > > > Key: METRON-2225 > URL: https://issues.apache.org/jira/browse/METRON-2225 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Michael Miklavcic >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We need to upgrade Solr from 6.6.2 to 7.4.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Assigned] (METRON-2225) Upgrade Solr
[ https://issues.apache.org/jira/browse/METRON-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic reassigned METRON-2225: - Assignee: Ryan Merriman (was: Michael Miklavcic) > Upgrade Solr > > > Key: METRON-2225 > URL: https://issues.apache.org/jira/browse/METRON-2225 > Project: Metron > Issue Type: Sub-task >Reporter: Ryan Merriman >Assignee: Ryan Merriman >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We need to upgrade Solr from 6.6.2 to 7.4.0 -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2230) Update typosquatting use case to reflect new OBJECT_GET max file size constraint property in global config
[
https://issues.apache.org/jira/browse/METRON-2230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913686#comment-16913686
]
Michael Miklavcic commented on METRON-2230:
---
The following config option needs to be added to global.json in order for the
[typsquatting|[https://github.com/apache/metron/tree/master/use-cases/typosquat_detection]]
use case to work.
{code:java}
"object.cache.max.file.size" : 500
{code}
Failure to provide the larger maximum file size will result in the following
error:
{code:java}
[Stellar]>>> BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'),
'gogle')
[!] Unable to parse:
BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), 'gogle') due to:
File at path '/tmp/reference/alexa10k_filter.ser' is larger than the configured
max file size of 1048576
org.apache.metron.stellar.dsl.ParseException: Unable to parse:
BLOOM_EXISTS(OBJECT_GET('/tmp/reference/alexa10k_filter.ser'), 'gogle') due to:
File at path '/tmp/reference/alexa10k_filter.ser' is larger than the configured
max file size of 1048576
at
org.apache.metron.stellar.common.BaseStellarProcessor.createException(BaseStellarProcessor.java:166)
at
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:154)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.executeStellar(DefaultStellarShellExecutor.java:407)
at
org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:257)
at
org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:359)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: File at path
'/tmp/reference/alexa10k_filter.ser' is larger than the configured max file
size of 1048576
at
org.apache.metron.enrichment.cache.ObjectCache$Loader.load(ObjectCache.java:72)
at
org.apache.metron.enrichment.cache.ObjectCache$Loader.load(ObjectCache.java:46)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3366)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$14(BoundedLocalCache.java:2039)
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:2037)
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:2020)
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:112)
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67)
at
org.apache.metron.enrichment.cache.ObjectCache.get(ObjectCache.java:82)
at
org.apache.metron.enrichment.stellar.ObjectGet.apply(ObjectGet.java:66)
at
org.apache.metron.stellar.common.StellarCompiler.lambda$exitTransformationFunc$13(StellarCompiler.java:664)
at
org.apache.metron.stellar.common.StellarCompiler$Expression.apply(StellarCompiler.java:259)
at
org.apache.metron.stellar.common.BaseStellarProcessor.parse(BaseStellarProcessor.java:151)
... 7 more
{code}
The default max file size is 1MB while the bloom filter in the use case is over
4MB.
{code:java}
# hdfs dfs -ls /tmp/reference/alexa10k_filter.ser
-rw-r--r-- 1 root hdfs4710254 2019-08-22 19:16
/tmp/reference/alexa10k_filter.ser
{code}
> Update typosquatting use case to reflect new OBJECT_GET max file size
> constraint property in global config
> --
>
> Key: METRON-2230
> URL: https://issues.apache.org/jira/browse/METRON-2230
> Project: Metron
> Issue Type: Bug
>Reporter: Michael Miklavcic
>Priority: Major
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (METRON-2230) Update typosquatting use case to reflect new OBJECT_GET max file size constraint property in global config
Michael Miklavcic created METRON-2230: - Summary: Update typosquatting use case to reflect new OBJECT_GET max file size constraint property in global config Key: METRON-2230 URL: https://issues.apache.org/jira/browse/METRON-2230 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2228) OBJECT_GET cache config options not documented anywhere
[ https://issues.apache.org/jira/browse/METRON-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913683#comment-16913683 ] Michael Miklavcic commented on METRON-2228: --- The typosquatting use case [https://github.com/apache/metron/tree/master/use-cases/typosquat_detection] should also be updated to specify that the global config option "object.cache.max.file.size" should be set to 500 in global.json. > OBJECT_GET cache config options not documented anywhere > --- > > Key: METRON-2228 > URL: https://issues.apache.org/jira/browse/METRON-2228 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Priority: Major > > This traces back to [https://github.com/apache/metron/pull/1399]. Request for > the global config documentation originally pointed out here > [https://github.com/apache/metron/pull/1399#discussion_r286653412]. We missed > it in the review cycle. The documentation for this feature belongs here > [https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration] > and here > [https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#object_get] > This is based on the inline doc here > [https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/stellar/ObjectGet.java#L48] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2229) Stellar shell should not output debugging detail by default
Michael Miklavcic created METRON-2229:
-
Summary: Stellar shell should not output debugging detail by
default
Key: METRON-2229
URL: https://issues.apache.org/jira/browse/METRON-2229
Project: Metron
Issue Type: Improvement
Reporter: Michael Miklavcic
When starting up the REPL the user is greeted with many lines of classpath and
other useful debugging detail. By default, we should not be printing all of
this to the CLI. This should be a command line option, similar to Maven's "-X"
command line switch.
{code:java}
[root@node1(127.0.0.1 192.168.66.121): ~]
# $METRON_HOME/bin/zk_load_configs.sh --mode PUSH -z $ZOOKEEPER -i
$METRON_HOME/config/zookeeper/
2019-08-22 20:25:50,625 INFO [main] imps.CuratorFrameworkImpl: Starting
2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
environment:host.name=node1
2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
environment:java.version=1.8.0_222
2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
environment:java.vendor=Oracle Corporation
2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.222.b10-0.el6_10.x86_64/jre
2019-08-22 20:25:50,647 INFO [main] zookeeper.ZooKeeper: Client
environment:java.class.path=/usr/metron/0.7.2/lib/metron-parsers-common-0.7.2-uber.jar:/usr/hdp/current/hbase-client/lib/hbase-server.jar:/usr/hdp/current/hbase-client/conf:/usr/jdk64/jdk1.8.0_112/lib/tools.jar:/usr/hdp/current/hbase-client:/usr/hdp/current/hbase-client/lib/activation-1.1.jar:/usr/hdp/current/hbase-client/lib/aopalliance-1.0.jar:/usr/hdp/current/hbase-client/lib/apacheds-i18n-2.0.0-M15.jar:/usr/hdp/current/hbase-client/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/hdp/current/hbase-client/lib/api-asn1-api-1.0.0-M20.jar:/usr/hdp/current/hbase-client/lib/api-util-1.0.0-M20.jar:/usr/hdp/current/hbase-client/lib/asm-3.1.jar:/usr/hdp/current/hbase-client/lib/avro-1.7.6.jar:/usr/hdp/current/hbase-client/lib/aws-java-sdk-core-1.10.6.jar:/usr/hdp/current/hbase-client/lib/aws-java-sdk-kms-1.10.6.jar:/usr/hdp/current/hbase-client/lib/aws-java-sdk-s3-1.10.6.jar:/usr/hdp/current/hbase-client/lib/azure-keyvault-core-0.8.0.jar:/usr/hdp/current/hbase-client/lib/azure-storage-5.4.0.jar:/usr/hdp/current/hbase-client/lib/commons-beanutils-1.7.0.jar:/usr/hdp/current/hbase-client/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/current/hbase-client/lib/commons-cli-1.2.jar:/usr/hdp/current/hbase-client/lib/commons-codec-1.9.jar:/usr/hdp/current/hbase-client/lib/commons-collections-3.2.2.jar:/usr/hdp/current/hbase-client/lib/commons-compress-1.4.1.jar:/usr/hdp/cur...
{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (METRON-2228) OBJECT_GET cache config options not documented anywhere
[ https://issues.apache.org/jira/browse/METRON-2228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913666#comment-16913666 ] Michael Miklavcic commented on METRON-2228: --- Note - this may also have to do with unintended side effects of switching from Guava's cache to Caffeine. New OBJECT_GET backing cache implementation: [https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/cache/ObjectCache.java#L21] Original implementation: [https://github.com/apache/metron/blob/14efe83e3a8d9ba7c2bef35061fc9829ee0e61f8/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/stellar/ObjectGet.java#L21] > OBJECT_GET cache config options not documented anywhere > --- > > Key: METRON-2228 > URL: https://issues.apache.org/jira/browse/METRON-2228 > Project: Metron > Issue Type: Bug >Reporter: Michael Miklavcic >Priority: Major > > This traces back to [https://github.com/apache/metron/pull/1399]. Request for > the global config documentation originally pointed out here > [https://github.com/apache/metron/pull/1399#discussion_r286653412]. We missed > it in the review cycle. The documentation for this feature belongs here > [https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration] > and here > [https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#object_get] > This is based on the inline doc here > [https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/stellar/ObjectGet.java#L48] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (METRON-2228) OBJECT_GET cache config options not documented anywhere
Michael Miklavcic created METRON-2228: - Summary: OBJECT_GET cache config options not documented anywhere Key: METRON-2228 URL: https://issues.apache.org/jira/browse/METRON-2228 Project: Metron Issue Type: Bug Reporter: Michael Miklavcic This traces back to [https://github.com/apache/metron/pull/1399]. Request for the global config documentation originally pointed out here [https://github.com/apache/metron/pull/1399#discussion_r286653412]. We missed it in the review cycle. The documentation for this feature belongs here [https://github.com/apache/metron/tree/master/metron-platform/metron-common#global-configuration] and here [https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#object_get] This is based on the inline doc here [https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/metron-enrichment-common/src/main/java/org/apache/metron/enrichment/stellar/ObjectGet.java#L48] -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Updated] (METRON-2212) Add debugging developer docs to hbase-server README
[ https://issues.apache.org/jira/browse/METRON-2212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2212: -- Fix Version/s: Next + 1 > Add debugging developer docs to hbase-server README > --- > > Key: METRON-2212 > URL: https://issues.apache.org/jira/browse/METRON-2212 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2226) Metron GUI testing on Centos7 requires explicit xvfb install
[ https://issues.apache.org/jira/browse/METRON-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913338#comment-16913338 ] Michael Miklavcic commented on METRON-2226: --- Is this issue exclusive to the Centos 7 box, or have you run into this on Centos 6 as well? > Metron GUI testing on Centos7 requires explicit xvfb install > > > Key: METRON-2226 > URL: https://issues.apache.org/jira/browse/METRON-2226 > Project: Metron > Issue Type: Bug >Reporter: Dale Richardson >Priority: Minor > > Just noticed setting up a recent centos7 to be a build box for Metron. > The headless GUI testing requires xvfb to be installed, and it does not > appear to be documented as a dependency anywhere. > The error appears during test for Metron-alerts as follows: > > *INFO*] > metron-alerts@0.7.2 cypress:run > /home/dale/metron/metron-interface/metron-alerts > [*INFO*] > cypress run > [*INFO*] > [*INFO*] [HPM] Proxy created: / -> http://localhost:4200 > [*INFO*] [HPM] Proxy created: / -> http://localhost:4200 > [*INFO*] Metron alerts ui is listening on > [*INFO*] http://127.0.0.1:4200 > [*INFO*] http://192.168.0.17:4200 > [*INFO*] It looks like this is your first time using Cypress: 3.1.0 > [*INFO*] > [*INFO*] [13:45:51] Verifying Cypress can run > /home/dale/.cache/Cypress/3.1.0/Cypress [started] > [*INFO*] [13:45:51] Verifying Cypress can run > /home/dale/.cache/Cypress/3.1.0/Cypress [failed] > [*INFO*] [13:45:51] → Your system is missing the dependency: XVFB > [*INFO*] > [*INFO*] Install XVFB and run Cypress again. > [*INFO*] > [*INFO*] Read our documentation on dependencies for more information: > [*INFO*] > [*INFO*] https://on.cypress.io/required-dependencies > [*INFO*] > [*INFO*] If you are using Docker, we provide containers with all required > dependencies installed. > [*INFO*] -- > [*INFO*] > [*INFO*] Caught error trying to run XVFB: "Your system is missing the > dependency: XVFB > [*INFO*] > [*INFO*] Install XVFB and run Cypress again. > [*INFO*] > [*INFO*] Read our documentation on dependencies for more information: > [*INFO*] > [*INFO*] https://on.cypress.io/required-dependencies > [*INFO*] > [*INFO*] If you are using Docker, we provide containers with all required > dependencies installed. > [*INFO*] -- > [*INFO*] > [*INFO*] Error: spawn Xvfb ENOENT > [*INFO*] -- > [*INFO*] > [*INFO*] Platform: linux (Centos - 7.6.1810) > [*INFO*] Cypress Version: 3.1.0" > [*INFO*] -- > [*INFO*] > [*INFO*] Platform: linux (Centos - 7.6.1810) > [*INFO*] Cypress Version: 3.1.0 > [*INFO*] Your system is missing the dependency: XVFB > [*INFO*] > [*INFO*] Install XVFB and run Cypress again. > [*INFO*] > [*INFO*] Read our documentation on dependencies for more information: > [*INFO*] > [*INFO*] https://on.cypress.io/required-dependencies > [*INFO*] > [*INFO*] If you are using Docker, we provide containers with all required > dependencies installed. > [*INFO*] -- > [*INFO*] > [*INFO*] Caught error trying to run XVFB: "Your system is missing the > dependency: XVFB > [*INFO*] > [*INFO*] Install XVFB and run Cypress again. > [*INFO*] > [*INFO*] Read our documentation on dependencies for more information: > [*INFO*] > [*INFO*] https://on.cypress.io/required-dependencies > [*INFO*] > [*INFO*] If you are using Docker, we provide containers with all required > dependencies installed. > [*INFO*] -- > [*INFO*] > [*INFO*] Error: spawn Xvfb ENOENT > [*INFO*] -- > [*INFO*] > [*INFO*] Platform: linux (Centos - 7.6.1810) > [*INFO*] Cypress Version: 3.1.0" > [*INFO*] -- > [*INFO*] > [*INFO*] Platform: linux (Centos - 7.6.1810) > [*INFO*] Cypress Version: 3.1.0 > [*ERROR*] npm ERR! code ELIFECYCLE > [*ERROR*] npm ERR! errno 1 > [*ERROR*] npm ERR! metron-alerts@0.7.2 cypress:run: `cypress run` > [*ERROR*] npm ERR! Exit status 1 > [*ERROR*] npm ERR! > [*ERROR*] npm ERR! Failed at the metron-alerts@0.7.2 cypress:run script. > [*ERROR*] npm ERR! This is probably not a problem with npm. There is likely > additional logging output above. > [*ERROR*] > [*ERROR*] npm ERR! A complete log of this run can be found in: > [*ERROR*] npm ERR! > /home/dale/.npm/_logs/2019-08-22T03_45_51_932Z-debug.log > [*ERROR*] ERROR: "cypress:run" exited with 1. > [*ERROR*] npm ERR! code ELIFECYCLE > [*ERROR*] npm ERR! errno 1 > [*ERROR*] npm ERR! metron-alerts@0.7.2 cypress:ci: `ng build --prod && run-p > --race start:ci cypress:run` > [*ERROR*] npm ERR! Exit status 1 > [*ERROR*] npm ERR! > [*ERROR*] npm ERR! Failed at the metron-alerts@0.7.2 cypress:ci script. > [*ERROR*] npm ERR! This is probably not a problem with npm. There is likely > additional logging output above. > [*ERROR*] > [*ERROR*] npm ERR! A complete log of this run can be found in: >
[jira] [Updated] (METRON-2076) Stellar timezone test failure
[
https://issues.apache.org/jira/browse/METRON-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2076:
--
Fix Version/s: Next + 1
> Stellar timezone test failure
> -
>
> Key: METRON-2076
> URL: https://issues.apache.org/jira/browse/METRON-2076
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Dale Richardson
>Assignee: Michael Miklavcic
>Priority: Minor
> Fix For: Next + 1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I notice that testDateFormatDefaultTimezone is failing on my machine (MacOS
> Mojave)
> public void testDateFormatDefaultTimezone() {
> Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ', epoch)");
> // Thu Aug 25 2016 11:27:10 Australian Eastern Standard Time (New South
> Wales)"
> assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
> 1)));
> //Australian Eastern Daylight Time (New South Wales)
> }
>
> # sudo systemsetup -gettimezone
> Time Zone: Australia/Sydney
>
> # echo `date -u` `date +%z`
> Mon 15 Apr 2019 12:05:06 UTC +1000
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (METRON-2076) Stellar timezone test failure
[
https://issues.apache.org/jira/browse/METRON-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2076:
-
Assignee: Dale Richardson (was: Michael Miklavcic)
> Stellar timezone test failure
> -
>
> Key: METRON-2076
> URL: https://issues.apache.org/jira/browse/METRON-2076
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Dale Richardson
>Assignee: Dale Richardson
>Priority: Minor
> Fix For: Next + 1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I notice that testDateFormatDefaultTimezone is failing on my machine (MacOS
> Mojave)
> public void testDateFormatDefaultTimezone() {
> Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ', epoch)");
> // Thu Aug 25 2016 11:27:10 Australian Eastern Standard Time (New South
> Wales)"
> assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
> 1)));
> //Australian Eastern Daylight Time (New South Wales)
> }
>
> # sudo systemsetup -gettimezone
> Time Zone: Australia/Sydney
>
> # echo `date -u` `date +%z`
> Mon 15 Apr 2019 12:05:06 UTC +1000
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (METRON-2076) Stellar timezone test failure
[
https://issues.apache.org/jira/browse/METRON-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2076:
-
Assignee: Dale Richardson
> Stellar timezone test failure
> -
>
> Key: METRON-2076
> URL: https://issues.apache.org/jira/browse/METRON-2076
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Dale Richardson
>Assignee: Dale Richardson
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I notice that testDateFormatDefaultTimezone is failing on my machine (MacOS
> Mojave)
> public void testDateFormatDefaultTimezone() {
> Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ', epoch)");
> // Thu Aug 25 2016 11:27:10 Australian Eastern Standard Time (New South
> Wales)"
> assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
> 1)));
> //Australian Eastern Daylight Time (New South Wales)
> }
>
> # sudo systemsetup -gettimezone
> Time Zone: Australia/Sydney
>
> # echo `date -u` `date +%z`
> Mon 15 Apr 2019 12:05:06 UTC +1000
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Assigned] (METRON-2076) Stellar timezone test failure
[
https://issues.apache.org/jira/browse/METRON-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic reassigned METRON-2076:
-
Assignee: Michael Miklavcic (was: Dale Richardson)
> Stellar timezone test failure
> -
>
> Key: METRON-2076
> URL: https://issues.apache.org/jira/browse/METRON-2076
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.0
>Reporter: Dale Richardson
>Assignee: Michael Miklavcic
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I notice that testDateFormatDefaultTimezone is failing on my machine (MacOS
> Mojave)
> public void testDateFormatDefaultTimezone() {
> Object result = run("DATE_FORMAT('EEE MMM dd hh:mm:ss ', epoch)");
> // Thu Aug 25 2016 11:27:10 Australian Eastern Standard Time (New South
> Wales)"
> assertTrue(result.toString().endsWith(TimeZone.getDefault().getDisplayName(true,
> 1)));
> //Australian Eastern Daylight Time (New South Wales)
> }
>
> # sudo systemsetup -gettimezone
> Time Zone: Australia/Sydney
>
> # echo `date -u` `date +%z`
> Mon 15 Apr 2019 12:05:06 UTC +1000
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Created] (METRON-2222) Deprecate Storm 1.x
Michael Miklavcic created METRON-: - Summary: Deprecate Storm 1.x Key: METRON- URL: https://issues.apache.org/jira/browse/METRON- Project: Metron Issue Type: Sub-task Reporter: Michael Miklavcic There isn't much to this task other than calling out the major Storm version deprecation once this upgrade makes its way to master. We should double-check that Upgrading.md reflects the current state of changes from this feature branch. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2211) [UI] Alerts UI should optionally render timestamp in local time
[ https://issues.apache.org/jira/browse/METRON-2211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911681#comment-16911681 ] Michael Miklavcic commented on METRON-2211: --- Linking this to 2147 and closing this as a duplicate > [UI] Alerts UI should optionally render timestamp in local time > --- > > Key: METRON-2211 > URL: https://issues.apache.org/jira/browse/METRON-2211 > Project: Metron > Issue Type: Improvement >Reporter: Shane Ardell >Priority: Major > > Currently, timestamps show in UTC time when the timestamp field is included > as a column header. This could be confusing for a user seeing alerts "from > the future" because their local time is behind UTC time. Even if there is no > confusion, a user might prefer seeing alerts based on their local time vs UTC. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (METRON-2217) Migrate current HBase client from HTableInterface to Table
[ https://issues.apache.org/jira/browse/METRON-2217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908592#comment-16908592 ] Michael Miklavcic commented on METRON-2217: --- We're currently supporting HBase 1.1.2 and leveraging the HTable and HTableInterface classes. We're currently undergoing a major upgrade from HDP 2.x to HDP 3.1. The new version of HBase supported is in the 2.x major revision, which has now completely removed client endpoints that had been deprecated in 1.x. See the attached links for more details on the interface deprecations. The purpose of this PR is to provide a limited, incremental change to our base HBase client library independent of the current HDP upgrade. The goal here is to enable an incremental approach that allows us to minimize making larger broad-sweeping architectural changes in conjunction with an already-risky major upgrade path. > Migrate current HBase client from HTableInterface to Table > -- > > Key: METRON-2217 > URL: https://issues.apache.org/jira/browse/METRON-2217 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (METRON-2217) Migrate current HBase client from HTableInterface to Table
Michael Miklavcic created METRON-2217: - Summary: Migrate current HBase client from HTableInterface to Table Key: METRON-2217 URL: https://issues.apache.org/jira/browse/METRON-2217 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (METRON-2212) Add debugging developer docs to hbase-server README
Michael Miklavcic created METRON-2212: - Summary: Add debugging developer docs to hbase-server README Key: METRON-2212 URL: https://issues.apache.org/jira/browse/METRON-2212 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Assignee: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (METRON-2203) Improve reverse-lookup of Stellar functions in the code base
Michael Miklavcic created METRON-2203: - Summary: Improve reverse-lookup of Stellar functions in the code base Key: METRON-2203 URL: https://issues.apache.org/jira/browse/METRON-2203 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic Stellar functions provide a powerful, flexible annotation API that enables developers to add context to function names by also including a namespace. This also makes it challenging at times to find the appropriate classes and static subclasses that correspond to a function. We should look into solutions to make it easier to trace back a function to its fully qualified classname. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (METRON-2195) Add defensive log level checks when constructing logs is expensive
[ https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2195: -- Fix Version/s: Next + 1 > Add defensive log level checks when constructing logs is expensive > -- > > Key: METRON-2195 > URL: https://issues.apache.org/jira/browse/METRON-2195 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Dale Richardson >Priority: Critical > Fix For: Next + 1 > > Time Spent: 2.5h > Remaining Estimate: 0h > > There are instances where we log, and output strings for json objects and > other things that are quite expensive. > These are done regardless of the log level being enabled and can increase > performance significantly: > https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77 > https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127 > We need to find places where this happens, and employ the best practice check > for the log level being enabled ( best practice if your parameters or log > message construction takes a lot of time ). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (METRON-2202) Add parameter validation for the stellar field validation functions
[
https://issues.apache.org/jira/browse/METRON-2202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2202:
--
Fix Version/s: Next + 1
> Add parameter validation for the stellar field validation functions
> ---
>
> Key: METRON-2202
> URL: https://issues.apache.org/jira/browse/METRON-2202
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Some of the field validation functions in the stellar returns true for empty
> parameters ,
> {code:java}
> [Stellar]>>> IS_DOMAIN()
> true
> [Stellar]>>> IS_EMAIL()
> true
> [Stellar]>>> IS_URL()
> true
> [Stellar]>>> IS_INTEGER()
> true
> {code}
> We should add a invalid number of arguments exception or should just return
> false.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Updated] (METRON-2197) Add debugging info output for Solr queries
[ https://issues.apache.org/jira/browse/METRON-2197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2197: -- Fix Version/s: Next + 1 > Add debugging info output for Solr queries > -- > > Key: METRON-2197 > URL: https://issues.apache.org/jira/browse/METRON-2197 > Project: Metron > Issue Type: Improvement >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Updated] (METRON-2189) Optimize imports in mpack python scripts
[ https://issues.apache.org/jira/browse/METRON-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Miklavcic updated METRON-2189: -- Fix Version/s: Next + 1 > Optimize imports in mpack python scripts > > > Key: METRON-2189 > URL: https://issues.apache.org/jira/browse/METRON-2189 > Project: Metron > Issue Type: Task >Reporter: Michael Miklavcic >Assignee: Michael Miklavcic >Priority: Major > Fix For: Next + 1 > > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (METRON-2195) Add defensive log level checks when constructing logs is expensive
[ https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896427#comment-16896427 ] Michael Miklavcic commented on METRON-2195: --- I created and linked a separate ticket for the general issue of addressing efficient serialization and deserialization. The 34 toJSONString instances I turned up do not have any overlap with this logging Jira. I think they can be addressed independently and each have their own potential solutions and benefits. > Add defensive log level checks when constructing logs is expensive > -- > > Key: METRON-2195 > URL: https://issues.apache.org/jira/browse/METRON-2195 > Project: Metron > Issue Type: Improvement >Reporter: Otto Fowler >Assignee: Dale Richardson >Priority: Critical > Time Spent: 1.5h > Remaining Estimate: 0h > > There are instances where we log, and output strings for json objects and > other things that are quite expensive. > These are done regardless of the log level being enabled and can increase > performance significantly: > https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77 > https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127 > We need to find places where this happens, and employ the best practice check > for the log level being enabled ( best practice if your parameters or log > message construction takes a lot of time ). -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (METRON-2198) Optimize JSON serialization and deserialization
Michael Miklavcic created METRON-2198: - Summary: Optimize JSON serialization and deserialization Key: METRON-2198 URL: https://issues.apache.org/jira/browse/METRON-2198 Project: Metron Issue Type: Improvement Reporter: Michael Miklavcic -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Commented] (METRON-2195) Add defensive log level checks when constructing logs is expensive
[
https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896363#comment-16896363
]
Michael Miklavcic commented on METRON-2195:
---
{quote}We have other on demand testing code for performance I think, this could
be like that code and not run as part of the build{quote}
[~otto] not an unreasonable suggestion, I'd be supportive of this approach as
well. I did this when developing the HLLP Stellar implementation, e.g. -
[https://github.com/apache/metron/blob/master/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HLLPMeasurement.java|https://github.com/apache/metron/blob/master/metron-analytics/metron-statistics/src/main/java/org/apache/metron/statistics/approximation/HLLPMeasurement.java]
> Add defensive log level checks when constructing logs is expensive
> --
>
> Key: METRON-2195
> URL: https://issues.apache.org/jira/browse/METRON-2195
> Project: Metron
> Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Dale Richardson
>Priority: Critical
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There are instances where we log, and output strings for json objects and
> other things that are quite expensive.
> These are done regardless of the log level being enabled and can increase
> performance significantly:
> https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
> We need to find places where this happens, and employ the best practice check
> for the log level being enabled ( best practice if your parameters or log
> message construction takes a lot of time ).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (METRON-2195) Add defensive log level checks when constructing logs is expensive
[
https://issues.apache.org/jira/browse/METRON-2195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16896360#comment-16896360
]
Michael Miklavcic commented on METRON-2195:
---
{quote}My next question is does anybody have a prioritised list of performance
sensitive areas I should be hunting through to update to the new logging
wrapper?
{quote}
I would look at the writers, for starters. Beyond that, I think it's going to
be a trial and error exercise. Minimally, address the HDFSWriter logging
statement in the original example. One option that's a bit brute force could be
doing a grep in the full code base on LOG.trace( and LOG.debug( and looking for
any obvious complex calls. I just ran this locally and trace appears to be
quite manageable. grep --include *.java -R LOG.trace .|wc -l
50
Debug came up with 264 results, but that's still not egregious to look for low
hanging fruit. I agree with your conclusion that we should not worry as much
about "error" logging, because those are typically already gated by try/catch
blocks.
One additional note about this API - I think we need to be conspicuously clear
that this is a temporary solution and add appropriate javadoc warnings to the
interface that references this ticket. We might also consider adopting the
Hadoop API annotation approach (i.e. @ Unstable), but that's probably overkill
for a private API. The last piece of this should be to create a Jira for
upgrading SLF4J. Here's an example for upgrading JUnit -
https://issues.apache.org/jira/browse/METRON-2037
> Add defensive log level checks when constructing logs is expensive
> --
>
> Key: METRON-2195
> URL: https://issues.apache.org/jira/browse/METRON-2195
> Project: Metron
> Issue Type: Improvement
>Reporter: Otto Fowler
>Assignee: Dale Richardson
>Priority: Critical
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There are instances where we log, and output strings for json objects and
> other things that are quite expensive.
> These are done regardless of the log level being enabled and can increase
> performance significantly:
> https://gist.github.com/mmiklavc/7fd6af13bfa0ca05d9b3f4e7806c8d77
> https://github.com/apache/metron/blob/master/metron-platform/metron-writer/metron-writer-storm/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java#L127
> We need to find places where this happens, and employ the best practice check
> for the log level being enabled ( best practice if your parameters or log
> message construction takes a lot of time ).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
