[jira] [Created] (METRON-1582) after installation of metron in ubuntu my system is hang
manisha tank created METRON-1582: Summary: after installation of metron in ubuntu my system is hang Key: METRON-1582 URL: https://issues.apache.org/jira/browse/METRON-1582 Project: Metron Issue Type: Bug Affects Versions: 0.4.2 Reporter: manisha tank Fix For: 0.4.2 I installed single node Apache metron in Ubuntu Operating system. Apache metron system require VM assigned 20 GB RAM, 100GB disk, and 2 CPUs. we already assign 32GB RAM in our system. I used all prerequisite require of metron. once installation complete I ingested sample squid logs inside apache metron. Once i ingested squid logs my sytem taked much more burden and loads. My full system is hang. I can't do any work in my system. Anybody have idea about this issue -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1583) issue regarding cisco asa logs
manisha tank created METRON-1583:
Summary: issue regarding cisco asa logs
Key: METRON-1583
URL: https://issues.apache.org/jira/browse/METRON-1583
Project: Metron
Issue Type: Bug
Affects Versions: 0.4.2
Reporter: manisha tank
Fix For: 0.4.2
I am trying to ingest cisco asa logs but I am facing some issue .
I have created log pattern below
CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
<%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
%\{GREEDYDATA:message}
CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
sample logs
Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL Server
172.19.83.105 not responding
Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped UDP
DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
length 932 bytes exceeds configured limit of 512 bytes
Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
configured rate is 10; Current average rate is 103 per second, max configured
rate is 5; Cumulative total count is 62196
Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
configured rate is 40; Current average rate is 119 per second, max configured
rate is 20; Cumulative total count is 71776
Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
with different initial sequence number
PFA error facing while inegesting cisco asa logs
!cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manisha tank updated METRON-1583:
-
Attachment: cisco_asa_logs_error.png
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_asa_logs_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1015: METRON-1544 Flaky test: org.apache.metron.stellar.common...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1015 lgtm, +1 ---
[jira] [Commented] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493834#comment-16493834
]
ASF GitHub Bot commented on METRON-1544:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1015
lgtm, +1
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1033: METRON-1579: Stellar should return the expression...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1033#discussion_r191513381
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
---
@@ -92,6 +96,19 @@ public String getStreamSubGroup(String enrichmentType,
String field) {
}
}
--- End diff --
Yeah, I agree. I'll move that.
---
[jira] [Commented] (METRON-1579) Stellar should return the expression that failed in the exception
[
https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493935#comment-16493935
]
ASF GitHub Bot commented on METRON-1579:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1033#discussion_r191513381
--- Diff:
metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/adapters/stellar/StellarAdapter.java
---
@@ -92,6 +96,19 @@ public String getStreamSubGroup(String enrichmentType,
String field) {
}
}
--- End diff --
Yeah, I agree. I'll move that.
> Stellar should return the expression that failed in the exception
> -
>
> Key: METRON-1579
> URL: https://issues.apache.org/jira/browse/METRON-1579
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> There are situations where we are not including the expression in the
> exception. Also, in stellar enrichments, we should include the relevant
> variable values in the exception to help with diagnosing issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1033: METRON-1579: Stellar should return the expression...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1033#discussion_r191513907
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/BaseStellarProcessor.java
---
@@ -143,7 +143,11 @@ public T parse(final String rule, final
VariableResolver variableResolver, final
try {
return clazz.cast(expression
.apply(new StellarCompiler.ExpressionState(context,
functionResolver, variableResolver)));
-}finally {
+}
--- End diff --
Yeah, agreed.
---
[GitHub] metron issue #1033: METRON-1579: Stellar should return the expression that f...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1033 @ottobackwards That makes sense. How do you like it now? ---
[jira] [Commented] (METRON-1579) Stellar should return the expression that failed in the exception
[
https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493939#comment-16493939
]
ASF GitHub Bot commented on METRON-1579:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1033#discussion_r191513907
--- Diff:
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/BaseStellarProcessor.java
---
@@ -143,7 +143,11 @@ public T parse(final String rule, final
VariableResolver variableResolver, final
try {
return clazz.cast(expression
.apply(new StellarCompiler.ExpressionState(context,
functionResolver, variableResolver)));
-}finally {
+}
--- End diff --
Yeah, agreed.
> Stellar should return the expression that failed in the exception
> -
>
> Key: METRON-1579
> URL: https://issues.apache.org/jira/browse/METRON-1579
> Project: Metron
> Issue Type: Improvement
>Reporter: Casey Stella
>Priority: Major
>
> There are situations where we are not including the expression in the
> exception. Also, in stellar enrichments, we should include the relevant
> variable values in the exception to help with diagnosing issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1579) Stellar should return the expression that failed in the exception
[ https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493940#comment-16493940 ] ASF GitHub Bot commented on METRON-1579: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1033 @ottobackwards That makes sense. How do you like it now? > Stellar should return the expression that failed in the exception > - > > Key: METRON-1579 > URL: https://issues.apache.org/jira/browse/METRON-1579 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > There are situations where we are not including the expression in the > exception. Also, in stellar enrichments, we should include the relevant > variable values in the exception to help with diagnosing issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (METRON-1584) Indexing Topology Crashes with Invalid Message
Nick Allen created METRON-1584:
--
Summary: Indexing Topology Crashes with Invalid Message
Key: METRON-1584
URL: https://issues.apache.org/jira/browse/METRON-1584
Project: Metron
Issue Type: Bug
Reporter: Nick Allen
Per Mohan Venkateshaiah:
I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see that
the random access indexing topology worker thread died and couldn't recover
until the kafka topic was deleted and recreated.
{code:java}
Caused by: java.lang.IllegalStateException: Unable to parse
adkadknalkda;LK;ad;Da;dD; due to null
at
org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
~[stormjar.jar:?]
at
org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
~[stormjar.jar:?]
at
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
~[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
... 6 more
Caused by: org.json.simple.parser.ParseException
at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
~[stormjar.jar:?]
at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
~[stormjar.jar:?]
at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
~[stormjar.jar:?]
at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
~[stormjar.jar:?]
at
org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
~[stormjar.jar:?]
at
org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
~[stormjar.jar:?]
at
org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
~[stormjar.jar:?]
at
org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
... 6 more
2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3] [ERROR]
Halting process: ("Worker died")
java.lang.RuntimeException: ("Worker died")
at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at
org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Shutting down worker
random_access_indexing-24-1527147389 703b5bf7-6c9d-46f3-8136-0c4877a69375 6700
2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Terminating messaging
context
2018-05-24 09:21:22.238 o.a.s.d.worker Thread-20 [INFO] Shutting down executors
2018-05-24 09:21:22.238 o.a.s.d.executor Thread-20 [INFO] Shutting down
executor
__metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
2018-05-24 09:21:22.239 o.a.s.util Thread-6-disruptor-executor[2 2]-send-queue
[INFO] Async loop interrupted!
2018-05-24 09:21:22.239 o.a.s.util
Thread-7-__metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink-executor[2
2] [INFO] Async loop interrupted!
2018-05-24 09:21:22.240 o.a.h.m.s.s.StormTimelineMetricsSink Thread-20 [INFO]
Stopping Storm Metrics Sink
2018-05-24 09:21:22.240 o.a.s.d.executor Thread-20 [INFO] Shut down executor
__metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
2018-05-24 09:21:22.241 o.a.s.d.executor Thread-20 [INFO] Shutting down
executor indexingBolt:[3 3]
{code}
--
This message was
[jira] [Assigned] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Allen reassigned METRON-1584:
--
Assignee: Nick Allen
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> 2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Shutting down worker
> random_access_indexing-24-1527147389 703b5bf7-6c9d-46f3-8136-0c4877a69375 6700
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Terminating messaging
> context
> 2018-05-24 09:21:22.238 o.a.s.d.worker Thread-20 [INFO] Shutting down
> executors
> 2018-05-24 09:21:22.238 o.a.s.d.executor Thread-20 [INFO] Shutting down
> executor
> __metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink:[2 2]
> 2018-05-24 09:21:22.239 o.a.s.util Thread-6-disruptor-executor[2
> 2]-send-queue [INFO] Async loop interrupted!
> 2018-05-24 09:21:22.239 o.a.s.util
> Thread-7-__metricsorg.apache.hadoop.metrics2.sink.storm.StormTimelineMetricsSink-executor
[GitHub] metron issue #1031: METRON-1577: Solr searches don't include the index of th...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1031 @merrimanr can you close this PR since it's been committed? ---
[jira] [Commented] (METRON-1577) Solr searches don't include the index of the result
[
https://issues.apache.org/jira/browse/METRON-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493968#comment-16493968
]
ASF GitHub Bot commented on METRON-1577:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1031
@merrimanr can you close this PR since it's been committed?
> Solr searches don't include the index of the result
> ---
>
> Key: METRON-1577
> URL: https://issues.apache.org/jira/browse/METRON-1577
> Project: Metron
> Issue Type: Sub-task
>Reporter: Ryan Merriman
>Assignee: Ryan Merriman
>Priority: Major
>
> For example
> {code:java}
> {
> "total": 370,
> "results": [
> {
> "id": "1dcf6e7e-9d16-477b-990e-e734bd400101",
> "source":
> {
> "guid": "1dcf6e7e-9d16-477b-990e-e734bd400101"
> },
> "score": 0,
> "index": null
> }
> ],
> "facetCounts": null
> }{code}
> We should also make sure that any other endpoints (if there are any) that
> return index, are populated properly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/1036 METRON-1584 Indexing Topology Crashes with Invalid Message If the indexing topology receives a message containing invalid JSON, the topology will crash. The topology needs to handle these types of errors and continue processing. ## Changes * Updated the `BulkWriterComponent` to allow tuples that do not contain valid JSON messages to be error'd out. * Updated `BulkMessageWriterBolt` to handle the case where a tuple does not contain a valid JSON message. * Updated the BulkMessageWriterBolt unit tests to validate that error'd messages are being ack'd and emitted on the correct error stream. Previously a mock was being used that hid a bug with the `BulkWriterComponent`. * Added a unit test for the case where an invalid message is sent as input to the indexing topology. ## Testing 1. Spin-up the development environment. 1. Ensure that messages are successfully being indexing by running the Service Check and/or validating the Alerts UI. 1. Write an invalid message to the "indexing" topic. 1. Ensure the invalid message is sent to the "error" topic. 1. Ensure the indexing topology did not crash. ## Pull Request Checklist - [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [ ] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? - [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [ ] Have you included steps or a guide to how the change may be verified and tested manually? - [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: - [ ] Have you written or updated unit tests and or integration tests to verify your changes? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1584 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1036.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1036 commit 765e05264b378a0a7608092e5842a31239418ae5 Author: Nick Allen Date: 2018-05-29T17:49:31Z METRON-1584 Indexing Topology Crashes with Invalid Message ---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493980#comment-16493980
]
ASF GitHub Bot commented on METRON-1584:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1036
METRON-1584 Indexing Topology Crashes with Invalid Message
If the indexing topology receives a message containing invalid JSON, the
topology will crash. The topology needs to handle these types of errors and
continue processing.
## Changes
* Updated the `BulkWriterComponent` to allow tuples that do not contain
valid JSON messages to be error'd out.
* Updated `BulkMessageWriterBolt` to handle the case where a tuple does not
contain a valid JSON message.
* Updated the BulkMessageWriterBolt unit tests to validate that error'd
messages are being ack'd and emitted on the correct error stream. Previously a
mock was being used that hid a bug with the `BulkWriterComponent`.
* Added a unit test for the case where an invalid message is sent as input
to the indexing topology.
## Testing
1. Spin-up the development environment.
1. Ensure that messages are successfully being indexing by running the
Service Check and/or validating the Alerts UI.
1. Write an invalid message to the "indexing" topic.
1. Ensure the invalid message is sent to the "error" topic.
1. Ensure the indexing topology did not crash.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1584
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1036.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1036
commit 765e05264b378a0a7608092e5842a31239418ae5
Author: Nick Allen
Date: 2018-05-29T17:49:31Z
METRON-1584 Indexing Topology Crashes with Invalid Message
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
[GitHub] metron issue #1015: METRON-1544 Flaky test: org.apache.metron.stellar.common...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1015 +1 for sure ---
[jira] [Commented] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493982#comment-16493982
]
ASF GitHub Bot commented on METRON-1544:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1015
+1 for sure
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1014 Should assignment be `=` or `:=`? I'm not super opinionated other than we have been using `:=` and we'd break existing installations if we move to `=`, right? ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16493999#comment-16493999 ] ASF GitHub Bot commented on METRON-1563: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1014 Should assignment be `=` or `:=`? I'm not super opinionated other than we have been using `:=` and we'd break existing installations if we move to `=`, right? > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1015: METRON-1544 Flaky test: org.apache.metron.stellar...
Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/1015 ---
[jira] [Commented] (METRON-1544) Flaky test: org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
[
https://issues.apache.org/jira/browse/METRON-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494013#comment-16494013
]
ASF GitHub Bot commented on METRON-1544:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1015
> Flaky test:
> org.apache.metron.stellar.common.CachingStellarProcessorTest#testCaching
>
>
> Key: METRON-1544
> URL: https://issues.apache.org/jira/browse/METRON-1544
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.3
> Environment: #uname -a
> Linux 60a83dc7a2ce 4.4.0-112-generic #135-Ubuntu SMP Fri Jan 19 11:48:36 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
>Reporter: Pravin Dsilva
>Assignee: Nick Allen
>Priority: Major
>
> command used: mvn test
> The test fails intermittently on master branch.
> {code:java}
> Running org.apache.metron.stellar.common.CachingStellarProcessorTest
> Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.054 sec <<<
> FAILURE! - in org.apache.metron.stellar.common.CachingStellarProcessorTest
> testCaching(org.apache.metron.stellar.common.CachingStellarProcessorTest)
> Time elapsed: 0.053 sec <<< FAILURE!
> java.lang.AssertionError: expected:<6> but was:<4>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.metron.stellar.common.CachingStellarProcessorTest.testCaching(CachingStellarProcessorTest.java:73)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1014 > Should assignment be = or :=? Ultimately, I'd like to see us move to supporting `=` as that is just way more intuitive for most people. Maybe that means supporting both `=` and `:=` for some period of time? It looks like `StellarAssignment` was not altered here, so maybe both are supported in this PR? Is that what you are thinking @ottobackwards ? I am not sure how this PR impacts all the places where Stellar is used and assignment might already occur; field transforms, enrichment, REPL, profiler, etc. There is some other way in which assignment is handled in at least some of these places. I imagine that we would need to update these to use the new mechanism here? Or maybe that is left to a future PR? ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494059#comment-16494059 ] ASF GitHub Bot commented on METRON-1563: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1014 > Should assignment be = or :=? Ultimately, I'd like to see us move to supporting `=` as that is just way more intuitive for most people. Maybe that means supporting both `=` and `:=` for some period of time? It looks like `StellarAssignment` was not altered here, so maybe both are supported in this PR? Is that what you are thinking @ottobackwards ? I am not sure how this PR impacts all the places where Stellar is used and assignment might already occur; field transforms, enrichment, REPL, profiler, etc. There is some other way in which assignment is handled in at least some of these places. I imagine that we would need to update these to use the new mechanism here? Or maybe that is left to a future PR? > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1014 > I am not sure how this PR impacts all the places where Stellar is used and assignment .. Or maybe that is left to a future PR? Sorry, its a little hard to digest that old PR, but I think from [this comment](https://github.com/apache/metron/pull/687#issuecomment-387840955), @ottobackwards is saying that all the other touch points would be updated in a future PR on this feature branch. ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494063#comment-16494063 ] ASF GitHub Bot commented on METRON-1563: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1014 > I am not sure how this PR impacts all the places where Stellar is used and assignment .. Or maybe that is left to a future PR? Sorry, its a little hard to digest that old PR, but I think from [this comment](https://github.com/apache/metron/pull/687#issuecomment-387840955), @ottobackwards is saying that all the other touch points would be updated in a future PR on this feature branch. > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1033: METRON-1579: Stellar should return the expression...
Github user cestella closed the pull request at: https://github.com/apache/metron/pull/1033 ---
[GitHub] metron issue #1021: METRON-1568: Stellar should have a _ special variable wh...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1021 +1 via inspection. ---
[GitHub] metron pull request #1033: METRON-1579: Stellar should return the expression...
GitHub user cestella reopened a pull request: https://github.com/apache/metron/pull/1033 METRON-1579: Stellar should return the expression that failed in the exception ## Contributor Comments There are situations where we are not including the expression in the exception. Also, in stellar enrichments, we should include the relevant variable values in the exception to help with diagnosing issues. You can try this out in the REPL as well as a stellar enrichment. Use a stellar expression which fails (e.g. `num_var / 0` where `num_var` is an integer field). ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron stellar_enrichment_exception Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1033.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1033 commit 28bd80f56be14003feb168292ac07ea510fb5943 Author: cstella Date: 2018-05-25T15:19:55Z Updating stellar to generate more helpful exceptions. commit 6946fdbefb7a3baf80beb0d68b86d64cd271f442 Author: cstella Date: 2018-05-25T18:26:05Z Updating error message for predicates commit 84f07ec0048c9144eb0dde06f5c3b8dff93dbc02 Author: cstella Date: 2018-05-25T18:47:13Z Missed ordinal test commit ac97b4f3a4dce0511afaab0875680ec5cd8152b3 Author: cstella Date: 2018-05-25T19:39:56Z missed stats test commit 167aae0faf745bbc7efe19fa1b1298e5632187d7 Author: cstella Date: 2018-05-25T20:14:24Z another missed test. commit 44ac3420a4406ed7ddd8524b4bf33c638269b643 Author: cstella Date: 2018-05-25T20:50:04Z And all the profiler tests. commit 71bc45e72c79efc5ac267e141f896c83ad3acc3d Author: cstella Date: 2018-05-25T21:26:23Z and assorted other tests commit 3384124bfdf332c6cf08a837f4ab7f07b4226e33 Author: cstella Date: 2018-05-29T17:44:44Z Moving things around for the palindrome ;) ---
[jira] [Commented] (METRON-1579) Stellar should return the expression that failed in the exception
[ https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494155#comment-16494155 ] ASF GitHub Bot commented on METRON-1579: Github user cestella closed the pull request at: https://github.com/apache/metron/pull/1033 > Stellar should return the expression that failed in the exception > - > > Key: METRON-1579 > URL: https://issues.apache.org/jira/browse/METRON-1579 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > There are situations where we are not including the expression in the > exception. Also, in stellar enrichments, we should include the relevant > variable values in the exception to help with diagnosing issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1568) Stellar should have a _ special variable which returns the message in map form
[ https://issues.apache.org/jira/browse/METRON-1568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494153#comment-16494153 ] ASF GitHub Bot commented on METRON-1568: Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1021 +1 via inspection. > Stellar should have a _ special variable which returns the message in map form > -- > > Key: METRON-1568 > URL: https://issues.apache.org/jira/browse/METRON-1568 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > In order to support functions which operate on the whole message, we should > have a special variable (_, keeping with the vaguely scala theme) which can > return the entire underlying message. This map should be immutable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1579) Stellar should return the expression that failed in the exception
[ https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494156#comment-16494156 ] ASF GitHub Bot commented on METRON-1579: GitHub user cestella reopened a pull request: https://github.com/apache/metron/pull/1033 METRON-1579: Stellar should return the expression that failed in the exception ## Contributor Comments There are situations where we are not including the expression in the exception. Also, in stellar enrichments, we should include the relevant variable values in the exception to help with diagnosing issues. You can try this out in the REPL as well as a stellar enrichment. Use a stellar expression which fails (e.g. `num_var / 0` where `num_var` is an integer field). ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For code changes: - [x] Have you included steps to reproduce the behavior or problem that is being changed or addressed? - [x] Have you included steps or a guide to how the change may be verified and tested manually? - [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && dev-utilities/build-utils/verify_licenses.sh ``` - [x] Have you written or updated unit tests and or integration tests to verify your changes? - [x] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [x] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/cestella/incubator-metron stellar_enrichment_exception Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/1033.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1033 commit 28bd80f56be14003feb168292ac07ea510fb5943 Author: cstella Date: 2018-05-25T15:19:55Z Updating stellar to generate more helpful exceptions. commit 6946fdbefb7a3baf80beb0d68b86d64cd271f442 Author: cstella Date: 2018-05-25T18:26:05Z Updating error message for predicates commit 84f07ec0048c9144eb0dde06f5c3b8dff93dbc02 Author: cstella Date: 2018-05-25T18:47:13Z Missed ordinal test commit ac97b4f3a4dce0511afaab0875680ec5cd8152b3 Author: cstella Date: 2018-05-25T19:39:56Z missed stats test commit 167aae0faf745bbc7efe19fa1b1298e5632187d7 Author: cstella Date: 2018-05-25T20:14:24Z another missed test. commit 44ac3420a4406ed7ddd8524b4bf33c638269b643 Author: cstella Date: 2018-05-25T20:50:04Z And all the profiler tests. commit 71bc45e72c79efc5ac267e141f896c83ad3acc3d Author: cstella Date: 2018-05-25T21:26:23Z and assorted other tests commit 3384124bfdf332c6cf08a837f4ab7f07b42
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1014 >Ultimately, I'd like to see us move to supporting = as that is just way more intuitive for most people. Maybe that means supporting both = and := for some period of time? In my opinion we should support both for at least a release. ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494185#comment-16494185 ] ASF GitHub Bot commented on METRON-1563: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1014 >Ultimately, I'd like to see us move to supporting = as that is just way more intuitive for most people. Maybe that means supporting both = and := for some period of time? In my opinion we should support both for at least a release. > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1014 Do we want to expand assignment to `:=` as part of this PR or make it a follow-on in this feature branch? I'm ok with either way. ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494207#comment-16494207 ] ASF GitHub Bot commented on METRON-1563: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1014 Do we want to expand assignment to `:=` as part of this PR or make it a follow-on in this feature branch? I'm ok with either way. > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #953: METRON-1472 Add stylelint support
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/953 -1 as this has become an abandoned PR. I will open a request to close this. ---
[GitHub] metron issue #951: METRON-1479 Add editorconfig - create guideline for code ...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/951 -1 as this has become an abandoned PR. I will open a request to close this PR. ---
[GitHub] metron issue #956: METRON-1474 Add normalizecss
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/956 -1 as this has become an abandoned PR. I will open a request to close this PR. ---
[GitHub] metron issue #952: METRON-1480 Add yarn as default build tool for the fronte...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/952 -1 as this has become an abandoned PR. I will open a request to close this. ---
[GitHub] metron issue #954: METRON-1478 Move type definitions to dev-dependencies
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/954 -1 as this has become an abandoned PR. I will open a request to close this. ---
[jira] [Commented] (METRON-1480) Add yarn as default build tool for the frontend
[ https://issues.apache.org/jira/browse/METRON-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494252#comment-16494252 ] ASF GitHub Bot commented on METRON-1480: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/952 -1 as this has become an abandoned PR. I will open a request to close this. > Add yarn as default build tool for the frontend > --- > > Key: METRON-1480 > URL: https://issues.apache.org/jira/browse/METRON-1480 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1474) Add normalizecss
[ https://issues.apache.org/jira/browse/METRON-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494255#comment-16494255 ] ASF GitHub Bot commented on METRON-1474: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/956 -1 as this has become an abandoned PR. I will open a request to close this PR. > Add normalizecss > > > Key: METRON-1474 > URL: https://issues.apache.org/jira/browse/METRON-1474 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Priority: Minor > > Add normalizecss to reduce css mistakes -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1479) Add editorconfig - create guideline for code formatting in the angular part of the app
[ https://issues.apache.org/jira/browse/METRON-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494254#comment-16494254 ] ASF GitHub Bot commented on METRON-1479: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/951 -1 as this has become an abandoned PR. I will open a request to close this PR. > Add editorconfig - create guideline for code formatting in the angular part > of the app > -- > > Key: METRON-1479 > URL: https://issues.apache.org/jira/browse/METRON-1479 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1472) Add stylelint support
[ https://issues.apache.org/jira/browse/METRON-1472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494251#comment-16494251 ] ASF GitHub Bot commented on METRON-1472: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/953 -1 as this has become an abandoned PR. I will open a request to close this. > Add stylelint support > - > > Key: METRON-1472 > URL: https://issues.apache.org/jira/browse/METRON-1472 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Priority: Minor > Original Estimate: 2h > Remaining Estimate: 2h > > Add stylelint support for better code quality -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (METRON-1478) Move type definitions to dev-dependencies
[ https://issues.apache.org/jira/browse/METRON-1478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494256#comment-16494256 ] ASF GitHub Bot commented on METRON-1478: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/954 -1 as this has become an abandoned PR. I will open a request to close this. > Move type definitions to dev-dependencies > - > > Key: METRON-1478 > URL: https://issues.apache.org/jira/browse/METRON-1478 > Project: Metron > Issue Type: Improvement >Reporter: Daniel Toth >Assignee: Daniel Toth >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1022: METRON-1569 Allow user to change field name conversion w...
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1022 All review feedback should have been addressed. Let me know if there is anything else @mmiklavc @cestella ---
[jira] [Commented] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494268#comment-16494268 ] ASF GitHub Bot commented on METRON-1569: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/1022 All review feedback should have been addressed. Let me know if there is anything else @mmiklavc @cestella > Allow user to change field name conversion when indexing to Elasticsearch > - > > Key: METRON-1569 > URL: https://issues.apache.org/jira/browse/METRON-1569 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > The `ElasticsearchWriter` has a mechanism to transform the field names of a > message before it is written to Elasticsearch. Right now this mechanism is > hard-coded to replace all '.' dots with ':' colons. > This mechanism was needed for Elasticsearch 2.x which did not allow dots in > field names. Now that Metron supports Elasticsearch 5.x this is no longer a > problem. > A user should be able to configure the field name transformation when writing > to Elasticsearch, as needed. > While it might have been simpler to just remove the de-dotting mechanism, > this would break backwards compatibility. Providing users with a means to > configure this mechanism provides them with an upgrade path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #830: METRON-1298: TimeRange Picker doesn't work on Safari
Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/830 -1 as this PR has been abandoned. ---
[jira] [Commented] (METRON-1298) TimeRange Picker doesn't work on Safari
[ https://issues.apache.org/jira/browse/METRON-1298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494270#comment-16494270 ] ASF GitHub Bot commented on METRON-1298: Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/830 -1 as this PR has been abandoned. > TimeRange Picker doesn't work on Safari > --- > > Key: METRON-1298 > URL: https://issues.apache.org/jira/browse/METRON-1298 > Project: Metron > Issue Type: Bug >Reporter: RaghuMitra >Assignee: RaghuMitra >Priority: Major > > Date Picker doesn't work on safari the request goes with a timestamp of NaN -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1022: METRON-1569 Allow user to change field name conve...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1022#discussion_r191574312
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/field/FieldNameConverters.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field;
+
+import org.apache.commons.lang.ClassUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+
+/**
+ * Enumerates a set of {@link FieldNameConverter} implementations.
+ *
+ * Provides shared instances of each {@link FieldNameConverter}.
+ *
+ * Allows the field name converter to be specified using a short-hand
+ * name, rather than the entire fully-qualified class name.
+ */
+public enum FieldNameConverters {
--- End diff --
Minor nit, can we have this implement `FieldNameConverter` so the enum can
be used directly without calling `get()`?
---
[jira] [Commented] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494273#comment-16494273
]
ASF GitHub Bot commented on METRON-1569:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1022#discussion_r191574312
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/field/FieldNameConverters.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field;
+
+import org.apache.commons.lang.ClassUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+
+/**
+ * Enumerates a set of {@link FieldNameConverter} implementations.
+ *
+ * Provides shared instances of each {@link FieldNameConverter}.
+ *
+ * Allows the field name converter to be specified using a short-hand
+ * name, rather than the entire fully-qualified class name.
+ */
+public enum FieldNameConverters {
--- End diff --
Minor nit, can we have this implement `FieldNameConverter` so the enum can
be used directly without calling `get()`?
> Allow user to change field name conversion when indexing to Elasticsearch
> -
>
> Key: METRON-1569
> URL: https://issues.apache.org/jira/browse/METRON-1569
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> The `ElasticsearchWriter` has a mechanism to transform the field names of a
> message before it is written to Elasticsearch. Right now this mechanism is
> hard-coded to replace all '.' dots with ':' colons.
> This mechanism was needed for Elasticsearch 2.x which did not allow dots in
> field names. Now that Metron supports Elasticsearch 5.x this is no longer a
> problem.
> A user should be able to configure the field name transformation when writing
> to Elasticsearch, as needed.
> While it might have been simpler to just remove the de-dotting mechanism,
> this would break backwards compatibility. Providing users with a means to
> configure this mechanism provides them with an upgrade path.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191582496
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
Does `ExceptionUtils.getRootCauseMessage()` return the whole stack trace or
just the root message? I think we'd like to keep the stack trace here.
---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494314#comment-16494314
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191582496
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
Does `ExceptionUtils.getRootCauseMessage()` return the whole stack trace or
just the root message? I think we'd like to keep the stack trace here.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191584626
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
}
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+tuples.forEach(t -> collector.ack(t));
--- End diff --
wow, we did we really not ack here before? That's a bug. good catch.
---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494320#comment-16494320
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191584626
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
}
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+tuples.forEach(t -> collector.ack(t));
--- End diff --
wow, we did we really not ack here before? That's a bug. good catch.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191585015
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+ /**
+ * Retrieves the JSON message contained in a tuple.
+ *
+ * @param tuple The tuple containing a JSON message.
+ * @return The JSON message contained in the tuple. If none, returns
null.
+ */
+ private JSONObject getMessag
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494322#comment-16494322
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191585015
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+ /*
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191585908
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
}
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+tuples.forEach(t -> collector.ack(t));
--- End diff --
Wait, @cestella isn't it already acked in line 123?
---
[GitHub] metron pull request #1022: METRON-1569 Allow user to change field name conve...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1022#discussion_r191585931
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/field/FieldNameConverters.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field;
+
+import org.apache.commons.lang.ClassUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+
+/**
+ * Enumerates a set of {@link FieldNameConverter} implementations.
+ *
+ * Provides shared instances of each {@link FieldNameConverter}.
+ *
+ * Allows the field name converter to be specified using a short-hand
+ * name, rather than the entire fully-qualified class name.
+ */
+public enum FieldNameConverters {
--- End diff --
Ok. Check 'er out.
---
[jira] [Commented] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[
https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494329#comment-16494329
]
ASF GitHub Bot commented on METRON-1569:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1022#discussion_r191585931
--- Diff:
metron-platform/metron-common/src/main/java/org/apache/metron/common/field/FieldNameConverters.java
---
@@ -0,0 +1,98 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.common.field;
+
+import org.apache.commons.lang.ClassUtils;
+import org.apache.commons.lang3.exception.ExceptionUtils;
+import org.apache.metron.common.configuration.writer.WriterConfiguration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.lang.invoke.MethodHandles;
+
+/**
+ * Enumerates a set of {@link FieldNameConverter} implementations.
+ *
+ * Provides shared instances of each {@link FieldNameConverter}.
+ *
+ * Allows the field name converter to be specified using a short-hand
+ * name, rather than the entire fully-qualified class name.
+ */
+public enum FieldNameConverters {
--- End diff --
Ok. Check 'er out.
> Allow user to change field name conversion when indexing to Elasticsearch
> -
>
> Key: METRON-1569
> URL: https://issues.apache.org/jira/browse/METRON-1569
> Project: Metron
> Issue Type: Improvement
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> The `ElasticsearchWriter` has a mechanism to transform the field names of a
> message before it is written to Elasticsearch. Right now this mechanism is
> hard-coded to replace all '.' dots with ':' colons.
> This mechanism was needed for Elasticsearch 2.x which did not allow dots in
> field names. Now that Metron supports Elasticsearch 5.x this is no longer a
> problem.
> A user should be able to configure the field name transformation when writing
> to Elasticsearch, as needed.
> While it might have been simpler to just remove the de-dotting mechanism,
> this would break backwards compatibility. Providing users with a means to
> configure this mechanism provides them with an upgrade path.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494328#comment-16494328
]
ASF GitHub Bot commented on METRON-1584:
Github user mmiklavc commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191585908
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
}
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+tuples.forEach(t -> collector.ack(t));
--- End diff --
Wait, @cestella isn't it already acked in line 123?
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0
[GitHub] metron issue #1022: METRON-1569 Allow user to change field name conversion w...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1022 Ok, I'm comfortable with the change now. +1 by inspection, pending @mmiklavc ---
[jira] [Commented] (METRON-1569) Allow user to change field name conversion when indexing to Elasticsearch
[ https://issues.apache.org/jira/browse/METRON-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494330#comment-16494330 ] ASF GitHub Bot commented on METRON-1569: Github user cestella commented on the issue: https://github.com/apache/metron/pull/1022 Ok, I'm comfortable with the change now. +1 by inspection, pending @mmiklavc > Allow user to change field name conversion when indexing to Elasticsearch > - > > Key: METRON-1569 > URL: https://issues.apache.org/jira/browse/METRON-1569 > Project: Metron > Issue Type: Improvement >Reporter: Nick Allen >Assignee: Nick Allen >Priority: Major > > The `ElasticsearchWriter` has a mechanism to transform the field names of a > message before it is written to Elasticsearch. Right now this mechanism is > hard-coded to replace all '.' dots with ':' colons. > This mechanism was needed for Elasticsearch 2.x which did not allow dots in > field names. Now that Metron supports Elasticsearch 5.x this is no longer a > problem. > A user should be able to configure the field name transformation when writing > to Elasticsearch, as needed. > While it might have been simpler to just remove the de-dotting mechanism, > this would break backwards compatibility. Providing users with a means to > configure this mechanism provides them with an upgrade path. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191587917
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+ /**
+ * Retrieves the JSON message contained in a tuple.
+ *
+ * @param tuple The tuple containing a JSON message.
+ * @return The JSON message contained in the tuple. If none, returns
null.
+ */
+ private JSONObject getMess
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494348#comment-16494348
]
ASF GitHub Bot commented on METRON-1584:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191587917
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191588328
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
We were not logging the the exception message or stack trace at all before;
just "Failing 8 tuples".
The error method was not being used as intended AFAIK. Having the
exception as the last parameter was just doing a `{}` substitution for a `{}`
that did not exist.
I can change this to log the stack trace, if you like.
---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494352#comment-16494352
]
ASF GitHub Bot commented on METRON-1584:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191588328
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
We were not logging the the exception message or stack trace at all before;
just "Failing 8 tuples".
The error method was not being used as intended AFAIK. Having the
exception as the last parameter was just doing a `{}` substitution for a `{}`
that did not exist.
I can change this to log the stack trace, if you like.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494355#comment-16494355
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191588785
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
}
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+tuples.forEach(t -> collector.ack(t));
--- End diff --
AH yes, it is, but this is a new error method. It feels that both `error`
method should be calling another `error`method to do the basics like `ack()`.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191588785
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
}
-tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+tuples.forEach(t -> collector.ack(t));
--- End diff --
AH yes, it is, but this is a new error method. It feels that both `error`
method should be calling another `error`method to do the basics like `ack()`.
---
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191588901
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+ /**
+ * Retrieves the JSON message contained in a tuple.
+ *
+ * @param tuple The tuple containing a JSON message.
+ * @return The JSON message contained in the tuple. If none, returns
null.
+ */
+ private JSONObject getMessag
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494357#comment-16494357
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191588901
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+ /*
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191590539
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+ /**
+ * Retrieves the JSON message contained in a tuple.
+ *
+ * @param tuple The tuple containing a JSON message.
+ * @return The JSON message contained in the tuple. If none, returns
null.
+ */
+ private JSONObject getMess
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494366#comment-16494366
]
ASF GitHub Bot commented on METRON-1584:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191590539
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
---
@@ -213,60 +214,144 @@ public void prepare(Map stormConf, TopologyContext
context, OutputCollector coll
@SuppressWarnings("unchecked")
@Override
public void execute(Tuple tuple) {
+
if (isTick(tuple)) {
- try {
-if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
- //WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
- LOG.debug("Flushing message queues older than their
batchTimeouts");
- getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
- new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
- , messageGetStrategy);
-}
- }
- catch(Exception e) {
-throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
- }
- finally {
-collector.ack(tuple);
- }
- return;
+ handleTick(tuple);
+
+} else {
+ handleMessage(tuple);
}
+ }
+
+ /**
+ * Handle a tuple containing a message; anything other than a tick tuple.
+ *
+ * @param tuple The tuple containing a message.
+ */
+ private void handleMessage(Tuple tuple) {
+try {
+
+ JSONObject message = getMessage(tuple);
+ if(message == null) {
+handleMissingMessage(tuple);
+return;
+ }
-try
-{
- JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
String sensorType = MessageUtils.getSensorType(message);
- LOG.trace("Writing enrichment message: {}", message);
- WriterConfiguration writerConfiguration =
configurationTransformation.apply(
- new IndexingWriterConfiguration(bulkMessageWriter.getName(),
getConfigurations()));
if(sensorType == null) {
-//sensor type somehow ended up being null. We want to error this
message directly.
-getWriterComponent().error("null"
- , new Exception("Sensor type is not specified
for message "
-+ message.toJSONString()
-)
- , ImmutableList.of(tuple)
- , messageGetStrategy
- );
- }
- else {
-if (writerConfiguration.isDefault(sensorType)) {
- //want to warn, but not fail the tuple
- collector.reportError(new Exception("WARNING: Default and
(likely) unoptimized writer config used for " + bulkMessageWriter.getName() + "
writer and sensor " + sensorType));
-}
-
-getWriterComponent().write(sensorType
-, tuple
-, message
-, bulkMessageWriter
-, writerConfiguration
-, messageGetStrategy
-);
+handleMissingSensorType(tuple, message);
+return;
}
+
+ writeMessage(tuple, message, sensorType);
+
+} catch (Exception e) {
+ throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
}
-catch(Exception e) {
+ }
+
+ /**
+ * Handles a tick tuple.
+ *
+ * @param tickTuple The tick tuple.
+ */
+ private void handleTick(Tuple tickTuple) {
+
+try {
+ if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+//WriterToBulkWriter doesn't allow batching, so no need to flush
on Tick.
+LOG.debug("Flushing message queues older than their
batchTimeouts");
+getWriterComponent().flushTimeouts(bulkMessageWriter,
configurationTransformation.apply(
+new
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+, messageGetStrategy);
+ }
+
+} catch(Exception e) {
throw new RuntimeException("This should have been caught in the
writerComponent. If you see this, file a JIRA", e);
+
+} finally {
+ collector.ack(tickTuple);
+}
+ }
+
+
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191592021
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
Ugh, yeah, so I think we refactored at some point to move to parameterized
logging statements and ran into
[this](https://www.slf4j.org/faq.html#paramException).
Perhaps we can use `LOG.error(String.format("Failing tuples; count=%d,
error=%s", ...), e);` and get the best of both worlds?
---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494376#comment-16494376
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191592021
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
Ugh, yeah, so I think we refactored at some point to move to parameterized
logging statements and ran into
[this](https://www.slf4j.org/faq.html#paramException).
Perhaps we can use `LOG.error(String.format("Failing tuples; count=%d,
error=%s", ...), e);` and get the best of both worlds?
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.ex
[GitHub] metron pull request #1036: METRON-1584 Indexing Topology Crashes with Invali...
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191596670
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
Sure will do.
---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494405#comment-16494405
]
ASF GitHub Bot commented on METRON-1584:
Github user nickwallen commented on a diff in the pull request:
https://github.com/apache/metron/pull/1036#discussion_r191596670
--- Diff:
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/BulkWriterComponent.java
---
@@ -115,15 +116,37 @@ public void commit(BulkWriterResponse response) {
}
public void error(String sensorType, Throwable e, Iterable
tuples, MessageGetStrategy messageGetStrategy) {
+
+if(!Iterables.isEmpty(tuples)) {
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
+}
tuples.forEach(t -> collector.ack(t));
MetronError error = new MetronError()
.withSensorType(sensorType)
.withErrorType(Constants.ErrorType.INDEXING_ERROR)
.withThrowable(e);
+tuples.forEach(t -> error.addRawMessage(messageGetStrategy.get(t)));
+ErrorUtils.handleError(collector, error);
+ }
+
+ /**
+ * Error a set of tuples that may not contain a valid message.
+ *
+ * Without a valid message, the source type is unknown.
+ * Without a valid message, the JSON message cannot be added to the
error.
+ *
+ * @param e The exception that occurred.
+ * @param tuples The tuples to error that may not contain valid messages.
+ */
+ public void error(Throwable e, Iterable tuples) {
+
if(!Iterables.isEmpty(tuples)) {
- LOG.error("Failing {} tuples", Iterables.size(tuples), e);
+ LOG.error("Failing tuples; count={}, error={}",
Iterables.size(tuples), ExceptionUtils.getRootCauseMessage(e));
--- End diff --
Sure will do.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.s
[GitHub] metron issue #1036: METRON-1584 Indexing Topology Crashes with Invalid Messa...
Github user mmiklavc commented on the issue: https://github.com/apache/metron/pull/1036 I want to take a step back a moment here. I think there are some useful, clarifying refactorings in this PR. I'm generally in favor of refactoring improvements e.g. "handleTick(tuple)" that sum up the code nicely. I also like adding sensible clarifying comments and making tests more clear. However, I'm not sure how I feel about mixing refactoring like this with the bug fix itself. It seems to me that the actual bug fix is roughly 5-10 lines of code. It makes it more difficult to determine code that's specific to the change in mind. I noticed this while reviewing https://github.com/apache/metron/pull/1015 as well. Might it be better to do separate refactoring PR's? I am certainly not suggesting we be pedantic about it - I'm not even clear how I would even suggest a reasonable rule to follow, to be honest. Part of me thinks "ok, broken window theory and boyscout rules - this is great stuff." But in general, I think that splitting the refactoring will allow reviewers to voice opinio ns specific to coding style and refactoring changes as distinct from a feature or bug fix itself. It's clear that we all have varying opinions on what reads best in our code, and I think splitting those discussions off will make the review process more expedient. ---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494406#comment-16494406
]
ASF GitHub Bot commented on METRON-1584:
Github user mmiklavc commented on the issue:
https://github.com/apache/metron/pull/1036
I want to take a step back a moment here. I think there are some useful,
clarifying refactorings in this PR. I'm generally in favor of refactoring
improvements e.g. "handleTick(tuple)" that sum up the code nicely. I also like
adding sensible clarifying comments and making tests more clear. However, I'm
not sure how I feel about mixing refactoring like this with the bug fix itself.
It seems to me that the actual bug fix is roughly 5-10 lines of code. It makes
it more difficult to determine code that's specific to the change in mind. I
noticed this while reviewing https://github.com/apache/metron/pull/1015 as
well. Might it be better to do separate refactoring PR's? I am certainly not
suggesting we be pedantic about it - I'm not even clear how I would even
suggest a reasonable rule to follow, to be honest. Part of me thinks "ok,
broken window theory and boyscout rules - this is great stuff." But in general,
I think that splitting the refactoring will allow reviewers to voice opinions
specific to coding style and refactoring changes as distinct from a feature or
bug fix itself. It's clear that we all have varying opinions on what reads best
in our code, and I think splitting those discussions off will make the review
process more expedient.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> 2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [stor
[GitHub] metron issue #1036: METRON-1584 Indexing Topology Crashes with Invalid Messa...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/1036 Yeah, I have mixed reactions. I think refactoring opportunistically is damned useful, but if it becomes excessive it can make it hard to review and make it hard for those with open branches in the same codebase. This seems like a broader discussion is merited, though. Can we maybe move this to a mailing list DISCUSS thread? This feels like one of those things we should hash out and should end up in the dev guidelines. ---
[jira] [Commented] (METRON-1584) Indexing Topology Crashes with Invalid Message
[
https://issues.apache.org/jira/browse/METRON-1584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494425#comment-16494425
]
ASF GitHub Bot commented on METRON-1584:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1036
Yeah, I have mixed reactions. I think refactoring opportunistically is
damned useful, but if it becomes excessive it can make it hard to review and
make it hard for those with open branches in the same codebase.
This seems like a broader discussion is merited, though. Can we maybe move
this to a mailing list DISCUSS thread? This feels like one of those things we
should hash out and should end up in the dev guidelines.
> Indexing Topology Crashes with Invalid Message
> --
>
> Key: METRON-1584
> URL: https://issues.apache.org/jira/browse/METRON-1584
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> Per Mohan Venkateshaiah:
> I published message "adkadknalkda;LK;ad;Da;dD;" to indexing topic , I see
> that the random access indexing topology worker thread died and couldn't
> recover until the kafka topic was deleted and recreated.
> {code:java}
> Caused by: java.lang.IllegalStateException: Unable to parse
> adkadknalkda;LK;ad;Da;dD; due to null
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:49)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> Caused by: org.json.simple.parser.ParseException
> at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:118)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:81)
> ~[stormjar.jar:?]
> at org.json.simple.parser.JSONParser.parse(JSONParser.java:75)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:47)
> ~[stormjar.jar:?]
> at
> org.apache.metron.common.message.JSONFromPosition.get(JSONFromPosition.java:25)
> ~[stormjar.jar:?]
> at
> org.apache.metron.writer.bolt.BulkMessageWriterBolt.execute(BulkMessageWriterBolt.java:237)
> ~[stormjar.jar:?]
> at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> ... 6 more
> 2018-05-24 09:21:22.236 o.a.s.util Thread-9-indexingBolt-executor[3 3]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
> 2018-05-24 09:21:22.237 o.a.s.d.worker Thread-20 [INFO] Shutting down worker
> random_access_indexing-24-1527147389 703b5bf7-6c9d-46f3-8136-0c4877a69375 6700
> 2018-05-24 0
[jira] [Commented] (METRON-1583) issue regarding cisco asa logs
[
https://issues.apache.org/jira/browse/METRON-1583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494581#comment-16494581
]
Simon Elliston Ball commented on METRON-1583:
-
Would it be possible to share the full log error message from the storm logs,
rather than the screenshot?
> issue regarding cisco asa logs
> --
>
> Key: METRON-1583
> URL: https://issues.apache.org/jira/browse/METRON-1583
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.4.2
>Reporter: manisha tank
>Priority: Major
> Fix For: 0.4.2
>
> Attachments: cisco_asa_logs_error.png
>
>
> I am trying to ingest cisco asa logs but I am facing some issue .
> I have created log pattern below
> CISCO_TAGGED_SYSLOG ^%\{SYSLOGTIMESTAMP} %\{SYSLOGHOST:sysloghost}
> <%\{POSINT:syslog_pri}>%\{CISCOTIMESTAMP}?: %%\{CISCOTAG:ciscotag}:
> %\{GREEDYDATA:message}
> CISCOTIMESTAMP %\{MONTH} +%\{MONTHDAY}(?: %\{YEAR})? %\{TIME}
> CISCOTAG [A-Z0-9]+-%\{INT}-(?:[A-Z0-9_]+)
> sample logs
> Oct 25 02:14:52 172.20.4.5 <163>Oct 24 2017 21:29:23: %ASA-3-304006: URL
> Server 172.19.83.105 not responding
> Oct 25 02:14:51 198.6.1.2 <164>Oct 24 2017 21:28:47: %ASA-4-410001: Dropped
> UDP DNS reply from outside:198.6.1.2/53 to inside:172.20.220.87/63887; packet
> length 932 bytes exceeds configured limit of 512 bytes
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:34: %ASA-4-733100: [
> Scanning] drop rate-1 exceeded. Current burst rate is 46 per second, max
> configured rate is 10; Current average rate is 103 per second, max configured
> rate is 5; Cumulative total count is 62196
> Oct 25 02:14:51 172.20.4.5 <164>Oct 24 2017 21:28:21: %ASA-4-733100: [
> SYSLOG 514] drop rate-1 exceeded. Current burst rate is 31 per second, max
> configured rate is 40; Current average rate is 119 per second, max configured
> rate is 20; Cumulative total count is 71776
>
> Oct 25 02:14:52 192.168.19.7 <164>Oct 24 2017 21:29:29: %ASA-4-419002:
> Duplicate TCP SYN from inside:192.168.19.7/64266 to outside:192.168.10.10/257
> with different initial sequence number
>
> PFA error facing while inegesting cisco asa logs
>
> !cisco_asa_logs_error.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1579) Stellar should return the expression that failed in the exception
[ https://issues.apache.org/jira/browse/METRON-1579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494601#comment-16494601 ] ASF GitHub Bot commented on METRON-1579: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1033 +1 I will say, that the exception building facilities and builders should be publicly exposed as part of stellar as a follow on. As part of my own antlr work on the syslog stuff, I am kind of fond of being able to allow expansion by deriving new listeners and visitors with separate implementations, the exception builders would be nice to have with that, if you follow. > Stellar should return the expression that failed in the exception > - > > Key: METRON-1579 > URL: https://issues.apache.org/jira/browse/METRON-1579 > Project: Metron > Issue Type: Improvement >Reporter: Casey Stella >Priority: Major > > There are situations where we are not including the expression in the > exception. Also, in stellar enrichments, we should include the relevant > variable values in the exception to help with diagnosing issues. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] metron issue #1033: METRON-1579: Stellar should return the expression that f...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1033 +1 I will say, that the exception building facilities and builders should be publicly exposed as part of stellar as a follow on. As part of my own antlr work on the syslog stuff, I am kind of fond of being able to allow expansion by deriving new listeners and visitors with separate implementations, the exception builders would be nice to have with that, if you follow. ---
[GitHub] metron issue #1014: METRON-1563 : Base Stellar assign for feature branch
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 Can I ask how we do ++, --, += etc with the := notation? ---
[jira] [Commented] (METRON-1563) Initial Feature Branch Work
[ https://issues.apache.org/jira/browse/METRON-1563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16494603#comment-16494603 ] ASF GitHub Bot commented on METRON-1563: Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/1014 Can I ask how we do ++, --, += etc with the := notation? > Initial Feature Branch Work > --- > > Key: METRON-1563 > URL: https://issues.apache.org/jira/browse/METRON-1563 > Project: Metron > Issue Type: Sub-task >Reporter: Otto Fowler >Assignee: Otto Fowler >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
