[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message
[
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596729#comment-16596729
]
ASF GitHub Bot commented on METRON-1751:
Github user asfgit closed the pull request at:
https://github.com/apache/metron/pull/1176
> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Async loop died! java.lang.RuntimeException:
> java.lang.NullPointerException at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by:
> java.lang.NullPointerException at java.lang.String.(String.java:491)
> ~[?:1.8.0_181] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
> ~[stormjar.jar:?] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
> ~[stormjar.jar:?] at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even if you start up the profiler again and run it
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message
[
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596721#comment-16596721
]
ASF GitHub Bot commented on METRON-1751:
Github user nickwallen commented on the issue:
https://github.com/apache/metron/pull/1176
Thanks for the review @cestella ! I ran this up in Full Dev and everything
looked proper.
> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Async loop died! java.lang.RuntimeException:
> java.lang.NullPointerException at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by:
> java.lang.NullPointerException at java.lang.String.(String.java:491)
> ~[?:1.8.0_181] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
> ~[stormjar.jar:?] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
> ~[stormjar.jar:?] at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even if you start up the profiler again and run it
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message
[
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16596408#comment-16596408
]
ASF GitHub Bot commented on METRON-1751:
Github user cestella commented on the issue:
https://github.com/apache/metron/pull/1176
+1 by inspection pending full-dev validation.
> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Async loop died! java.lang.RuntimeException:
> java.lang.NullPointerException at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by:
> java.lang.NullPointerException at java.lang.String.(String.java:491)
> ~[?:1.8.0_181] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
> ~[stormjar.jar:?] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
> ~[stormjar.jar:?] at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even if you start up the profiler again and run it
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message
[
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16593699#comment-16593699
]
ASF GitHub Bot commented on METRON-1751:
GitHub user nickwallen opened a pull request:
https://github.com/apache/metron/pull/1176
METRON-1751 Storm Profiler dies when consuming null message
The Storm Profiler dies if a null message is consumed from the input Kafka
topic. The Profiler should skip past any null messages and continue processing.
## Testing
1. Create a "hello-world" profile using the Storm Profiler as outlined in
the README.
1. Push a null message on to the "indexing" Kafka topic.
1. Make sure the Storm topology does not die and that a profile measurement
is flushed.
## Pull Request Checklist
- [ ] Is there a JIRA ticket associated with this PR? If not one needs to
be created at [Metron
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [ ] Does your PR title start with METRON- where is the JIRA
number you are trying to resolve? Pay particular attention to the hyphen "-"
character.
- [ ] Has your PR been rebased against the latest commit within the target
branch (typically master)?
- [ ] Have you included steps to reproduce the behavior or problem that is
being changed or addressed?
- [ ] Have you included steps or a guide to how the change may be verified
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been
executed in the root metron folder via:
- [ ] Have you written or updated unit tests and or integration tests to
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] Have you verified the basic functionality of the build by building
and running locally with Vagrant full-dev environment or the equivalent?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/metron METRON-1751
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/metron/pull/1176.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1176
commit 5123f28ea76ac47d97772d2d4bf0c193aa156697
Author: Nick Allen
Date: 2018-08-27T13:56:28Z
METRON-1751 Storm Profiler dies when consuming null message
> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
> Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7]
> [ERROR] Async loop died! java.lang.RuntimeException:
> java.lang.NullPointerException at
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484)
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by:
> java.lang.NullPointerException at java.lang.String.(String.java:491)
> ~[?:1.8.0_181] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
> ~[stormjar.jar:?] at
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
> ~[stormjar.jar:?] at
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
> ~[storm-core-
