(Moving thread to dev@ to discuss versions not yet released) Nick,
I suspect you might be hitting a bug in 1.0.2 that might be fixed in one of the next releases. Can you try setting the version of storm-kafka-client to 1.1.0-SNAPSHOT (leave other storm components at 1.0.2)? If that fixes the issue, let us know on [email protected] <mailto:[email protected]> so we can back port the fix. -Taylor > On Nov 18, 2016, at 2:10 PM, Cuneo, Nicholas <[email protected]> wrote: > > Hello, > > We are working with storm 1.0.2 and using Kafka client to subscribe to kafka > topics to retrieve data. Randomly when our topologies are starting up we > receive a null pointer exception which is killing the topology. We can’t > seem to identify why this happens, or what measures we can take to prevent > it. I’m considering raising a ticket with storm to check for null to prevent > the topology from crashing. > > 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died! > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > Caused by: java.lang.NullPointerException > at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) > ~[stormjar.jar:?] > at > org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > ~[storm-core-1.0.2.jar:1.0.2] > ... 7 more > 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > Caused by: java.lang.NullPointerException > at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) > ~[stormjar.jar:?] > at > org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$fn__7990$tuple_action_fn__7996.invoke(executor.clj:536) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > ~[storm-core-1.0.2.jar:1.0.2] > ... 7 more > 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died") > java.lang.RuntimeException: ("Worker died") > at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?] > at > org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) > [storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$mk_executor_data$fn__7875$fn__7876.invoke(executor.clj:274) > [storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > > The method and line number in question below: > > @Override > public void ack(Object messageId) { > final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId; > if (!consumerAutoCommitMode) { // Only need to keep track of acked > tuples if commits are not done automatically > acked.get(msgId.getTopicPartition()).add(msgId); > } > emitted.remove(msgId); > } > > Restarting the topology sometimes fixes the issue but sometimes does not. I > don’t know if there’s some kind of race condition occurring. We have about > 15 topologies all connecting to kafka the same way, and this doesn’t happen > consistently between them. > > Does anyone have any insight as to why this occurs or how to prevent it on > our end? > > > Thanks, > > Nick Cuneo / Software Engineer, Cloud / Enterprise Software > Tel: +1 949 517 4802 / Mobile: +1 949 243 4952 > 3 Ada / Irvine, CA 92618 / USA > [email protected] <mailto:[email protected]> / www.tyco.com > <http://www.tyco.com/> > <image003.png> > > > This email (including any attachments) may contain information that is > private or business confidential. If you received this email in error, please > delete it from your system without copying it and notify sender by reply > email so that our records can be corrected > > > > This e-mail contains privileged and confidential information intended for the > use of the addressees named above. If you are not the intended recipient of > this e-mail, you are hereby notified that you must not disseminate, copy or > take any action in respect of any information contained in it. If you have > received this e-mail in error, please notify the sender immediately by e-mail > and immediately destroy this e-mail and its attachments.
