I hope this ends up in the right thread... (it's not entirely obvious to me how to reply to messages sent before I subscribed)
Nick, you're very likely hitting a bug fixed here https://github.com/apache/storm/pull/1696. 1.1.0 isn't released yet, so if you need the fix now you need to build the spout code yourself. There is a 1.x version of the fix here https://github.com/srdo/storm/tree/STORM-2104-1.x. Clone the repository and run "mvn clean install -DskipTests" in the root of the project. Then update your project to use 1.1.0-SNAPSHOT for storm-kafka-client. 2016-11-20 16:18 GMT+01:00 <[email protected]>: > > dev Digest of: get.46228 > > Topics (messages 46228 through 46228) > > Re: NullPointerException on startup > 46228 by: Cuneo, Nicholas > > Administrivia: > > > --- Administrative commands for the dev list --- > > I can handle administrative requests automatically. Please > do not send them to the list address! Instead, send > your message to the correct command address: > > To subscribe to the list, send a message to: > <[email protected]> > > To remove your address from the list, send a message to: > <[email protected]> > > Send mail to the following for info and FAQ for this list: > <[email protected]> > <[email protected]> > > Similar addresses exist for the digest list: > <[email protected]> > <[email protected]> > > To get messages 123 through 145 (a maximum of 100 per request), mail: > <[email protected]> > > To get an index with subject and author for messages 123-456 , mail: > <[email protected]> > > They are always returned as sets of 100, max 2000 per request, > so you'll actually get 100-499. > > To receive all messages with the same subject as message 12345, > send a short message to: > <[email protected]> > > The messages should contain one line or word of text to avoid being > treated as sp@m, but I will ignore their content. > Only the ADDRESS you send to is important. > > You can start a subscription for an alternate address, > for example "[email protected]", just add a hyphen and your > address (with '=' instead of '@') after the command word: > <[email protected]> > > To stop subscription for this address, mail: > <[email protected]> > > In both cases, I'll send a confirmation message to that address. When > you receive it, simply reply to it to complete your subscription. > > If despite following these instructions, you do not get the > desired results, please contact my owner at > [email protected]. Please be patient, my owner is a > lot slower than I am ;-) > > --- Enclosed is a copy of the request I received. > > Return-Path: <[email protected]> > Received: (qmail 4249 invoked by uid 99); 20 Nov 2016 15:18:44 -0000 > Received: from pnap-us-west-generic-nat.apache.org (HELO > spamd3-us-west.apache.org) (209.188.14.142) > by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Nov 2016 15:18:44 > +0000 > Received: from localhost (localhost [127.0.0.1]) > by spamd3-us-west.apache.org (ASF Mail Server at > spamd3-us-west.apache.org) with ESMTP id 451B0180538 > for <[email protected]>; Sun, 20 Nov 2016 15:18:44 > +0000 (UTC) > X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org > X-Spam-Flag: NO > X-Spam-Score: 4.742 > X-Spam-Level: **** > X-Spam-Status: No, score=4.742 tagged_above=-999 required=6.31 > tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, > EMPTY_MESSAGE=2.344, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, > RCVD_IN_MSPIKE_H2=-0.001, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] > autolearn=disabled > Authentication-Results: spamd3-us-west.apache.org (amavisd-new); > dkim=pass (2048-bit key) header.d=gmail.com > Received: from mx1-lw-eu.apache.org ([10.40.0.8]) > by localhost (spamd3-us-west.apache.org [10.40.0.10]) > (amavisd-new, port 10024) > with ESMTP id rLreHScnI0Pn for <[email protected]>; > Sun, 20 Nov 2016 15:18:43 +0000 (UTC) > Received: from mail-qk0-f176.google.com (mail-qk0-f176.google.com > [209.85.220.176]) > by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) > with ESMTPS id 298E25F1A1 > for <[email protected]>; Sun, 20 Nov 2016 15:18:43 > +0000 (UTC) > Received: by mail-qk0-f176.google.com with SMTP id n204so322548508qke.2 > for <[email protected]>; Sun, 20 Nov 2016 07:18:43 > -0800 (PST) > DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=gmail.com; s=20120113; > h=mime-version:from:date:message-id:subject:to; > bh=F12Hx9VOR79q2gJqgHEkVU41OZvdidNlPzZnqSofeQY=; > b=EHHlk+nIXTMc3i/7inugAgxJ9P7z7KRekUl+ > linLaf9HbQprAbRXxoSyo5jgSO0bgC > n+hxmf+fiowpxWs4wGQPNoL+ORjz/C5xqQVTSnGAE1XypTe1osr60Mdojt5 > cDaoC8plh > J0QtjGENpmvyDjzH/3WcOxucZ0Kl0dBRoZAUL2XCAipXQyk > KwHbMLjWAbllcsnzY463W > kE1B5rDplv96aHeVxZ4vb4ZqhZDT2GWg4+xg+PE43DYrpyDaRAnlO1ML4ZPs4mvI+ > 9F5 > JLT2rNF476ptYU/3d6Tfrr6zCVVsmogdseMNQyLt8SZSL > edVQFZZuSHA0MBFY1AUUqsM > 1K3g== > X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; > d=1e100.net; s=20130820; > h=x-gm-message-state:mime-version:from:date:message-id:subject:to; > bh=F12Hx9VOR79q2gJqgHEkVU41OZvdidNlPzZnqSofeQY=; > b=cj6cmMfk7UuYJjz+2SOshAHT2WFTsBTvYl16hatKcty+ > BF6Ke1nTFadN+zJKz7BP3y > /01uaGEUiN3W2Tbh9FHPQa8tmrR5cczzN4/ofX0C6JSKJgcTs1buI6ZHplwN6g+ > 3Hz/U > powCug1pvWV9TEmPdsH9I81oQfL1RZAOkpaWYQTO7KBdPoX8aPlV5SOtqXRe > wMBMH7JP > 5yYivOX1J/+Q/hQ34Mkw2KxlXAVlqm9KsKFwxTsZvpA > G7yZf/x1RCjrm7acfyqCcnW+4 > lWXtYNdWRNN9KGiwaVMA2yqHJzjDCVQh4UxZeDY9dz26pq3llk2YI+j+ > jp05ZcGYQ/W1 > VC0g== > X-Gm-Message-State: AKaTC00hrmDNQuCxlqf0ze7PLbfKgle2Cql/ > FkhAr2z21UkKBTjFANcLjAecey54eRxuhx4Pvnj8L66mHIBp1A== > X-Received: by 10.55.200.75 with SMTP id c72mr11712058qkj.169. > 1479655119250; > Sun, 20 Nov 2016 07:18:39 -0800 (PST) > MIME-Version: 1.0 > Received: by 10.12.160.197 with HTTP; Sun, 20 Nov 2016 07:18:38 -0800 (PST) > From: =?UTF-8?Q?Stig_D=C3=B8ssing?= <[email protected]> > Date: Sun, 20 Nov 2016 16:18:38 +0100 > Message-ID: <CAG09ER1EFVF3vMfnVqkp4xoHHX0K0Jo2i59B7cV1rwHWMq40pQ@mail. > gmail.com> > Subject: > To: [email protected] > Content-Type: multipart/alternative; boundary=001a1146e82cc5e9da0541bd0d6a > > > ---------------------------------------------------------------------- > > > > ---------- Videresendt meddelelse ---------- > From: "Cuneo, Nicholas" <[email protected]> > To: "P. Taylor Goetz" <[email protected]>, "[email protected]" < > [email protected]> > Cc: > Date: Fri, 18 Nov 2016 22:12:18 +0000 > Subject: RE: NullPointerException on startup > I’m having trouble finding that version, is it located in a separate > repository? > > Thanks, > Nick > > From: P. Taylor Goetz [mailto:[email protected]] > Sent: Friday, November 18, 2016 12:40 PM > To: [email protected]; Cuneo, Nicholas > Subject: Re: NullPointerException on startup > > (Moving thread to dev@ to discuss versions not yet released) > > Nick, > > I suspect you might be hitting a bug in 1.0.2 that might be fixed in one > of the next releases. > > Can you try setting the version of storm-kafka-client to 1.1.0-SNAPSHOT > (leave other storm components at 1.0.2)? > > If that fixes the issue, let us know on [email protected]<mailto:de > [email protected]> so we can back port the fix. > > -Taylor > > On Nov 18, 2016, at 2:10 PM, Cuneo, Nicholas <[email protected]<mailto: > [email protected]>> wrote: > > Hello, > > We are working with storm 1.0.2 and using Kafka client to subscribe to > kafka topics to retrieve data. Randomly when our topologies are starting > up we receive a null pointer exception which is killing the topology. We > can’t seem to identify why this happens, or what measures we can take to > prevent it. I’m considering raising a ticket with storm to check for null > to prevent the topology from crashing. > > 2016-11-17 23:11:05.366 o.a.s.util [ERROR] Async loop died! > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > Caused by: java.lang.NullPointerException > at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) > ~[stormjar.jar:?] > at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.daemon.executor$fn__7990$tuple_ > action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > ~[storm-core-1.0.2.jar:1.0.2] > ... 7 more > 2016-11-17 23:11:05.379 o.a.s.d.executor [ERROR] > java.lang.RuntimeException: java.lang.NullPointerException > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatch(DisruptorQueue.java:420) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.disruptor$consume_batch.invoke(disruptor.clj:69) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$fn__7990$fn__8005$fn__8036.invoke(executor.clj:628) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:484) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > Caused by: java.lang.NullPointerException > at org.apache.storm.kafka.spout.KafkaSpout.ack(KafkaSpout.java:316) > ~[stormjar.jar:?] > at org.apache.storm.daemon.executor$ack_spout_msg.invoke(executor.clj:448) > ~[storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.daemon.executor$fn__7990$tuple_ > action_fn__7996.invoke(executor.clj:536) ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.daemon.executor$mk_task_receiver$fn__7979.invoke(executor.clj:464) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.disruptor$clojure_handler$reify__7492.onEvent(disruptor.clj:40) > ~[storm-core-1.0.2.jar:1.0.2] > at > org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) > ~[storm-core-1.0.2.jar:1.0.2] > ... 7 more > 2016-11-17 23:11:05.473 o.a.s.util [ERROR] Halting process: ("Worker died") > java.lang.RuntimeException: ("Worker died") > at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?] > at org.apache.storm.daemon.worker$fn__8663$fn__8664.invoke(worker.clj:765) > [storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.daemon.executor$mk_executor_data$fn__ > 7875$fn__7876.invoke(executor.clj:274) [storm-core-1.0.2.jar:1.0.2] > at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:494) > [storm-core-1.0.2.jar:1.0.2] > at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91] > > The method and line number in question below: > > @Override > public void ack(Object messageId) { > final KafkaSpoutMessageId msgId = (KafkaSpoutMessageId) messageId; > if (!consumerAutoCommitMode) { // Only need to keep track of > acked tuples if commits are not done automatically > acked.get(msgId.getTopicPartition()).add(msgId); > } > emitted.remove(msgId); > } > > Restarting the topology sometimes fixes the issue but sometimes does not. > I don’t know if there’s some kind of race condition occurring. We have > about 15 topologies all connecting to kafka the same way, and this doesn’t > happen consistently between them. > > Does anyone have any insight as to why this occurs or how to prevent it on > our end? > > > Thanks, > > Nick Cuneo / Software Engineer, Cloud / Enterprise Software > Tel: +1 949 517 4802 / Mobile: +1 949 243 4952 > 3 Ada / Irvine, CA 92618 / USA > [email protected]<mailto:[email protected]> / www.tyco.com< > http://www.tyco.com/> > <image003.png> > > This email (including any attachments) may contain information that is > private or business confidential. If you received this email in error, > please delete it from your system without copying it and notify sender by > reply email so that our records can be corrected > > > ________________________________ > > This e-mail contains privileged and confidential information intended for > the use of the addressees named above. If you are not the intended > recipient of this e-mail, you are hereby notified that you must not > disseminate, copy or take any action in respect of any information > contained in it. If you have received this e-mail in error, please notify > the sender immediately by e-mail and immediately destroy this e-mail and > its attachments. > > > ________________________________ > > This e-mail contains privileged and confidential information intended for > the use of the addressees named above. If you are not the intended > recipient of this e-mail, you are hereby notified that you must not > disseminate, copy or take any action in respect of any information > contained in it. If you have received this e-mail in error, please notify > the sender immediately by e-mail and immediately destroy this e-mail and > its attachments. > >
