I am using the latest, 0.9.3 and I don't use the shell bolts. I don't have any explicit multi-threading in my bolts, either. The issue seems like some kind of memory leak and storm is running out of memory. When I monitor the processes, I don't see any memory reclaimed even after the tuples have reached the final destination (the remote web servers). I did synchronize the collector, and ack/fail the tuples where necessary, but the issue persists.
On Mon, Jan 5, 2015 at 9:01 AM, Bobby Evans <ev...@yahoo-inc.com> wrote: > What version of storm are you using? Are any of the bolts shell bolts? > There is a known issue where this can happen if two shell bolts share an > executor, because they are multi-threaded. > > - Bobby > > > On Sunday, January 4, 2015 10:19 PM, clay teahouse < > clayteaho...@gmail.com> wrote: > > > Hi All, > I have the following topology > spout -> Bolt1 --> Bolt2 > > Neither bolts are async or multi-threaded. Bolt2 uses http client to make > post/put requests to a web server. Both bolts ack the tuples before exiting > the execute. > > The topology runs fine for a while under a load of about 50MB/minute. > After about 10-15 minutes the topology dies with the following error. Any > hint what the issue might be? I'd appreciate any feedback. > > java.lang.RuntimeException: java.lang.NullPointerException > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$consume_loop_STAR_$fn__759.invoke(disruptor.clj:94) > ~[storm-core-0.9.3.jar:0.9.3] > at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) > ~[storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: java.lang.NullPointerException: null > at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na] > at > backtype.storm.daemon.worker$mk_transfer_fn$fn__3548.invoke(worker.clj:129) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3282.invoke(executor.clj:258) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$clojure_handler$reify__746.onEvent(disruptor.clj:58) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > ~[storm-core-0.9.3.jar:0.9.3] > ... 6 common frames omitted > 1059203 [Thread-76-disruptor-executor[34 34]-send-queue] ERROR > backtype.storm.daemon.executor - > java.lang.RuntimeException: java.lang.NullPointerException > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:128) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:99) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:80) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$consume_loop_STAR_$fn__759.invoke(disruptor.clj:94) > ~[storm-core-0.9.3.jar:0.9.3] > at backtype.storm.util$async_loop$fn__458.invoke(util.clj:463) > ~[storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > Caused by: java.lang.NullPointerException: null > at clojure.lang.RT.intCast(RT.java:1087) ~[clojure-1.5.1.jar:na] > at > backtype.storm.daemon.worker$mk_transfer_fn$fn__3548.invoke(worker.clj:129) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$start_batch_transfer__GT_worker_handler_BANG_$fn__3282.invoke(executor.clj:258) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.disruptor$clojure_handler$reify__746.onEvent(disruptor.clj:58) > ~[storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:125) > ~[storm-core-0.9.3.jar:0.9.3] > ... 6 common frames omitted > 1060135 [Thread-76-disruptor-executor[34 34]-send-queue] ERROR > backtype.storm.util - Halting process: ("Worker died") > java.lang.RuntimeException: ("Worker died") > at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.RestFn.invoke(RestFn.java:423) > [clojure-1.5.1.jar:na] > at > backtype.storm.daemon.worker$fn__3807$fn__3808.invoke(worker.clj:452) > [storm-core-0.9.3.jar:0.9.3] > at > backtype.storm.daemon.executor$mk_executor_data$fn__3273$fn__3274.invoke(executor.clj:240) > [storm-core-0.9.3.jar:0.9.3] > at backtype.storm.util$async_loop$fn__458.invoke(util.clj:473) > [storm-core-0.9.3.jar:0.9.3] > at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] > at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65] > > > >