Hi,
*Version:*
Storm : 0.10.0
Streamparse : 2.1.4
I am running a storm topology with a python streamparse "sparse run".
This topology stops executing in the middle and throw an exception
158031 [pool-37-thread-1] ERROR b.s.s.ShellSpout - Halting process:
> ShellSpout died.
> java.lang.RuntimeException: *subprocess heartbeat timeout*
> at
> backtype.storm.spout.ShellSpout$SpoutHeartbeatTimerTask.run(ShellSpout.java:261)
> [storm-core-0.10.0.jar:0.10.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_40]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_40]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
> 158036 [pool-37-thread-1] ERROR b.s.d.executor -
> java.lang.RuntimeException: subprocess heartbeat timeout
> at
> backtype.storm.spout.ShellSpout$SpoutHeartbeatTimerTask.run(ShellSpout.java:261)
> [storm-core-0.10.0.jar:0.10.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [?:1.8.0_40]
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> [?:1.8.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> [?:1.8.0_40]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> [?:1.8.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_40]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_40]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
>
This occurs randomly
I can't able to trace back to the problem
Maybe if spout takes too long to process, then streamparse can't
acknowledge the heartbeat in time
changed "supervisor.worker.timeout.secs" from 30 to 600
Still the topology breaks. Is there any other options here?
Thanks,
Ram.
