> Supervisor log at the time of freeze looks like below > > 2017-07-12 14:38:46.712 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started
There are two situations where you would see those messages: When a topology is first deployed, and when a worker has died and is being restarted. I suspect the latter. Have you looked at the worker logs for any indication that the workers might be crashing and what might be causing it? What components are involved in you’re topology? -Taylor > On Jul 12, 2017, at 5:26 AM, Sreeram <[email protected]> wrote: > > Hi, > > I am observing that my storm topology intermediately freezes and does > not continue to process tuples from Kafka. This happens frequently and > when it happens this freeze lasts for 5 to 15 minutes. No content is > written to any of the worker log files during this time. > > The version of storm I use is 1.0.2 and Kafka version is 0.9.0. > > Any suggestions to solve the issue ? > > Thanks, > Sreeram > > Supervisor log at the time of freeze looks like below > > 2017-07-12 14:38:46.712 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:47.212 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:47.712 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:48.213 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:48.713 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:49.213 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:49.713 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:50.214 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > 2017-07-12 14:38:50.714 o.a.s.d.supervisor [INFO] > d8958816-5bc8-449e-94e3-87ddbb2c3d02 still hasn't started > > > Thread stacks (sample) > Most of worker threads during this freeze period look like one of the > below two stack traces. > > Thread 104773: (state = BLOCKED) > - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; > information may be imprecise) > - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, > long) @bci=20, line=215 (Compiled frame) > - > java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.util.concurrent.SynchronousQueue$TransferStack$SNode, > boolean, long) @bci=160, line=460 (Compil > ed frame) > - > java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.lang.Object, > boolean, long) @bci=102, line=362 (Compiled frame) > - java.util.concurrent.SynchronousQueue.poll(long, > java.util.concurrent.TimeUnit) @bci=11, line=941 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=134, > line=1066 (Compiled frame) > - > java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) > @bci=26, line=1127 (Compiled frame) > - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, > line=617 (Compiled frame) > - java.lang.Thread.run() @bci=11, line=745 (Compiled frame) > > Thread 147495: (state = IN_NATIVE) > - sun.nio.ch.EPollArrayWrapper.epollWait(long, int, long, int) @bci=0 > (Compiled frame; information may be imprecise) > - sun.nio.ch.EPollArrayWrapper.poll(long) @bci=18, line=269 (Compiled frame) > - sun.nio.ch.EPollSelectorImpl.doSelect(long) @bci=28, line=93 (Compiled > frame) > - sun.nio.ch.SelectorImpl.lockAndDoSelect(long) @bci=37, line=86 > (Compiled frame) > - sun.nio.ch.SelectorImpl.select(long) @bci=30, line=97 (Compiled frame) > - org.apache.kafka.common.network.Selector.select(long) @bci=35, > line=425 (Compiled frame) > - org.apache.kafka.common.network.Selector.poll(long) @bci=81, > line=254 (Compiled frame) > - org.apache.kafka.clients.NetworkClient.poll(long, long) @bci=84, > line=270 (Compiled frame) > - org.apache.kafka.clients.producer.internals.Sender.run(long) > @bci=343, line=216 (Compiled frame) > - org.apache.kafka.clients.producer.internals.Sender.run() @bci=27, > line=128 (Interpreted frame) > - java.lang.Thread.run() @bci=11, line=745 (Compiled frame)
