I wanted to look at bolt stack specifically. With the names removed, it is difficult to figure out. can you tell me what are the bolts/spouts busy in? Are they very slow? The timer which publishes metric tick is blocked because the receive queue of some executor is full.
Some related details are captured here - https://issues.apache.org/jira/browse/STORM-972 On Mon, Apr 11, 2016 at 2:28 PM, Denis DEBARBIEUX <ddebarbi...@norsys.fr> wrote: > Jungtaek, > > Thanks for your help. > > Please find enclose the jstack file: > - I remove by hand all name related to my project (spout, bolt...). > - The running jdk was openJDK (not an oracle JDK). > > I also increase the buffer size (topology.executor.receive.buffer.size and > topology.executor.send.buffer.size) to 32768 and lauch my topology on 2 > workers. Then the dead lock occurs after 3 days. At this moment 24577 > points have been send. > > Please feel free to contact me if you need further information. > > Denis > > Le 29/03/2016 11:11, Jungtaek Lim a écrit : > > Denis, > > Could you share your jstack output which contains locked information? > And I'm not sure but you may hit STORM-1027 > <https://issues.apache.org/jira/browse/STORM-1027> which is resolved at > 0.10.0 and 0.9.6. > > Thanks, > Jungtaek Lim (HeartSaVioR) > > 2016년 3월 29일 (화) 오후 5:41, Denis DEBARBIEUX < <ddebarbi...@norsys.fr> > ddebarbi...@norsys.fr> 님이 작성: > >> Hi all, >> >> I profile my code and I see that the thread <BoldName>MetricConsumer is >> locked and never unlocked. It is why my metrics are never published. >> >> More over, the problem is related to buffer size: >> * topology.executor.receive.buffer.size=16 and >> topology.executor.send.buffer.size=16 implies a blocking after 4 minutes. >> * with a size=512, I am able to run my test for more than 1 hour. >> >> >> Thanks for your help. >> >> Denis >> >> Le 18/03/2016 22:38, Abhishek Agarwal a écrit : >> >> Do you get this problem if you use only the LoggingMetricsConsumer? >> >> Excuse typos >> On Mar 18, 2016 9:38 PM, "Denis DEBARBIEUX" <ddebarbi...@norsys.fr> >> wrote: >> >>> Dear all, >>> >>> I have a question about metrics and how they are consumed: >>> I launch my topology. During 3 or 4 minutes, I receive my metrics. Then >>> no more logs. >>> It's look like the process that is consuming the metrics has a dead lock >>> or is blocked by a time out. >>> >>> Here comes some information about the context: >>> - I am using storm 0.9.4. >>> - Some metrics are defined by defaut (eg emit-count.default) some other >>> are customed. >>> - I am sure that my topology is still running. >>> - According to Storm UI, the number of messages emiting by my bolts is >>> increasing. In other words, metrics are still computing and send somewhere. >>> - I try two instances of LoggingMetricsConsumer: in file >>> (backtype.storm.metric.LoggingMetricsConsumer) and in graphite >>> (com.verisign.storm.metrics.GraphiteMetricsConsumer). >>> - I can reproduce it on 2 different machines. >>> - No error are logged (worker, metric, supervisor). >>> - I am using storm since 1 year and I never seen this problem. >>> >>> Thanks for your help. >>> >>> Denis >>> >>> --- >>> L'absence de virus dans ce courrier électronique a été vérifiée par le >>> logiciel antivirus Avast. >>> https://www.avast.com/antivirus >>> >>> >> >> >> ------------------------------ >> [image: Avast logo] >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> >> >> L'absence de virus dans ce courrier électronique a été vérifiée par le >> logiciel antivirus Avast. >> >> >> <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> >> www.avast.com >> >> > > > ------------------------------ > [image: Avast logo] > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > > L'absence de virus dans ce courrier électronique a été vérifiée par le > logiciel antivirus Avast. > www.avast.com > <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> > > -- Regards, Abhishek Agarwal
