Hello, Thank you for your replies!
On Fri, Mar 3, 2017 at 1:36 AM, Jungtaek Lim <kabh...@gmail.com> wrote: > Each executor has backup unbounded buffer for queue since without that > deadlock may occur. > > 'topology.max.spout.pending' was introduced much earlier than > backpressure. It was only way to throttle, and still valid for > non-backpressure-activated topology. > Backpressure doesn't work smoothly so having good value of max spout > pending is still better than relying on backpressure. (Indeed we disable > backpressure by default.) It should be addressed. > > - Jungtaek Lim (HeartSaVioR) > > 2017년 3월 3일 (금) 오전 7:42, Erik Weathers <eweath...@groupon.com>님이 작성: > > The miguno blog post is a bit out of date, it predates the switch from > ZeroMQ to Netty as the communication layer between workers. > > Notably, netty has an unbounded buffer (at least in version 0.9.6): > > - https://github.com/apache/storm/blob/v0.9.6/storm-core/ > src/jvm/backtype/storm/messaging/netty/Server.java#L97 > > <https://github.com/apache/storm/blob/v0.9.6/storm-core/src/jvm/backtype/storm/messaging/netty/Server.java#L97> > > > On Thu, Mar 2, 2017 at 1:28 PM, David Koch <ogd...@googlemail.com> wrote: > > Hi, > > The Storm documentation mentions setting topology.max.spout.pending as a > way of preventing "queue explosion"[1]. What is meant by this? Tuples > piling up and eventually causing out of memory exceptions? If I understand > correctly, the topology's queue and buffer sizes are all limited [2] - so > at what point could something explode even without limiting the maximum > number of pending tuples and/or the back pressure mechanism activated. > > Thanks, > > David > > [1] http://storm.apache.org/releases/1.0.0/Running- > topologies-on-a-production-cluster.html > [2] http://www.michael-noll.com/blog/2013/06/21/ > understanding-storm-internal-message-buffers/ > > >
