[ https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368525#comment-15368525 ]
Robert Joseph Evans commented on STORM-1949: -------------------------------------------- [~roshan_naik], When writing to ZK every 100 ms we will check to be sure that the ZK state matches the local worker state, just in case there is a race condition and they got off some how. So yes we should have a backup thread to poll ZK for backpressure in case we get off some how with the triggers. As for the load on ZooKeeper yes, that too is one of my concerns, and we at Yahoo have been holding off turning it on by default until it has stabilized a bit and until we have moved all of our clusters to pacemaker. We expect to turn it on fairly soon. How expensive is it though? If you are running zookeeper in the default configuration it can be very expensive because you are constrained by a single disk's IOps, and if you have lots of throttling going on then those IOps can get eaten up fairly quickly. If you have the flush turned off, then it is going to be constrained by the write log to the disk, or by the network whichever saturates first. If it is the write log then you get about 80MB/sec of writes, this is just writing a small amount of metadata, so it should not have that big of an impact, but I have not really measured it. If the network then reads will likely saturate first because of the listings, but again this is a relatively small amount of metadata, so it should have a relatively small impact. If you want to turn it off in defaults.yaml that is fine, just file a separate JIRA for that. If you have a better solution that will reduce the load on ZK that would be great, again file another JIRA for that and we can discuss details there. I am skeptical that we will get anywhere near as good of a throughput if we put in a delay of some sort, and I think we would start to run into issues with heaps filling up if we put in too much of a delay. > Backpressure can cause spout to stop emitting and stall topology > ---------------------------------------------------------------- > > Key: STORM-1949 > URL: https://issues.apache.org/jira/browse/STORM-1949 > Project: Apache Storm > Issue Type: Bug > Reporter: Roshan Naik > > Problem can be reproduced by this [Word count > topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java] > within a IDE. > I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt > instances. > The problem is more easily reproduced with WC topology as it causes an > explosion of tuples due to splitting a sentence tuple into word tuples. As > the bolts have to process more tuples than the spout is producing, spout > needs to operate slower. > The amount of time it takes for the topology to stall can vary.. but > typically under 10 mins. > *My theory:* I suspect there is a race condition in the way ZK is being > utilized to enable/disable back pressure. When congested (i.e pressure > exceeds high water mark), the bolt's worker records this congested situation > in ZK by creating a node. Once the congestion is reduced below the low water > mark, it deletes this node. > The spout's worker has setup a watch on the parent node, expecting a callback > whenever there is change in the child nodes. On receiving the callback the > spout's worker lists the parent node to check if there are 0 or more child > nodes.... it is essentially trying to figure out the nature of state change > in ZK to determine whether to throttle or not. Subsequently it setsup > another watch in ZK to keep an eye on future changes. > When there are multiple bolts, there can be rapid creation/deletion of these > ZK nodes. Between the time the worker receives a callback and sets up the > next watch.. many changes may have undergone in ZK which will go unnoticed by > the spout. > The condition that the bolts are no longer congested may not get noticed as a > result of this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)