[
https://issues.apache.org/jira/browse/STORM-1949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368525#comment-15368525
]
Robert Joseph Evans commented on STORM-1949:
--------------------------------------------
[~roshan_naik],
When writing to ZK every 100 ms we will check to be sure that the ZK state
matches the local worker state, just in case there is a race condition and they
got off some how. So yes we should have a backup thread to poll ZK for
backpressure in case we get off some how with the triggers.
As for the load on ZooKeeper yes, that too is one of my concerns, and we at
Yahoo have been holding off turning it on by default until it has stabilized a
bit and until we have moved all of our clusters to pacemaker. We expect to
turn it on fairly soon. How expensive is it though? If you are running
zookeeper in the default configuration it can be very expensive because you are
constrained by a single disk's IOps, and if you have lots of throttling going
on then those IOps can get eaten up fairly quickly. If you have the flush
turned off, then it is going to be constrained by the write log to the disk, or
by the network whichever saturates first. If it is the write log then you get
about 80MB/sec of writes, this is just writing a small amount of metadata, so
it should not have that big of an impact, but I have not really measured it.
If the network then reads will likely saturate first because of the listings,
but again this is a relatively small amount of metadata, so it should have a
relatively small impact.
If you want to turn it off in defaults.yaml that is fine, just file a separate
JIRA for that. If you have a better solution that will reduce the load on ZK
that would be great, again file another JIRA for that and we can discuss
details there. I am skeptical that we will get anywhere near as good of a
throughput if we put in a delay of some sort, and I think we would start to run
into issues with heaps filling up if we put in too much of a delay.
> Backpressure can cause spout to stop emitting and stall topology
> ----------------------------------------------------------------
>
> Key: STORM-1949
> URL: https://issues.apache.org/jira/browse/STORM-1949
> Project: Apache Storm
> Issue Type: Bug
> Reporter: Roshan Naik
>
> Problem can be reproduced by this [Word count
> topology|https://github.com/hortonworks/storm/blob/perftopos1.x/examples/storm-starter/src/jvm/org/apache/storm/starter/perf/FileReadWordCountTopo.java]
> within a IDE.
> I ran it with 1 spout instance, 2 splitter bolt instances, 2 counter bolt
> instances.
> The problem is more easily reproduced with WC topology as it causes an
> explosion of tuples due to splitting a sentence tuple into word tuples. As
> the bolts have to process more tuples than the spout is producing, spout
> needs to operate slower.
> The amount of time it takes for the topology to stall can vary.. but
> typically under 10 mins.
> *My theory:* I suspect there is a race condition in the way ZK is being
> utilized to enable/disable back pressure. When congested (i.e pressure
> exceeds high water mark), the bolt's worker records this congested situation
> in ZK by creating a node. Once the congestion is reduced below the low water
> mark, it deletes this node.
> The spout's worker has setup a watch on the parent node, expecting a callback
> whenever there is change in the child nodes. On receiving the callback the
> spout's worker lists the parent node to check if there are 0 or more child
> nodes.... it is essentially trying to figure out the nature of state change
> in ZK to determine whether to throttle or not. Subsequently it setsup
> another watch in ZK to keep an eye on future changes.
> When there are multiple bolts, there can be rapid creation/deletion of these
> ZK nodes. Between the time the worker receives a callback and sets up the
> next watch.. many changes may have undergone in ZK which will go unnoticed by
> the spout.
> The condition that the bolts are no longer congested may not get noticed as a
> result of this.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
