[
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14716636#comment-14716636
]
ASF GitHub Bot commented on STORM-886:
--------------------------------------
Github user revans2 commented on the pull request:
https://github.com/apache/storm/pull/700#issuecomment-135422890
@arunmahadevan It is the intention to throttle the entire topology, all
spouts. This is what Heron does and is intended to be a last resort, which is
better then nothing, but not a truly final solution. The reason for this is
that from the level of a single queue it is very difficult to know what is
causing that congestion. STORM-907 is intended as a follow on that will analyze
the topology, determine where there are loops and provide more of true
backpressure. But in the case of a loop, and storm does support loops, you
have no way to determine which upstream part is causing slowness. And in fact
it could be the bolt itself, and it needs to increase it's parallelism.
> Automatic Back Pressure
> -----------------------
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Robert Joseph Evans
> Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG
> since different components may have unmatched tuple processing speed.
> Currently, the tuples may get dropped if the downstream components can not
> process as quickly, thereby causing a waste of network bandwidth and
> processing capability. In addition, it is difficult to tune the
> max.spout.pending parameter for best backpressure performance. Therefore, an
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that does not rely on acking or max
> spout pending. Instead spouts throttle not only when max.spout.pending is
> hit, but also if any bolt has gone over a high water mark in their input
> queue, and has not yet gone below a low water mark again. There is a lot of
> room for potential improvement here around control theory and having spouts
> only respond to downstream bolts backing up, but a simple bang-bang
> controller like this is a great start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
