[
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhuo Liu updated STORM-886:
---------------------------
Description:
This new feature is aimed for automatic flow control through the topology DAG
since different components may have unmatched tuple processing speed.
Currently, the tuples may get dropped if the downstream components can not
process as quickly, thereby causing a waste of network bandwidth and processing
capability. In addition, it is difficult to tune the max.spout.pending
parameter for best backpressure performance. Another big motivation is that
using max.spout.pending for flow control forces users to enable acking, which
does not make sense.
Therefore, an automatic back pressure scheme is highly desirable.
Heron proposed a form of back pressure that does not rely on acking or max
spout pending. Instead spouts throttle not only when max.spout.pending is hit,
but also if any bolt has gone over a high water mark in their input queue, and
has not yet gone below a low water mark again. There is a lot of room for
potential improvement here around control theory and having spouts only respond
to downstream bolts backing up, but a simple bang-bang controller like this is
a great start.
was:
This new feature is aimed for automatic flow control through the topology DAG
since different components may have unmatched tuple processing speed.
Currently, the tuples may get dropped if the downstream components can not
process as quickly, thereby causing a waste of network bandwidth and processing
capability. In addition, it is difficult to tune the max.spout.pending
parameter for best backpressure performance. Therefore, an automatic back
pressure scheme is highly desirable.
Heron proposed a form of back pressure that does not rely on acking or max
spout pending. Instead spouts throttle not only when max.spout.pending is hit,
but also if any bolt has gone over a high water mark in their input queue, and
has not yet gone below a low water mark again. There is a lot of room for
potential improvement here around control theory and having spouts only respond
to downstream bolts backing up, but a simple bang-bang controller like this is
a great start.
> Automatic Back Pressure
> -----------------------
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-core
> Reporter: Robert Joseph Evans
> Assignee: Zhuo Liu
> Fix For: 0.11.0
>
> Attachments: aSimpleExampleOfBackpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG
> since different components may have unmatched tuple processing speed.
> Currently, the tuples may get dropped if the downstream components can not
> process as quickly, thereby causing a waste of network bandwidth and
> processing capability. In addition, it is difficult to tune the
> max.spout.pending parameter for best backpressure performance. Another big
> motivation is that using max.spout.pending for flow control forces users to
> enable acking, which does not make sense.
> Therefore, an automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that does not rely on acking or max
> spout pending. Instead spouts throttle not only when max.spout.pending is
> hit, but also if any bolt has gone over a high water mark in their input
> queue, and has not yet gone below a low water mark again. There is a lot of
> room for potential improvement here around control theory and having spouts
> only respond to downstream bolts backing up, but a simple bang-bang
> controller like this is a great start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
