GitHub user zhuoliu opened a pull request:
https://github.com/apache/storm/pull/700
[STORM-886] Automatic Back Pressure (ABP)
This new feature is aimed for automatic flow control through the topology
DAG since different components may have unmatched tuple processing speed.
Currently, the tuples may get dropped if the downstream components can not
process as quickly, thereby causing a waste of network bandwidth and processing
capability. In addition, it is difficult to tune the max.spout.pending
parameter for best backpressure performance. Therefore, an automatic back
pressure scheme is highly desirable.
Recently, Heron proposed a form of back pressure that does not rely on
acking or max spout pending. Instead spouts throttle not only when
max.spout.pending is hit, but also if any bolt has gone over a high water mark
in their input queue, and has not yet gone below a low water mark again. There
is a lot of room for potential improvement here around control theory and
having spouts only respond to downstream bolts backing up, but a simple
bang-bang controller like this is a great start.
Our ABP scheme implements a light-weight yet efficient back pressure
scheme. It monitors certain queues in executors and exploits the callback
schemes on ZooKeeper and disruptor queue for a fast-responding (in a push
manner) flow control.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhuoliu/storm 886
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/700.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #700
----
commit fc2fcf41b8977b833d7f7f883783263e3b422c45
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-18T18:47:39Z
Initial commit for YSTORM-1949 automatic backpressure
commit 3316a70b314474c900cf319b355f13f3567febfa
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-21T21:15:31Z
Temp version that can do both throttle and back-to-normal,
WordCountTopology changed, may revert it later
commit 5c5f34d4f63df08d69871b1b0d395a96768283e1
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-21T21:24:25Z
re-enable the recurring of throttle check to zk in worker
commit 52615833e1b88ac55b91000687a4d9123faf9662
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-25T18:06:31Z
Use ephemeral-node for worker backpressure
commit 21a314b3b19f79693ce1bb7257d42961266a0afa
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-25T21:04:51Z
Make backpressure configurable
commit ae194073612477636651efc2668a4623a4785020
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-25T21:55:44Z
use disruptor/notify-backpressure-checker
commit 7996116072e45b13267b86045cef6d777cf3fb04
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-25T22:35:14Z
Remove debug message for daemons
commit fdd75728ce51ac52f48ad091db4a623ba1fd7f0c
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-26T02:26:10Z
Minor cleanup
commit ae2ed251c79c640854f336ca4e1b13e3dd7aabe5
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-26T02:29:07Z
Minor
commit 8c31ea6b081a21459d9dff7a0687fd6de9bfe8bb
Author: zhuol <zh...@yahoo-inc.com>
Date: 2015-08-26T16:04:48Z
Decomment tests
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
