GitHub user zhuoliu opened a pull request: https://github.com/apache/storm/pull/700
[STORM-886] Automatic Back Pressure (ABP) This new feature is aimed for automatic flow control through the topology DAG since different components may have unmatched tuple processing speed. Currently, the tuples may get dropped if the downstream components can not process as quickly, thereby causing a waste of network bandwidth and processing capability. In addition, it is difficult to tune the max.spout.pending parameter for best backpressure performance. Therefore, an automatic back pressure scheme is highly desirable. Recently, Heron proposed a form of back pressure that does not rely on acking or max spout pending. Instead spouts throttle not only when max.spout.pending is hit, but also if any bolt has gone over a high water mark in their input queue, and has not yet gone below a low water mark again. There is a lot of room for potential improvement here around control theory and having spouts only respond to downstream bolts backing up, but a simple bang-bang controller like this is a great start. Our ABP scheme implements a light-weight yet efficient back pressure scheme. It monitors certain queues in executors and exploits the callback schemes on ZooKeeper and disruptor queue for a fast-responding (in a push manner) flow control. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhuoliu/storm 886 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/storm/pull/700.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #700 ---- commit fc2fcf41b8977b833d7f7f883783263e3b422c45 Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-18T18:47:39Z Initial commit for YSTORM-1949 automatic backpressure commit 3316a70b314474c900cf319b355f13f3567febfa Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-21T21:15:31Z Temp version that can do both throttle and back-to-normal, WordCountTopology changed, may revert it later commit 5c5f34d4f63df08d69871b1b0d395a96768283e1 Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-21T21:24:25Z re-enable the recurring of throttle check to zk in worker commit 52615833e1b88ac55b91000687a4d9123faf9662 Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-25T18:06:31Z Use ephemeral-node for worker backpressure commit 21a314b3b19f79693ce1bb7257d42961266a0afa Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-25T21:04:51Z Make backpressure configurable commit ae194073612477636651efc2668a4623a4785020 Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-25T21:55:44Z use disruptor/notify-backpressure-checker commit 7996116072e45b13267b86045cef6d777cf3fb04 Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-25T22:35:14Z Remove debug message for daemons commit fdd75728ce51ac52f48ad091db4a623ba1fd7f0c Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-26T02:26:10Z Minor cleanup commit ae2ed251c79c640854f336ca4e1b13e3dd7aabe5 Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-26T02:29:07Z Minor commit 8c31ea6b081a21459d9dff7a0687fd6de9bfe8bb Author: zhuol <zh...@yahoo-inc.com> Date: 2015-08-26T16:04:48Z Decomment tests ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---