[
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715354#comment-14715354
]
ASF GitHub Bot commented on STORM-886:
--------------------------------------
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38022357
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -114,12 +114,34 @@
(fast-list-iter [[task tuple :as pair] tuple-batch]
(.serialize serializer tuple)))
+(defn- mk-backpressure-handler [executors]
+ "make a handler that checks and updates worker's backpressure flag"
+ (disruptor/backpressure-handler
+ (fn [worker]
+ (let [storm-id (:storm-id worker)
+ assignment-id (:assignment-id worker)
+ port (:port worker)
+ storm-cluster-state (:storm-cluster-state worker)]
+ (if executors
+ (if (reduce #(or %1 %2) (map #(.get-backpressure-flag %1)
executors))
+ (reset! (:backpressure worker) true) ;; at least one
executor has set backpressure
+ (reset! (:backpressure worker) false))) ;; no executor has
backpressure set
+ ;; update the worker's backpressure flag to zookeeper here
+ (.worker-backpressure! storm-cluster-state storm-id assignment-id
port @(:backpressure worker))
+ ))))
+
(defn mk-transfer-fn [worker]
(let [local-tasks (-> worker :task-ids set)
local-transfer (:transfer-local-fn worker)
- ^DisruptorQueue transfer-queue (:transfer-queue worker)
+ transfer-queue (:transfer-queue worker)
--- End diff --
Why did we drop the type hint?
> Automatic Back Pressure
> -----------------------
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Robert Joseph Evans
> Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG
> since different components may have unmatched tuple processing speed.
> Currently, the tuples may get dropped if the downstream components can not
> process as quickly, thereby causing a waste of network bandwidth and
> processing capability. In addition, it is difficult to tune the
> max.spout.pending parameter for best backpressure performance. Therefore, an
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that does not rely on acking or max
> spout pending. Instead spouts throttle not only when max.spout.pending is
> hit, but also if any bolt has gone over a high water mark in their input
> queue, and has not yet gone below a low water mark again. There is a lot of
> room for potential improvement here around control theory and having spouts
> only respond to downstream bolts backing up, but a simple bang-bang
> controller like this is a great start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
