[ https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715354#comment-14715354 ]
ASF GitHub Bot commented on STORM-886: -------------------------------------- Github user revans2 commented on a diff in the pull request: https://github.com/apache/storm/pull/700#discussion_r38022357 --- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj --- @@ -114,12 +114,34 @@ (fast-list-iter [[task tuple :as pair] tuple-batch] (.serialize serializer tuple))) +(defn- mk-backpressure-handler [executors] + "make a handler that checks and updates worker's backpressure flag" + (disruptor/backpressure-handler + (fn [worker] + (let [storm-id (:storm-id worker) + assignment-id (:assignment-id worker) + port (:port worker) + storm-cluster-state (:storm-cluster-state worker)] + (if executors + (if (reduce #(or %1 %2) (map #(.get-backpressure-flag %1) executors)) + (reset! (:backpressure worker) true) ;; at least one executor has set backpressure + (reset! (:backpressure worker) false))) ;; no executor has backpressure set + ;; update the worker's backpressure flag to zookeeper here + (.worker-backpressure! storm-cluster-state storm-id assignment-id port @(:backpressure worker)) + )))) + (defn mk-transfer-fn [worker] (let [local-tasks (-> worker :task-ids set) local-transfer (:transfer-local-fn worker) - ^DisruptorQueue transfer-queue (:transfer-queue worker) + transfer-queue (:transfer-queue worker) --- End diff -- Why did we drop the type hint? > Automatic Back Pressure > ----------------------- > > Key: STORM-886 > URL: https://issues.apache.org/jira/browse/STORM-886 > Project: Apache Storm > Issue Type: Improvement > Reporter: Robert Joseph Evans > Assignee: Zhuo Liu > Attachments: an simple example for backpressure.png, backpressure.png > > > This new feature is aimed for automatic flow control through the topology DAG > since different components may have unmatched tuple processing speed. > Currently, the tuples may get dropped if the downstream components can not > process as quickly, thereby causing a waste of network bandwidth and > processing capability. In addition, it is difficult to tune the > max.spout.pending parameter for best backpressure performance. Therefore, an > automatic back pressure scheme is highly desirable. > Heron proposed a form of back pressure that does not rely on acking or max > spout pending. Instead spouts throttle not only when max.spout.pending is > hit, but also if any bolt has gone over a high water mark in their input > queue, and has not yet gone below a low water mark again. There is a lot of > room for potential improvement here around control theory and having spouts > only respond to downstream bolts backing up, but a simple bang-bang > controller like this is a great start. -- This message was sent by Atlassian JIRA (v6.3.4#6332)