[
https://issues.apache.org/jira/browse/STORM-886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14715368#comment-14715368
]
ASF GitHub Bot commented on STORM-886:
--------------------------------------
Github user revans2 commented on a diff in the pull request:
https://github.com/apache/storm/pull/700#discussion_r38023405
--- Diff: storm-core/src/clj/backtype/storm/daemon/worker.clj ---
@@ -510,15 +554,21 @@
))
)
credentials (atom initial-credentials)
- check-credentials-changed (fn []
- (let [new-creds (.credentials
(:storm-cluster-state worker) storm-id nil)]
- (when-not (= new-creds @credentials)
;;This does not have to be atomic, worst case we update when one is not needed
- (AuthUtils/updateSubject subject
auto-creds new-creds)
- (dofor [e @executors]
(.credentials-changed e new-creds))
- (reset! credentials new-creds))))
- ]
- (.credentials (:storm-cluster-state worker) storm-id (fn [args]
(check-credentials-changed)))
- (schedule-recurring (:refresh-credentials-timer worker) 0 (conf
TASK-CREDENTIALS-POLL-SECS) check-credentials-changed)
+ check-credentials-throttle-changed (fn []
+ (let [callback (fn cb [&
ignored]
+ (let
[throttle-on (.topology-backpressure (:storm-cluster-state worker) storm-id cb)]
+ (reset!
(:throttle-on worker) throttle-on)))
+ new-throttle-on (if
((:storm-conf worker) TOPOLOGY-BACKPRESSURE-ENABLE)
+
(.topology-backpressure (:storm-cluster-state worker) storm-id callback) nil)
+ new-creds (.credentials
(:storm-cluster-state worker) storm-id nil)]
+ (if ((:storm-conf worker)
TOPOLOGY-BACKPRESSURE-ENABLE)
+ (reset! (:throttle-on
worker) new-throttle-on))
+ (when-not (= new-creds
@credentials) ;;This does not have to be atomic, worst case we update when one
is not needed
+ (AuthUtils/updateSubject
subject auto-creds new-creds)
+ (dofor [e @executors]
(.credentials-changed e new-creds))
+ (reset! credentials
new-creds))))]
+ (.credentials (:storm-cluster-state worker) storm-id (fn [args]
(check-credentials-throttle-changed)))
--- End diff --
I think I would prefer to have two functions instead of just one. Both can
be scheduled on the same timer, but keeping them separate seems cleaner.
> Automatic Back Pressure
> -----------------------
>
> Key: STORM-886
> URL: https://issues.apache.org/jira/browse/STORM-886
> Project: Apache Storm
> Issue Type: Improvement
> Reporter: Robert Joseph Evans
> Assignee: Zhuo Liu
> Attachments: an simple example for backpressure.png, backpressure.png
>
>
> This new feature is aimed for automatic flow control through the topology DAG
> since different components may have unmatched tuple processing speed.
> Currently, the tuples may get dropped if the downstream components can not
> process as quickly, thereby causing a waste of network bandwidth and
> processing capability. In addition, it is difficult to tune the
> max.spout.pending parameter for best backpressure performance. Therefore, an
> automatic back pressure scheme is highly desirable.
> Heron proposed a form of back pressure that does not rely on acking or max
> spout pending. Instead spouts throttle not only when max.spout.pending is
> hit, but also if any bolt has gone over a high water mark in their input
> queue, and has not yet gone below a low water mark again. There is a lot of
> room for potential improvement here around control theory and having spouts
> only respond to downstream bolts backing up, but a simple bang-bang
> controller like this is a great start.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
