[
https://issues.apache.org/jira/browse/BEAM-6540?focusedWorklogId=194869&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-194869
]
ASF GitHub Bot logged work on BEAM-6540:
----------------------------------------
Author: ASF GitHub Bot
Created on: 06/Feb/19 00:01
Start Date: 06/Feb/19 00:01
Worklog Time Spent: 10m
Work Description: dustin12 commented on pull request #7664: [BEAM-6540]
Windmill quota piping
URL: https://github.com/apache/beam/pull/7664#discussion_r254095787
##########
File path:
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/GrpcWindmillServer.java
##########
@@ -1416,4 +1458,62 @@ public void close() throws IOException {
stream.close();
}
}
+
+ /**
+ * A stopwatch used to track the amount of time spent throttled due to
Resource Exhausted errors.
+ * Throttle time is cumulative for all three rpcs types but not for all
streams. So if GetWork and
+ * CommitWork are both blocked for x, totalTime will be 2x. However, if 2
GetWork streams are both
+ * blocked for x totalTime will be x. All methods are thread safe.
+ */
+ private static class ThrottleTimer {
+
+ // This is -1 if not currently being throttled or the time in
+ // milliseconds when throttling for this type started.
+ private Long startTime = Long.valueOf(-1);
+ // This is the collected total throttle times since the last poll.
Throttle times are
+ // reported as a delta so this is cleared whenever it gets reported.
+ private Long totalTime = Long.valueOf(0);
+
+ /**
+ * Starts the timer if it has not been started and does nothing if it has
already been started.
+ */
+ public synchronized void start() {
+ if (!throttled()) { // This timer is not started yet so start it now.
+ startTime = Instant.now().getMillis();
+ }
+ }
+
+ /** Stops the timer if it has been started and does nothing if it has not
been started. */
+ public synchronized void stop() {
+ if (throttled()) { // This timer has been started already so stop it now.
+ totalTime = totalTime + getCurrentThrottleTime();
+ resetStartTime();
+ }
+ }
+
+ /** Returns if the specified type is currently being throttled */
+ public synchronized boolean throttled() {
+ return startTime != -1;
+ }
+
+ /** Returns the combined total of all throttle times and resets those
times to 0. */
+ public synchronized long getAndResetThrottleTime() {
+ if (throttled()) {
+ stop();
+ start();
+ }
+ Long toReturn = totalTime;
+ totalTime = Long.valueOf(0);
+ return toReturn;
+ }
+
+ private synchronized long getCurrentThrottleTime() {
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 194869)
Time Spent: 7h (was: 6h 50m)
> Autoscaling should be aware of Streaming RPC Quota
> ---------------------------------------------------
>
> Key: BEAM-6540
> URL: https://issues.apache.org/jira/browse/BEAM-6540
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Affects Versions: 2.11.0
> Reporter: Dustin Rhodes
> Assignee: Tyler Akidau
> Priority: Major
> Fix For: 2.11.0
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Streaming Windmill Service introduces quota for the shared windmill workers.
> Autoscaling needs to be aware of throttling due to this quota in order to not
> upscale. This PR adds in that reporting.
>
> It also introduces the flag --EnableStreamingEngine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
