[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=294318&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294318 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 14/Aug/19 01:22 Start Date: 14/Aug/19 01:22 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 294318) Time Spent: 2h 10m (was: 2h) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=294184&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294184 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 13/Aug/19 21:05 Start Date: 13/Aug/19 21:05 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#issuecomment-521008896 LGTM. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 294184) Time Spent: 1h 50m (was: 1h 40m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=294185&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-294185 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 13/Aug/19 21:05 Start Date: 13/Aug/19 21:05 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#issuecomment-521008937 Retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 294185) Time Spent: 2h (was: 1h 50m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=289987&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-289987 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 06/Aug/19 20:58 Start Date: 06/Aug/19 20:58 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#discussion_r311271852 ## File path: sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py ## @@ -20,10 +20,17 @@ from __future__ import absolute_import +import logging +import time Review comment: Thanks for the comment. Updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 289987) Time Spent: 1h 40m (was: 1.5h) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=288334&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-288334 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 02/Aug/19 23:57 Start Date: 02/Aug/19 23:57 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#discussion_r310327778 ## File path: sdks/python/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py ## @@ -20,10 +20,17 @@ from __future__ import absolute_import +import logging +import time Review comment: I believe this class is auto-generated. So this change might get reverted next time we update the client. We should at least add a test that prevents this and move most of the changes here to another file (prob. everything other than "retry_func=xxx". This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 288334) Time Spent: 1.5h (was: 1h 20m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=286098&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-286098 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 31/Jul/19 17:49 Start Date: 31/Jul/19 17:49 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#discussion_r309351299 ## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java ## @@ -66,6 +66,8 @@ // TODO: Move throttle time Metric to a dedicated namespace. protected static final String DATASTORE_THROTTLE_TIME_NAMESPACE = "org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$DatastoreWriterFn"; + protected static final String HTTP_CLIENT_API_THROTTLE_TIME_NAMESPACE = Review comment: Yes, it just followed the same convention for compatibility. Moving to a dedicated namespace is on TODO list. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 286098) Time Spent: 1h 20m (was: 1h 10m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=285382&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-285382 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 31/Jul/19 00:52 Start Date: 31/Jul/19 00:52 Worklog Time Spent: 10m Work Description: ihji commented on issue #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#issuecomment-516649941 @chamikaramj PTAL. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 285382) Time Spent: 1h 10m (was: 1h) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=285225&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-285225 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 30/Jul/19 20:07 Start Date: 30/Jul/19 20:07 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#issuecomment-516575857 @ihji please let me know when the comments have been addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 285225) Time Spent: 1h (was: 50m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=281922&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-281922 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 24/Jul/19 12:43 Start Date: 24/Jul/19 12:43 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#discussion_r306787866 ## File path: runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/BatchModeExecutionContext.java ## @@ -66,6 +66,8 @@ // TODO: Move throttle time Metric to a dedicated namespace. protected static final String DATASTORE_THROTTLE_TIME_NAMESPACE = "org.apache.beam.sdk.io.gcp.datastore.DatastoreV1$DatastoreWriterFn"; + protected static final String HTTP_CLIENT_API_THROTTLE_TIME_NAMESPACE = Review comment: This seems an odd explicit namespace to use, is it for backwards compatibility reasons? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 281922) Time Spent: 50m (was: 40m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=281367&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-281367 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 23/Jul/19 22:41 Start Date: 23/Jul/19 22:41 Worklog Time Spent: 10m Work Description: chamikaramj commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#discussion_r306550488 ## File path: sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/RetryHttpRequestInitializer.java ## @@ -68,6 +68,8 @@ private final BackOff ioExceptionBackOff; private final BackOff unsuccessfulResponseBackOff; private final Set ignoredResponseCodes; +private final Counter throttlingSeconds = +Metrics.counter(LoggingHttpBackOffHandler.class, "cumulativeThrottlingSeconds"); Review comment: Please add a comment describing what this is. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 281367) Time Spent: 40m (was: 0.5h) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=271233&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-271233 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 02/Jul/19 21:23 Start Date: 02/Jul/19 21:23 Worklog Time Spent: 10m Work Description: ihji commented on issue #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#issuecomment-507849939 R: @chamikaramj This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 271233) Time Spent: 0.5h (was: 20m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=270513&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-270513 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 01/Jul/19 23:06 Start Date: 01/Jul/19 23:06 Worklog Time Spent: 10m Work Description: ihji commented on issue #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973#issuecomment-507456416 Throttling time is now delivered to Dataflow as a part of WorkItemStatus message in Java SDK. Whether it's actually used for autoscaling algorithm (or just ignored) is still unsure. For Python SDK, we're recording the throttling time metric but it's not reported to Dataflow yet (we need to modify Dataflow Python worker and it's not open-sourced) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 270513) Time Spent: 20m (was: 10m) > report GCS throttling time to Dataflow autoscaler > - > > Key: BEAM-7667 > URL: https://issues.apache.org/jira/browse/BEAM-7667 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > report GCS throttling time to Dataflow autoscaler. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-7667) report GCS throttling time to Dataflow autoscaler
[ https://issues.apache.org/jira/browse/BEAM-7667?focusedWorklogId=270429&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-270429 ] ASF GitHub Bot logged work on BEAM-7667: Author: ASF GitHub Bot Created on: 01/Jul/19 19:08 Start Date: 01/Jul/19 19:08 Worklog Time Spent: 10m Work Description: ihji commented on pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler URL: https://github.com/apache/beam/pull/8973 report GCS throttling time to Dataflow autoscaler. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python3_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [