[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[
https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=176772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176772
]
ASF GitHub Bot logged work on BEAM-6197:
Author: ASF GitHub Bot
Created on: 18/Dec/18 22:11
Start Date: 18/Dec/18 22:11
Worklog Time Spent: 10m
Work Description: aaltay closed pull request #7235: [BEAM-6197] Log time
for Dataflow GCS upload of staged files + add a test program
URL: https://github.com/apache/beam/pull/7235
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index c0b831c6e547..572136c45f76 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -109,6 +109,8 @@ test {
def dataflowProject = project.findProperty('dataflowProject') ?:
'apache-beam-testing'
def dataflowValidatesTempRoot = project.findProperty('dataflowTempRoot') ?:
'gs://temp-storage-for-validates-runner-tests/'
def dataflowPostCommitTempRoot = project.findProperty('dataflowTempRoot') ?:
'gs://temp-storage-for-end-to-end-tests'
+def dataflowUploadTemp = project.findProperty('dataflowTempRoot') ?:
'gs://temp-storage-for-upload-tests'
+def testFilesToStage = project.findProperty('filesToStage') ?: 'test.txt'
def dataflowLegacyWorkerJar = project.findProperty('dataflowWorkerJar') ?:
project(":beam-runners-google-cloud-dataflow-java-legacy-worker").shadowJar.archivePath
def dataflowFnApiWorkerJar = project.findProperty('dataflowWorkerJar') ?:
project(":beam-runners-google-cloud-dataflow-java-fn-api-worker").shadowJar.archivePath
@@ -440,3 +442,11 @@ createJavaExamplesArchetypeValidationTask(type:
'MobileGaming',
gcsBucket: gcsBucket,
bqDataset: bqDataset,
pubsubTopic: pubsubTopic)
+
+// Standalone task for testing GCS upload, use with -PfilesToStage and
-PdataflowTempRoot.
+task GCSUpload(type: JavaExec) {
+ main = 'org.apache.beam.runners.dataflow.util.GCSUploadMain'
+ classpath = sourceSets.test.runtimeClasspath
+ args "--stagingLocation=${dataflowUploadTemp}/staging",
+"--filesToStage=${testFilesToStage}"
+}
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
index a8fba4bab676..b919922a1f5e 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java
@@ -64,6 +64,8 @@
import org.apache.beam.sdk.util.MoreFutures;
import org.apache.beam.sdk.util.ZipFiles;
import org.joda.time.Duration;
+import org.joda.time.Instant;
+import org.joda.time.Seconds;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
@@ -316,6 +318,7 @@ public DataflowPackage stageToFile(
"Uploading {} files from PipelineOptions.filesToStage to staging
location to "
+ "prepare for execution.",
classpathElements.size());
+Instant start = Instant.now();
if (classpathElements.size() > SANE_CLASSPATH_SIZE) {
LOG.warn(
@@ -386,10 +389,12 @@ public DataflowPackage stageToFile(
}
} while (!finished);
List stagedPackages = MoreFutures.get(stagingFutures);
+ Instant done = Instant.now();
LOG.info(
- "Staging files complete: {} files cached, {} files newly uploaded",
+ "Staging files complete: {} files cached, {} files newly uploaded in
{} seconds",
numCached.get(),
- numUploaded.get());
+ numUploaded.get(),
+ Seconds.secondsBetween(start, done).getSeconds());
return stagedPackages;
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
diff --git
a/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/GCSUploadMain.java
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/GCSUploadMain.java
new file mode 100644
index ..1783fd63f026
--- /dev/null
+++
b/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/GCSUploadMain.java
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ *
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=176133=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176133 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 17/Dec/18 16:57 Start Date: 17/Dec/18 16:57 Worklog Time Spent: 10m Work Description: alanmyrvold edited a comment on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447917234 @aaltay The remaining Portable_Python PreCommit test is failing consistently on master, https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ and tracked by https://issues.apache.org/jira/browse/BEAM-6242 But it just passed for this PR https://builds.apache.org/job/beam_PreCommit_Portable_Python_Phrase/45/ This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 176133) Time Spent: 2h 40m (was: 2.5h) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 2h 40m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=176135=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176135 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 17/Dec/18 16:57 Start Date: 17/Dec/18 16:57 Worklog Time Spent: 10m Work Description: alanmyrvold removed a comment on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447588323 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 176135) Time Spent: 3h (was: 2h 50m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 3h > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=176134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176134 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 17/Dec/18 16:57 Start Date: 17/Dec/18 16:57 Worklog Time Spent: 10m Work Description: alanmyrvold removed a comment on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447539110 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 176134) Time Spent: 2h 50m (was: 2h 40m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 2h 50m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=176132=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176132 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 17/Dec/18 16:55 Start Date: 17/Dec/18 16:55 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447917234 @aaltay The remaining Portable_Python PreCommit test is failing consistently on master, https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/ and tracked by https://issues.apache.org/jira/browse/BEAM-6242 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 176132) Time Spent: 2.5h (was: 2h 20m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 2.5h > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=176122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-176122 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 17/Dec/18 16:44 Start Date: 17/Dec/18 16:44 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447913222 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 176122) Time Spent: 2h 20m (was: 2h 10m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 2h 20m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175725=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175725 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 15/Dec/18 18:28 Start Date: 15/Dec/18 18:28 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447588323 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175725) Time Spent: 2h 10m (was: 2h) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 2h 10m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175675=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175675 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 15/Dec/18 05:30 Start Date: 15/Dec/18 05:30 Worklog Time Spent: 10m Work Description: alanmyrvold removed a comment on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447515127 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175675) Time Spent: 2h (was: 1h 50m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 2h > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175674 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 15/Dec/18 05:30 Start Date: 15/Dec/18 05:30 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447539110 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175674) Time Spent: 1h 50m (was: 1h 40m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 1h 50m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175638=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175638 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 15/Dec/18 00:08 Start Date: 15/Dec/18 00:08 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447515309 Thank you. I can merge it after the last test passes. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175638) Time Spent: 1h 40m (was: 1.5h) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 1h 40m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175635=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175635 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 15/Dec/18 00:07 Start Date: 15/Dec/18 00:07 Worklog Time Spent: 10m Work Description: alanmyrvold removed a comment on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447511793 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175635) Time Spent: 1h 20m (was: 1h 10m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 1h 20m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175636 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 15/Dec/18 00:07 Start Date: 15/Dec/18 00:07 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447515127 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175636) Time Spent: 1.5h (was: 1h 20m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 1.5h > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175622 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 14/Dec/18 23:45 Start Date: 14/Dec/18 23:45 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447511793 Run Portable_Python PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175622) Time Spent: 1h 10m (was: 1h) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 1h 10m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=175557=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-175557 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 14/Dec/18 19:20 Start Date: 14/Dec/18 19:20 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-447427556 > @alanmyrvold do you know why tests are failing? Yes. checkstyle was failing due to the format of the class comment. Fixed the comment and re-trying This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 175557) Time Spent: 1h (was: 50m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 1h > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=174382=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-174382 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 12/Dec/18 02:03 Start Date: 12/Dec/18 02:03 Worklog Time Spent: 10m Work Description: aaltay commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-446433808 Run Java PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 174382) Time Spent: 40m (was: 0.5h) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=173144=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173144 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 08/Dec/18 01:59 Start Date: 08/Dec/18 01:59 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-445420634 @aaltay PTAL? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 173144) Time Spent: 0.5h (was: 20m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=173143=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173143 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 08/Dec/18 01:56 Start Date: 08/Dec/18 01:56 Worklog Time Spent: 10m Work Description: alanmyrvold commented on issue #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files + add a test program URL: https://github.com/apache/beam/pull/7235#issuecomment-445420359 Run Spotless PreCommit This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 173143) Time Spent: 20m (was: 10m) > Log time for Dataflow GCS upload of staged files > > > Key: BEAM-6197 > URL: https://issues.apache.org/jira/browse/BEAM-6197 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow >Reporter: Alan Myrvold >Assignee: Alan Myrvold >Priority: Minor > Time Spent: 20m > Remaining Estimate: 0h > > Would be nice to collect timing in the logs of Dataflow GCS upload of staged > files -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work logged] (BEAM-6197) Log time for Dataflow GCS upload of staged files
[ https://issues.apache.org/jira/browse/BEAM-6197?focusedWorklogId=173140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-173140 ] ASF GitHub Bot logged work on BEAM-6197: Author: ASF GitHub Bot Created on: 08/Dec/18 01:52 Start Date: 08/Dec/18 01:52 Worklog Time Spent: 10m Work Description: alanmyrvold opened a new pull request #7235: [BEAM-6197] Log time for Dataflow GCS upload of staged files URL: https://github.com/apache/beam/pull/7235 Create a log message for time taken for Dataflow GCS upload of staged files. Add a standalone test program to test the GCS upload without running a pipeline. Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 173140) Time Spent: 10m Remaining Estimate: 0h > Log time for
