damccorm commented on code in PR #41258:
URL: https://github.com/apache/beam/pull/41258#discussion_r4559708666
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java:
##########
@@ -122,11 +123,33 @@ private CompletionStage<PackageAttributes>
computePackageAttributes(
() -> PackageAttributes.forFileToStage(source, hash, dest,
stagingPath), executorService);
}
+ /**
+ * Maximum age of a previously staged file before it is re-uploaded. Shared
test and staging
+ * buckets often configure GCS lifecycle deletion rules (for example, 7 days
on {@code
+ * gs://temp-storage-for-validates-runner-tests}), so re-uploading files
older than 1 day
+ * refreshes their object creation timestamp well before lifecycle deletion
can race with job
+ * execution.
+ */
+ static final Duration MAX_STAGED_FILE_AGE = Duration.standardDays(1);
+
private boolean alreadyStaged(PackageAttributes attributes) throws
IOException {
try {
- long remoteLength =
-
FileSystems.matchSingleFileSpec(attributes.getDestination().getLocation()).sizeBytes();
- return remoteLength == attributes.getSize();
+ Metadata remoteMetadata =
+
FileSystems.matchSingleFileSpec(attributes.getDestination().getLocation());
+ if (remoteMetadata.sizeBytes() != attributes.getSize()) {
+ return false;
+ }
+ long lastModifiedMillis = remoteMetadata.lastModifiedMillis();
+ if (lastModifiedMillis > 0
+ && Instant.now().getMillis() - lastModifiedMillis >
MAX_STAGED_FILE_AGE.getMillis()) {
Review Comment:
Instead of checking the if the file age is older than 1 day, can we figure
out what the file's TTL is somehow and compare to that?
If not, we should find a different way of handling this (probably staging
artifacts with unique names in our tests)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
