damccorm commented on code in PR #41258:
URL: https://github.com/apache/beam/pull/41258#discussion_r4559708666
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/PackageUtil.java:
##########
@@ -122,11 +123,33 @@ private CompletionStage<PackageAttributes>
computePackageAttributes(
() -> PackageAttributes.forFileToStage(source, hash, dest,
stagingPath), executorService);
}
+ /**
+ * Maximum age of a previously staged file before it is re-uploaded. Shared
test and staging
+ * buckets often configure GCS lifecycle deletion rules (for example, 7 days
on {@code
+ * gs://temp-storage-for-validates-runner-tests}), so re-uploading files
older than 1 day
+ * refreshes their object creation timestamp well before lifecycle deletion
can race with job
+ * execution.
+ */
+ static final Duration MAX_STAGED_FILE_AGE = Duration.standardDays(1);
+
private boolean alreadyStaged(PackageAttributes attributes) throws
IOException {
try {
- long remoteLength =
-
FileSystems.matchSingleFileSpec(attributes.getDestination().getLocation()).sizeBytes();
- return remoteLength == attributes.getSize();
+ Metadata remoteMetadata =
+
FileSystems.matchSingleFileSpec(attributes.getDestination().getLocation());
+ if (remoteMetadata.sizeBytes() != attributes.getSize()) {
+ return false;
+ }
+ long lastModifiedMillis = remoteMetadata.lastModifiedMillis();
+ if (lastModifiedMillis > 0
+ && Instant.now().getMillis() - lastModifiedMillis >
MAX_STAGED_FILE_AGE.getMillis()) {
Review Comment:
Instead of checking the if the file age is older than 1 day, can we figure
out what the file's TTL is somehow and compare to that?
If not, we should find a different way of handling this (probably staging
artifacts with unique names in our tests). The purpose of the staging directory
is to reuse previously staged artifacts (even if they were staged a long time
ago)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
