damccorm opened a new pull request, #41258: URL: https://github.com/apache/beam/pull/41258
## Summary Fixes recurring `Staged package ... at location 'gs://temp-storage-for-validates-runner-tests/staging/...' is inaccessible` failures across Dataflow PostCommit suites: - https://github.com/apache/beam/actions/runs/43313768013 (`PostCommit Java Examples Dataflow V2`) - https://github.com/apache/beam/actions/runs/43301179015 (`PostCommit Java ValidatesRunner Dataflow Streaming TagEncodingV2`) - https://github.com/apache/beam/actions/runs/42756757787 (`PostCommit Java ValidatesRunner Dataflow V2 Streaming`) ### Root Cause `gs://temp-storage-for-validates-runner-tests` configures a 7-day GCS Object Lifecycle `Delete` policy (`age: 7`), while Dataflow ValidatesRunner and Examples suites share a common staging prefix (`gs://temp-storage-for-validates-runner-tests/staging`). Unchanged dependency and Beam module JARs (restored from Gradle's build cache with identical SHA-256 hashes) keep the same staged object name in `staging/`. Because `PackageUtil.alreadyStaged` previously only checked `remoteMetadata.sizeBytes() == attributes.getSize()` without checking object age, unchanged JARs were never re-uploaded until GCS's 7-day lifecycle sweep deleted them. Any test suite that checked `alreadyStaged` shortly before the 7-day lifecycle deletion (or cached the staged file list in `GcsStager.STAGED_FILES_CACHE`) then submitted Dataflow jobs referencing just-deleted objects in `staging/`. ### Fix - Update `PackageUtil.alreadyStaged` to also inspect `remoteMetadata.lastModifiedMillis()` and re-stage existing objects older than 1 day (`MAX_STAGED_FILE_AGE = Duration.standardDays(1)`). - Re-uploading active JARs once per day refreshes their GCS object generation and creation/modification timestamps well before the 7-day lifecycle rule can delete them, while still reusing cached objects across runs within each 24-hour window and allowing obsolete JARs to expire after 7 days. - Add `PackageUtilTest.testPackageUploadIsNotSkippedWhenStagedFileIsExpired` to verify that expired staged objects are re-uploaded. ## Verification - Local unit tests: `./gradlew :runners:google-cloud-dataflow-java:test --tests org.apache.beam.runners.dataflow.util.PackageUtilTest :runners:google-cloud-dataflow-java:spotlessCheck` - Triggered `PostCommit Java Examples Dataflow V2` on this branch: https://github.com/apache/beam/actions/runs/43335161591 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
