reuvenlax commented on PR #40321: URL: https://github.com/apache/beam/pull/40321#issuecomment-6616157460
This isn't just a known issue, we strongly tell customers not to put lifecycles on staging buckets, as many things can break (e.g. autoscaling breaks if the artifacts are gone, etc.). What is our GCS lifecycle today on this directory? Would it work to change the directory so that we roll over to a new directory every hour? Or make sure that we otherwise keep the directory stable for a group of test runs (e.g. single PostCommit run) but let it change after that? On Tue, Nov 3, 2026 at 12:29 PM Yi Hu ***@***.***> wrote: > *Abacn* left a comment (apache/beam#40321) > <https://github.com/apache/beam/pull/40321#issuecomment-6614697502> > > This caused another flaky pattern (#40727 > <https://github.com/apache/beam/issues/40727>) see > https://github.com/apache/beam/actions/runs/41591069535 > > Many tests failing with "java.lang.RuntimeException: Staged package > HikariCP-4.0.3-fAJK7_HBBjV210RTUT-d5kR9jmJNF_jifzCi6XaIxsk.jar at location > 'gs://temp-storage-for-validates-runner-tests/staging/HikariCP-4.0.3-fAJK7_HBBjV210RTUT-d5kR9jmJNF_jifzCi6XaIxsk.jar' > is inaccessible..." > > it changed per-job staging directories to a single static directory shared > forever across all runs. > > In the middle, these jars get deleted by GCS Lifecycle Management. > However, alreadyStaged() cache the results up to 30 min: > https://github.com/apache/beam/blob/e9247147131e066ae1b5abdc3156b2eaf03a6032/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/GcsStager.java#L57 > > Beam artifact staging not work well with gcs lifecycle is a known issue > (internal tracker: b/525510729 > <https://buganizer.corp.google.com/525510729>). This change exposed that > issue > > — > Reply to this email directly, view it on GitHub > <https://github.com/apache/beam/pull/40321?email_source=notifications&email_token=AFAYJVOAR35XTCWO3VCEDVL5XI54NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMNRRGQ3DSNZVGAZKM4TFMFZW63VMON2GC5DFL5RWQYLOM5S2KZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6614697502>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AFAYJVKEH4OOUNJNEP7F4JT5XI54NAVCNFSNUABEKJSXA33TNF2G64TZHM2TAOJQGQZDINJ3JFZXG5LFHM2TMMJZG42TKOBWGCQXMAQ> > . > Triage notifications, keep track of coding agent tasks and review pull > requests on the go with GitHub Mobile for iOS > <https://github.com/notifications/mobile/ios/AFAYJVIJ7YZLNS35PQONKUT5XI54NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMNRRGQ3DSNZVGAZKM4TFMFZW63VMON2GC5DFL5RWQYLOM5S2KZLWMVXHJKTGN5XXIZLSL5UW64Y> > and Android > <https://github.com/notifications/mobile/android/AFAYJVNRKHNVBVMKXWR7KPT5XI54NA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMNRRGQ3DSNZVGAZKM4TFMFZW63VMON2GC5DFL5RWQYLOM5S2KZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>. > Download it today! > You are receiving this because you modified the open/close state.Message > ID: ***@***.***> > -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
