I reached out to the Apache Infra team in their slack channel and have yet to hear back. Given that it's getting pretty late on a Friday here (I'm located in East US), it is unlikely that I will have a workaround by the time the snapshot is removed by the retention policy; to ensure continued functionality over the weekend and the start of next week, I recommend pinning to a later snapshot.
Thanks, Danny On Fri, Jul 22, 2022 at 2:17 PM Daniel Thevessen <dant...@google.com> wrote: > I don't know if this helps, but I found one directory that has retained an > old build from 2017 even, the one for 0.7.0-SNAPSHOT > <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/0.7.0-SNAPSHOT/>. > Every other snapshots directory is empty. > Is it possible they don't get deleted if they're in use? The sonatype > documentation you linked mentions a "last downloaded" criterion, I'm not > sure if it is enabled though. > I haven't been able to find a matching cleanup script in the Beam repo or > a config file in the snapshots directory, it must be configured somewhere > else. > > If we could somehow get (1) or (2) to work that would be extremely > helpful, changing the dependency back to 2.41.0-SNAPSHOT comes with risks > and difficulties for us. > > > > On Fri, Jul 22, 2022 at 8:10 AM Danny McCormick <dannymccorm...@google.com> > wrote: > >> > I think you could change the TTL on the Jenkins side (That sound right >> to you Danny?) but I'm not sure we could preserve a specific snapshot >> without keeping all of them... >> >> I'm admittedly not particularly well versed in this area of our infra, >> but after some digging I don't see a way to do this. I'm still piecing >> together how our release infra works, but at a high level we use the Maven >> publish plugin [1] to publish >> snapshots to the ASF managed Nexus snapshots repo (which is shared across >> projects). As best I can tell, there isn't a way to configure retention >> from the repo side, and it is likely managed by a repo-wide cleanup policy >> [3]. I don't have the right permission set to actually view cleanup >> policies, so I can't say that with 100% confidence, but I do see that >> other projects seem to be limited to a smallish number of snapshot versions >> as well [4] [5] [6] which points to a repo-wide policy, and I think that's >> a common way to handle things like this. >> >> If someone with more experience with our releases wants to chime in and >> confirm that understanding or say I'm wrong, I'd appreciate it. Or, if I'm >> right and someone actually has access to modify the cleanup policy (maybe >> @Kenneth >> Knowles <k...@google.com>) and can just change it, that would be awesome! >> >> Options for moving forward: >> 1) If someone has permissions, change the retention policy for the time >> being. >> 2) Engage infra and see if (a) they can confirm the setup is indeed what >> I described and (b) if they can help temporarily change the retention >> policy for Beam. >> 3) Daniel, you could change how you're consuming the artifacts. The >> shortest term version of this (which would buy more time to figure out a >> solution) is just updating to a more recent snapshot, but you could also >> just pin to the latest snapshot version (obviously has its risks), or >> mirror the dependencies into your own maven repo. You could also just pin >> to 2.40 if this is a regression. >> >> Obviously any solution that isn't pinning to 2.40 is operating in an >> unsafe mode since we haven't performed the usual level of validation that >> accompanies a release. >> >> Thanks, >> Danny >> >> [1] https://docs.gradle.org/current/userguide/publishing_maven.html >> [2] https://infra.apache.org/publishing-maven-artifacts.html >> [3] >> https://help.sonatype.com/repomanager3/nexus-repository-administration/repository-management/cleanup-policies >> [4] >> https://repository.apache.org/content/groups/snapshots/org/apache/avro/avro-compiler/1.12.0-SNAPSHOT/ >> [5] >> https://repository.apache.org/content/groups/snapshots/org/apache/flink/flink-architecture-tests-base/1.15-SNAPSHOT/ >> [6] >> https://repository.apache.org/content/groups/snapshots/org/apache/iotdb/client-cpp-example/0.14.0-SNAPSHOT/ >> >> On Thu, Jul 21, 2022 at 8:54 PM Evan Galpin <egal...@apache.org> wrote: >> >>> Admittedly this is potentially self-serving, but I feel there could be >>> mutual benefit. >>> >>> I have a similar situation where I want to use pre-release version of >>> beam-sdks-java-io-google-cloud-platform. Though I’ve been having >>> trouble doing so, a possible alternative solution to using the nightly >>> snapshots might be building beam-sdks-java-io-google-cloud-platform >>> from source and including the resulting jar as part of the pipeline >>> deployment. I’ve successfully done this for direct runner, but not Dataflow >>> runner. >>> >>> Perhaps some others on the thread might be able to shed light on this >>> technique (only if applicable to solving the original problem, as I don’t >>> intend to thread-hijack). >>> >>> - Evan >>> >>> On Thu, Jul 21, 2022 at 19:45 Byron Ellis via dev <dev@beam.apache.org> >>> wrote: >>> >>>> I think you could change the TTL on the Jenkins side (That sound right >>>> to you Danny?) but I'm not sure we could preserve a specific snapshot >>>> without keeping all of them... >>>> >>>> On Thu, Jul 21, 2022 at 4:16 PM Ahmet Altay <al...@google.com> wrote: >>>> >>>>> Thank you for the email Daniel. >>>>> >>>>> Adding people who could help: @Kenneth Knowles <k...@google.com> @Danny >>>>> McCormick <dannymccorm...@google.com> @Chamikara Jayalath >>>>> <chamik...@google.com> @John Casey <johnjca...@google.com> @Byron >>>>> Ellis <byronel...@google.com> >>>>> >>>>> On Thu, Jul 21, 2022, 4:14 PM Daniel Thevessen via dev < >>>>> dev@beam.apache.org> wrote: >>>>> >>>>>> Hi all, >>>>>> >>>>>> The Java Firestore connector had a bug >>>>>> <https://github.com/apache/beam/issues/22089> recently that needed >>>>>> to be fixed. Until the fix can be released as part of 2.41.0, we need one >>>>>> of the daily snapshot builds >>>>>> <https://repository.apache.org/content/groups/snapshots/org/apache/beam/beam-sdks-java-io-google-cloud-platform/2.41.0-SNAPSHOT/> >>>>>> as a safe version to use. This is the one from Jul 15 >>>>>> (2.41.0-20220715.201105-31), which we have checked to be working >>>>>> correctly >>>>>> and some have already switched to. >>>>>> Unfortunately it looks like these builds get cleared out after a >>>>>> while for storage reasons. Would it be possible to extend the TTL on just >>>>>> that build, at least until 2.41.0 is released? I'm guessing this would >>>>>> just >>>>>> be a settings change for whoever owns the Snapshots repository. >>>>>> The change is in beam-sdks-java-io-google-cloud-platform, but I'm >>>>>> fairly certain the other Java packages need to be kept as well for >>>>>> compatibility. >>>>>> >>>>>> This is relatively urgent, it looks like the TTL might be weekly so >>>>>> that build will be deleted on Saturday. >>>>>> >>>>>> Thanks, >>>>>> Daniel Thevessen >>>>>> >>>>>> >>>>>> > > -- > > Daniel Thevessen | Site Reliability Engineer > > Firestore SRE > San Francisco, CA, USA | +1 (415) 373-7762 <(415)%20373-7762> >
