Thanks a lot for your responses. I didn't know that you can explicitly refer to the timestamped snapshots of the artifacts. The limitation to the last 2 snapshots means that a push to flink-shaded can break our main CI? This sounds very fragile to me, given that the setup itself is probably a bit uncommon and difficult to understand.
Maybe we should add an automated check to flink-shaded that warns if a PR would break Flink's CI? (by checking out flink and scanning the poms for references to a timestamp-to-be-deleted) Or we ask Infra to keep more than two snapshots for flink-shaded? On Mon, Apr 12, 2021 at 4:41 PM Chesnay Schepler <ches...@apache.org> wrote: > a) yes. > b) maven by default adds a timestamp to snapshot artifacts that we can > use. The apache repository retains the last 2 snapshots, so we do need > to keep things in sync a fair bit, but there are rarely commits made in > flink-shaded that I don't think this will be a problem. > c) a-SNAPSHOT-uniquesuffix => a.0 > > On 4/12/2021 3:07 PM, Robert Metzger wrote: > > Thanks a lot for your proposal, I'm generally open to the idea > > > > I have a few questions: > > a) Does this mean that we are changing flink-shaded to deploy snapshot > > artifacts to Apache's snapshot maven repository, and change Flink's > parent > > pom to point to this snapshot repo? > > b) How do you plan to generate the unique SNAPSHOT version on CI? Will we > > increment the version on every push to flink-shaded:master ? > > c) How do the unique SNAPSHOT versions relate to the final release > versions? > > > > > > > > > > On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kna...@apache.org> > wrote: > > > >> Sounds good. +1 > >> > >> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ches...@apache.org> > >> wrote: > >> > >>> Hello all, > >>> > >>> I would like to propose a change in how the Flink master interacts with > >>> Flink-shaded. > >>> > >>> TL;DR: Release snapshot artifacts for flink-shaded, and have the Flink > >>> master rely on specific snapshot versions for earlier dependency bumps. > >>> > >>> > >>> Aa a project we have come to the general conclusion that dependencies > >>> should be bumped as early in the release cycle as possible. This both > >>> prevents cases where some undefined amount of work is still waiting for > >>> as when we want to release the next version (working against the goal > of > >>> always being in a releasable state), and it gives us more time to > >>> evaluate the stability and performance of system. Finally it gives us > >>> ample time to look for alternatives if an issue is found. > >>> > >>> Currently, this conclusion is at odds with how we handle flink-shaded. > >>> Flink has always relied on flink-shaded artifacts that went through a > >>> proper release cycle. However, since we want to create as few releases > >>> as possible due to the overhead/noise/etc., flink-shaded releases are > >>> typically relegated to the end of the release cycle. > >>> This is particularly troublesome since flink-shaded dependencies are > >>> used in the core of Flink, and hence usage of them cannot be avoided. > >>> > >>> As a compromise between these 2 goals I propose the following: > >>> - we deploy SNAPSHOT artifacts for flink-shaded for every change made > >>> - every deployed artifact has a unique version, that is automatically > >>> set via maven (=> no overhead on our side) > >>> - once such an artifact is released we update the Flink dependency to > >>> point to this _specific_ flink-shaded snapshot artifact > >>> - to be clear, this is a manual step, which implies that things > >>> cannot break all of a sudden because something was pushed to > flink-shaded > >>> - once the Flink release cycle ends, we publish a proper flink-shaded > >>> release, and change the Flink dependency in the release branch > >> accordingly > >>> This should give us the best of both worlds: We have as few releases as > >>> necessary (at most 1 per Flink release cycle), but can update the > >>> dependencies in Flink as soon as possible. > >>> Furthermore, this can also be considered a test run for how multiple > >>> repos with the same release cycle could be developed in sync with each > >>> other. > >>> > >>> Let me know what you think. > >>> > >>> Regards, > >>> > >>> Chesnay > >>> > >>> > >> -- > >> > >> Konstantin Knauf > >> > >> https://twitter.com/snntrable > >> > >> https://github.com/knaufk > >> > >
