Thanks for creating this proposal Chesnay. I do understand the problem you want to fix.
What I am wondering is why we don't release flink-shaded more often. Does the release process cause too much overhead? If this is the case, then we could look into what is causing the overhead and whether we can improve the situation. Concerning the noise, I personally don't see it as a problem. My main concern is that it can easily slip our minds to change the flink-shaded SNAPSHOT version to a non SNAPSHOT version and that it introduces another manual step. If we forget to change the version, then the Flink release does not build against a stable set of dependencies. Moreover, I also second Robert's concern that a single commit to flink-shaded can then break downstream projects (Flink in this case) if we rely on the SNAPSHOT builds. Having to scan poms for some references sounds like an indicator that this might not be the most straight forward approach. Cheers, Till On Tue, Apr 13, 2021 at 9:26 AM Robert Metzger <rmetz...@apache.org> wrote: > Thanks a lot for your responses. > > I didn't know that you can explicitly refer to the timestamped snapshots of > the artifacts. The limitation to the last 2 snapshots means that a push to > flink-shaded can break our main CI? This sounds very fragile to me, given > that the setup itself is probably a bit uncommon and difficult to > understand. > > Maybe we should add an automated check to flink-shaded that warns if a PR > would break Flink's CI? (by checking out flink and scanning the poms for > references to a timestamp-to-be-deleted) > Or we ask Infra to keep more than two snapshots for flink-shaded? > > > > On Mon, Apr 12, 2021 at 4:41 PM Chesnay Schepler <ches...@apache.org> > wrote: > > > a) yes. > > b) maven by default adds a timestamp to snapshot artifacts that we can > > use. The apache repository retains the last 2 snapshots, so we do need > > to keep things in sync a fair bit, but there are rarely commits made in > > flink-shaded that I don't think this will be a problem. > > c) a-SNAPSHOT-uniquesuffix => a.0 > > > > On 4/12/2021 3:07 PM, Robert Metzger wrote: > > > Thanks a lot for your proposal, I'm generally open to the idea > > > > > > I have a few questions: > > > a) Does this mean that we are changing flink-shaded to deploy snapshot > > > artifacts to Apache's snapshot maven repository, and change Flink's > > parent > > > pom to point to this snapshot repo? > > > b) How do you plan to generate the unique SNAPSHOT version on CI? Will > we > > > increment the version on every push to flink-shaded:master ? > > > c) How do the unique SNAPSHOT versions relate to the final release > > versions? > > > > > > > > > > > > > > > On Mon, Apr 12, 2021 at 1:48 PM Konstantin Knauf <kna...@apache.org> > > wrote: > > > > > >> Sounds good. +1 > > >> > > >> On Mon, Apr 12, 2021 at 1:23 PM Chesnay Schepler <ches...@apache.org> > > >> wrote: > > >> > > >>> Hello all, > > >>> > > >>> I would like to propose a change in how the Flink master interacts > with > > >>> Flink-shaded. > > >>> > > >>> TL;DR: Release snapshot artifacts for flink-shaded, and have the > Flink > > >>> master rely on specific snapshot versions for earlier dependency > bumps. > > >>> > > >>> > > >>> Aa a project we have come to the general conclusion that dependencies > > >>> should be bumped as early in the release cycle as possible. This both > > >>> prevents cases where some undefined amount of work is still waiting > for > > >>> as when we want to release the next version (working against the goal > > of > > >>> always being in a releasable state), and it gives us more time to > > >>> evaluate the stability and performance of system. Finally it gives us > > >>> ample time to look for alternatives if an issue is found. > > >>> > > >>> Currently, this conclusion is at odds with how we handle > flink-shaded. > > >>> Flink has always relied on flink-shaded artifacts that went through a > > >>> proper release cycle. However, since we want to create as few > releases > > >>> as possible due to the overhead/noise/etc., flink-shaded releases are > > >>> typically relegated to the end of the release cycle. > > >>> This is particularly troublesome since flink-shaded dependencies are > > >>> used in the core of Flink, and hence usage of them cannot be avoided. > > >>> > > >>> As a compromise between these 2 goals I propose the following: > > >>> - we deploy SNAPSHOT artifacts for flink-shaded for every change made > > >>> - every deployed artifact has a unique version, that is automatically > > >>> set via maven (=> no overhead on our side) > > >>> - once such an artifact is released we update the Flink dependency to > > >>> point to this _specific_ flink-shaded snapshot artifact > > >>> - to be clear, this is a manual step, which implies that things > > >>> cannot break all of a sudden because something was pushed to > > flink-shaded > > >>> - once the Flink release cycle ends, we publish a proper flink-shaded > > >>> release, and change the Flink dependency in the release branch > > >> accordingly > > >>> This should give us the best of both worlds: We have as few releases > as > > >>> necessary (at most 1 per Flink release cycle), but can update the > > >>> dependencies in Flink as soon as possible. > > >>> Furthermore, this can also be considered a test run for how multiple > > >>> repos with the same release cycle could be developed in sync with > each > > >>> other. > > >>> > > >>> Let me know what you think. > > >>> > > >>> Regards, > > >>> > > >>> Chesnay > > >>> > > >>> > > >> -- > > >> > > >> Konstantin Knauf > > >> > > >> https://twitter.com/snntrable > > >> > > >> https://github.com/knaufk > > >> > > > > >
