A small addendum: The project currently enforces only binary compatibility for classes which are annotated with @Public. The StreamingFileSink is annotated with @PublicEvolving.
I am not sure whether the community properly defined what kind of stability/compatibility guarantees we provide for @PublicEvolving classes. We can derive two follow-up discussions from this: 1) Should the StreamingFileSink be made @Public? Should other @PublicEvolving classes be promoted? What is the process here? 2) Should we enforce binary compatibility of @PublicEvolving classes between bug fix releases and allow breaking changes between minor/major releases? Cheers, Till On Wed, May 13, 2020 at 2:30 PM Ufuk Celebi <u...@apache.org> wrote: > Thanks for the analysis Till. > > 1/ I think breaking binary compatibility is acceptable between minor > releases (1.10 -> 1.11) as you suggested since the API is marked > as @PublicEvolving. > > 2/ I'm quite torn about how to proceed with the 1.10.x patch release, but > slightly leaning towards the "pragmatic" solution of keeping the change and > adding a big warning to the 1.10.1 release announcement*. What do others > think? If we do want to revert the change and follow up with a 1.10.2 > release we should do it _before_ we announce 1.10.1 publicly. Doing it > after 1.10.1 has been announced would only cause more friction for users. > > – Ufuk > > *I hope we don't lose user trust with this. I really would like users to be > able to upgrade to the latest patch release without hesitation. > > On Wed, May 13, 2020 at 11:45 AM Till Rohrmann <trohrm...@apache.org> > wrote: > > > Thanks for reporting this issue Seth. This is indeed a big problem. > > > > I looked into the problem and it seems we have the following situation: > > > > # 1.9.x --> 1.10.0 > > > > There is an API breaking change between 1.9.x and 1.10 due FLINK-13864 > > because it introduced another generic parameter. I expected only few > people > > will be affected by this because one would have to store the builder in a > > variable. > > > > 1.9.x is binary compatible with 1.10.0. > > > > # 1.10.0 -> 1.10.1 > > > > There is no API breaking change between 1.10.0 and 1.10.1. > > > > I changed the return type of the StreamingFileSink.forRowFormat and > > .forBulkFormat to a subtype of StreamingFileSink.BulkFormatBuilder in > > FLINK-16684. This causes the java.lang.NoSuchMethodError and the binary > > incompatibility between 1.10.0 and 1.10.1. This is should not happen for > > bug fix releases. > > > > W/o FLINK-16684, the StreamingFileSink builders cannot be used with > Flink's > > Scala API. > > > > # Options > > > > I think we should keep the fix for 1.11.0 and add a release note that we > > are violating binary compatibility between 1.10 and 1.11. > > > > Now the question is what to do with the 1.10.x branch: > > > > a) We can revert the change to re-establish binary compatibility between > > 1.9.x, 1.10.0 and 1.10.2 but not between 1.10.1 and 1.10.2. This would > also > > imply that we cannot use the StreamingFileSink builders with the Scala > API. > > > > b) Keep the change and add a release note that our users need to > recompile > > their jobs. This would allow Flink 1.10.x with x >= 1 users to use > > StreamingFileSink builders with the Scala API. > > > > I am not entirely sure which need weighs more: binary compatibility > between > > bug fix releases or a working API (small subset of it). Another aspect to > > consider is how many people will migrate from 1.10.0 to 1.10.1 compared > to > > 1.y to 1.10.1 with y <= 9. If the former is very small then one might > make > > a case for keeping the change. > > > > What do the others think? > > > > https://issues.apache.org/jira/browse/FLINK-13864 > > https://issues.apache.org/jira/browse/FLINK-16684 > > > > Cheers, > > Till > > > > On Tue, May 12, 2020 at 7:26 PM Thomas Weise <t...@apache.org> wrote: > > > > > We also noticed that and had to make an adjustment downstream. > > > > > > It would be good to mention this in the release notes (if that's not > > > already the case). > > > > > > Thomas > > > > > > > > > On Tue, May 12, 2020 at 10:06 AM Seth Wiesman <sjwies...@gmail.com> > > wrote: > > > > > > > Hi Everyone, > > > > > > > > I realize I'm about 7 hours late but I just realized there is a > > breaking > > > > API change in 1.10.1. FLINK-16684 changes the API of the streaming > file > > > > sink in a binary incompatible way. Since the release has been > approved > > > I'm > > > > not sure the correct course of action but I wanted to bring this to > the > > > > communities attention. > > > > > > > > Seth > > > > > > > > https://issues.apache.org/jira/browse/FLINK-16684 > > > > > > > > > >
