How would we enforce this change? That we would enable different, more strict compatibility checks on the `release-1.x` branches?
Piotrek > On 16 May 2020, at 08:33, Congxian Qiu <qcx978132...@gmail.com> wrote: > > Sorry for the late jump in. > > +1 to keep the compatibility of @PublicEvolving between minor > releases(x.y.a -> x.y.b), as a user I always think this as a bug-fix > release, break the compatibility between minor releases may give users a > surprise. > > As the previous emails said, how and when will a @PublicEvolving > become @Public, and I'm not sure if we can have a technical solution to > keep such a rule. (In my opinion, check such things -- change > @PublicEvolving to @Public -- manually may not so easy) > > Best > Congxian > > > Till Rohrmann <trohrm...@apache.org> 于2020年5月15日周五 下午9:18写道： > >> The vote thread can be found here >> >> https://lists.apache.org/thread.html/rc58099fb0e31d0eac951a7bbf7f8bda8b7b65c9ed0c04622f5333745%40%3Cdev.flink.apache.org%3E >> . >> >> Cheers, >> Till >> >> On Fri, May 15, 2020 at 3:03 PM Till Rohrmann <trohrm...@apache.org> >> wrote: >> >>> I completely agree that there are many other aspect of our guarantees and >>> processes around the @Public and @PublicEvolving classes which need to be >>> discussed and properly defined. For the sake of keeping this discussion >>> thread narrowly scoped, I would suggest to start a separate discussion >>> about the following points (not exhaustive): >>> >>> - What should be annotated with @Public and @PublicEvolving? >>> - Process for transforming @PublicEvolving into @Public; How to ensure >>> that @PublicEvolving will eventually be promoted to @Public? >>> - Process of retiring a @Public/@PublicEvolving API >>> >>> I will start a vote thread about the change I proposed here which is to >>> ensure API and binary compatibility for @PublicEvolving classes between >>> bugfix releases (x.y.z and x.y.u). >>> >>> Cheers, >>> Till >>> >>> On Fri, May 15, 2020 at 6:33 AM Zhu Zhu <reed...@gmail.com> wrote: >>> >>>> +1 for "API + binary compatibility for @PublicEvolving classes for all >> bug >>>> fix >>>> releases in a minor release (x.y.z is compatible to x.y.u)" >>>> >>>> This @PublicEnvolving would then be a hard limit to changes. >>>> So it's important to rethink the policy towards using it, as Stephan >>>> proposed. >>>> >>>> I think any Flink interfaces that are visible to users should be >>>> explicitly >>>> marked as @Public or @PublicEnvolving. >>>> Any other interfaces should not be marked as @Public/@PublicEnvolving. >>>> This would be essential for us to check whether we are breaking any user >>>> faced interfaces unexpectedly. >>>> The only exception would be the case that we had to expose a >> method/class >>>> due to implementation limitations, it should be explicitly marked it >>>> as @Internal. >>>> >>>> Thanks, >>>> Zhu Zhu >>>> >>>> Yun Tang <myas...@live.com> 于2020年5月15日周五 上午11:41写道： >>>> >>>>> +1 for this idea, and I also like Xintong's suggestion to make it >>>>> explicitly when the @PublicEvolving API could upgrade to @Public API. >>>>> If we have the rule to upgrade API stable level but not define the >> clear >>>>> timeline, I'm afraid not everyone have the enthusiasm to upgrade this. >>>>> >>>>> The minor suggestion is that I think two major release (which is x.y.0 >>>> as >>>>> Chesnay clarified) might be a bit quick. From the release history [1], >>>>> Flink bump major version every 3 ~ 6 months and two major release gap >>>>> could only be at least half a year. >>>>> I think half a year might be a bit too frequent for users to collect >>>>> enough feedbacks, and upgrading API stable level every 3 major >> versions >>>>> should be better. >>>>> >>>>> [1] https://flink.apache.org/downloads.html#flink >>>>> >>>>> Best >>>>> Yun Tang >>>>> >>>>> >>>>> ________________________________ >>>>> From: Xintong Song <tonysong...@gmail.com> >>>>> Sent: Friday, May 15, 2020 11:04 >>>>> To: dev <dev@flink.apache.org> >>>>> Subject: Re: [DISCUSS] Stability guarantees for @PublicEvolving >> classes >>>>> >>>>> ### Documentation on API compatibility policies >>>>> >>>>> Do we have any formal documentation about the API compatibility >>>> policies? >>>>> The only things I found are: >>>>> >>>>> - In the release announcement (take 1.10.0 as an example) [1]: >>>>> "This version is API-compatible with previous 1.x releases for APIs >>>>> annotated with the @Public annotation." >>>>> - JavaDoc for Public [2] and PublicEvolving [3]. >>>>> >>>>> I think we might have a formal documentation, clearly state our >> policies >>>>> for API compatibility. >>>>> >>>>> - What does the annotations mean >>>>> - In what circumstance would the APIs remain compatible / become >>>>> incompatible >>>>> - How do APIs retire (e.g., first deprecated then removed?) >>>>> >>>>> Maybe there is already such kind of documentation that I overlooked? >> If >>>> so, >>>>> we probably want to make it more explicit and easy-to-find. >>>>> >>>>> ### @Public vs. @PublicEvolving for new things >>>>> >>>>> I share Stephan's concern that, with @PublicEvolving used for every >> new >>>>> feature and rarely upgraded to @Public, we are practically making no >>>>> compatibility guarantee between minor versions (x.y.* / x.z.*). On the >>>>> other hand, I think in many circumstances we do need some time to >>>> collect >>>>> feedbacks for new features before we have enough confidence to make >> the >>>>> commitment that our APIs are stable. Therefore, it makes more sense to >>>> me >>>>> to first make new features @PublicEvolving and then upgrade to @Public >>>> in >>>>> the next one or two releases (unless there's a good reason to further >>>>> postpone it). >>>>> >>>>> I think the key point is how do we make sure the @PublicEvolving >>>> features >>>>> upgrade to @Public. Maybe we can add a parameter to indicate the >>>> expected >>>>> upgrading version. E.g., a new feature introduced in release 1.10.0 >>>> might >>>>> be annotated as @PublicEvolving("1.12.0"), indicating that it is >>>> expected >>>>> to be upgraded to @Public in release 1.12.0. We can check the >>>> annotations >>>>> against the version automatically, forcing to either upgrad the >> feature >>>>> to @Public or explicitly postpone it by modifying the annotation >>>> parameter >>>>> (if there's a good reason). >>>>> >>>>> Additionally, we can do the similar for deprecated features / APIs, >>>>> reminding us to remove things annotated as @Deprecated at certain >> time. >>>>> >>>>> Thank you~ >>>>> >>>>> Xintong Song >>>>> >>>>> >>>>> [1] https://flink.apache.org/news/2020/02/11/release-1.10.0.html >>>>> >>>>> [2] >>>>> >>>>> >>>> >> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/annotation/Public.html >>>>> >>>>> [3] >>>>> >>>>> >>>> >> https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/annotation/PublicEvolving.html >>>>> >>>>> >>>>> >>>>> >>>>> On Thu, May 14, 2020 at 8:22 PM Stephan Ewen <se...@apache.org> >> wrote: >>>>> >>>>>> I just want to throw in that we also need to rethink our policy >>>> towards >>>>>> using @PublicEvolving. >>>>>> >>>>>> We often introduce this easily (for every new feature) and rarely >>>> (almost >>>>>> never) upgrade it to @Public. This kind of leads the idea behind >>>> stable >>>>> API >>>>>> guarantees ad absurdum. >>>>>> >>>>>> I would suggest that we make @PublicEvolving an exception that >> needs a >>>>> good >>>>>> reason rather than for everything that is new (when we don't want to >>>> be >>>>>> bothered with thinking about compatibility). >>>>>> >>>>>> >>>>>> On Thu, May 14, 2020 at 1:05 PM Xintong Song <tonysong...@gmail.com >>> >>>>>> wrote: >>>>>> >>>>>>> Thanks for the clarification. >>>>>>> +1 for keeping the current guarantees for @Public. >>>>>>> >>>>>>> Thank you~ >>>>>>> >>>>>>> Xintong Song >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, May 14, 2020 at 6:07 PM Till Rohrmann < >> trohrm...@apache.org >>>>> >>>>>>> wrote: >>>>>>> >>>>>>>> Sorry for the confusion. @Public classes are guaranteed to be >>>> stable >>>>>>>> between releases x.y.z and x.u.v (minor and bug fix release; >>>> naming >>>>> is >>>>>>>> indeed a bit off here) and we can break it with major releases >>>> (x.0.0 >>>>>> and >>>>>>>> y.0.0). >>>>>>>> >>>>>>>> @Tison I would like to make what to include in the public API, >>>> hence >>>>>> what >>>>>>>> to annotate with @Public and @PublicEvolving, a separate >>>> discussion. >>>>>>>> >>>>>>>> Cheers, >>>>>>>> Till >>>>>>>> >>>>>>>> On Thu, May 14, 2020 at 11:48 AM tison <wander4...@gmail.com> >>>> wrote: >>>>>>>> >>>>>>>>> Thanks for starting this discussion! >>>>>>>>> >>>>>>>>> I agree turn on japicmp on PublicEvolving among bugfix releases >>>> is a >>>>>> nit >>>>>>>>> win. >>>>>>>>> >>>>>>>>> @Xintong Song <tonysong...@gmail.com> I think @Public >> guarantee >>>> is >>>>>> good >>>>>>>>> enough, the problem is a reachable 2.0 plan. >>>>>>>>> >>>>>>>>> My concern is more on classes that have no annotation but our >>>>>> developers >>>>>>>>> regard as "something that should be stable". Previously I was >>>>> required >>>>>>> to >>>>>>>>> keep compatibility of ClusterClient & HighAvailabilityServices >>>>> because >>>>>>>>> they >>>>>>>>> might be depended on by user. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> tison. >>>>>>>>> >>>>>>>>> >>>>>>>>> Dawid Wysakowicz <dwysakow...@apache.org> 于2020年5月14日周四 >>>> 下午5:08写道： >>>>>>>>> >>>>>>>>>> I also like the proposal for keeping the binary compatibility >>>> of >>>>>>>>>> @PublicEvolving for bugfix releases. >>>>>>>>>> >>>>>>>>>> As for the @Public classes I think the current guarantees are >>>> good >>>>>>>>> enough. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> >>>>>>>>>> Dawid >>>>>>>>>> >>>>>>>>>> On 14/05/2020 10:49, Jingsong Li wrote: >>>>>>>>>>> Thanks Till for starting this discussion. >>>>>>>>>>> >>>>>>>>>>> +1 for enabling the japicmp-maven-plugin for >> @PublicEvolving >>>> for >>>>>> bug >>>>>>>>> fix >>>>>>>>>>> releases. >>>>>>>>>>> Bug fix should just be user imperceptible bug fix. Should >> not >>>>>> affect >>>>>>>>> API >>>>>>>>>>> and binary compatibility. >>>>>>>>>>> >>>>>>>>>>> And even PublicEvolving api change for "y" release, we >> should >>>>>> expose >>>>>>>>> it >>>>>>>>>> in >>>>>>>>>>> dev mail list for discussing or a FLIP? >>>>>>>>>>> >>>>>>>>>>> BTW, public api can be changed by major releases? In >>>> annotation >>>>>>>>> comments: >>>>>>>>>>> "Only major releases (1.0, 2.0, 3.0) can break interfaces >>>> with >>>>>> this >>>>>>>>>>> annotation". >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Jingsong Lee >>>>>>>>>>> >>>>>>>>>>> On Thu, May 14, 2020 at 4:30 PM Till Rohrmann < >>>>>> trohrm...@apache.org >>>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Dear community, >>>>>>>>>>>> >>>>>>>>>>>> in the latest 1.10.1 bug fix release I introduced a binary >>>>>>>>> incompatible >>>>>>>>>>>> change to a class which is annotated with @PublicEvolving >>>> [1]. >>>>>>> While >>>>>>>>>> this >>>>>>>>>>>> change is technically ok since we only provide API and >>>> binary >>>>>>>>>> compatibility >>>>>>>>>>>> for @Public classes across releases, it raised the >> question >>>>>> whether >>>>>>>>> we >>>>>>>>>>>> can't do better. >>>>>>>>>>>> >>>>>>>>>>>> For our users it might be surprising and really annoying >>>> that >>>>>> they >>>>>>>>>> cannot >>>>>>>>>>>> simply upgrade to the latest bug fix release without >>>>> recompiling >>>>>>> the >>>>>>>>>>>> program or even having to change the source code of an >>>>>>> application. I >>>>>>>>>>>> believe we would provide a much better experience if we >>>> ensured >>>>>>> that >>>>>>>>> bug >>>>>>>>>>>> fix releases maintain API and binary compatibility also >> for >>>>>>>>>> @PublicEvolving >>>>>>>>>>>> classes. Hence my proposal would be to tighten the >> stability >>>>>>>>> guarantees >>>>>>>>>> the >>>>>>>>>>>> following way: >>>>>>>>>>>> >>>>>>>>>>>> * API + binary compatibility for @Public classes across >> all >>>>>>> releases >>>>>>>>>> (x.y.z >>>>>>>>>>>> is compatible to u.v.w) >>>>>>>>>>>> * API + binary compatibility for @PublicEvolving classes >> for >>>>> all >>>>>>> bug >>>>>>>>> fix >>>>>>>>>>>> releases in a minor release (x.y.z is compatible to x.y.u) >>>>>>>>>>>> >>>>>>>>>>>> This would entail that we can change @PublicEvolving >> classes >>>>> only >>>>>>>>> across >>>>>>>>>>>> minor/major releases. >>>>>>>>>>>> >>>>>>>>>>>> Practically this would mean that we enable the >>>>>> japicmp-maven-plugin >>>>>>>>>>>> for @PublicEvolving for bug fix releases. >>>>>>>>>>>> >>>>>>>>>>>> What do you think? >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >> https://lists.apache.org/thread.html/r293768d13d08149d756e0bf91be52372edb444c317535d1d5a496c3e%40%3Cdev.flink.apache.org%3E >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Till >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >>
