Hi Xintong, Thanks for the summary, most of the points that you agreed upon also make sense to me.
> 2. Dropping deprecated Public APIs in minor releases, or demoting APIs > from Public to PublicEvolving / Experimental / Retired in any version bump, > are not good practices. I hope we can can go beyond calling it “not a good practice" and reach consensus that we will not demote or remove public APIs in minor releases and (technically) enforce this rule for all contributions. I think it’s a reasonable expectation for stability from a project as mature as Flink. > I'm personally in favor of option 1. As the migration-period rule is newly > proposed, I think it's fair to make exceptions for cases where we already > missed the best chance for planning the deprecation. Moreover, I do believe > having a quick major version bump does confuse users. Yes, we can explain > to users that bumping from 2.x to 3.0 does not cost anything other than the > removal of an deprecated API. But having to explain this itself is an > indicator that it might be confusing for users. > > > Becket, on the other hand, prefers option 2. From my understanding, his > major point is that a quick major version bump causes barely any actual > lose on users, while in option 1 not providing the migration period or > providing a shorter on is an actual lose on users. (@Becket, please correct > me if I'm mistaken.) > I would be open to both options and maybe postpone this decision until we have the complete roadmap for Flink 2.0. We would have a better overview about the technical consequences, e.g. how hard it would be to offer and maintain both APIs side by side, how well we get both APIs separated, and whether or not we will be able to use the full potential of our breaking changes before the DataStream API is completely removed. One example for the last point I can think of would be moving from lazy to eager state declaration. If we believe that it is reasonable to support all planned changes while maintaining both APIs, I'm leaning towards option 2. Best, Stefan > On 20. Jun 2023, at 14:43, Xintong Song <tonysong...@gmail.com> wrote: > > Becket and I had an offline voice call earlier today. We have reached > consensus on some of the arguments, but still hold different opinions on > some others. I'd like to post the outcome here for transparency. > > We both agree that: > > 1. Providing a migration period would be beneficial for our users, and > we should do that > 2. Dropping deprecated Public APIs in minor releases, or demoting APIs > from Public to PublicEvolving / Experimental / Retired in any version bump, > are not good practices. > 3. Ideally, with this FLIP, developers should be more careful and plan > API changes ahead. That means： > 1. Be more careful with designing APIs and promoting them to Public, > so that they won't be changed / removed very soon, and also the > maintenance overhead for keeping them after deprecation should be > affordable. (I believe this also aligns with John's opinion.) > 2. Plan the deprecation / removal of old APIs that we don't want to > carry to the next major version earlier, so that they'll have the > 2-minor-release migration period before the next major version bump. > 4. A practical situation is that, we are not ready for deprecating the > DataStream API, nor afford to carry it for another couple of years. > 1. We cannot deprecate it now, because the new ProcessFunction API > (the planned replacement for DataStream API) is not yet ready. We haven't > planned the ProcessFunction API earlier because the original > plan was to do > in-place changes directly on the DataStream API in release 2.0, until the > smooth migration period is proposed. Before this, the only guarantees we > provide for Public APIs is that they'll stay compatible until the next > major version, which led to the understanding the in-place changes on > Public APIs should be fine. > 2. We also cannot afford carrying the DataStream API for another > couple of years. One of the reasons that we want to refactor / replace > DataStream API is that it exposes and depends on too many > Flink's internal > runtime implementations, which has already limited / blocked plans for > Flink's internal improvements for many times. Also due to the wide and > transitive dependencies on the internals, maintaining two sets of lower > APIs itself is expensive and would be better to keep it short. This is > admittedly a historical design flaw which should be avoided in future. > > What we haven't reached consensus is about how to deal with the current > situation around the DataStream API. There are two options. > > > 1. We say this FLIP is enforced starting release 2.0. For current 1.x > APIs, we provide a migration period with best effort, while allowing > exceptions for immediate removal in 2.0. That means we will still try with > best effort to get the ProcessFuncion API ready and deprecate the > DataStream API in 1.x, but will also be allowed to remove DataStream API in > 2.0 if it's not deprecated 2 minor releases before the major version bump. > 2. We strictly follow the process in this FLIP, and will quickly bump > the major version from 2.x to 3.0 once the migration period for DataStream > API is reached. > > I'm personally in favor of option 1. As the migration-period rule is newly > proposed, I think it's fair to make exceptions for cases where we already > missed the best chance for planning the deprecation. Moreover, I do believe > having a quick major version bump does confuse users. Yes, we can explain > to users that bumping from 2.x to 3.0 does not cost anything other than the > removal of an deprecated API. But having to explain this itself is an > indicator that it might be confusing for users. > > > Becket, on the other hand, prefers option 2. From my understanding, his > major point is that a quick major version bump causes barely any actual > lose on users, while in option 1 not providing the migration period or > providing a shorter on is an actual lose on users. (@Becket, please correct > me if I'm mistaken.) > > > And we'd like to hear more feedback from the community. > > > Best, > > Xintong > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcep...@apache.org > <mailto:vvcep...@apache.org>> wrote: > >> Hi Becket and Xintong, >> >> I hope you don't mind if I chime in a little. >> >> Once an API is marked Public, we're committing to support it until it's >> deprecated, and once it's deprecated, to leave it in place for at least two >> minor releases and then only remove it in the following major release. >> >> As a user, I would be dismayed to discover that the project maintainers >> demoted a Public API to PublicEvolving or Experimental in order to violate >> the spirit of the deprecation period for Public APIs. Likewise, I'd view a >> rapid sequence of minor releases for the purpose of dropping deprecated >> APIs in the next major release as another violation of the spirit of our >> guarantees. I'm reminded of this xkcd: >> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_ >> >> I'm glad we're gaming out these situations before we institute this FLIP, >> and I hope we all converge on a clear understanding of what we're >> guaranteeing. From where I'm sitting, the essence of this FLIP is >> specifically to prevent us from doing whatever we want and instead to >> commit to providing a suitable notice period to users who rely on Public >> APIs. >> >> It honestly shouldn't matter what the maintenance overhead is (or, rather, >> evaluating the maintenance overhead should be a consideration before we >> promote an API to Public to begin with). My experience in other projects is >> that regretting a feature so much you want to drop it asap is extremely >> rare, and that living with the pain for a little while until the >> deprecation period expires will probably make us better engineers. >> >> With that all said, one thing that is not a violation is to propose a new >> major release in order to drop especially burdensome deprecated APIs that >> have already enjoyed their full deprecation period. Those really bad "case >> A" features will be very rare. >> >> Looking over the FLIP again, I see a reference to the release >> documentation, but not to the build tooling support for the Deprecated >> annotation, so it might help to share this scenario: The way this will all >> come together in practice is that, by giving good stability guarantees, we >> will encourage our users to bump minor versions when they are released, >> which means that they will see compiler warnings immediately if any of the >> APIs they've used are deprecated. In fact, I'd recommend for them to use >> the `-Werror` javac flag to be sure that they don't miss those warnings. >> Given our release cadence, this means that they will have about six months >> to migrate off of any deprecated APIs without being forced to avoid >> updates. And they really need to be able to adopt further minor updates in >> the series, since they may contain further improvements that actually >> facilitate migrating off of the deprecated APIs. >> >> The compiler warnings are a far more reliable mechanism to advertise >> deprecation than release notes, since users' own builds will notify them >> right away about only the deprecations that are relevant to them, but it >> does rely on users feeling ok to bump up minor versions in a timely >> fashion. If we consistently burn them by not observing the stability >> guarantees, they'll try to avoid updates instead, and the whole thing falls >> apart. >> >> Thanks again, >> -John >> >> On 2023/06/19 10:43:05 Becket Qin wrote: >>> Hi Xintong, >>> >>> Let's compare the following cases: >>> >>> A. If the maintenance overhead of the deprecated API is high, and we want >>> to remove it after two minor releases. Then there are two options: >>> A1: Demote the API in the major version bump and remove the code >> with a >>> minor version bump. >>> A2: Do exactly the same as 1-A, except that when removing the code, >>> bump up the major version. So this might be a short major version. >>> >>> B. If the maintenance overhead of the deprecated API is not high, >> therefore >>> keeping it for a long time is affordable. There are also two options: >>> B1: Same as A-1, demote the API in the major version bump and remove >>> the code with a minor version bump. >>> B2: Keep the API for all the minor versions in the major version, and >>> only remove the code in the next major version. >>> >>> For case B, do we agree that B2 is the way to go? >>> >>> For case A: >>> The following stays the same for A1 and A2: >>> - users will lose the API after two minor leases. So the migration >> period >>> is the same for A-1 and A-2. >>> - A1 and A2 will have exactly the same code after the removal of >>> deprecated API, the only difference is versioning. >>> - To move forward, users need to move to the next minor version in A1, >> or >>> to the next major version in A2. Because the code are the same, the >> actual >>> effort to upgrade is the same for A1 and A2. >>> >>> The differences between A-1 and A-2 are: >>> - A1 allows keeping the major version release cadence. A-2 will have a >>> short major version release. >>> - A1 breaks the well understood API semantic, while A-2 does not. >>> >>> From what I see, since there is no well establish standard regarding how >>> long a major version should be released, A short major version release is >>> potential and emotional. It is not ideal but does not have material >>> downsides compared with A1. I did not hear anyone complaining about Kafka >>> only has two 1.x release. However, A1 actually breaks the well understood >>> API semantic, which has more material impact. >>> >>> Also, I'd imagine 90% or more of the Public APIs should fall into case B. >>> So, short major versions should be very occasional. I'd be very concerned >>> if the reason we choose A1 is simply because we cannot afford >> maintaining a >>> bunch of deprecated APIs until the next major version. This indicates >> that >>> the actual problem we need to solve is to lower the maintenance overhead >> of >>> deprecated APIs, so that we are comfortable to keep them longer. As John >>> and I mentioned earlier, there are ways to achieve this and we need to >>> learn how to do it in Flink. Otherwise, our discussion about versioning >>> here does not bring much value, because we will end up with a bunch of >>> short-lived APIs which upset our users, no matter how we version the >>> releases. >>> >>> So, if there are concrete examples that you think will block us from >>> keeping API stability with affordable cost, let's take a look together >> and >>> see if that can be improved. >>> >>> Thanks, >>> >>> Jiangjie (Becket) Qin >>> >>> >>> From what I see, >>> >>> >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <tonysong...@gmail.com >>> <mailto:tonysong...@gmail.com>> >> wrote: >>> >>>>> >>>>> The part I don't understand is if we are willing to have a migration >>>>> period, and do a minor version bump to remove an API, what do we >> lose to >>>> do >>>>> a major version bump instead, so we don't break the common versioning >>>>> semantic? >>>>> >>>> >>>> I think we are talking about the cases where a major version bump >> happens >>>> before a deprecated Public API reaches its migration period. So >> removing >>>> the API with another major version bump means we have two consecutive >> major >>>> versions in a very short time. And as previously mentioned, having >> frequent >>>> major version bumps would weaken the value of the commitment "Public >> API >>>> stay compatible within a major version". I think users also have >>>> expectations about how long a major version should live, which should >> be at >>>> least a couple of years rather than 1-2 minor releases. >>>> >>>> This is another option, but I think it is very likely more expensive >> than >>>>> simply bumping the major version. And I imagine there will be >> questions >>>>> like "why this feature is in 1.20, but does not exist in 2.0"? >>>>> >>>> >>>> Yes, it's more expensive than simply bumping the major version, but >> IMHO >>>> it's probably cheaper than carrying the API until the next major >> version if >>>> we don't bump the major version very soon. And see my reply above on >> why >>>> not bumping immediately. >>>> >>>> Best, >>>> >>>> Xintong >>>> >>>> >>>> >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket....@gmail.com >>>> <mailto:becket....@gmail.com>> >> wrote: >>>> >>>>> Hi Xintong, >>>>> >>>>> Please see the replies below. >>>>> >>>>> I see your point, that users would feel surprised if they find >> things no >>>>>> longer work when upgrading to another 2.x minor release. However, >> I'd >>>>> like >>>>>> to point out that PublicEvolving APIs would have the similar >> problem >>>>>> anyway. So the question is, how do we catch users' attention and >> make >>>>> sure >>>>>> they are aware that the Public APIs in 1.x may no longer be Public >> in >>>>> 2.0. >>>>>> There're various ways to do that, e.g., release notes, warnings in >>>> logs, >>>>>> etc. >>>>> >>>>> First of all, I am not a fan of removing PublicEvolving APIs in minor >>>>> version changes. Personally speaking, I want them to be also removed >> in >>>>> major version changes. >>>>> >>>>> Empirically, I rarely see projects with complex API semantic work >> well. >>>>> Many, if not most, users don't read docs / release notes / warnings >> in >>>>> logs. I don't think that is their fault. Typically an application >>>> developer >>>>> will have to deal with dozens of libraries maintained by various >>>>> communities / groups. It is just too difficult for them to keep >> track of >>>>> all the specific semantics each project puts there. If you think >> about >>>> it, >>>>> how many of Flink developers read all the release notes of Guava, >> Apache >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of >> them? >>>>> One would probably try bumping the dependency version and see if >> there is >>>>> an exception and solve them case by case. And if it is a minor >> version >>>>> bump, one probably does not expect an exception at all. Another >> example >>>> is >>>>> that in Flink we still have so many usage of deprecated methods all >> over >>>>> the place, and I strongly doubt everyone knows when the source code >> of >>>>> these methods are deprecated and when they should be removed. >>>>> >>>>> So, the most important principle of API is simple and intuitive. The >>>>> versioning semantic is a simple and universally accepted API >> stability >>>>> standard. If we ourselves as Flink developers are relying on this >> for our >>>>> own dependencies. I don't think we can expect more from our users. >>>>> >>>>> Another possible alternative: whenever there's a deprecated Public >> API >>>> that >>>>>> reaches a major version bump before the migration period, and we >> also >>>>> don't >>>>>> want to carry it for all the next major release series, we may >> consider >>>>>> releasing more minor releases for the previous major version after >> the >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump >> to >>>> 2.0, >>>>>> we can release one more 1.20 after 2.0. That should provide users >>>> another >>>>>> choice rather than upgrading to 2.0, while satisfying the >>>> 2-minor-release >>>>>> migration period. >>>>> >>>>> This is another option, but I think it is very likely more expensive >> than >>>>> simply bumping the major version. And I imagine there will be >> questions >>>>> like "why this feature is in 1.20, but does not exist in 2.0"? >>>>> >>>>> I think my major point is, we should not carry APIs deprecated in a >>>>>> previous major version along all the next major version series. I'd >>>> like >>>>> to >>>>>> try giving users more commitments, i.e. the migration period, as >> long >>>> as >>>>> it >>>>>> does not prevent us from making breaking changes. If it doesn't >> work, >>>> I'd >>>>>> be in favor of not providing the migration period, but fallback to >> only >>>>>> guarantee the compatibility within the major version. >>>>> >>>>> The part I don't understand is if we are willing to have a migration >>>>> period, and do a minor version bump to remove an API, what do we >> lose to >>>> do >>>>> a major version bump instead, so we don't break the common versioning >>>>> semantic? >>>>> >>>>> Thanks, >>>>> >>>>> Jiangjie (Becket) Qin >>>>> >>>>> >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <tonysong...@gmail.com >>>>> <mailto:tonysong...@gmail.com>> >>>>> wrote: >>>>> >>>>>>> >>>>>>> As an end user who only uses Public APIs, if I don't change my >> code >>>> at >>>>>>> all, my expectation is the following: >>>>>>> 1. Upgrading from 1.x to 2.x may have issues. >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine >> with >>>>> all >>>>>>> the 2.x versions. >>>>>>> Actually I think there are some dependency version resolution >>>> policies >>>>>> out >>>>>>> there which picks the highest minor version when the dependencies >>>> pull >>>>> in >>>>>>> multiple minor versions of the same jar, which may be broken if >> we >>>>> remove >>>>>>> the API in minor releases. >>>>>>> >>>>>> >>>>>> I see your point, that users would feel surprised if they find >> things >>>> no >>>>>> longer work when upgrading to another 2.x minor release. However, >> I'd >>>>> like >>>>>> to point out that PublicEvolving APIs would have the similar >> problem >>>>>> anyway. So the question is, how do we catch users' attention and >> make >>>>> sure >>>>>> they are aware that the Public APIs in 1.x may no longer be Public >> in >>>>> 2.0. >>>>>> There're various ways to do that, e.g., release notes, warnings in >>>> logs, >>>>>> etc. >>>>>> >>>>>> Another possible alternative: whenever there's a deprecated Public >> API >>>>> that >>>>>> reaches a major version bump before the migration period, and we >> also >>>>> don't >>>>>> want to carry it for all the next major release series, we may >> consider >>>>>> releasing more minor releases for the previous major version after >> the >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump >> to >>>> 2.0, >>>>>> we can release one more 1.20 after 2.0. That should provide users >>>> another >>>>>> choice rather than upgrading to 2.0, while satisfying the >>>> 2-minor-release >>>>>> migration period. >>>>>> >>>>>> I think my major point is, we should not carry APIs deprecated in a >>>>>> previous major version along all the next major version series. I'd >>>> like >>>>> to >>>>>> try giving users more commitments, i.e. the migration period, as >> long >>>> as >>>>> it >>>>>> does not prevent us from making breaking changes. If it doesn't >> work, >>>> I'd >>>>>> be in favor of not providing the migration period, but fallback to >> only >>>>>> guarantee the compatibility within the major version. >>>>>> >>>>>> Best, >>>>>> >>>>>> Xintong >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vvcep...@apache.org >>>>>> <mailto:vvcep...@apache.org> >>> >>>>> wrote: >>>>>> >>>>>>> Hi Becket, >>>>>>> >>>>>>> Thanks for the reply! I’d like to continue the conversation about >>>>>>> compatibility outside of this FLIP thread, but for now, I can >> accept >>>>> your >>>>>>> decision. It’s certainly an improvement. >>>>>>> >>>>>>> Thanks again, >>>>>>> John >>>>>>> >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote: >>>>>>>> Hi John, >>>>>>>> >>>>>>>> Completely agree with all you said. >>>>>>>> >>>>>>>> Can we consider only dropping deprecated APIs in major releases >>>>> across >>>>>>> the >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs >> are >>>> by >>>>>>>>> definition less stable, but it seems like this should be >> reflected >>>>> in >>>>>>> the >>>>>>>>> required deprecation period alone. I.e. that we must keep them >>>>> around >>>>>>> for >>>>>>>>> at least zero or one minor release, not that we can drop them >> in a >>>>>>> minor or >>>>>>>>> patch release. >>>>>>>> >>>>>>>> Personally speaking, I would love to do this, for exactly the >>>> reason >>>>>> you >>>>>>>> mentioned. However, I did not propose this due to the following >>>>>> reasons: >>>>>>>> >>>>>>>> 1. I am hesitating a little bit about changing the accepted >> FLIPs >>>> too >>>>>>> soon. >>>>>>>> 2. More importantly, to avoid slowing down our development. At >> this >>>>>>> point, >>>>>>>> Flink still lacks some design / routines to support good API >>>>>>> evolvability / >>>>>>>> extensibility. Just like you said, it takes some time to be >> good at >>>>>> this. >>>>>>>> In this case, my concern is that only removing Experimental / >>>>>>>> PublicEvolving APIs in major version changes may result in too >> much >>>>>>>> overhead and dramatically slow down the development of Flink. >> So, I >>>>> was >>>>>>>> thinking that we can start with the current status. Hopefully >> after >>>>> we >>>>>>> are >>>>>>>> more comfortable with the maintenance overhead of deprecated >> APIs, >>>> we >>>>>> can >>>>>>>> then have a stronger guarantee for Experimental / >> PublicEvolving >>>>> APIs. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Jiangjie (Becket) Qin >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler < >> vvcep...@apache.org <mailto:vvcep...@apache.org>> >>>>>>> wrote: >>>>>>>> >>>>>>>>> Hi Becket, >>>>>>>>> >>>>>>>>> Thanks for this FLIP! Having a deprecation process is really >>>>>> important. >>>>>>> I >>>>>>>>> understand some people’s concerns about the additional burden >> for >>>>>>> project >>>>>>>>> maintainers, but my personal experience with Kafka has been >> that >>>>> it’s >>>>>>> very >>>>>>>>> liveable and that it’s well worth the benefit to users. In >> fact, >>>>> users >>>>>>>>> being able to confidently upgrade is also a benefit to >>>> maintainers, >>>>> as >>>>>>> we >>>>>>>>> will get fewer questions from people stuck on very old >> versions. >>>>>>>>> >>>>>>>>> One question: >>>>>>>>> Can we consider only dropping deprecated APIs in major >> releases >>>>> across >>>>>>> the >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs >> are >>>> by >>>>>>>>> definition less stable, but it seems like this should be >> reflected >>>>> in >>>>>>> the >>>>>>>>> required deprecation period alone. I.e. that we must keep them >>>>> around >>>>>>> for >>>>>>>>> at least zero or one minor release, not that we can drop them >> in a >>>>>>> minor or >>>>>>>>> patch release. >>>>>>>>> >>>>>>>>> The advantage of forbidding the removal of any API in minor or >>>> patch >>>>>>>>> releases is that users will get a strong guarantee that they >> can >>>>> bump >>>>>>> the >>>>>>>>> minor or patch version and still be able to compile, or even >> just >>>>>>> re-link >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run >> time. >>>>> This >>>>>>> is a >>>>>>>>> binary guarantee: if we allow removing even Experimental APIs >>>>> outside >>>>>>> of >>>>>>>>> major releases, users can no longer confidently upgrade. >>>>>>>>> >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points: >>>>>>>>> * I’d use the official Deprecated annotation instead of >>>> introducing >>>>>> our >>>>>>>>> own flavor (Retired, etc), since Deprecated is well integrated >>>> into >>>>>>> build >>>>>>>>> tools and IDEs. >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it >> seems >>>>>>>>> orthogonal, and something that should probably be taken >>>> case-by-case >>>>>>>>> anyway. >>>>>>>>> * Aside from deprecation and removal, there have been some >>>>> discussions >>>>>>>>> about how to evolve APIs and behavior in compatible ways. >> This is >>>>>>> somewhat >>>>>>>>> of an art, and if folks haven’t wrestled with it before, it’ll >>>> take >>>>>> some >>>>>>>>> time to become good at it. I feel like this topic should also >> be >>>>>>> orthogonal >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a >> simple >>>>>> policy >>>>>>> not >>>>>>>>> to break existing user programs, and leave the “how” up to >>>>>> implementers >>>>>>> and >>>>>>>>> reviewers. >>>>>>>>> >>>>>>>>> Thanks again, >>>>>>>>> John >>>>>>>>> >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: >>>>>>>>>> Hi All, >>>>>>>>>> >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a >> great >>>>> idea. >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e. >>>> @PublicEvolving >>>>>> --(2 >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of >> flexibility >>>>>>> without >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to >> change >>>>>> between >>>>>>>>> major >>>>>>>>>> releases. Changing annotations is acceptable and provides >>>>> additional >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself >> are >>>> not >>>>>>>>> changed. >>>>>>>>>> >>>>>>>>>> I had similar thoughts when I was facing those issues. I >> want to >>>>>> move >>>>>>> one >>>>>>>>>> step further and suggest introducing one more annotation >>>> @Retired. >>>>>>>>>> >>>>>>>>>> Not like the @PublicRetired which is a compromise of >> downgrading >>>>>>> @Public >>>>>>>>> to >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java >>>> standard >>>>>>>>>> @deprecated should be used in the early stage of the >> deprecation >>>>>>> process >>>>>>>>>> and doesn't really meet our requirement. Since Java does not >>>> allow >>>>>> us >>>>>>> to >>>>>>>>>> extend annotation, I think it would be feasible to have the >> new >>>>>>> @Retired >>>>>>>>> to >>>>>>>>>> help us monitor and manage the deprecation process, house >>>>> cleaning, >>>>>>> etc. >>>>>>>>>> >>>>>>>>>> Some ideas could be(open for discussion): >>>>>>>>>> >>>>>>>>>> @Retired: >>>>>>>>>> >>>>>>>>>> 1. There must be a replacement with functionality >> compatibility >>>>>> before >>>>>>>>> APIs >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes >> on >>>> the >>>>> ML >>>>>>> are >>>>>>>>>> mandatory (a FLIP is recommended). >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor >> release >>>>>>> sharply >>>>>>>>>> (using ArchUnit to force it, needs further check whether it >> is >>>>>>> possible). >>>>>>>>>> Devs who marked them as @Retired are responsible to remove >> them. >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired >> are >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal -> >> @Retired >>>>>> could >>>>>>>>> also >>>>>>>>>> be used if it can increase user-friendliness or >>>> dev-friendliness, >>>>>> but >>>>>>> not >>>>>>>>>> mandatory. >>>>>>>>>> 4. Some variables will be defined in @Retired to support the >>>>>>> deprecation >>>>>>>>>> process management. Further extension is possible, since the >>>>>>> annotation >>>>>>>>> is >>>>>>>>>> built by us. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Best regards, >>>>>>>>>> Jing >>>>>>>>>> >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin < >>>> becket....@gmail.com <mailto:becket....@gmail.com> >>>>>> >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Xintong, >>>>>>>>>>> >>>>>>>>>>> Thanks for the explanation. Please see the replies inline >>>> below. >>>>>>>>>>> >>>>>>>>>>> I agree. And from my understanding, demoting a Public API >> is >>>>> also a >>>>>>>>> kind of >>>>>>>>>>>> such change, just like removing one, which can only >> happen >>>> with >>>>>>> major >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public >>>> APIs >>>>>>>>> anytime, >>>>>>>>>>> but >>>>>>>>>>>> only in the case major version bumps happen before >> reaching >>>> the >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting >> would >>>> be a >>>>>>> weaker >>>>>>>>>>>> change compared to removing the API immediately upon >> major >>>>>> version >>>>>>>>> bumps, >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release >>>>>> migration >>>>>>>>>>> period. >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` >> sounds >>>>>>> against >>>>>>>>>>>> conventions, we may introduce a new annotation if >> necessary, >>>>>> e.g., >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. >>>>>>>>>>> >>>>>>>>>>> As an end user who only uses Public APIs, if I don't >> change my >>>>> code >>>>>>> at >>>>>>>>> all, >>>>>>>>>>> my expectation is the following: >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues. >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am >> fine >>>>>> with >>>>>>> all >>>>>>>>>>> the 2.x versions. >>>>>>>>>>> Actually I think there are some dependency version >> resolution >>>>>>> policies >>>>>>>>> out >>>>>>>>>>> there which picks the highest minor version when the >>>> dependencies >>>>>>> pull >>>>>>>>> in >>>>>>>>>>> multiple minor versions of the same jar, which may be >> broken if >>>>> we >>>>>>>>> remove >>>>>>>>>>> the API in minor releases. >>>>>>>>>>> >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that >> we >>>>> bump >>>>>> up >>>>>>>>> the >>>>>>>>>>>> major version whenever a breaking change is needed. >> However, >>>>> this >>>>>>> also >>>>>>>>>>>> weakens the value of the commitment that public APIs will >>>> stay >>>>>>> stable >>>>>>>>>>>> within the major release series, as the series can end >>>> anytime. >>>>>>> IMHO, >>>>>>>>>>> short >>>>>>>>>>>> major release series are not something "make the end >> users >>>>>> happy", >>>>>>> but >>>>>>>>>>>> backdoors that allow us as the developers to make >> frequent >>>>>> breaking >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we >> can >>>>>> still >>>>>>>>> have >>>>>>>>>>>> longer major release series, while only allowing Public >> APIs >>>>>>>>> deprecated >>>>>>>>>>> at >>>>>>>>>>>> the end of the previous major version to be removed in >> the >>>> next >>>>>>> major >>>>>>>>>>>> version. >>>>>>>>>>> >>>>>>>>>>> I totally agree that frequent major version bumps are not >>>> ideal, >>>>>> but >>>>>>>>> here >>>>>>>>>>> we are comparing it with a minor version bump which >> removes a >>>>>> Public >>>>>>>>> API. >>>>>>>>>>> So the context is that we have already decided to remove >> this >>>>>> Public >>>>>>> API >>>>>>>>>>> while keeping everything else backwards compatible. I >> think a >>>>> major >>>>>>>>> version >>>>>>>>>>> bump is a commonly understood signal here, compared with a >>>> minor >>>>>>> version >>>>>>>>>>> change. From end users' perspective, for those who are not >>>>>> impacted, >>>>>>> in >>>>>>>>>>> this case upgrading a major version is not necessarily more >>>>>> involved >>>>>>>>> than >>>>>>>>>>> upgrading a minor version - both should be as smooth as a >>>>>> dependency >>>>>>>>>>> version change. For those who are impacted, they will lose >> the >>>>>> Public >>>>>>>>> API >>>>>>>>>>> anyways and a major version bump ensures there is no >> surprise. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> >>>>>>>>>>> Jiangjie (Becket) Qin >>>>>>>>>>> >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song < >>>>>> tonysong...@gmail.com <mailto:tonysong...@gmail.com> >>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Public API is a well defined common concept, and one of >> its >>>>>>>>>>>>> convention is that it only changes with a major version >>>>> change. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I agree. And from my understanding, demoting a Public >> API is >>>>>> also a >>>>>>>>> kind >>>>>>>>>>>> of such change, just like removing one, which can only >> happen >>>>>> with >>>>>>>>> major >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public >>>> APIs >>>>>>>>> anytime, >>>>>>>>>>> but >>>>>>>>>>>> only in the case major version bumps happen before >> reaching >>>> the >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting >> would >>>> be a >>>>>>> weaker >>>>>>>>>>>> change compared to removing the API immediately upon >> major >>>>>> version >>>>>>>>> bumps, >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release >>>>>> migration >>>>>>>>>>> period. >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` >> sounds >>>>>>> against >>>>>>>>>>>> conventions, we may introduce a new annotation if >> necessary, >>>>>> e.g., >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. >>>>>>>>>>>> >>>>>>>>>>>> But it should be >>>>>>>>>>>>> completely OK to bump up the major version if we really >> want >>>>> to >>>>>>> get >>>>>>>>> rid >>>>>>>>>>> of >>>>>>>>>>>>> a public API, right? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" >> that we >>>>>> bump >>>>>>> up >>>>>>>>> the >>>>>>>>>>>> major version whenever a breaking change is needed. >> However, >>>>> this >>>>>>> also >>>>>>>>>>>> weakens the value of the commitment that public APIs will >>>> stay >>>>>>> stable >>>>>>>>>>>> within the major release series, as the series can end >>>> anytime. >>>>>>> IMHO, >>>>>>>>>>> short >>>>>>>>>>>> major release series are not something "make the end >> users >>>>>> happy", >>>>>>> but >>>>>>>>>>>> backdoors that allow us as the developers to make >> frequent >>>>>> breaking >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we >> can >>>>>> still >>>>>>>>> have >>>>>>>>>>>> longer major release series, while only allowing Public >> APIs >>>>>>>>> deprecated >>>>>>>>>>> at >>>>>>>>>>>> the end of the previous major version to be removed in >> the >>>> next >>>>>>> major >>>>>>>>>>>> version. >>>>>>>>>>>> >>>>>>>>>>>> Given our track record I would prefer a regular cycle >> (1-2 >>>>> years) >>>>>>> to >>>>>>>>>>>>> force us to think about this whole topic, and not put it >>>> again >>>>>> to >>>>>>> the >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on >>>> when >>>>>>>>> breaking >>>>>>>>>>>>> changes can be made. >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time >> for new >>>>>> major >>>>>>>>>>>> versions, or longer if there's no breaking changes >> needed. >>>> That >>>>>>> makes >>>>>>>>> 1-2 >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving >> us >>>> more >>>>>>> time >>>>>>>>> to >>>>>>>>>>>> prepare the major release if needed. >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> >>>>>>>>>>>> Xintong >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler < >>>>>>> ches...@apache.org <mailto:ches...@apache.org> >>>>>>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor >>>>> versions >>>>>>> for >>>>>>>>>>>>> previous >>>>>>>>>>>>>> major versions. >>>>>>>>>>>>> >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how >> viable >>>> that >>>>>> is >>>>>>> in >>>>>>>>>>>>> practice. >>>>>>>>>>>>> >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping >> support >>>> for >>>>>> Java >>>>>>>>> 8/11, >>>>>>>>>>>>> which may very well be a problem for our current users. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion >> about >>>>>> Public >>>>>>>>> API >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake >> in >>>> the >>>>>>> NOTES >>>>>>>>> part >>>>>>>>>>>>> for >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed >> it. >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we >> are on >>>>> the >>>>>>> same >>>>>>>>>>>>> page on that note. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote: >>>>>>>>>>>>>> But it should be >>>>>>>>>>>>>> completely OK to bump up the major version if we >> really >>>> want >>>>>> to >>>>>>> get >>>>>>>>>>> rid >>>>>>>>>>>>> of >>>>>>>>>>>>>> a public API, right? >>>>>>>>>>>>> >>>>>>>>>>>>> Technically yes, but look at how long it took to get us >> to >>>>> 2.0. >>>>>> ;) >>>>>>>>>>>>> >>>>>>>>>>>>> There's a separate discussion to be had on the cadence >> of >>>>> major >>>>>>>>> releases >>>>>>>>>>>>> going forward, and there seem to be different opinions >> on >>>>> that. >>>>>>>>>>>>> >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between >>>> major >>>>>>> ones, >>>>>>>>>>>>> that for us means that users have to potentially deal >> with >>>>>>> breaking >>>>>>>>>>>>> changes every 6 months, which seems like a lot. >>>>>>>>>>>>> >>>>>>>>>>>>> Given our track record I would prefer a regular cycle >> (1-2 >>>>>> years) >>>>>>> to >>>>>>>>>>>>> force us to think about this whole topic, and not put it >>>> again >>>>>> to >>>>>>> the >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on >>>> when >>>>>>>>> breaking >>>>>>>>>>>>> changes can be made. >>>>>>>>>>>>> >>>>>>>>>>>>> But again, maybe this should be in a separate thread. >>>>>>>>>>>>> >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote: >>>>>>>>>>>>>> Do you have an example of behavioral change in mind? >> Not >>>>> sure >>>>>> I >>>>>>>>> fully >>>>>>>>>>>>>> understand the concern for behavioral change here. >>>>>>>>>>>>> >>>>>>>>>>>>> This could be a lot of things. It can be performance in >>>>> certain >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly) >> relied >>>>> upon >>>>>>>>>>>>> (https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I), >>>>>>>>>>>>> a semantic change to some >> API. >>>>>>>>>>>>> >>>>>>>>>>>>> For a concrete example, consider the job submission. A >> few >>>>>>> releases >>>>>>>>> back >>>>>>>>>>>>> we made changes such that the initialization of the job >>>> master >>>>>>>>> happens >>>>>>>>>>>>> asynchronously. >>>>>>>>>>>>> This meant the job submission call returns sooner, and >> the >>>> job >>>>>>> state >>>>>>>>>>>>> enum was extended to cover this state. >>>>>>>>>>>>> API-wise we consider this a compatible change, but the >>>>> observed >>>>>>>>> behavior >>>>>>>>>>>>> may be different. >>>>>>>>>>>>> >>>>>>>>>>>>> Metrics are another example; I believe over time we >> changed >>>>> what >>>>>>> some >>>>>>>>>>>>> metrics returned a few times.
