Hi Becket and Xintong, I hope you don't mind if I chime in a little.
Once an API is marked Public, we're committing to support it until it's deprecated, and once it's deprecated, to leave it in place for at least two minor releases and then only remove it in the following major release. As a user, I would be dismayed to discover that the project maintainers demoted a Public API to PublicEvolving or Experimental in order to violate the spirit of the deprecation period for Public APIs. Likewise, I'd view a rapid sequence of minor releases for the purpose of dropping deprecated APIs in the next major release as another violation of the spirit of our guarantees. I'm reminded of this xkcd: https://xkcd.com/1494/ I'm glad we're gaming out these situations before we institute this FLIP, and I hope we all converge on a clear understanding of what we're guaranteeing. >From where I'm sitting, the essence of this FLIP is specifically to prevent us from doing whatever we want and instead to commit to providing a suitable notice period to users who rely on Public APIs. It honestly shouldn't matter what the maintenance overhead is (or, rather, evaluating the maintenance overhead should be a consideration before we promote an API to Public to begin with). My experience in other projects is that regretting a feature so much you want to drop it asap is extremely rare, and that living with the pain for a little while until the deprecation period expires will probably make us better engineers. With that all said, one thing that is not a violation is to propose a new major release in order to drop especially burdensome deprecated APIs that have already enjoyed their full deprecation period. Those really bad "case A" features will be very rare. Looking over the FLIP again, I see a reference to the release documentation, but not to the build tooling support for the Deprecated annotation, so it might help to share this scenario: The way this will all come together in practice is that, by giving good stability guarantees, we will encourage our users to bump minor versions when they are released, which means that they will see compiler warnings immediately if any of the APIs they've used are deprecated. In fact, I'd recommend for them to use the `-Werror` javac flag to be sure that they don't miss those warnings. Given our release cadence, this means that they will have about six months to migrate off of any deprecated APIs without being forced to avoid updates. And they really need to be able to adopt further minor updates in the series, since they may contain further improvements that actually facilitate migrating off of the deprecated APIs. The compiler warnings are a far more reliable mechanism to advertise deprecation than release notes, since users' own builds will notify them right away about only the deprecations that are relevant to them, but it does rely on users feeling ok to bump up minor versions in a timely fashion. If we consistently burn them by not observing the stability guarantees, they'll try to avoid updates instead, and the whole thing falls apart. Thanks again, -John On 2023/06/19 10:43:05 Becket Qin wrote: > Hi Xintong, > > Let's compare the following cases: > > A. If the maintenance overhead of the deprecated API is high, and we want > to remove it after two minor releases. Then there are two options: > A1: Demote the API in the major version bump and remove the code with a > minor version bump. > A2: Do exactly the same as 1-A, except that when removing the code, > bump up the major version. So this might be a short major version. > > B. If the maintenance overhead of the deprecated API is not high, therefore > keeping it for a long time is affordable. There are also two options: > B1: Same as A-1, demote the API in the major version bump and remove > the code with a minor version bump. > B2: Keep the API for all the minor versions in the major version, and > only remove the code in the next major version. > > For case B, do we agree that B2 is the way to go? > > For case A: > The following stays the same for A1 and A2: > - users will lose the API after two minor leases. So the migration period > is the same for A-1 and A-2. > - A1 and A2 will have exactly the same code after the removal of > deprecated API, the only difference is versioning. > - To move forward, users need to move to the next minor version in A1, or > to the next major version in A2. Because the code are the same, the actual > effort to upgrade is the same for A1 and A2. > > The differences between A-1 and A-2 are: > - A1 allows keeping the major version release cadence. A-2 will have a > short major version release. > - A1 breaks the well understood API semantic, while A-2 does not. > > From what I see, since there is no well establish standard regarding how > long a major version should be released, A short major version release is > potential and emotional. It is not ideal but does not have material > downsides compared with A1. I did not hear anyone complaining about Kafka > only has two 1.x release. However, A1 actually breaks the well understood > API semantic, which has more material impact. > > Also, I'd imagine 90% or more of the Public APIs should fall into case B. > So, short major versions should be very occasional. I'd be very concerned > if the reason we choose A1 is simply because we cannot afford maintaining a > bunch of deprecated APIs until the next major version. This indicates that > the actual problem we need to solve is to lower the maintenance overhead of > deprecated APIs, so that we are comfortable to keep them longer. As John > and I mentioned earlier, there are ways to achieve this and we need to > learn how to do it in Flink. Otherwise, our discussion about versioning > here does not bring much value, because we will end up with a bunch of > short-lived APIs which upset our users, no matter how we version the > releases. > > So, if there are concrete examples that you think will block us from > keeping API stability with affordable cost, let's take a look together and > see if that can be improved. > > Thanks, > > Jiangjie (Becket) Qin > > > From what I see, > > > On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <tonysong...@gmail.com> wrote: > > > > > > > The part I don't understand is if we are willing to have a migration > > > period, and do a minor version bump to remove an API, what do we lose to > > do > > > a major version bump instead, so we don't break the common versioning > > > semantic? > > > > > > > I think we are talking about the cases where a major version bump happens > > before a deprecated Public API reaches its migration period. So removing > > the API with another major version bump means we have two consecutive major > > versions in a very short time. And as previously mentioned, having frequent > > major version bumps would weaken the value of the commitment "Public API > > stay compatible within a major version". I think users also have > > expectations about how long a major version should live, which should be at > > least a couple of years rather than 1-2 minor releases. > > > > This is another option, but I think it is very likely more expensive than > > > simply bumping the major version. And I imagine there will be questions > > > like "why this feature is in 1.20, but does not exist in 2.0"? > > > > > > > Yes, it's more expensive than simply bumping the major version, but IMHO > > it's probably cheaper than carrying the API until the next major version if > > we don't bump the major version very soon. And see my reply above on why > > not bumping immediately. > > > > Best, > > > > Xintong > > > > > > > > On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket....@gmail.com> wrote: > > > > > Hi Xintong, > > > > > > Please see the replies below. > > > > > > I see your point, that users would feel surprised if they find things no > > > > longer work when upgrading to another 2.x minor release. However, I'd > > > like > > > > to point out that PublicEvolving APIs would have the similar problem > > > > anyway. So the question is, how do we catch users' attention and make > > > sure > > > > they are aware that the Public APIs in 1.x may no longer be Public in > > > 2.0. > > > > There're various ways to do that, e.g., release notes, warnings in > > logs, > > > > etc. > > > > > > First of all, I am not a fan of removing PublicEvolving APIs in minor > > > version changes. Personally speaking, I want them to be also removed in > > > major version changes. > > > > > > Empirically, I rarely see projects with complex API semantic work well. > > > Many, if not most, users don't read docs / release notes / warnings in > > > logs. I don't think that is their fault. Typically an application > > developer > > > will have to deal with dozens of libraries maintained by various > > > communities / groups. It is just too difficult for them to keep track of > > > all the specific semantics each project puts there. If you think about > > it, > > > how many of Flink developers read all the release notes of Guava, Apache > > > Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of them? > > > One would probably try bumping the dependency version and see if there is > > > an exception and solve them case by case. And if it is a minor version > > > bump, one probably does not expect an exception at all. Another example > > is > > > that in Flink we still have so many usage of deprecated methods all over > > > the place, and I strongly doubt everyone knows when the source code of > > > these methods are deprecated and when they should be removed. > > > > > > So, the most important principle of API is simple and intuitive. The > > > versioning semantic is a simple and universally accepted API stability > > > standard. If we ourselves as Flink developers are relying on this for our > > > own dependencies. I don't think we can expect more from our users. > > > > > > Another possible alternative: whenever there's a deprecated Public API > > that > > > > reaches a major version bump before the migration period, and we also > > > don't > > > > want to carry it for all the next major release series, we may consider > > > > releasing more minor releases for the previous major version after the > > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump to > > 2.0, > > > > we can release one more 1.20 after 2.0. That should provide users > > another > > > > choice rather than upgrading to 2.0, while satisfying the > > 2-minor-release > > > > migration period. > > > > > > This is another option, but I think it is very likely more expensive than > > > simply bumping the major version. And I imagine there will be questions > > > like "why this feature is in 1.20, but does not exist in 2.0"? > > > > > > I think my major point is, we should not carry APIs deprecated in a > > > > previous major version along all the next major version series. I'd > > like > > > to > > > > try giving users more commitments, i.e. the migration period, as long > > as > > > it > > > > does not prevent us from making breaking changes. If it doesn't work, > > I'd > > > > be in favor of not providing the migration period, but fallback to only > > > > guarantee the compatibility within the major version. > > > > > > The part I don't understand is if we are willing to have a migration > > > period, and do a minor version bump to remove an API, what do we lose to > > do > > > a major version bump instead, so we don't break the common versioning > > > semantic? > > > > > > Thanks, > > > > > > Jiangjie (Becket) Qin > > > > > > > > > On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > > > > > > > As an end user who only uses Public APIs, if I don't change my code > > at > > > > > all, my expectation is the following: > > > > > 1. Upgrading from 1.x to 2.x may have issues. > > > > > 2. If I can upgrade from 1.x to 2.x without an issue, I am fine with > > > all > > > > > the 2.x versions. > > > > > Actually I think there are some dependency version resolution > > policies > > > > out > > > > > there which picks the highest minor version when the dependencies > > pull > > > in > > > > > multiple minor versions of the same jar, which may be broken if we > > > remove > > > > > the API in minor releases. > > > > > > > > > > > > > I see your point, that users would feel surprised if they find things > > no > > > > longer work when upgrading to another 2.x minor release. However, I'd > > > like > > > > to point out that PublicEvolving APIs would have the similar problem > > > > anyway. So the question is, how do we catch users' attention and make > > > sure > > > > they are aware that the Public APIs in 1.x may no longer be Public in > > > 2.0. > > > > There're various ways to do that, e.g., release notes, warnings in > > logs, > > > > etc. > > > > > > > > Another possible alternative: whenever there's a deprecated Public API > > > that > > > > reaches a major version bump before the migration period, and we also > > > don't > > > > want to carry it for all the next major release series, we may consider > > > > releasing more minor releases for the previous major version after the > > > > bump. E.g., an Public API is deprecated in 1.19, and then we bump to > > 2.0, > > > > we can release one more 1.20 after 2.0. That should provide users > > another > > > > choice rather than upgrading to 2.0, while satisfying the > > 2-minor-release > > > > migration period. > > > > > > > > I think my major point is, we should not carry APIs deprecated in a > > > > previous major version along all the next major version series. I'd > > like > > > to > > > > try giving users more commitments, i.e. the migration period, as long > > as > > > it > > > > does not prevent us from making breaking changes. If it doesn't work, > > I'd > > > > be in favor of not providing the migration period, but fallback to only > > > > guarantee the compatibility within the major version. > > > > > > > > Best, > > > > > > > > Xintong > > > > > > > > > > > > > > > > On Mon, Jun 19, 2023 at 10:48 AM John Roesler <vvcep...@apache.org> > > > wrote: > > > > > > > > > Hi Becket, > > > > > > > > > > Thanks for the reply! I’d like to continue the conversation about > > > > > compatibility outside of this FLIP thread, but for now, I can accept > > > your > > > > > decision. It’s certainly an improvement. > > > > > > > > > > Thanks again, > > > > > John > > > > > > > > > > On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote: > > > > > > Hi John, > > > > > > > > > > > > Completely agree with all you said. > > > > > > > > > > > > Can we consider only dropping deprecated APIs in major releases > > > across > > > > > the > > > > > >> board? I understand that Experimental and PublicEvolving APIs are > > by > > > > > >> definition less stable, but it seems like this should be reflected > > > in > > > > > the > > > > > >> required deprecation period alone. I.e. that we must keep them > > > around > > > > > for > > > > > >> at least zero or one minor release, not that we can drop them in a > > > > > minor or > > > > > >> patch release. > > > > > > > > > > > > Personally speaking, I would love to do this, for exactly the > > reason > > > > you > > > > > > mentioned. However, I did not propose this due to the following > > > > reasons: > > > > > > > > > > > > 1. I am hesitating a little bit about changing the accepted FLIPs > > too > > > > > soon. > > > > > > 2. More importantly, to avoid slowing down our development. At this > > > > > point, > > > > > > Flink still lacks some design / routines to support good API > > > > > evolvability / > > > > > > extensibility. Just like you said, it takes some time to be good at > > > > this. > > > > > > In this case, my concern is that only removing Experimental / > > > > > > PublicEvolving APIs in major version changes may result in too much > > > > > > overhead and dramatically slow down the development of Flink. So, I > > > was > > > > > > thinking that we can start with the current status. Hopefully after > > > we > > > > > are > > > > > > more comfortable with the maintenance overhead of deprecated APIs, > > we > > > > can > > > > > > then have a stronger guarantee for Experimental / PublicEvolving > > > APIs. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jun 18, 2023 at 6:44 AM John Roesler <vvcep...@apache.org> > > > > > wrote: > > > > > > > > > > > >> Hi Becket, > > > > > >> > > > > > >> Thanks for this FLIP! Having a deprecation process is really > > > > important. > > > > > I > > > > > >> understand some people’s concerns about the additional burden for > > > > > project > > > > > >> maintainers, but my personal experience with Kafka has been that > > > it’s > > > > > very > > > > > >> liveable and that it’s well worth the benefit to users. In fact, > > > users > > > > > >> being able to confidently upgrade is also a benefit to > > maintainers, > > > as > > > > > we > > > > > >> will get fewer questions from people stuck on very old versions. > > > > > >> > > > > > >> One question: > > > > > >> Can we consider only dropping deprecated APIs in major releases > > > across > > > > > the > > > > > >> board? I understand that Experimental and PublicEvolving APIs are > > by > > > > > >> definition less stable, but it seems like this should be reflected > > > in > > > > > the > > > > > >> required deprecation period alone. I.e. that we must keep them > > > around > > > > > for > > > > > >> at least zero or one minor release, not that we can drop them in a > > > > > minor or > > > > > >> patch release. > > > > > >> > > > > > >> The advantage of forbidding the removal of any API in minor or > > patch > > > > > >> releases is that users will get a strong guarantee that they can > > > bump > > > > > the > > > > > >> minor or patch version and still be able to compile, or even just > > > > > re-link > > > > > >> and know that they won’t face “MethodDef” exceptions at run time. > > > This > > > > > is a > > > > > >> binary guarantee: if we allow removing even Experimental APIs > > > outside > > > > > of > > > > > >> major releases, users can no longer confidently upgrade. > > > > > >> > > > > > >> Aside from that, I’d share my 2 cents on a couple of points: > > > > > >> * I’d use the official Deprecated annotation instead of > > introducing > > > > our > > > > > >> own flavor (Retired, etc), since Deprecated is well integrated > > into > > > > > build > > > > > >> tools and IDEs. > > > > > >> * I wouldn’t worry about a demotion process in this FLIP; it seems > > > > > >> orthogonal, and something that should probably be taken > > case-by-case > > > > > >> anyway. > > > > > >> * Aside from deprecation and removal, there have been some > > > discussions > > > > > >> about how to evolve APIs and behavior in compatible ways. This is > > > > > somewhat > > > > > >> of an art, and if folks haven’t wrestled with it before, it’ll > > take > > > > some > > > > > >> time to become good at it. I feel like this topic should also be > > > > > orthogonal > > > > > >> to this FLIP, but FWIW, my suggestion would be to adopt a simple > > > > policy > > > > > not > > > > > >> to break existing user programs, and leave the “how” up to > > > > implementers > > > > > and > > > > > >> reviewers. > > > > > >> > > > > > >> Thanks again, > > > > > >> John > > > > > >> > > > > > >> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > > > > > >> > Hi All, > > > > > >> > > > > > > >> > The @Public -> @PublicEvolving proposed by Xintong is a great > > > idea. > > > > > >> > Especially, after he suggest @PublicRetired, i.e. > > @PublicEvolving > > > > --(2 > > > > > >> > minor release)--> @Public --> @deprecated --(1 major > > > > > >> > release)--> @PublicRetired. It will provide a lot of flexibility > > > > > without > > > > > >> > breaking any rules we had. @Public APIs are allowed to change > > > > between > > > > > >> major > > > > > >> > releases. Changing annotations is acceptable and provides > > > additional > > > > > >> > tolerance i.e. user-friendliness, since the APIs themself are > > not > > > > > >> changed. > > > > > >> > > > > > > >> > I had similar thoughts when I was facing those issues. I want to > > > > move > > > > > one > > > > > >> > step further and suggest introducing one more annotation > > @Retired. > > > > > >> > > > > > > >> > Not like the @PublicRetired which is a compromise of downgrading > > > > > @Public > > > > > >> to > > > > > >> > @PublicEvolving. As I mentioned earlier in my reply, Java > > standard > > > > > >> > @deprecated should be used in the early stage of the deprecation > > > > > process > > > > > >> > and doesn't really meet our requirement. Since Java does not > > allow > > > > us > > > > > to > > > > > >> > extend annotation, I think it would be feasible to have the new > > > > > @Retired > > > > > >> to > > > > > >> > help us monitor and manage the deprecation process, house > > > cleaning, > > > > > etc. > > > > > >> > > > > > > >> > Some ideas could be(open for discussion): > > > > > >> > > > > > > >> > @Retired: > > > > > >> > > > > > > >> > 1. There must be a replacement with functionality compatibility > > > > before > > > > > >> APIs > > > > > >> > can be marked as @Retired, i.e. DISCUSS and VOTE processes on > > the > > > ML > > > > > are > > > > > >> > mandatory (a FLIP is recommended). > > > > > >> > 2. APIs marked as @Retired will be removed after 1 minor release > > > > > sharply > > > > > >> > (using ArchUnit to force it, needs further check whether it is > > > > > possible). > > > > > >> > Devs who marked them as @Retired are responsible to remove them. > > > > > >> > 3. Both @Public -> @Retired and @PublicEvolving -> @Retired are > > > > > >> > recommended. @Experimental -> @Retired and @Internal -> @Retired > > > > could > > > > > >> also > > > > > >> > be used if it can increase user-friendliness or > > dev-friendliness, > > > > but > > > > > not > > > > > >> > mandatory. > > > > > >> > 4. Some variables will be defined in @Retired to support the > > > > > deprecation > > > > > >> > process management. Further extension is possible, since the > > > > > annotation > > > > > >> is > > > > > >> > built by us. > > > > > >> > > > > > > >> > > > > > > >> > Best regards, > > > > > >> > Jing > > > > > >> > > > > > > >> > On Fri, Jun 16, 2023 at 10:31 AM Becket Qin < > > becket....@gmail.com > > > > > > > > > >> wrote: > > > > > >> > > > > > > >> >> Hi Xintong, > > > > > >> >> > > > > > >> >> Thanks for the explanation. Please see the replies inline > > below. > > > > > >> >> > > > > > >> >> I agree. And from my understanding, demoting a Public API is > > > also a > > > > > >> kind of > > > > > >> >> > such change, just like removing one, which can only happen > > with > > > > > major > > > > > >> >> > version bumps. I'm not proposing to allow demoting Public > > APIs > > > > > >> anytime, > > > > > >> >> but > > > > > >> >> > only in the case major version bumps happen before reaching > > the > > > > > >> >> > 2-minor-release migration period. Actually, demoting would > > be a > > > > > weaker > > > > > >> >> > change compared to removing the API immediately upon major > > > > version > > > > > >> bumps, > > > > > >> >> > in order to keep the commitment about the 2-minor-release > > > > migration > > > > > >> >> period. > > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds > > > > > against > > > > > >> >> > conventions, we may introduce a new annotation if necessary, > > > > e.g., > > > > > >> >> > `@PublicRetiring`, to avoid confusions. > > > > > >> >> > > > > > >> >> As an end user who only uses Public APIs, if I don't change my > > > code > > > > > at > > > > > >> all, > > > > > >> >> my expectation is the following: > > > > > >> >> 1. Upgrading from 1.x to 2.x may have issues. > > > > > >> >> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine > > > > with > > > > > all > > > > > >> >> the 2.x versions. > > > > > >> >> Actually I think there are some dependency version resolution > > > > > policies > > > > > >> out > > > > > >> >> there which picks the highest minor version when the > > dependencies > > > > > pull > > > > > >> in > > > > > >> >> multiple minor versions of the same jar, which may be broken if > > > we > > > > > >> remove > > > > > >> >> the API in minor releases. > > > > > >> >> > > > > > >> >> I'm not sure about this. Yes, it's completely "legal" that we > > > bump > > > > up > > > > > >> the > > > > > >> >> > major version whenever a breaking change is needed. However, > > > this > > > > > also > > > > > >> >> > weakens the value of the commitment that public APIs will > > stay > > > > > stable > > > > > >> >> > within the major release series, as the series can end > > anytime. > > > > > IMHO, > > > > > >> >> short > > > > > >> >> > major release series are not something "make the end users > > > > happy", > > > > > but > > > > > >> >> > backdoors that allow us as the developers to make frequent > > > > breaking > > > > > >> >> > changes. On the contrary, with the demoting approach, we can > > > > still > > > > > >> have > > > > > >> >> > longer major release series, while only allowing Public APIs > > > > > >> deprecated > > > > > >> >> at > > > > > >> >> > the end of the previous major version to be removed in the > > next > > > > > major > > > > > >> >> > version. > > > > > >> >> > > > > > >> >> I totally agree that frequent major version bumps are not > > ideal, > > > > but > > > > > >> here > > > > > >> >> we are comparing it with a minor version bump which removes a > > > > Public > > > > > >> API. > > > > > >> >> So the context is that we have already decided to remove this > > > > Public > > > > > API > > > > > >> >> while keeping everything else backwards compatible. I think a > > > major > > > > > >> version > > > > > >> >> bump is a commonly understood signal here, compared with a > > minor > > > > > version > > > > > >> >> change. From end users' perspective, for those who are not > > > > impacted, > > > > > in > > > > > >> >> this case upgrading a major version is not necessarily more > > > > involved > > > > > >> than > > > > > >> >> upgrading a minor version - both should be as smooth as a > > > > dependency > > > > > >> >> version change. For those who are impacted, they will lose the > > > > Public > > > > > >> API > > > > > >> >> anyways and a major version bump ensures there is no surprise. > > > > > >> >> > > > > > >> >> Thanks, > > > > > >> >> > > > > > >> >> Jiangjie (Becket) Qin > > > > > >> >> > > > > > >> >> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song < > > > > tonysong...@gmail.com > > > > > > > > > > > >> >> wrote: > > > > > >> >> > > > > > >> >> > Public API is a well defined common concept, and one of its > > > > > >> >> >> convention is that it only changes with a major version > > > change. > > > > > >> >> >> > > > > > >> >> > > > > > > >> >> > I agree. And from my understanding, demoting a Public API is > > > > also a > > > > > >> kind > > > > > >> >> > of such change, just like removing one, which can only happen > > > > with > > > > > >> major > > > > > >> >> > version bumps. I'm not proposing to allow demoting Public > > APIs > > > > > >> anytime, > > > > > >> >> but > > > > > >> >> > only in the case major version bumps happen before reaching > > the > > > > > >> >> > 2-minor-release migration period. Actually, demoting would > > be a > > > > > weaker > > > > > >> >> > change compared to removing the API immediately upon major > > > > version > > > > > >> bumps, > > > > > >> >> > in order to keep the commitment about the 2-minor-release > > > > migration > > > > > >> >> period. > > > > > >> >> > If the concern is that `@Public` -> `@PublicEvolving` sounds > > > > > against > > > > > >> >> > conventions, we may introduce a new annotation if necessary, > > > > e.g., > > > > > >> >> > `@PublicRetiring`, to avoid confusions. > > > > > >> >> > > > > > > >> >> > But it should be > > > > > >> >> >> completely OK to bump up the major version if we really want > > > to > > > > > get > > > > > >> rid > > > > > >> >> of > > > > > >> >> >> a public API, right? > > > > > >> >> >> > > > > > >> >> > > > > > > >> >> > I'm not sure about this. Yes, it's completely "legal" that we > > > > bump > > > > > up > > > > > >> the > > > > > >> >> > major version whenever a breaking change is needed. However, > > > this > > > > > also > > > > > >> >> > weakens the value of the commitment that public APIs will > > stay > > > > > stable > > > > > >> >> > within the major release series, as the series can end > > anytime. > > > > > IMHO, > > > > > >> >> short > > > > > >> >> > major release series are not something "make the end users > > > > happy", > > > > > but > > > > > >> >> > backdoors that allow us as the developers to make frequent > > > > breaking > > > > > >> >> > changes. On the contrary, with the demoting approach, we can > > > > still > > > > > >> have > > > > > >> >> > longer major release series, while only allowing Public APIs > > > > > >> deprecated > > > > > >> >> at > > > > > >> >> > the end of the previous major version to be removed in the > > next > > > > > major > > > > > >> >> > version. > > > > > >> >> > > > > > > >> >> > Given our track record I would prefer a regular cycle (1-2 > > > years) > > > > > to > > > > > >> >> >> force us to think about this whole topic, and not put it > > again > > > > to > > > > > the > > > > > >> >> >> wayside and giving us (and users) a clear expectation on > > when > > > > > >> breaking > > > > > >> >> >> changes can be made. > > > > > >> >> >> > > > > > >> >> > > > > > > >> >> > +1. I personally think 2-3 years would be a good time for new > > > > major > > > > > >> >> > versions, or longer if there's no breaking changes needed. > > That > > > > > makes > > > > > >> 1-2 > > > > > >> >> > year a perfect time to revisit the topic, while leaving us > > more > > > > > time > > > > > >> to > > > > > >> >> > prepare the major release if needed. > > > > > >> >> > > > > > > >> >> > Best, > > > > > >> >> > > > > > > >> >> > Xintong > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler < > > > > > ches...@apache.org > > > > > >> > > > > > > >> >> > wrote: > > > > > >> >> > > > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote: > > > > > >> >> >> > It would be valuable if we can avoid releasing minor > > > versions > > > > > for > > > > > >> >> >> previous > > > > > >> >> >> > major versions. > > > > > >> >> >> > > > > > >> >> >> On paper, /absolutely /agree, but I'm not sure how viable > > that > > > > is > > > > > in > > > > > >> >> >> practice. > > > > > >> >> >> > > > > > >> >> >> On the current 2.0 agenda is potentially dropping support > > for > > > > Java > > > > > >> 8/11, > > > > > >> >> >> which may very well be a problem for our current users. > > > > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> On 13/06/2023 17:26, Becket Qin wrote: > > > > > >> >> >> > Thanks for the feedback and sorry for the confusion about > > > > Public > > > > > >> API > > > > > >> >> >> > deprecation. I just noticed that there was a mistake in > > the > > > > > NOTES > > > > > >> part > > > > > >> >> >> for > > > > > >> >> >> > Public API due to a copy-paste error... I just fixed it. > > > > > >> >> >> I'm very relieved to hear that. Glad to hear that we are on > > > the > > > > > same > > > > > >> >> >> page on that note. > > > > > >> >> >> > > > > > >> >> >> > > > > > >> >> >> On 15/06/2023 15:20, Becket Qin wrote: > > > > > >> >> >> > But it should be > > > > > >> >> >> > completely OK to bump up the major version if we really > > want > > > > to > > > > > get > > > > > >> >> rid > > > > > >> >> >> of > > > > > >> >> >> > a public API, right? > > > > > >> >> >> > > > > > >> >> >> Technically yes, but look at how long it took to get us to > > > 2.0. > > > > ;) > > > > > >> >> >> > > > > > >> >> >> There's a separate discussion to be had on the cadence of > > > major > > > > > >> releases > > > > > >> >> >> going forward, and there seem to be different opinions on > > > that. > > > > > >> >> >> > > > > > >> >> >> If we take the Kafka example of 2 minor releases between > > major > > > > > ones, > > > > > >> >> >> that for us means that users have to potentially deal with > > > > > breaking > > > > > >> >> >> changes every 6 months, which seems like a lot. > > > > > >> >> >> > > > > > >> >> >> Given our track record I would prefer a regular cycle (1-2 > > > > years) > > > > > to > > > > > >> >> >> force us to think about this whole topic, and not put it > > again > > > > to > > > > > the > > > > > >> >> >> wayside and giving us (and users) a clear expectation on > > when > > > > > >> breaking > > > > > >> >> >> changes can be made. > > > > > >> >> >> > > > > > >> >> >> But again, maybe this should be in a separate thread. > > > > > >> >> >> > > > > > >> >> >> On 14/06/2023 11:37, Becket Qin wrote: > > > > > >> >> >> > Do you have an example of behavioral change in mind? Not > > > sure > > > > I > > > > > >> fully > > > > > >> >> >> > understand the concern for behavioral change here. > > > > > >> >> >> > > > > > >> >> >> This could be a lot of things. It can be performance in > > > certain > > > > > >> >> >> edge-cases, a bug fix that users (maybe unknowingly) relied > > > upon > > > > > >> >> >> (https://xkcd.com/1172/), a semantic change to some API. > > > > > >> >> >> > > > > > >> >> >> For a concrete example, consider the job submission. A few > > > > > releases > > > > > >> back > > > > > >> >> >> we made changes such that the initialization of the job > > master > > > > > >> happens > > > > > >> >> >> asynchronously. > > > > > >> >> >> This meant the job submission call returns sooner, and the > > job > > > > > state > > > > > >> >> >> enum was extended to cover this state. > > > > > >> >> >> API-wise we consider this a compatible change, but the > > > observed > > > > > >> behavior > > > > > >> >> >> may be different. > > > > > >> >> >> > > > > > >> >> >> Metrics are another example; I believe over time we changed > > > what > > > > > some > > > > > >> >> >> metrics returned a few times. > > > > > >> >> >> > > > > > >> >> > > > > > > >> >> > > > > > >> > > > > > > > > > > > > > > >
