Thanks much for the input, John, Stefan and Jing. I think Xingtong has well summarized the pros and cons of the two options. Let's collect a few more opinions here and we can move forward with the one more people prefer.
Thanks, Jiangjie (Becket) Qin On Wed, Jun 21, 2023 at 3:20 AM Jing Ge <j...@ververica.com.invalid> wrote: > Hi all, > > Thanks Xingtong for the summary. If I could only choose one of the given > two options, I would go with option 1. I understood that option 2 worked > great with Kafka. But the bridge release will still confuse users and my > gut feeling is that many users will skip 2.0 and be waiting for 3.0 or even > 3.x. And since fewer users will use Flink 2.x, the development focus will > be on Flink 3.0 with the fact that the current Flink release is 1.17 and we > are preparing 2.0 release. That is weird for me. > > THB, I would not name the change from @Public to @Retired as a demotion. > The purpose of @Retire is to extend the API lifecycle with one more stage, > like in the real world, people born, studied, graduated, worked, and > retired. Afaiu from the previous discussion, there are two rules we'd like > to follow simultaneously: > > 1. Public APIs can only be changed between major releases. > 2. A smooth migration phase should be offered to users, i.e. at least 2 > minor releases after APIs are marked as @deprecated. There should be new > APIs as the replacement. > > Agree, those rules are good to improve the user friendliness. Issues we > discussed are rising because we want to fulfill both of them. If we take > care of deprecation very seriously, APIs can be marked as @Deprecated, only > when the new APIs as the replacement provide all functionalities the > deprecated APIs have. In an ideal case without critical bugs that might > stop users adopting the new APIs. Otherwise the expected "replacement" will > not happen. Users will still stick to the deprecated APIs, because the new > APIs can not be used. For big features, it will need at least 4 minor > releases(ideal case), i.e. 2+ years to remove deprecated APIs: > > - 1st minor release to build the new APIs as the replacement and waiting > for feedback. It might be difficult to mark the old API as deprecated in > this release, because we are not sure if the new APIs could cover 100% > functionalities. > - In the lucky case, mark all old APIs as deprecated in the 2nd minor > release. (I would even suggest having the new APIs released at least for > two minor releases before marking it as deprecated to make sure they can > really replace the old APIs, in case we care more about smooth migration) > - 3rd minor release for the migration period > - In another lucky case, the 4th release is a major release, the > deprecated APIs could be removed. > > The above described scenario works only in an ideal case. In reality, it > might take longer to get the new APIs ready and mark the old API > deprecated. Furthermore, if the 4th release is not a major release, we will > have to maintain both APIs for many further minor releases. The question is > how to know the next major release in advance, especially 4 minor releases' > period, i.e. more than 2 years in advance? Given that Flink contains many > modules, it is difficult to ask devs to create a 2-3 years deprecation plan > for each case. In case we want to build major releases at a fast pace, > let's say every two years, it means devs must plan any API deprecation > right after each major release. Afaiac, it is quite difficult. > > The major issue is, afaiu, if we follow rule 2, we have to keep all @Public > APIs, e.g. DataStream, that are not marked as deprecated yet, to 2.0. Then > we have to follow rule 1 to keep it unchanged until we have 3.0. That is > why @Retired is useful to give devs more flexibility and still fulfill both > rules. Let's check it with examples: > > - we have @Public DataStream API in 1.18. It will not be marked > as @Deprecated, because the new APIs as the replacement are not ready. > - we keep the DataStream API itself unchanged in 2.0, but change the > annotation from @Public to @Retire. New APIs will be introduced too. In > this case, Rule 1 is ok, since public API is allowed to change between > major releases. Only changing annotation is the minimal change we could do > and it does not break rule 1. Rule 2 is ok too, since the DataStream APIs > work exactly the same. Attention: the change of @Public -> @Retired can > only be done between major releases, because @Public APIs can only be > changed between major releases. > - in 2.1, DataStream API will be marked as deprecated. > - in 2.2, DataStream will be kept for the migration period. > - in 2.3, DataStream will be removed. > > Becket mentioned previously (please correct me if I didn't understand it > correctly) that users might not check the changes of annotation and the > upgrade process might not be smooth. I was wondering, if users don't pay > attention to that, they will also not pay attention to @deprecated. The > migration will not be smooth too even when we stick to the Java > standard @deprecated. Let's go throw it with examples: > > Scenario 1: with @Retired > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing should be done > for users who still want to use DataStream. The migration is smooth. But > they should be aware of the annotation changes, i.e. @Public -> @Retired > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed. Since it is > a major release upgrade, breaking changes are expected. > Case 3: users upgrade from 2.0 to 2.3 - DataStream is removed. Since APIs > are marked as @Retired, breaking changes are expected too. > > In addition to fulfilling both rules, there are two more benefits of this > option: > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1. > 2. no bridge release is required. DataStream API could be deprecated in 2.x > without breaking any of the two rules. @Retired provides enough flexibility > for future deprecations and upgrades. > > The only issue I can find is that users might be surprised in case 3, if > they ignore the previous @Public -> @Retired change. But for such users, > they will ignore @deprecated annotation too and will have issues with > pure @deprecated > annotation too: > > Scenario 2: with @deprecated > > The example will be slightly different from scenario 1, because @Public > APIs can only be removed between major releases. As an example, DataStream > API will be marked as deprecated in 1.19, kept in 1.20, and then removed in > 2.0 > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not marked as > deprecated are removed between major releases. Rule 1 is ok. The 2 minor > releases period defined in rule 2 sounds like a smooth migration plan, but > for users in this case, it is still not smooth. > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade process with > deprecated APIs. But for those users who ignore annotation changes, will > have the same issue mentioned in case 1. > > I understood that, compared to scenario 1, users, who ignore @Public > -> @Retired > changes, will not face any breaking changes, after they migrated to 2.0. > But the cost of it is bridge releases. There might be many bridge releases > in the future. As I mentioned previously, it is not easy to find the right > timing to kick off the deprecation process of APIs before the next expected > major release and 2-3 years in advance. > > Scenario 3: @PublicEvolving deprecation for users who ignore @deprecated > annotation changes > > Sample API marked as deprecated in 1.19, kept in 1.20, removed in 1.21 > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no smooth > migration. > Case 2: users upgrade from 1.20 to 1.21 - same breaking change, > since @deprecated is ignored > > In summary, for users who care about the annotation change, they will get > more benefits with @Retired. For users who ignore annotation changes, they > will be always facing non-smooth upgrades even using pure standard > @deprecated > annotation/process. > > For the case that IDEs support @deprecated, that is true. There are no such > supports for homebuilt annotation. But the intention is to let users be > aware of it. There will be other ways to do it for users who care about > annotation changes. For users who do not care, there should be no > difference between with tool support or without. Using @Retired with > @deprecation > together is at least not worse than only using @deprecation alone. And > there are additional benefits with @Retired. > > Just my 2 cents and looking forward to your feedback, especially different > opinions that will help me understand the issue better. Thanks! > > > Best regards, > Jing > > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter > <srich...@confluent.io.invalid> wrote: > > > Hi Xintong, > > > > Thanks for the summary, most of the points that you agreed upon also make > > sense to me. > > > > > 2. Dropping deprecated Public APIs in minor releases, or demoting > APIs > > > from Public to PublicEvolving / Experimental / Retired in any version > > bump, > > > are not good practices. > > > > I hope we can can go beyond calling it “not a good practice" and reach > > consensus that we will not demote or remove public APIs in minor releases > > and (technically) enforce this rule for all contributions. I think it’s a > > reasonable expectation for stability from a project as mature as Flink. > > > > > I'm personally in favor of option 1. As the migration-period rule is > > newly > > > proposed, I think it's fair to make exceptions for cases where we > already > > > missed the best chance for planning the deprecation. Moreover, I do > > believe > > > having a quick major version bump does confuse users. Yes, we can > explain > > > to users that bumping from 2.x to 3.0 does not cost anything other than > > the > > > removal of an deprecated API. But having to explain this itself is an > > > indicator that it might be confusing for users. > > > > > > > > > Becket, on the other hand, prefers option 2. From my understanding, his > > > major point is that a quick major version bump causes barely any actual > > > lose on users, while in option 1 not providing the migration period or > > > providing a shorter on is an actual lose on users. (@Becket, please > > correct > > > me if I'm mistaken.) > > > > > > > I would be open to both options and maybe postpone this decision until we > > have the complete roadmap for Flink 2.0. We would have a better overview > > about the technical consequences, e.g. how hard it would be to offer and > > maintain both APIs side by side, how well we get both APIs separated, and > > whether or not we will be able to use the full potential of our breaking > > changes before the DataStream API is completely removed. One example for > > the last point I can think of would be moving from lazy to eager state > > declaration. If we believe that it is reasonable to support all planned > > changes while maintaining both APIs, I'm leaning towards option 2. > > > > Best, > > Stefan > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <tonysong...@gmail.com> wrote: > > > > > > Becket and I had an offline voice call earlier today. We have reached > > > consensus on some of the arguments, but still hold different opinions > on > > > some others. I'd like to post the outcome here for transparency. > > > > > > We both agree that: > > > > > > 1. Providing a migration period would be beneficial for our users, > and > > > we should do that > > > 2. Dropping deprecated Public APIs in minor releases, or demoting > APIs > > > from Public to PublicEvolving / Experimental / Retired in any version > > bump, > > > are not good practices. > > > 3. Ideally, with this FLIP, developers should be more careful and > plan > > > API changes ahead. That means： > > > 1. Be more careful with designing APIs and promoting them to > Public, > > > so that they won't be changed / removed very soon, and also the > > > maintenance overhead for keeping them after deprecation should be > > > affordable. (I believe this also aligns with John's opinion.) > > > 2. Plan the deprecation / removal of old APIs that we don't want > to > > > carry to the next major version earlier, so that they'll have the > > > 2-minor-release migration period before the next major version > bump. > > > 4. A practical situation is that, we are not ready for deprecating > the > > > DataStream API, nor afford to carry it for another couple of years. > > > 1. We cannot deprecate it now, because the new ProcessFunction API > > > (the planned replacement for DataStream API) is not yet ready. We > > haven't > > > planned the ProcessFunction API earlier because the original > > > plan was to do > > > in-place changes directly on the DataStream API in release 2.0, > > until the > > > smooth migration period is proposed. Before this, the only > > guarantees we > > > provide for Public APIs is that they'll stay compatible until the > > next > > > major version, which led to the understanding the in-place changes > > on > > > Public APIs should be fine. > > > 2. We also cannot afford carrying the DataStream API for another > > > couple of years. One of the reasons that we want to refactor / > > replace > > > DataStream API is that it exposes and depends on too many > > > Flink's internal > > > runtime implementations, which has already limited / blocked plans > > for > > > Flink's internal improvements for many times. Also due to the wide > > and > > > transitive dependencies on the internals, maintaining two sets of > > lower > > > APIs itself is expensive and would be better to keep it short. > This > > is > > > admittedly a historical design flaw which should be avoided in > > future. > > > > > > What we haven't reached consensus is about how to deal with the current > > > situation around the DataStream API. There are two options. > > > > > > > > > 1. We say this FLIP is enforced starting release 2.0. For current 1.x > > > APIs, we provide a migration period with best effort, while allowing > > > exceptions for immediate removal in 2.0. That means we will still try > > with > > > best effort to get the ProcessFuncion API ready and deprecate the > > > DataStream API in 1.x, but will also be allowed to remove DataStream > > API in > > > 2.0 if it's not deprecated 2 minor releases before the major version > > bump. > > > 2. We strictly follow the process in this FLIP, and will quickly bump > > > the major version from 2.x to 3.0 once the migration period for > > DataStream > > > API is reached. > > > > > > I'm personally in favor of option 1. As the migration-period rule is > > newly > > > proposed, I think it's fair to make exceptions for cases where we > already > > > missed the best chance for planning the deprecation. Moreover, I do > > believe > > > having a quick major version bump does confuse users. Yes, we can > explain > > > to users that bumping from 2.x to 3.0 does not cost anything other than > > the > > > removal of an deprecated API. But having to explain this itself is an > > > indicator that it might be confusing for users. > > > > > > > > > Becket, on the other hand, prefers option 2. From my understanding, his > > > major point is that a quick major version bump causes barely any actual > > > lose on users, while in option 1 not providing the migration period or > > > providing a shorter on is an actual lose on users. (@Becket, please > > correct > > > me if I'm mistaken.) > > > > > > > > > And we'd like to hear more feedback from the community. > > > > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <vvcep...@apache.org > > <mailto:vvcep...@apache.org>> wrote: > > > > > >> Hi Becket and Xintong, > > >> > > >> I hope you don't mind if I chime in a little. > > >> > > >> Once an API is marked Public, we're committing to support it until > it's > > >> deprecated, and once it's deprecated, to leave it in place for at > least > > two > > >> minor releases and then only remove it in the following major release. > > >> > > >> As a user, I would be dismayed to discover that the project > maintainers > > >> demoted a Public API to PublicEvolving or Experimental in order to > > violate > > >> the spirit of the deprecation period for Public APIs. Likewise, I'd > > view a > > >> rapid sequence of minor releases for the purpose of dropping > deprecated > > >> APIs in the next major release as another violation of the spirit of > our > > >> guarantees. I'm reminded of this xkcd: > > > https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_ > > >> > > >> I'm glad we're gaming out these situations before we institute this > > FLIP, > > >> and I hope we all converge on a clear understanding of what we're > > >> guaranteeing. From where I'm sitting, the essence of this FLIP is > > >> specifically to prevent us from doing whatever we want and instead to > > >> commit to providing a suitable notice period to users who rely on > Public > > >> APIs. > > >> > > >> It honestly shouldn't matter what the maintenance overhead is (or, > > rather, > > >> evaluating the maintenance overhead should be a consideration before > we > > >> promote an API to Public to begin with). My experience in other > > projects is > > >> that regretting a feature so much you want to drop it asap is > extremely > > >> rare, and that living with the pain for a little while until the > > >> deprecation period expires will probably make us better engineers. > > >> > > >> With that all said, one thing that is not a violation is to propose a > > new > > >> major release in order to drop especially burdensome deprecated APIs > > that > > >> have already enjoyed their full deprecation period. Those really bad > > "case > > >> A" features will be very rare. > > >> > > >> Looking over the FLIP again, I see a reference to the release > > >> documentation, but not to the build tooling support for the Deprecated > > >> annotation, so it might help to share this scenario: The way this will > > all > > >> come together in practice is that, by giving good stability > guarantees, > > we > > >> will encourage our users to bump minor versions when they are > released, > > >> which means that they will see compiler warnings immediately if any of > > the > > >> APIs they've used are deprecated. In fact, I'd recommend for them to > use > > >> the `-Werror` javac flag to be sure that they don't miss those > warnings. > > >> Given our release cadence, this means that they will have about six > > months > > >> to migrate off of any deprecated APIs without being forced to avoid > > >> updates. And they really need to be able to adopt further minor > updates > > in > > >> the series, since they may contain further improvements that actually > > >> facilitate migrating off of the deprecated APIs. > > >> > > >> The compiler warnings are a far more reliable mechanism to advertise > > >> deprecation than release notes, since users' own builds will notify > them > > >> right away about only the deprecations that are relevant to them, but > it > > >> does rely on users feeling ok to bump up minor versions in a timely > > >> fashion. If we consistently burn them by not observing the stability > > >> guarantees, they'll try to avoid updates instead, and the whole thing > > falls > > >> apart. > > >> > > >> Thanks again, > > >> -John > > >> > > >> On 2023/06/19 10:43:05 Becket Qin wrote: > > >>> Hi Xintong, > > >>> > > >>> Let's compare the following cases: > > >>> > > >>> A. If the maintenance overhead of the deprecated API is high, and we > > want > > >>> to remove it after two minor releases. Then there are two options: > > >>> A1: Demote the API in the major version bump and remove the code > > >> with a > > >>> minor version bump. > > >>> A2: Do exactly the same as 1-A, except that when removing the > code, > > >>> bump up the major version. So this might be a short major version. > > >>> > > >>> B. If the maintenance overhead of the deprecated API is not high, > > >> therefore > > >>> keeping it for a long time is affordable. There are also two options: > > >>> B1: Same as A-1, demote the API in the major version bump and > remove > > >>> the code with a minor version bump. > > >>> B2: Keep the API for all the minor versions in the major version, > > and > > >>> only remove the code in the next major version. > > >>> > > >>> For case B, do we agree that B2 is the way to go? > > >>> > > >>> For case A: > > >>> The following stays the same for A1 and A2: > > >>> - users will lose the API after two minor leases. So the migration > > >> period > > >>> is the same for A-1 and A-2. > > >>> - A1 and A2 will have exactly the same code after the removal of > > >>> deprecated API, the only difference is versioning. > > >>> - To move forward, users need to move to the next minor version in > A1, > > >> or > > >>> to the next major version in A2. Because the code are the same, the > > >> actual > > >>> effort to upgrade is the same for A1 and A2. > > >>> > > >>> The differences between A-1 and A-2 are: > > >>> - A1 allows keeping the major version release cadence. A-2 will > have a > > >>> short major version release. > > >>> - A1 breaks the well understood API semantic, while A-2 does not. > > >>> > > >>> From what I see, since there is no well establish standard regarding > > how > > >>> long a major version should be released, A short major version > release > > is > > >>> potential and emotional. It is not ideal but does not have material > > >>> downsides compared with A1. I did not hear anyone complaining about > > Kafka > > >>> only has two 1.x release. However, A1 actually breaks the well > > understood > > >>> API semantic, which has more material impact. > > >>> > > >>> Also, I'd imagine 90% or more of the Public APIs should fall into > case > > B. > > >>> So, short major versions should be very occasional. I'd be very > > concerned > > >>> if the reason we choose A1 is simply because we cannot afford > > >> maintaining a > > >>> bunch of deprecated APIs until the next major version. This indicates > > >> that > > >>> the actual problem we need to solve is to lower the maintenance > > overhead > > >> of > > >>> deprecated APIs, so that we are comfortable to keep them longer. As > > John > > >>> and I mentioned earlier, there are ways to achieve this and we need > to > > >>> learn how to do it in Flink. Otherwise, our discussion about > versioning > > >>> here does not bring much value, because we will end up with a bunch > of > > >>> short-lived APIs which upset our users, no matter how we version the > > >>> releases. > > >>> > > >>> So, if there are concrete examples that you think will block us from > > >>> keeping API stability with affordable cost, let's take a look > together > > >> and > > >>> see if that can be improved. > > >>> > > >>> Thanks, > > >>> > > >>> Jiangjie (Becket) Qin > > >>> > > >>> > > >>> From what I see, > > >>> > > >>> > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <tonysong...@gmail.com > > <mailto:tonysong...@gmail.com>> > > >> wrote: > > >>> > > >>>>> > > >>>>> The part I don't understand is if we are willing to have a > migration > > >>>>> period, and do a minor version bump to remove an API, what do we > > >> lose to > > >>>> do > > >>>>> a major version bump instead, so we don't break the common > versioning > > >>>>> semantic? > > >>>>> > > >>>> > > >>>> I think we are talking about the cases where a major version bump > > >> happens > > >>>> before a deprecated Public API reaches its migration period. So > > >> removing > > >>>> the API with another major version bump means we have two > consecutive > > >> major > > >>>> versions in a very short time. And as previously mentioned, having > > >> frequent > > >>>> major version bumps would weaken the value of the commitment "Public > > >> API > > >>>> stay compatible within a major version". I think users also have > > >>>> expectations about how long a major version should live, which > should > > >> be at > > >>>> least a couple of years rather than 1-2 minor releases. > > >>>> > > >>>> This is another option, but I think it is very likely more expensive > > >> than > > >>>>> simply bumping the major version. And I imagine there will be > > >> questions > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"? > > >>>>> > > >>>> > > >>>> Yes, it's more expensive than simply bumping the major version, but > > >> IMHO > > >>>> it's probably cheaper than carrying the API until the next major > > >> version if > > >>>> we don't bump the major version very soon. And see my reply above on > > >> why > > >>>> not bumping immediately. > > >>>> > > >>>> Best, > > >>>> > > >>>> Xintong > > >>>> > > >>>> > > >>>> > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <becket....@gmail.com > > <mailto:becket....@gmail.com>> > > >> wrote: > > >>>> > > >>>>> Hi Xintong, > > >>>>> > > >>>>> Please see the replies below. > > >>>>> > > >>>>> I see your point, that users would feel surprised if they find > > >> things no > > >>>>>> longer work when upgrading to another 2.x minor release. However, > > >> I'd > > >>>>> like > > >>>>>> to point out that PublicEvolving APIs would have the similar > > >> problem > > >>>>>> anyway. So the question is, how do we catch users' attention and > > >> make > > >>>>> sure > > >>>>>> they are aware that the Public APIs in 1.x may no longer be Public > > >> in > > >>>>> 2.0. > > >>>>>> There're various ways to do that, e.g., release notes, warnings in > > >>>> logs, > > >>>>>> etc. > > >>>>> > > >>>>> First of all, I am not a fan of removing PublicEvolving APIs in > minor > > >>>>> version changes. Personally speaking, I want them to be also > removed > > >> in > > >>>>> major version changes. > > >>>>> > > >>>>> Empirically, I rarely see projects with complex API semantic work > > >> well. > > >>>>> Many, if not most, users don't read docs / release notes / warnings > > >> in > > >>>>> logs. I don't think that is their fault. Typically an application > > >>>> developer > > >>>>> will have to deal with dozens of libraries maintained by various > > >>>>> communities / groups. It is just too difficult for them to keep > > >> track of > > >>>>> all the specific semantics each project puts there. If you think > > >> about > > >>>> it, > > >>>>> how many of Flink developers read all the release notes of Guava, > > >> Apache > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a version of > > >> them? > > >>>>> One would probably try bumping the dependency version and see if > > >> there is > > >>>>> an exception and solve them case by case. And if it is a minor > > >> version > > >>>>> bump, one probably does not expect an exception at all. Another > > >> example > > >>>> is > > >>>>> that in Flink we still have so many usage of deprecated methods all > > >> over > > >>>>> the place, and I strongly doubt everyone knows when the source code > > >> of > > >>>>> these methods are deprecated and when they should be removed. > > >>>>> > > >>>>> So, the most important principle of API is simple and intuitive. > The > > >>>>> versioning semantic is a simple and universally accepted API > > >> stability > > >>>>> standard. If we ourselves as Flink developers are relying on this > > >> for our > > >>>>> own dependencies. I don't think we can expect more from our users. > > >>>>> > > >>>>> Another possible alternative: whenever there's a deprecated Public > > >> API > > >>>> that > > >>>>>> reaches a major version bump before the migration period, and we > > >> also > > >>>>> don't > > >>>>>> want to carry it for all the next major release series, we may > > >> consider > > >>>>>> releasing more minor releases for the previous major version after > > >> the > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump > > >> to > > >>>> 2.0, > > >>>>>> we can release one more 1.20 after 2.0. That should provide users > > >>>> another > > >>>>>> choice rather than upgrading to 2.0, while satisfying the > > >>>> 2-minor-release > > >>>>>> migration period. > > >>>>> > > >>>>> This is another option, but I think it is very likely more > expensive > > >> than > > >>>>> simply bumping the major version. And I imagine there will be > > >> questions > > >>>>> like "why this feature is in 1.20, but does not exist in 2.0"? > > >>>>> > > >>>>> I think my major point is, we should not carry APIs deprecated in a > > >>>>>> previous major version along all the next major version series. > I'd > > >>>> like > > >>>>> to > > >>>>>> try giving users more commitments, i.e. the migration period, as > > >> long > > >>>> as > > >>>>> it > > >>>>>> does not prevent us from making breaking changes. If it doesn't > > >> work, > > >>>> I'd > > >>>>>> be in favor of not providing the migration period, but fallback to > > >> only > > >>>>>> guarantee the compatibility within the major version. > > >>>>> > > >>>>> The part I don't understand is if we are willing to have a > migration > > >>>>> period, and do a minor version bump to remove an API, what do we > > >> lose to > > >>>> do > > >>>>> a major version bump instead, so we don't break the common > versioning > > >>>>> semantic? > > >>>>> > > >>>>> Thanks, > > >>>>> > > >>>>> Jiangjie (Becket) Qin > > >>>>> > > >>>>> > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song < > tonysong...@gmail.com > > <mailto:tonysong...@gmail.com>> > > >>>>> wrote: > > >>>>> > > >>>>>>> > > >>>>>>> As an end user who only uses Public APIs, if I don't change my > > >> code > > >>>> at > > >>>>>>> all, my expectation is the following: > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am fine > > >> with > > >>>>> all > > >>>>>>> the 2.x versions. > > >>>>>>> Actually I think there are some dependency version resolution > > >>>> policies > > >>>>>> out > > >>>>>>> there which picks the highest minor version when the dependencies > > >>>> pull > > >>>>> in > > >>>>>>> multiple minor versions of the same jar, which may be broken if > > >> we > > >>>>> remove > > >>>>>>> the API in minor releases. > > >>>>>>> > > >>>>>> > > >>>>>> I see your point, that users would feel surprised if they find > > >> things > > >>>> no > > >>>>>> longer work when upgrading to another 2.x minor release. However, > > >> I'd > > >>>>> like > > >>>>>> to point out that PublicEvolving APIs would have the similar > > >> problem > > >>>>>> anyway. So the question is, how do we catch users' attention and > > >> make > > >>>>> sure > > >>>>>> they are aware that the Public APIs in 1.x may no longer be Public > > >> in > > >>>>> 2.0. > > >>>>>> There're various ways to do that, e.g., release notes, warnings in > > >>>> logs, > > >>>>>> etc. > > >>>>>> > > >>>>>> Another possible alternative: whenever there's a deprecated Public > > >> API > > >>>>> that > > >>>>>> reaches a major version bump before the migration period, and we > > >> also > > >>>>> don't > > >>>>>> want to carry it for all the next major release series, we may > > >> consider > > >>>>>> releasing more minor releases for the previous major version after > > >> the > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and then we bump > > >> to > > >>>> 2.0, > > >>>>>> we can release one more 1.20 after 2.0. That should provide users > > >>>> another > > >>>>>> choice rather than upgrading to 2.0, while satisfying the > > >>>> 2-minor-release > > >>>>>> migration period. > > >>>>>> > > >>>>>> I think my major point is, we should not carry APIs deprecated in > a > > >>>>>> previous major version along all the next major version series. > I'd > > >>>> like > > >>>>> to > > >>>>>> try giving users more commitments, i.e. the migration period, as > > >> long > > >>>> as > > >>>>> it > > >>>>>> does not prevent us from making breaking changes. If it doesn't > > >> work, > > >>>> I'd > > >>>>>> be in favor of not providing the migration period, but fallback to > > >> only > > >>>>>> guarantee the compatibility within the major version. > > >>>>>> > > >>>>>> Best, > > >>>>>> > > >>>>>> Xintong > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler < > vvcep...@apache.org > > <mailto:vvcep...@apache.org> > > >>> > > >>>>> wrote: > > >>>>>> > > >>>>>>> Hi Becket, > > >>>>>>> > > >>>>>>> Thanks for the reply! I’d like to continue the conversation about > > >>>>>>> compatibility outside of this FLIP thread, but for now, I can > > >> accept > > >>>>> your > > >>>>>>> decision. It’s certainly an improvement. > > >>>>>>> > > >>>>>>> Thanks again, > > >>>>>>> John > > >>>>>>> > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote: > > >>>>>>>> Hi John, > > >>>>>>>> > > >>>>>>>> Completely agree with all you said. > > >>>>>>>> > > >>>>>>>> Can we consider only dropping deprecated APIs in major releases > > >>>>> across > > >>>>>>> the > > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs > > >> are > > >>>> by > > >>>>>>>>> definition less stable, but it seems like this should be > > >> reflected > > >>>>> in > > >>>>>>> the > > >>>>>>>>> required deprecation period alone. I.e. that we must keep them > > >>>>> around > > >>>>>>> for > > >>>>>>>>> at least zero or one minor release, not that we can drop them > > >> in a > > >>>>>>> minor or > > >>>>>>>>> patch release. > > >>>>>>>> > > >>>>>>>> Personally speaking, I would love to do this, for exactly the > > >>>> reason > > >>>>>> you > > >>>>>>>> mentioned. However, I did not propose this due to the following > > >>>>>> reasons: > > >>>>>>>> > > >>>>>>>> 1. I am hesitating a little bit about changing the accepted > > >> FLIPs > > >>>> too > > >>>>>>> soon. > > >>>>>>>> 2. More importantly, to avoid slowing down our development. At > > >> this > > >>>>>>> point, > > >>>>>>>> Flink still lacks some design / routines to support good API > > >>>>>>> evolvability / > > >>>>>>>> extensibility. Just like you said, it takes some time to be > > >> good at > > >>>>>> this. > > >>>>>>>> In this case, my concern is that only removing Experimental / > > >>>>>>>> PublicEvolving APIs in major version changes may result in too > > >> much > > >>>>>>>> overhead and dramatically slow down the development of Flink. > > >> So, I > > >>>>> was > > >>>>>>>> thinking that we can start with the current status. Hopefully > > >> after > > >>>>> we > > >>>>>>> are > > >>>>>>>> more comfortable with the maintenance overhead of deprecated > > >> APIs, > > >>>> we > > >>>>>> can > > >>>>>>>> then have a stronger guarantee for Experimental / > > >> PublicEvolving > > >>>>> APIs. > > >>>>>>>> > > >>>>>>>> Thanks, > > >>>>>>>> > > >>>>>>>> Jiangjie (Becket) Qin > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler < > > >> vvcep...@apache.org <mailto:vvcep...@apache.org>> > > >>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi Becket, > > >>>>>>>>> > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process is really > > >>>>>> important. > > >>>>>>> I > > >>>>>>>>> understand some people’s concerns about the additional burden > > >> for > > >>>>>>> project > > >>>>>>>>> maintainers, but my personal experience with Kafka has been > > >> that > > >>>>> it’s > > >>>>>>> very > > >>>>>>>>> liveable and that it’s well worth the benefit to users. In > > >> fact, > > >>>>> users > > >>>>>>>>> being able to confidently upgrade is also a benefit to > > >>>> maintainers, > > >>>>> as > > >>>>>>> we > > >>>>>>>>> will get fewer questions from people stuck on very old > > >> versions. > > >>>>>>>>> > > >>>>>>>>> One question: > > >>>>>>>>> Can we consider only dropping deprecated APIs in major > > >> releases > > >>>>> across > > >>>>>>> the > > >>>>>>>>> board? I understand that Experimental and PublicEvolving APIs > > >> are > > >>>> by > > >>>>>>>>> definition less stable, but it seems like this should be > > >> reflected > > >>>>> in > > >>>>>>> the > > >>>>>>>>> required deprecation period alone. I.e. that we must keep them > > >>>>> around > > >>>>>>> for > > >>>>>>>>> at least zero or one minor release, not that we can drop them > > >> in a > > >>>>>>> minor or > > >>>>>>>>> patch release. > > >>>>>>>>> > > >>>>>>>>> The advantage of forbidding the removal of any API in minor or > > >>>> patch > > >>>>>>>>> releases is that users will get a strong guarantee that they > > >> can > > >>>>> bump > > >>>>>>> the > > >>>>>>>>> minor or patch version and still be able to compile, or even > > >> just > > >>>>>>> re-link > > >>>>>>>>> and know that they won’t face “MethodDef” exceptions at run > > >> time. > > >>>>> This > > >>>>>>> is a > > >>>>>>>>> binary guarantee: if we allow removing even Experimental APIs > > >>>>> outside > > >>>>>>> of > > >>>>>>>>> major releases, users can no longer confidently upgrade. > > >>>>>>>>> > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple of points: > > >>>>>>>>> * I’d use the official Deprecated annotation instead of > > >>>> introducing > > >>>>>> our > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well integrated > > >>>> into > > >>>>>>> build > > >>>>>>>>> tools and IDEs. > > >>>>>>>>> * I wouldn’t worry about a demotion process in this FLIP; it > > >> seems > > >>>>>>>>> orthogonal, and something that should probably be taken > > >>>> case-by-case > > >>>>>>>>> anyway. > > >>>>>>>>> * Aside from deprecation and removal, there have been some > > >>>>> discussions > > >>>>>>>>> about how to evolve APIs and behavior in compatible ways. > > >> This is > > >>>>>>> somewhat > > >>>>>>>>> of an art, and if folks haven’t wrestled with it before, it’ll > > >>>> take > > >>>>>> some > > >>>>>>>>> time to become good at it. I feel like this topic should also > > >> be > > >>>>>>> orthogonal > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to adopt a > > >> simple > > >>>>>> policy > > >>>>>>> not > > >>>>>>>>> to break existing user programs, and leave the “how” up to > > >>>>>> implementers > > >>>>>>> and > > >>>>>>>>> reviewers. > > >>>>>>>>> > > >>>>>>>>> Thanks again, > > >>>>>>>>> John > > >>>>>>>>> > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > > >>>>>>>>>> Hi All, > > >>>>>>>>>> > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong is a > > >> great > > >>>>> idea. > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e. > > >>>> @PublicEvolving > > >>>>>> --(2 > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 major > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot of > > >> flexibility > > >>>>>>> without > > >>>>>>>>>> breaking any rules we had. @Public APIs are allowed to > > >> change > > >>>>>> between > > >>>>>>>>> major > > >>>>>>>>>> releases. Changing annotations is acceptable and provides > > >>>>> additional > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs themself > > >> are > > >>>> not > > >>>>>>>>> changed. > > >>>>>>>>>> > > >>>>>>>>>> I had similar thoughts when I was facing those issues. I > > >> want to > > >>>>>> move > > >>>>>>> one > > >>>>>>>>>> step further and suggest introducing one more annotation > > >>>> @Retired. > > >>>>>>>>>> > > >>>>>>>>>> Not like the @PublicRetired which is a compromise of > > >> downgrading > > >>>>>>> @Public > > >>>>>>>>> to > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my reply, Java > > >>>> standard > > >>>>>>>>>> @deprecated should be used in the early stage of the > > >> deprecation > > >>>>>>> process > > >>>>>>>>>> and doesn't really meet our requirement. Since Java does not > > >>>> allow > > >>>>>> us > > >>>>>>> to > > >>>>>>>>>> extend annotation, I think it would be feasible to have the > > >> new > > >>>>>>> @Retired > > >>>>>>>>> to > > >>>>>>>>>> help us monitor and manage the deprecation process, house > > >>>>> cleaning, > > >>>>>>> etc. > > >>>>>>>>>> > > >>>>>>>>>> Some ideas could be(open for discussion): > > >>>>>>>>>> > > >>>>>>>>>> @Retired: > > >>>>>>>>>> > > >>>>>>>>>> 1. There must be a replacement with functionality > > >> compatibility > > >>>>>> before > > >>>>>>>>> APIs > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE processes > > >> on > > >>>> the > > >>>>> ML > > >>>>>>> are > > >>>>>>>>>> mandatory (a FLIP is recommended). > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1 minor > > >> release > > >>>>>>> sharply > > >>>>>>>>>> (using ArchUnit to force it, needs further check whether it > > >> is > > >>>>>>> possible). > > >>>>>>>>>> Devs who marked them as @Retired are responsible to remove > > >> them. > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving -> @Retired > > >> are > > >>>>>>>>>> recommended. @Experimental -> @Retired and @Internal -> > > >> @Retired > > >>>>>> could > > >>>>>>>>> also > > >>>>>>>>>> be used if it can increase user-friendliness or > > >>>> dev-friendliness, > > >>>>>> but > > >>>>>>> not > > >>>>>>>>>> mandatory. > > >>>>>>>>>> 4. Some variables will be defined in @Retired to support the > > >>>>>>> deprecation > > >>>>>>>>>> process management. Further extension is possible, since the > > >>>>>>> annotation > > >>>>>>>>> is > > >>>>>>>>>> built by us. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>>> Best regards, > > >>>>>>>>>> Jing > > >>>>>>>>>> > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin < > > >>>> becket....@gmail.com <mailto:becket....@gmail.com> > > >>>>>> > > >>>>>>>>> wrote: > > >>>>>>>>>> > > >>>>>>>>>>> Hi Xintong, > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks for the explanation. Please see the replies inline > > >>>> below. > > >>>>>>>>>>> > > >>>>>>>>>>> I agree. And from my understanding, demoting a Public API > > >> is > > >>>>> also a > > >>>>>>>>> kind of > > >>>>>>>>>>>> such change, just like removing one, which can only > > >> happen > > >>>> with > > >>>>>>> major > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public > > >>>> APIs > > >>>>>>>>> anytime, > > >>>>>>>>>>> but > > >>>>>>>>>>>> only in the case major version bumps happen before > > >> reaching > > >>>> the > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting > > >> would > > >>>> be a > > >>>>>>> weaker > > >>>>>>>>>>>> change compared to removing the API immediately upon > > >> major > > >>>>>> version > > >>>>>>>>> bumps, > > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release > > >>>>>> migration > > >>>>>>>>>>> period. > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` > > >> sounds > > >>>>>>> against > > >>>>>>>>>>>> conventions, we may introduce a new annotation if > > >> necessary, > > >>>>>> e.g., > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > >>>>>>>>>>> > > >>>>>>>>>>> As an end user who only uses Public APIs, if I don't > > >> change my > > >>>>> code > > >>>>>>> at > > >>>>>>>>> all, > > >>>>>>>>>>> my expectation is the following: > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an issue, I am > > >> fine > > >>>>>> with > > >>>>>>> all > > >>>>>>>>>>> the 2.x versions. > > >>>>>>>>>>> Actually I think there are some dependency version > > >> resolution > > >>>>>>> policies > > >>>>>>>>> out > > >>>>>>>>>>> there which picks the highest minor version when the > > >>>> dependencies > > >>>>>>> pull > > >>>>>>>>> in > > >>>>>>>>>>> multiple minor versions of the same jar, which may be > > >> broken if > > >>>>> we > > >>>>>>>>> remove > > >>>>>>>>>>> the API in minor releases. > > >>>>>>>>>>> > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" that > > >> we > > >>>>> bump > > >>>>>> up > > >>>>>>>>> the > > >>>>>>>>>>>> major version whenever a breaking change is needed. > > >> However, > > >>>>> this > > >>>>>>> also > > >>>>>>>>>>>> weakens the value of the commitment that public APIs will > > >>>> stay > > >>>>>>> stable > > >>>>>>>>>>>> within the major release series, as the series can end > > >>>> anytime. > > >>>>>>> IMHO, > > >>>>>>>>>>> short > > >>>>>>>>>>>> major release series are not something "make the end > > >> users > > >>>>>> happy", > > >>>>>>> but > > >>>>>>>>>>>> backdoors that allow us as the developers to make > > >> frequent > > >>>>>> breaking > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we > > >> can > > >>>>>> still > > >>>>>>>>> have > > >>>>>>>>>>>> longer major release series, while only allowing Public > > >> APIs > > >>>>>>>>> deprecated > > >>>>>>>>>>> at > > >>>>>>>>>>>> the end of the previous major version to be removed in > > >> the > > >>>> next > > >>>>>>> major > > >>>>>>>>>>>> version. > > >>>>>>>>>>> > > >>>>>>>>>>> I totally agree that frequent major version bumps are not > > >>>> ideal, > > >>>>>> but > > >>>>>>>>> here > > >>>>>>>>>>> we are comparing it with a minor version bump which > > >> removes a > > >>>>>> Public > > >>>>>>>>> API. > > >>>>>>>>>>> So the context is that we have already decided to remove > > >> this > > >>>>>> Public > > >>>>>>> API > > >>>>>>>>>>> while keeping everything else backwards compatible. I > > >> think a > > >>>>> major > > >>>>>>>>> version > > >>>>>>>>>>> bump is a commonly understood signal here, compared with a > > >>>> minor > > >>>>>>> version > > >>>>>>>>>>> change. From end users' perspective, for those who are not > > >>>>>> impacted, > > >>>>>>> in > > >>>>>>>>>>> this case upgrading a major version is not necessarily more > > >>>>>> involved > > >>>>>>>>> than > > >>>>>>>>>>> upgrading a minor version - both should be as smooth as a > > >>>>>> dependency > > >>>>>>>>>>> version change. For those who are impacted, they will lose > > >> the > > >>>>>> Public > > >>>>>>>>> API > > >>>>>>>>>>> anyways and a major version bump ensures there is no > > >> surprise. > > >>>>>>>>>>> > > >>>>>>>>>>> Thanks, > > >>>>>>>>>>> > > >>>>>>>>>>> Jiangjie (Becket) Qin > > >>>>>>>>>>> > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song < > > >>>>>> tonysong...@gmail.com <mailto:tonysong...@gmail.com> > > >>>>>>>> > > >>>>>>>>>>> wrote: > > >>>>>>>>>>> > > >>>>>>>>>>>> Public API is a well defined common concept, and one of > > >> its > > >>>>>>>>>>>>> convention is that it only changes with a major version > > >>>>> change. > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> I agree. And from my understanding, demoting a Public > > >> API is > > >>>>>> also a > > >>>>>>>>> kind > > >>>>>>>>>>>> of such change, just like removing one, which can only > > >> happen > > >>>>>> with > > >>>>>>>>> major > > >>>>>>>>>>>> version bumps. I'm not proposing to allow demoting Public > > >>>> APIs > > >>>>>>>>> anytime, > > >>>>>>>>>>> but > > >>>>>>>>>>>> only in the case major version bumps happen before > > >> reaching > > >>>> the > > >>>>>>>>>>>> 2-minor-release migration period. Actually, demoting > > >> would > > >>>> be a > > >>>>>>> weaker > > >>>>>>>>>>>> change compared to removing the API immediately upon > > >> major > > >>>>>> version > > >>>>>>>>> bumps, > > >>>>>>>>>>>> in order to keep the commitment about the 2-minor-release > > >>>>>> migration > > >>>>>>>>>>> period. > > >>>>>>>>>>>> If the concern is that `@Public` -> `@PublicEvolving` > > >> sounds > > >>>>>>> against > > >>>>>>>>>>>> conventions, we may introduce a new annotation if > > >> necessary, > > >>>>>> e.g., > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > >>>>>>>>>>>> > > >>>>>>>>>>>> But it should be > > >>>>>>>>>>>>> completely OK to bump up the major version if we really > > >> want > > >>>>> to > > >>>>>>> get > > >>>>>>>>> rid > > >>>>>>>>>>> of > > >>>>>>>>>>>>> a public API, right? > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely "legal" > > >> that we > > >>>>>> bump > > >>>>>>> up > > >>>>>>>>> the > > >>>>>>>>>>>> major version whenever a breaking change is needed. > > >> However, > > >>>>> this > > >>>>>>> also > > >>>>>>>>>>>> weakens the value of the commitment that public APIs will > > >>>> stay > > >>>>>>> stable > > >>>>>>>>>>>> within the major release series, as the series can end > > >>>> anytime. > > >>>>>>> IMHO, > > >>>>>>>>>>> short > > >>>>>>>>>>>> major release series are not something "make the end > > >> users > > >>>>>> happy", > > >>>>>>> but > > >>>>>>>>>>>> backdoors that allow us as the developers to make > > >> frequent > > >>>>>> breaking > > >>>>>>>>>>>> changes. On the contrary, with the demoting approach, we > > >> can > > >>>>>> still > > >>>>>>>>> have > > >>>>>>>>>>>> longer major release series, while only allowing Public > > >> APIs > > >>>>>>>>> deprecated > > >>>>>>>>>>> at > > >>>>>>>>>>>> the end of the previous major version to be removed in > > >> the > > >>>> next > > >>>>>>> major > > >>>>>>>>>>>> version. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Given our track record I would prefer a regular cycle > > >> (1-2 > > >>>>> years) > > >>>>>>> to > > >>>>>>>>>>>>> force us to think about this whole topic, and not put it > > >>>> again > > >>>>>> to > > >>>>>>> the > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on > > >>>> when > > >>>>>>>>> breaking > > >>>>>>>>>>>>> changes can be made. > > >>>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good time > > >> for new > > >>>>>> major > > >>>>>>>>>>>> versions, or longer if there's no breaking changes > > >> needed. > > >>>> That > > >>>>>>> makes > > >>>>>>>>> 1-2 > > >>>>>>>>>>>> year a perfect time to revisit the topic, while leaving > > >> us > > >>>> more > > >>>>>>> time > > >>>>>>>>> to > > >>>>>>>>>>>> prepare the major release if needed. > > >>>>>>>>>>>> > > >>>>>>>>>>>> Best, > > >>>>>>>>>>>> > > >>>>>>>>>>>> Xintong > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay Schepler < > > >>>>>>> ches...@apache.org <mailto:ches...@apache.org> > > >>>>>>>>>> > > >>>>>>>>>>>> wrote: > > >>>>>>>>>>>> > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing minor > > >>>>> versions > > >>>>>>> for > > >>>>>>>>>>>>> previous > > >>>>>>>>>>>>>> major versions. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure how > > >> viable > > >>>> that > > >>>>>> is > > >>>>>>> in > > >>>>>>>>>>>>> practice. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially dropping > > >> support > > >>>> for > > >>>>>> Java > > >>>>>>>>> 8/11, > > >>>>>>>>>>>>> which may very well be a problem for our current users. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the confusion > > >> about > > >>>>>> Public > > >>>>>>>>> API > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a mistake > > >> in > > >>>> the > > >>>>>>> NOTES > > >>>>>>>>> part > > >>>>>>>>>>>>> for > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just fixed > > >> it. > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear that we > > >> are on > > >>>>> the > > >>>>>>> same > > >>>>>>>>>>>>> page on that note. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote: > > >>>>>>>>>>>>>> But it should be > > >>>>>>>>>>>>>> completely OK to bump up the major version if we > > >> really > > >>>> want > > >>>>>> to > > >>>>>>> get > > >>>>>>>>>>> rid > > >>>>>>>>>>>>> of > > >>>>>>>>>>>>>> a public API, right? > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Technically yes, but look at how long it took to get us > > >> to > > >>>>> 2.0. > > >>>>>> ;) > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> There's a separate discussion to be had on the cadence > > >> of > > >>>>> major > > >>>>>>>>> releases > > >>>>>>>>>>>>> going forward, and there seem to be different opinions > > >> on > > >>>>> that. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor releases between > > >>>> major > > >>>>>>> ones, > > >>>>>>>>>>>>> that for us means that users have to potentially deal > > >> with > > >>>>>>> breaking > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Given our track record I would prefer a regular cycle > > >> (1-2 > > >>>>>> years) > > >>>>>>> to > > >>>>>>>>>>>>> force us to think about this whole topic, and not put it > > >>>> again > > >>>>>> to > > >>>>>>> the > > >>>>>>>>>>>>> wayside and giving us (and users) a clear expectation on > > >>>> when > > >>>>>>>>> breaking > > >>>>>>>>>>>>> changes can be made. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> But again, maybe this should be in a separate thread. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote: > > >>>>>>>>>>>>>> Do you have an example of behavioral change in mind? > > >> Not > > >>>>> sure > > >>>>>> I > > >>>>>>>>> fully > > >>>>>>>>>>>>>> understand the concern for behavioral change here. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> This could be a lot of things. It can be performance in > > >>>>> certain > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe unknowingly) > > >> relied > > >>>>> upon > > >>>>>>>>>>>>> ( > > > https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I > ), > > a semantic change to some > > >> API. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> For a concrete example, consider the job submission. A > > >> few > > >>>>>>> releases > > >>>>>>>>> back > > >>>>>>>>>>>>> we made changes such that the initialization of the job > > >>>> master > > >>>>>>>>> happens > > >>>>>>>>>>>>> asynchronously. > > >>>>>>>>>>>>> This meant the job submission call returns sooner, and > > >> the > > >>>> job > > >>>>>>> state > > >>>>>>>>>>>>> enum was extended to cover this state. > > >>>>>>>>>>>>> API-wise we consider this a compatible change, but the > > >>>>> observed > > >>>>>>>>> behavior > > >>>>>>>>>>>>> may be different. > > >>>>>>>>>>>>> > > >>>>>>>>>>>>> Metrics are another example; I believe over time we > > >> changed > > >>>>> what > > >>>>>>> some > > >>>>>>>>>>>>> metrics returned a few times. > > > > >
