Hi Martijn, Thanks for the reply. Regarding the behavioral stability guarantee, by definition, an "API" always consists of the signature + behavior. So, the behavior is a native part of an API. Therefore, behavioral changes should be treated with the same guarantee as signature changes.
+1 on having toolings to enforce the conventions. Thanks, Jiangjie (Becket) Qin On Wed, Jun 28, 2023 at 5:09 AM Martijn Visser <martijnvis...@apache.org> wrote: > Hi all, > > Thanks for the lively and good discussion. Given the length of the > discussion, I skimmed through and then did a deep dive on the latest state > of the FLIP. I think the FLIP is overall in a good state and ready to bring > to a vote. > > One thing that I did notice while skimming through the discussions is that > I think there are some follow-ups that could be worthy of a FLIP and a > discussion. For example, if/what/how does the Flink community offer any > behavioral stability guarantees, or other types of expectations and > guarantees. I also do think that we must have tooling in place to implement > this FLIP (and also FLIP-196 and FLIP-197), to avoid that we're not > creating a policy on paper, but also have the means to enforce it. Last but > not least, I echo Jark's point that we can't estimate maintenance cost with > a concrete design and code/POC. For me, that means that a contributor can > choose to propose a deviation, but that the contributor would need to > explicitly mention it in the FLIP and get it discussed/voted on as part of > the FLIP process. But the starting point is as defined in this and other > relevant FLIPs. > > Best regards, > > Martijn > > On Tue, Jun 27, 2023 at 3:38 AM Becket Qin <becket....@gmail.com> wrote: > > > Hi Xintong, Jark and Jing, > > > > Thanks for the reply. Yes, we can only mark the DataStream API as > > @Deprecated after the ProcessFunction API is fully functional and mature. > > > > It is a fair point that the condition of marking a @Public API as > > deprecated should also be a part of this FLIP. I just added that to the > > FLIP wiki. This is probably more of a clarification on the existing > > convention, rather than a change. > > > > It looks like we are on the same page now for this FLIP. If so, I'll > start > > a VOTE thread in two days. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Mon, Jun 26, 2023 at 8:09 PM Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > > > > > > Considering DataStream API is the most fundamental and complex API of > > > > Flink, I think it is worth a longer time than the general process for > > the > > > > deprecation period to wait for the new API be mature. > > > > > > > > > > This inspires me. In this specific case, compared to how long should > > > DataStream API be removed after deprecation, it's probably more > important > > > to answer the question how long would ProcessFunction API become mature > > and > > > stable after being introduced. According to FLIP-197[1], it requires 4 > > > minor releases by default to promote an @Experimental API to @Public. > And > > > for ProcessFunction API, which aims to replace DataStream API as one of > > the > > > most fundamental API of Flink, I'd expect this to take at least the > > default > > > time, or even longer. And we probably should wait until we believe > > > ProcessFunction API is stable to mark DataStream API as deprecated, > > rather > > > than as soon as it's introduced. Assuming we introduce the > > ProcessFunction > > > API in 2.0, that means we would need to wait for 6 minor releases (4 > for > > > the new API to become stable, and 2 for the migration period) to remove > > > DataStream API, which is ~2.5 year (assuming 5 months / minor release), > > > which sounds acceptable for another major version bump. > > > > > > To wrap things up, it seems to me, sadly, that anyway we cannot avoid > the > > > overhead for maintaining both DataStream & ProcessFunction APIs for at > > > least 6 minor releases. > > > > > > Best, > > > > > > Xintong > > > > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > > > > > > > > > > > > On Mon, Jun 26, 2023 at 5:41 PM Jing Ge <j...@ververica.com.invalid> > > > wrote: > > > > > > > Hi all, > > > > > > > > Just want to make sure we are on the same page. There is another > > > example[1] > > > > I was aware of recently that shows why more factors need to be taken > > care > > > > of than just the migration period. Thanks Galen for your hint. > > > > > > > > To put it simply, the concern about API deprecation is not that > > > deprecated > > > > APIs have been removed too early (min migration period is required). > > The > > > > major concern is that APIs are marked as deprecated for a (too) long > > > time, > > > > much longer than the migration period discussed in this thread, > afaik. > > > > Since there is no clear picture/definition, no one knows when to do > the > > > > migration for users(after the migration period has expired) and when > to > > > > remove deprecated APIs for Flink developers. > > > > > > > > Based on all the information I knew, there are two kinds of obstacles > > > that > > > > will and should block the deprecation process: > > > > > > > > 1. Lack of functionalities in new APIs. It happens e.g. with the > > > > SourceFunction to FLIP-27 Source migration. Users who rely on those > > > > functions can not migrate to new APIs. > > > > 2. new APIs have critical bugs. An example could be found at [1]. > Users > > > > have to stick to the deprecated APIs. > > > > > > > > Since FLIP-321 is focusing on the API deprecation process, those > > blocking > > > > issues deserve attention and should be put into the FLIP. The current > > > FLIP > > > > seems to only focus on migration periods. If we consider those > blocking > > > > issues as orthogonal issues that are beyond the scope of this > > discussion, > > > > does it make sense to change the FLIP title to something like > > "Introduce > > > > minimum migration periods of API deprecation process"? > > > > > > > > Best regards, > > > > Jing > > > > > > > > [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc > > > > > > > > On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <imj...@gmail.com> wrote: > > > > > > > > > I agree with Jingsong and Becket. > > > > > > > > > > Look at the legacy SourceFunction (a small part of DataStream API), > > > > > the SourceFunction is still not and can't be marked deprecated[1] > > until > > > > > now after the new Source was released 2 years ago, because the new > > > Source > > > > > still can't fully consume the abilities of legacy API. Considering > > > > > DataStream > > > > > API is the most fundamental and complex API of Flink, I think it is > > > worth > > > > > a longer time than the general process for the deprecation period > to > > > > > wait for the new API be mature. The above 2 options sound a bit of > > rush > > > > > for such a widely used API. > > > > > > > > > > I fully understand the concern of maintenance overhead, but it's a > > bit > > > > hard > > > > > for others to estimate maintenance costs without a concrete design > > and > > > > code > > > > > of the new ProcessFunction API. I agree with Becket that maybe we > can > > > > > re-evaluate the API deprecation process once we have the new > > > > > ProcessFunction > > > > > API. If the maintenance is indeed huge, I think it is reasonable to > > > have > > > > a > > > > > special rule for this case at that time. > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > > > > > [1]: https://issues.apache.org/jira/browse/FLINK-28045 > > > > > > > > > > On Sun, 25 Jun 2023 at 16:22, Becket Qin <becket....@gmail.com> > > wrote: > > > > > > > > > > > Hi Jingsong, > > > > > > > > > > > > Thanks for the reply. I completely agree with you. > > > > > > > > > > > > The above 2 options are based on the assumption that the > community > > > > cannot > > > > > > afford to maintain the deprecated DataStream API for long. I'd > say > > we > > > > > > should try everything we can to maintain it for as much time as > > > > possible. > > > > > > DataStream API is actually the most used API in Flink by so many > > > users > > > > at > > > > > > this point. Removing it any time soon will dramatically hurt our > > > users. > > > > > So > > > > > > ideally we should keep it for at least 2 years after deprecation, > > if > > > > not > > > > > > more. > > > > > > > > > > > > The prohibitively high maintenance overhead is just an > assumption. > > > > > > Personally speaking, I don't feel this assumption is necessarily > > > true. > > > > We > > > > > > should re-evaluate once we have the new ProcessFunction API in > > place. > > > > > > Without the code it is hard to tell for sure. I am actually kind > of > > > > > > optimistic about the maintenance cost. > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > > > > > > > > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li < > > jingsongl...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Thanks Becket and all for your discussion. > > > > > > > > > > > > > > > 1. We say this FLIP is enforced starting release 2.0. For > > current > > > > 1.x > > > > > > > APIs, > > > > > > > we provide a migration period with best effort, while allowing > > > > > exceptions > > > > > > > for immediate removal in 2.0. That means we will still try with > > > best > > > > > > effort > > > > > > > to get the ProcessFuncion API ready and deprecate the > DataStream > > > API > > > > in > > > > > > > 1.x, but will also be allowed to remove DataStream API in 2.0 > if > > > it's > > > > > not > > > > > > > deprecated 2 minor releases before the major version bump. > > > > > > > > > > > > > > > 2. We strictly follow the process in this FLIP, and will > > quickly > > > > bump > > > > > > the > > > > > > > major version from 2.x to 3.0 once the migration period for > > > > DataStream > > > > > > API > > > > > > > is reached. > > > > > > > > > > > > > > Sorry, I didn't read the previous detailed discussion because > the > > > > > > > discussion list was so long. > > > > > > > > > > > > > > I don't really like either of these options. > > > > > > > > > > > > > > Considering that DataStream is such an important API, can we > > offer > > > a > > > > > > third > > > > > > > option: > > > > > > > > > > > > > > 3. Maintain the DataStream API throughout 2.X and remove it > until > > > > 3.x. > > > > > > But > > > > > > > there's no need to assume that 2.X is a short version, it's > > still a > > > > > > normal > > > > > > > major version. > > > > > > > > > > > > > > Best, > > > > > > > Jingsong > > > > > > > > > > > > > > Becket Qin <becket....@gmail.com>于2023年6月22日 周四16:02写道： > > > > > > > > > > > > > > > Thanks much for the input, John, Stefan and Jing. > > > > > > > > > > > > > > > > I think Xingtong has well summarized the pros and cons of the > > two > > > > > > > options. > > > > > > > > Let's collect a few more opinions here and we can move > forward > > > with > > > > > the > > > > > > > one > > > > > > > > more people prefer. > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > Jiangjie (Becket) Qin > > > > > > > > > > > > > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge > > > <j...@ververica.com.invalid > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > Thanks Xingtong for the summary. If I could only choose one > > of > > > > the > > > > > > > given > > > > > > > > > two options, I would go with option 1. I understood that > > > option 2 > > > > > > > worked > > > > > > > > > great with Kafka. But the bridge release will still confuse > > > users > > > > > and > > > > > > > my > > > > > > > > > gut feeling is that many users will skip 2.0 and be waiting > > for > > > > 3.0 > > > > > > or > > > > > > > > even > > > > > > > > > 3.x. And since fewer users will use Flink 2.x, the > > development > > > > > focus > > > > > > > will > > > > > > > > > be on Flink 3.0 with the fact that the current Flink > release > > is > > > > > 1.17 > > > > > > > and > > > > > > > > we > > > > > > > > > are preparing 2.0 release. That is weird for me. > > > > > > > > > > > > > > > > > > THB, I would not name the change from @Public to @Retired > as > > a > > > > > > > demotion. > > > > > > > > > The purpose of @Retire is to extend the API lifecycle with > > one > > > > more > > > > > > > > stage, > > > > > > > > > like in the real world, people born, studied, graduated, > > > worked, > > > > > and > > > > > > > > > retired. Afaiu from the previous discussion, there are two > > > rules > > > > > we'd > > > > > > > > like > > > > > > > > > to follow simultaneously: > > > > > > > > > > > > > > > > > > 1. Public APIs can only be changed between major releases. > > > > > > > > > 2. A smooth migration phase should be offered to users, > i.e. > > at > > > > > > least 2 > > > > > > > > > minor releases after APIs are marked as @deprecated. There > > > should > > > > > be > > > > > > > new > > > > > > > > > APIs as the replacement. > > > > > > > > > > > > > > > > > > Agree, those rules are good to improve the user > friendliness. > > > > > Issues > > > > > > we > > > > > > > > > discussed are rising because we want to fulfill both of > them. > > > If > > > > we > > > > > > > take > > > > > > > > > care of deprecation very seriously, APIs can be marked as > > > > > > @Deprecated, > > > > > > > > only > > > > > > > > > when the new APIs as the replacement provide all > > > functionalities > > > > > the > > > > > > > > > deprecated APIs have. In an ideal case without critical > bugs > > > that > > > > > > might > > > > > > > > > stop users adopting the new APIs. Otherwise the expected > > > > > > "replacement" > > > > > > > > will > > > > > > > > > not happen. Users will still stick to the deprecated APIs, > > > > because > > > > > > the > > > > > > > > new > > > > > > > > > APIs can not be used. For big features, it will need at > > least 4 > > > > > minor > > > > > > > > > releases(ideal case), i.e. 2+ years to remove deprecated > > APIs: > > > > > > > > > > > > > > > > > > - 1st minor release to build the new APIs as the > replacement > > > and > > > > > > > waiting > > > > > > > > > for feedback. It might be difficult to mark the old API as > > > > > deprecated > > > > > > > in > > > > > > > > > this release, because we are not sure if the new APIs could > > > cover > > > > > > 100% > > > > > > > > > functionalities. > > > > > > > > > - In the lucky case, mark all old APIs as deprecated in > the > > > 2nd > > > > > > minor > > > > > > > > > release. (I would even suggest having the new APIs released > > at > > > > > least > > > > > > > for > > > > > > > > > two minor releases before marking it as deprecated to make > > sure > > > > > they > > > > > > > can > > > > > > > > > really replace the old APIs, in case we care more about > > smooth > > > > > > > migration) > > > > > > > > > - 3rd minor release for the migration period > > > > > > > > > - In another lucky case, the 4th release is a major > release, > > > the > > > > > > > > > deprecated APIs could be removed. > > > > > > > > > > > > > > > > > > The above described scenario works only in an ideal case. > In > > > > > reality, > > > > > > > it > > > > > > > > > might take longer to get the new APIs ready and mark the > old > > > API > > > > > > > > > deprecated. Furthermore, if the 4th release is not a major > > > > release, > > > > > > we > > > > > > > > will > > > > > > > > > have to maintain both APIs for many further minor releases. > > The > > > > > > > question > > > > > > > > is > > > > > > > > > how to know the next major release in advance, especially 4 > > > minor > > > > > > > > releases' > > > > > > > > > period, i.e. more than 2 years in advance? Given that Flink > > > > > contains > > > > > > > many > > > > > > > > > modules, it is difficult to ask devs to create a 2-3 years > > > > > > deprecation > > > > > > > > plan > > > > > > > > > for each case. In case we want to build major releases at a > > > fast > > > > > > pace, > > > > > > > > > let's say every two years, it means devs must plan any API > > > > > > deprecation > > > > > > > > > right after each major release. Afaiac, it is quite > > difficult. > > > > > > > > > > > > > > > > > > The major issue is, afaiu, if we follow rule 2, we have to > > keep > > > > all > > > > > > > > @Public > > > > > > > > > APIs, e.g. DataStream, that are not marked as deprecated > yet, > > > to > > > > > 2.0. > > > > > > > > Then > > > > > > > > > we have to follow rule 1 to keep it unchanged until we have > > > 3.0. > > > > > That > > > > > > > is > > > > > > > > > why @Retired is useful to give devs more flexibility and > > still > > > > > > fulfill > > > > > > > > both > > > > > > > > > rules. Let's check it with examples: > > > > > > > > > > > > > > > > > > - we have @Public DataStream API in 1.18. It will not be > > marked > > > > > > > > > as @Deprecated, because the new APIs as the replacement are > > not > > > > > > ready. > > > > > > > > > - we keep the DataStream API itself unchanged in 2.0, but > > > change > > > > > the > > > > > > > > > annotation from @Public to @Retire. New APIs will be > > introduced > > > > > too. > > > > > > In > > > > > > > > > this case, Rule 1 is ok, since public API is allowed to > > change > > > > > > between > > > > > > > > > major releases. Only changing annotation is the minimal > > change > > > we > > > > > > could > > > > > > > > do > > > > > > > > > and it does not break rule 1. Rule 2 is ok too, since the > > > > > DataStream > > > > > > > APIs > > > > > > > > > work exactly the same. Attention: the change of @Public -> > > > > > @Retired > > > > > > > can > > > > > > > > > only be done between major releases, because @Public APIs > can > > > > only > > > > > be > > > > > > > > > changed between major releases. > > > > > > > > > - in 2.1, DataStream API will be marked as deprecated. > > > > > > > > > - in 2.2, DataStream will be kept for the migration period. > > > > > > > > > - in 2.3, DataStream will be removed. > > > > > > > > > > > > > > > > > > Becket mentioned previously (please correct me if I didn't > > > > > understand > > > > > > > it > > > > > > > > > correctly) that users might not check the changes of > > annotation > > > > and > > > > > > the > > > > > > > > > upgrade process might not be smooth. I was wondering, if > > users > > > > > don't > > > > > > > pay > > > > > > > > > attention to that, they will also not pay attention to > > > > @deprecated. > > > > > > The > > > > > > > > > migration will not be smooth too even when we stick to the > > Java > > > > > > > > > standard @deprecated. Let's go throw it with examples: > > > > > > > > > > > > > > > > > > Scenario 1: with @Retired > > > > > > > > > > > > > > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing > > > should > > > > > be > > > > > > > done > > > > > > > > > for users who still want to use DataStream. The migration > is > > > > > smooth. > > > > > > > But > > > > > > > > > they should be aware of the annotation changes, i.e. > @Public > > -> > > > > > > > @Retired > > > > > > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is > > removed. > > > > > Since > > > > > > > it > > > > > > > > is > > > > > > > > > a major release upgrade, breaking changes are expected. > > > > > > > > > Case 3: users upgrade from 2.0 to 2.3 - DataStream is > > removed. > > > > > Since > > > > > > > > APIs > > > > > > > > > are marked as @Retired, breaking changes are expected too. > > > > > > > > > > > > > > > > > > In addition to fulfilling both rules, there are two more > > > benefits > > > > > of > > > > > > > this > > > > > > > > > option: > > > > > > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case > 1. > > > > > > > > > 2. no bridge release is required. DataStream API could be > > > > > deprecated > > > > > > in > > > > > > > > 2.x > > > > > > > > > without breaking any of the two rules. @Retired provides > > enough > > > > > > > > flexibility > > > > > > > > > for future deprecations and upgrades. > > > > > > > > > > > > > > > > > > The only issue I can find is that users might be surprised > in > > > > case > > > > > 3, > > > > > > > if > > > > > > > > > they ignore the previous @Public -> @Retired change. But > for > > > such > > > > > > > users, > > > > > > > > > they will ignore @deprecated annotation too and will have > > > issues > > > > > with > > > > > > > > > pure @deprecated > > > > > > > > > annotation too: > > > > > > > > > > > > > > > > > > Scenario 2: with @deprecated > > > > > > > > > > > > > > > > > > The example will be slightly different from scenario 1, > > because > > > > > > @Public > > > > > > > > > APIs can only be removed between major releases. As an > > example, > > > > > > > > DataStream > > > > > > > > > API will be marked as deprecated in 1.19, kept in 1.20, and > > > then > > > > > > > removed > > > > > > > > in > > > > > > > > > 2.0 > > > > > > > > > > > > > > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not > > > marked > > > > > as > > > > > > > > > deprecated are removed between major releases. Rule 1 is > ok. > > > The > > > > 2 > > > > > > > minor > > > > > > > > > releases period defined in rule 2 sounds like a smooth > > > migration > > > > > > plan, > > > > > > > > but > > > > > > > > > for users in this case, it is still not smooth. > > > > > > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade > > > process > > > > > > with > > > > > > > > > deprecated APIs. But for those users who ignore annotation > > > > changes, > > > > > > > will > > > > > > > > > have the same issue mentioned in case 1. > > > > > > > > > > > > > > > > > > I understood that, compared to scenario 1, users, who > ignore > > > > > @Public > > > > > > > > > -> @Retired > > > > > > > > > changes, will not face any breaking changes, after they > > > migrated > > > > to > > > > > > > 2.0. > > > > > > > > > But the cost of it is bridge releases. There might be many > > > bridge > > > > > > > > releases > > > > > > > > > in the future. As I mentioned previously, it is not easy to > > > find > > > > > the > > > > > > > > right > > > > > > > > > timing to kick off the deprecation process of APIs before > the > > > > next > > > > > > > > expected > > > > > > > > > major release and 2-3 years in advance. > > > > > > > > > > > > > > > > > > Scenario 3: @PublicEvolving deprecation for users who > ignore > > > > > > > @deprecated > > > > > > > > > annotation changes > > > > > > > > > > > > > > > > > > Sample API marked as deprecated in 1.19, kept in 1.20, > > removed > > > in > > > > > > 1.21 > > > > > > > > > > > > > > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, > no > > > > > smooth > > > > > > > > > migration. > > > > > > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking > > change, > > > > > > > > > since @deprecated is ignored > > > > > > > > > > > > > > > > > > In summary, for users who care about the annotation change, > > > they > > > > > will > > > > > > > get > > > > > > > > > more benefits with @Retired. For users who ignore > annotation > > > > > changes, > > > > > > > > they > > > > > > > > > will be always facing non-smooth upgrades even using pure > > > > standard > > > > > > > > > @deprecated > > > > > > > > > annotation/process. > > > > > > > > > > > > > > > > > > For the case that IDEs support @deprecated, that is true. > > There > > > > are > > > > > > no > > > > > > > > such > > > > > > > > > supports for homebuilt annotation. But the intention is to > > let > > > > > users > > > > > > be > > > > > > > > > aware of it. There will be other ways to do it for users > who > > > care > > > > > > about > > > > > > > > > annotation changes. For users who do not care, there should > > be > > > no > > > > > > > > > difference between with tool support or without. Using > > @Retired > > > > > with > > > > > > > > > @deprecation > > > > > > > > > together is at least not worse than only using @deprecation > > > > alone. > > > > > > And > > > > > > > > > there are additional benefits with @Retired. > > > > > > > > > > > > > > > > > > Just my 2 cents and looking forward to your feedback, > > > especially > > > > > > > > different > > > > > > > > > opinions that will help me understand the issue better. > > Thanks! > > > > > > > > > > > > > > > > > > > > > > > > > > > Best regards, > > > > > > > > > Jing > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter > > > > > > > > > <srich...@confluent.io.invalid> wrote: > > > > > > > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > > > > > > > Thanks for the summary, most of the points that you > agreed > > > upon > > > > > > also > > > > > > > > make > > > > > > > > > > sense to me. > > > > > > > > > > > > > > > > > > > > > 2. Dropping deprecated Public APIs in minor releases, > > or > > > > > > demoting > > > > > > > > > APIs > > > > > > > > > > > from Public to PublicEvolving / Experimental / > Retired > > in > > > > any > > > > > > > > version > > > > > > > > > > bump, > > > > > > > > > > > are not good practices. > > > > > > > > > > > > > > > > > > > > I hope we can can go beyond calling it “not a good > > practice" > > > > and > > > > > > > reach > > > > > > > > > > consensus that we will not demote or remove public APIs > in > > > > minor > > > > > > > > releases > > > > > > > > > > and (technically) enforce this rule for all > contributions. > > I > > > > > think > > > > > > > > it’s a > > > > > > > > > > reasonable expectation for stability from a project as > > mature > > > > as > > > > > > > Flink. > > > > > > > > > > > > > > > > > > > > > I'm personally in favor of option 1. As the > > > migration-period > > > > > rule > > > > > > > is > > > > > > > > > > newly > > > > > > > > > > > proposed, I think it's fair to make exceptions for > cases > > > > where > > > > > we > > > > > > > > > already > > > > > > > > > > > missed the best chance for planning the deprecation. > > > > Moreover, > > > > > I > > > > > > do > > > > > > > > > > believe > > > > > > > > > > > having a quick major version bump does confuse users. > > Yes, > > > we > > > > > can > > > > > > > > > explain > > > > > > > > > > > to users that bumping from 2.x to 3.0 does not cost > > > anything > > > > > > other > > > > > > > > than > > > > > > > > > > the > > > > > > > > > > > removal of an deprecated API. But having to explain > this > > > > itself > > > > > > is > > > > > > > an > > > > > > > > > > > indicator that it might be confusing for users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becket, on the other hand, prefers option 2. From my > > > > > > understanding, > > > > > > > > his > > > > > > > > > > > major point is that a quick major version bump causes > > > barely > > > > > any > > > > > > > > actual > > > > > > > > > > > lose on users, while in option 1 not providing the > > > migration > > > > > > period > > > > > > > > or > > > > > > > > > > > providing a shorter on is an actual lose on users. > > > (@Becket, > > > > > > please > > > > > > > > > > correct > > > > > > > > > > > me if I'm mistaken.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would be open to both options and maybe postpone this > > > > decision > > > > > > > until > > > > > > > > we > > > > > > > > > > have the complete roadmap for Flink 2.0. We would have a > > > better > > > > > > > > overview > > > > > > > > > > about the technical consequences, e.g. how hard it would > be > > > to > > > > > > offer > > > > > > > > and > > > > > > > > > > maintain both APIs side by side, how well we get both > APIs > > > > > > separated, > > > > > > > > and > > > > > > > > > > whether or not we will be able to use the full potential > of > > > our > > > > > > > > breaking > > > > > > > > > > changes before the DataStream API is completely removed. > > One > > > > > > example > > > > > > > > for > > > > > > > > > > the last point I can think of would be moving from lazy > to > > > > eager > > > > > > > state > > > > > > > > > > declaration. If we believe that it is reasonable to > support > > > all > > > > > > > planned > > > > > > > > > > changes while maintaining both APIs, I'm leaning towards > > > option > > > > > 2. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Stefan > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song < > > > > tonysong...@gmail.com > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > Becket and I had an offline voice call earlier today. > We > > > have > > > > > > > reached > > > > > > > > > > > consensus on some of the arguments, but still hold > > > different > > > > > > > opinions > > > > > > > > > on > > > > > > > > > > > some others. I'd like to post the outcome here for > > > > > transparency. > > > > > > > > > > > > > > > > > > > > > > We both agree that: > > > > > > > > > > > > > > > > > > > > > > 1. Providing a migration period would be beneficial > for > > > our > > > > > > > users, > > > > > > > > > and > > > > > > > > > > > we should do that > > > > > > > > > > > 2. Dropping deprecated Public APIs in minor releases, > > or > > > > > > demoting > > > > > > > > > APIs > > > > > > > > > > > from Public to PublicEvolving / Experimental / > Retired > > in > > > > any > > > > > > > > version > > > > > > > > > > bump, > > > > > > > > > > > are not good practices. > > > > > > > > > > > 3. Ideally, with this FLIP, developers should be more > > > > careful > > > > > > and > > > > > > > > > plan > > > > > > > > > > > API changes ahead. That means： > > > > > > > > > > > 1. Be more careful with designing APIs and > promoting > > > > them > > > > > to > > > > > > > > > Public, > > > > > > > > > > > so that they won't be changed / removed very soon, > > and > > > > > also > > > > > > > the > > > > > > > > > > > maintenance overhead for keeping them after > > > deprecation > > > > > > should > > > > > > > > be > > > > > > > > > > > affordable. (I believe this also aligns with > John's > > > > > > opinion.) > > > > > > > > > > > 2. Plan the deprecation / removal of old APIs that > > we > > > > > don't > > > > > > > want > > > > > > > > > to > > > > > > > > > > > carry to the next major version earlier, so that > > > they'll > > > > > > have > > > > > > > > the > > > > > > > > > > > 2-minor-release migration period before the next > > major > > > > > > version > > > > > > > > > bump. > > > > > > > > > > > 4. A practical situation is that, we are not ready > for > > > > > > > deprecating > > > > > > > > > the > > > > > > > > > > > DataStream API, nor afford to carry it for another > > couple > > > > of > > > > > > > years. > > > > > > > > > > > 1. We cannot deprecate it now, because the new > > > > > > ProcessFunction > > > > > > > > API > > > > > > > > > > > (the planned replacement for DataStream API) is > not > > > yet > > > > > > ready. > > > > > > > > We > > > > > > > > > > haven't > > > > > > > > > > > planned the ProcessFunction API earlier because > the > > > > > original > > > > > > > > > > > plan was to do > > > > > > > > > > > in-place changes directly on the DataStream API in > > > > release > > > > > > > 2.0, > > > > > > > > > > until the > > > > > > > > > > > smooth migration period is proposed. Before this, > > the > > > > only > > > > > > > > > > guarantees we > > > > > > > > > > > provide for Public APIs is that they'll stay > > > compatible > > > > > > until > > > > > > > > the > > > > > > > > > > next > > > > > > > > > > > major version, which led to the understanding the > > > > in-place > > > > > > > > changes > > > > > > > > > > on > > > > > > > > > > > Public APIs should be fine. > > > > > > > > > > > 2. We also cannot afford carrying the DataStream > API > > > for > > > > > > > another > > > > > > > > > > > couple of years. One of the reasons that we want > to > > > > > > refactor / > > > > > > > > > > replace > > > > > > > > > > > DataStream API is that it exposes and depends on > too > > > > many > > > > > > > > > > > Flink's internal > > > > > > > > > > > runtime implementations, which has already > limited / > > > > > blocked > > > > > > > > plans > > > > > > > > > > for > > > > > > > > > > > Flink's internal improvements for many times. Also > > due > > > > to > > > > > > the > > > > > > > > wide > > > > > > > > > > and > > > > > > > > > > > transitive dependencies on the internals, > > maintaining > > > > two > > > > > > sets > > > > > > > > of > > > > > > > > > > lower > > > > > > > > > > > APIs itself is expensive and would be better to > keep > > > it > > > > > > short. > > > > > > > > > This > > > > > > > > > > is > > > > > > > > > > > admittedly a historical design flaw which should > be > > > > > avoided > > > > > > in > > > > > > > > > > future. > > > > > > > > > > > > > > > > > > > > > > What we haven't reached consensus is about how to deal > > with > > > > the > > > > > > > > current > > > > > > > > > > > situation around the DataStream API. There are two > > options. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. We say this FLIP is enforced starting release 2.0. > > For > > > > > > current > > > > > > > > 1.x > > > > > > > > > > > APIs, we provide a migration period with best effort, > > > while > > > > > > > > allowing > > > > > > > > > > > exceptions for immediate removal in 2.0. That means > we > > > will > > > > > > still > > > > > > > > try > > > > > > > > > > with > > > > > > > > > > > best effort to get the ProcessFuncion API ready and > > > > deprecate > > > > > > the > > > > > > > > > > > DataStream API in 1.x, but will also be allowed to > > remove > > > > > > > > DataStream > > > > > > > > > > API in > > > > > > > > > > > 2.0 if it's not deprecated 2 minor releases before > the > > > > major > > > > > > > > version > > > > > > > > > > bump. > > > > > > > > > > > 2. We strictly follow the process in this FLIP, and > > will > > > > > > quickly > > > > > > > > bump > > > > > > > > > > > the major version from 2.x to 3.0 once the migration > > > period > > > > > for > > > > > > > > > > DataStream > > > > > > > > > > > API is reached. > > > > > > > > > > > > > > > > > > > > > > I'm personally in favor of option 1. As the > > > migration-period > > > > > rule > > > > > > > is > > > > > > > > > > newly > > > > > > > > > > > proposed, I think it's fair to make exceptions for > cases > > > > where > > > > > we > > > > > > > > > already > > > > > > > > > > > missed the best chance for planning the deprecation. > > > > Moreover, > > > > > I > > > > > > do > > > > > > > > > > believe > > > > > > > > > > > having a quick major version bump does confuse users. > > Yes, > > > we > > > > > can > > > > > > > > > explain > > > > > > > > > > > to users that bumping from 2.x to 3.0 does not cost > > > anything > > > > > > other > > > > > > > > than > > > > > > > > > > the > > > > > > > > > > > removal of an deprecated API. But having to explain > this > > > > itself > > > > > > is > > > > > > > an > > > > > > > > > > > indicator that it might be confusing for users. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Becket, on the other hand, prefers option 2. From my > > > > > > understanding, > > > > > > > > his > > > > > > > > > > > major point is that a quick major version bump causes > > > barely > > > > > any > > > > > > > > actual > > > > > > > > > > > lose on users, while in option 1 not providing the > > > migration > > > > > > period > > > > > > > > or > > > > > > > > > > > providing a shorter on is an actual lose on users. > > > (@Becket, > > > > > > please > > > > > > > > > > correct > > > > > > > > > > > me if I'm mistaken.) > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > And we'd like to hear more feedback from the community. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler < > > > > > > vvcep...@apache.org > > > > > > > > > > <mailto:vvcep...@apache.org>> wrote: > > > > > > > > > > > > > > > > > > > > > >> Hi Becket and Xintong, > > > > > > > > > > >> > > > > > > > > > > >> I hope you don't mind if I chime in a little. > > > > > > > > > > >> > > > > > > > > > > >> Once an API is marked Public, we're committing to > > support > > > it > > > > > > until > > > > > > > > > it's > > > > > > > > > > >> deprecated, and once it's deprecated, to leave it in > > place > > > > for > > > > > > at > > > > > > > > > least > > > > > > > > > > two > > > > > > > > > > >> minor releases and then only remove it in the > following > > > > major > > > > > > > > release. > > > > > > > > > > >> > > > > > > > > > > >> As a user, I would be dismayed to discover that the > > > project > > > > > > > > > maintainers > > > > > > > > > > >> demoted a Public API to PublicEvolving or Experimental > > in > > > > > order > > > > > > to > > > > > > > > > > violate > > > > > > > > > > >> the spirit of the deprecation period for Public APIs. > > > > > Likewise, > > > > > > > I'd > > > > > > > > > > view a > > > > > > > > > > >> rapid sequence of minor releases for the purpose of > > > dropping > > > > > > > > > deprecated > > > > > > > > > > >> APIs in the next major release as another violation of > > the > > > > > > spirit > > > > > > > of > > > > > > > > > our > > > > > > > > > > >> guarantees. I'm reminded of this xkcd: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_ > > > > > > > > > > >> > > > > > > > > > > >> I'm glad we're gaming out these situations before we > > > > institute > > > > > > > this > > > > > > > > > > FLIP, > > > > > > > > > > >> and I hope we all converge on a clear understanding of > > > what > > > > > > we're > > > > > > > > > > >> guaranteeing. From where I'm sitting, the essence of > > this > > > > FLIP > > > > > > is > > > > > > > > > > >> specifically to prevent us from doing whatever we want > > and > > > > > > instead > > > > > > > > to > > > > > > > > > > >> commit to providing a suitable notice period to users > > who > > > > rely > > > > > > on > > > > > > > > > Public > > > > > > > > > > >> APIs. > > > > > > > > > > >> > > > > > > > > > > >> It honestly shouldn't matter what the maintenance > > overhead > > > > is > > > > > > (or, > > > > > > > > > > rather, > > > > > > > > > > >> evaluating the maintenance overhead should be a > > > > consideration > > > > > > > before > > > > > > > > > we > > > > > > > > > > >> promote an API to Public to begin with). My experience > > in > > > > > other > > > > > > > > > > projects is > > > > > > > > > > >> that regretting a feature so much you want to drop it > > asap > > > > is > > > > > > > > > extremely > > > > > > > > > > >> rare, and that living with the pain for a little while > > > until > > > > > the > > > > > > > > > > >> deprecation period expires will probably make us > better > > > > > > engineers. > > > > > > > > > > >> > > > > > > > > > > >> With that all said, one thing that is not a violation > is > > > to > > > > > > > propose > > > > > > > > a > > > > > > > > > > new > > > > > > > > > > >> major release in order to drop especially burdensome > > > > > deprecated > > > > > > > APIs > > > > > > > > > > that > > > > > > > > > > >> have already enjoyed their full deprecation period. > > Those > > > > > really > > > > > > > bad > > > > > > > > > > "case > > > > > > > > > > >> A" features will be very rare. > > > > > > > > > > >> > > > > > > > > > > >> Looking over the FLIP again, I see a reference to the > > > > release > > > > > > > > > > >> documentation, but not to the build tooling support > for > > > the > > > > > > > > Deprecated > > > > > > > > > > >> annotation, so it might help to share this scenario: > The > > > way > > > > > > this > > > > > > > > will > > > > > > > > > > all > > > > > > > > > > >> come together in practice is that, by giving good > > > stability > > > > > > > > > guarantees, > > > > > > > > > > we > > > > > > > > > > >> will encourage our users to bump minor versions when > > they > > > > are > > > > > > > > > released, > > > > > > > > > > >> which means that they will see compiler warnings > > > immediately > > > > > if > > > > > > > any > > > > > > > > of > > > > > > > > > > the > > > > > > > > > > >> APIs they've used are deprecated. In fact, I'd > recommend > > > for > > > > > > them > > > > > > > to > > > > > > > > > use > > > > > > > > > > >> the `-Werror` javac flag to be sure that they don't > miss > > > > those > > > > > > > > > warnings. > > > > > > > > > > >> Given our release cadence, this means that they will > > have > > > > > about > > > > > > > six > > > > > > > > > > months > > > > > > > > > > >> to migrate off of any deprecated APIs without being > > forced > > > > to > > > > > > > avoid > > > > > > > > > > >> updates. And they really need to be able to adopt > > further > > > > > minor > > > > > > > > > updates > > > > > > > > > > in > > > > > > > > > > >> the series, since they may contain further > improvements > > > that > > > > > > > > actually > > > > > > > > > > >> facilitate migrating off of the deprecated APIs. > > > > > > > > > > >> > > > > > > > > > > >> The compiler warnings are a far more reliable > mechanism > > to > > > > > > > advertise > > > > > > > > > > >> deprecation than release notes, since users' own > builds > > > will > > > > > > > notify > > > > > > > > > them > > > > > > > > > > >> right away about only the deprecations that are > relevant > > > to > > > > > > them, > > > > > > > > but > > > > > > > > > it > > > > > > > > > > >> does rely on users feeling ok to bump up minor > versions > > > in a > > > > > > > timely > > > > > > > > > > >> fashion. If we consistently burn them by not observing > > the > > > > > > > stability > > > > > > > > > > >> guarantees, they'll try to avoid updates instead, and > > the > > > > > whole > > > > > > > > thing > > > > > > > > > > falls > > > > > > > > > > >> apart. > > > > > > > > > > >> > > > > > > > > > > >> Thanks again, > > > > > > > > > > >> -John > > > > > > > > > > >> > > > > > > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote: > > > > > > > > > > >>> Hi Xintong, > > > > > > > > > > >>> > > > > > > > > > > >>> Let's compare the following cases: > > > > > > > > > > >>> > > > > > > > > > > >>> A. If the maintenance overhead of the deprecated API > is > > > > high, > > > > > > and > > > > > > > > we > > > > > > > > > > want > > > > > > > > > > >>> to remove it after two minor releases. Then there are > > two > > > > > > > options: > > > > > > > > > > >>> A1: Demote the API in the major version bump and > > > remove > > > > > the > > > > > > > code > > > > > > > > > > >> with a > > > > > > > > > > >>> minor version bump. > > > > > > > > > > >>> A2: Do exactly the same as 1-A, except that when > > > > removing > > > > > > the > > > > > > > > > code, > > > > > > > > > > >>> bump up the major version. So this might be a short > > major > > > > > > > version. > > > > > > > > > > >>> > > > > > > > > > > >>> B. If the maintenance overhead of the deprecated API > is > > > not > > > > > > high, > > > > > > > > > > >> therefore > > > > > > > > > > >>> keeping it for a long time is affordable. There are > > also > > > > two > > > > > > > > options: > > > > > > > > > > >>> B1: Same as A-1, demote the API in the major > version > > > > bump > > > > > > and > > > > > > > > > remove > > > > > > > > > > >>> the code with a minor version bump. > > > > > > > > > > >>> B2: Keep the API for all the minor versions in the > > > major > > > > > > > > version, > > > > > > > > > > and > > > > > > > > > > >>> only remove the code in the next major version. > > > > > > > > > > >>> > > > > > > > > > > >>> For case B, do we agree that B2 is the way to go? > > > > > > > > > > >>> > > > > > > > > > > >>> For case A: > > > > > > > > > > >>> The following stays the same for A1 and A2: > > > > > > > > > > >>> - users will lose the API after two minor leases. So > > the > > > > > > > migration > > > > > > > > > > >> period > > > > > > > > > > >>> is the same for A-1 and A-2. > > > > > > > > > > >>> - A1 and A2 will have exactly the same code after > the > > > > > removal > > > > > > of > > > > > > > > > > >>> deprecated API, the only difference is versioning. > > > > > > > > > > >>> - To move forward, users need to move to the next > > minor > > > > > > version > > > > > > > in > > > > > > > > > A1, > > > > > > > > > > >> or > > > > > > > > > > >>> to the next major version in A2. Because the code are > > the > > > > > same, > > > > > > > the > > > > > > > > > > >> actual > > > > > > > > > > >>> effort to upgrade is the same for A1 and A2. > > > > > > > > > > >>> > > > > > > > > > > >>> The differences between A-1 and A-2 are: > > > > > > > > > > >>> - A1 allows keeping the major version release > cadence. > > > A-2 > > > > > > will > > > > > > > > > have a > > > > > > > > > > >>> short major version release. > > > > > > > > > > >>> - A1 breaks the well understood API semantic, while > > A-2 > > > > does > > > > > > > not. > > > > > > > > > > >>> > > > > > > > > > > >>> From what I see, since there is no well establish > > > standard > > > > > > > > regarding > > > > > > > > > > how > > > > > > > > > > >>> long a major version should be released, A short > major > > > > > version > > > > > > > > > release > > > > > > > > > > is > > > > > > > > > > >>> potential and emotional. It is not ideal but does not > > > have > > > > > > > material > > > > > > > > > > >>> downsides compared with A1. I did not hear anyone > > > > complaining > > > > > > > about > > > > > > > > > > Kafka > > > > > > > > > > >>> only has two 1.x release. However, A1 actually breaks > > the > > > > > well > > > > > > > > > > understood > > > > > > > > > > >>> API semantic, which has more material impact. > > > > > > > > > > >>> > > > > > > > > > > >>> Also, I'd imagine 90% or more of the Public APIs > should > > > > fall > > > > > > into > > > > > > > > > case > > > > > > > > > > B. > > > > > > > > > > >>> So, short major versions should be very occasional. > I'd > > > be > > > > > very > > > > > > > > > > concerned > > > > > > > > > > >>> if the reason we choose A1 is simply because we > cannot > > > > afford > > > > > > > > > > >> maintaining a > > > > > > > > > > >>> bunch of deprecated APIs until the next major > version. > > > This > > > > > > > > indicates > > > > > > > > > > >> that > > > > > > > > > > >>> the actual problem we need to solve is to lower the > > > > > maintenance > > > > > > > > > > overhead > > > > > > > > > > >> of > > > > > > > > > > >>> deprecated APIs, so that we are comfortable to keep > > them > > > > > > longer. > > > > > > > As > > > > > > > > > > John > > > > > > > > > > >>> and I mentioned earlier, there are ways to achieve > this > > > and > > > > > we > > > > > > > need > > > > > > > > > to > > > > > > > > > > >>> learn how to do it in Flink. Otherwise, our > discussion > > > > about > > > > > > > > > versioning > > > > > > > > > > >>> here does not bring much value, because we will end > up > > > > with a > > > > > > > bunch > > > > > > > > > of > > > > > > > > > > >>> short-lived APIs which upset our users, no matter how > > we > > > > > > version > > > > > > > > the > > > > > > > > > > >>> releases. > > > > > > > > > > >>> > > > > > > > > > > >>> So, if there are concrete examples that you think > will > > > > block > > > > > us > > > > > > > > from > > > > > > > > > > >>> keeping API stability with affordable cost, let's > take > > a > > > > look > > > > > > > > > together > > > > > > > > > > >> and > > > > > > > > > > >>> see if that can be improved. > > > > > > > > > > >>> > > > > > > > > > > >>> Thanks, > > > > > > > > > > >>> > > > > > > > > > > >>> Jiangjie (Becket) Qin > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> From what I see, > > > > > > > > > > >>> > > > > > > > > > > >>> > > > > > > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song < > > > > > > > > tonysong...@gmail.com > > > > > > > > > > <mailto:tonysong...@gmail.com>> > > > > > > > > > > >> wrote: > > > > > > > > > > >>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> The part I don't understand is if we are willing to > > > have > > > > a > > > > > > > > > migration > > > > > > > > > > >>>>> period, and do a minor version bump to remove an > API, > > > > what > > > > > do > > > > > > > we > > > > > > > > > > >> lose to > > > > > > > > > > >>>> do > > > > > > > > > > >>>>> a major version bump instead, so we don't break the > > > > common > > > > > > > > > versioning > > > > > > > > > > >>>>> semantic? > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> I think we are talking about the cases where a major > > > > version > > > > > > > bump > > > > > > > > > > >> happens > > > > > > > > > > >>>> before a deprecated Public API reaches its migration > > > > period. > > > > > > So > > > > > > > > > > >> removing > > > > > > > > > > >>>> the API with another major version bump means we > have > > > two > > > > > > > > > consecutive > > > > > > > > > > >> major > > > > > > > > > > >>>> versions in a very short time. And as previously > > > > mentioned, > > > > > > > having > > > > > > > > > > >> frequent > > > > > > > > > > >>>> major version bumps would weaken the value of the > > > > commitment > > > > > > > > "Public > > > > > > > > > > >> API > > > > > > > > > > >>>> stay compatible within a major version". I think > users > > > > also > > > > > > have > > > > > > > > > > >>>> expectations about how long a major version should > > live, > > > > > which > > > > > > > > > should > > > > > > > > > > >> be at > > > > > > > > > > >>>> least a couple of years rather than 1-2 minor > > releases. > > > > > > > > > > >>>> > > > > > > > > > > >>>> This is another option, but I think it is very > likely > > > more > > > > > > > > expensive > > > > > > > > > > >> than > > > > > > > > > > >>>>> simply bumping the major version. And I imagine > there > > > > will > > > > > be > > > > > > > > > > >> questions > > > > > > > > > > >>>>> like "why this feature is in 1.20, but does not > exist > > > in > > > > > > 2.0"? > > > > > > > > > > >>>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> Yes, it's more expensive than simply bumping the > major > > > > > > version, > > > > > > > > but > > > > > > > > > > >> IMHO > > > > > > > > > > >>>> it's probably cheaper than carrying the API until > the > > > next > > > > > > major > > > > > > > > > > >> version if > > > > > > > > > > >>>> we don't bump the major version very soon. And see > my > > > > reply > > > > > > > above > > > > > > > > on > > > > > > > > > > >> why > > > > > > > > > > >>>> not bumping immediately. > > > > > > > > > > >>>> > > > > > > > > > > >>>> Best, > > > > > > > > > > >>>> > > > > > > > > > > >>>> Xintong > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> > > > > > > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin < > > > > > > > becket....@gmail.com > > > > > > > > > > <mailto:becket....@gmail.com>> > > > > > > > > > > >> wrote: > > > > > > > > > > >>>> > > > > > > > > > > >>>>> Hi Xintong, > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Please see the replies below. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> I see your point, that users would feel surprised > if > > > they > > > > > > find > > > > > > > > > > >> things no > > > > > > > > > > >>>>>> longer work when upgrading to another 2.x minor > > > release. > > > > > > > > However, > > > > > > > > > > >> I'd > > > > > > > > > > >>>>> like > > > > > > > > > > >>>>>> to point out that PublicEvolving APIs would have > the > > > > > similar > > > > > > > > > > >> problem > > > > > > > > > > >>>>>> anyway. So the question is, how do we catch users' > > > > > attention > > > > > > > and > > > > > > > > > > >> make > > > > > > > > > > >>>>> sure > > > > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no > > > longer > > > > > be > > > > > > > > Public > > > > > > > > > > >> in > > > > > > > > > > >>>>> 2.0. > > > > > > > > > > >>>>>> There're various ways to do that, e.g., release > > notes, > > > > > > > warnings > > > > > > > > in > > > > > > > > > > >>>> logs, > > > > > > > > > > >>>>>> etc. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> First of all, I am not a fan of removing > > PublicEvolving > > > > > APIs > > > > > > in > > > > > > > > > minor > > > > > > > > > > >>>>> version changes. Personally speaking, I want them > to > > be > > > > > also > > > > > > > > > removed > > > > > > > > > > >> in > > > > > > > > > > >>>>> major version changes. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Empirically, I rarely see projects with complex API > > > > > semantic > > > > > > > work > > > > > > > > > > >> well. > > > > > > > > > > >>>>> Many, if not most, users don't read docs / release > > > notes > > > > / > > > > > > > > warnings > > > > > > > > > > >> in > > > > > > > > > > >>>>> logs. I don't think that is their fault. Typically > an > > > > > > > application > > > > > > > > > > >>>> developer > > > > > > > > > > >>>>> will have to deal with dozens of libraries > maintained > > > by > > > > > > > various > > > > > > > > > > >>>>> communities / groups. It is just too difficult for > > them > > > > to > > > > > > keep > > > > > > > > > > >> track of > > > > > > > > > > >>>>> all the specific semantics each project puts there. > > If > > > > you > > > > > > > think > > > > > > > > > > >> about > > > > > > > > > > >>>> it, > > > > > > > > > > >>>>> how many of Flink developers read all the release > > notes > > > > of > > > > > > > Guava, > > > > > > > > > > >> Apache > > > > > > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you > > upgrade a > > > > > > version > > > > > > > > of > > > > > > > > > > >> them? > > > > > > > > > > >>>>> One would probably try bumping the dependency > version > > > and > > > > > see > > > > > > > if > > > > > > > > > > >> there is > > > > > > > > > > >>>>> an exception and solve them case by case. And if it > > is > > > a > > > > > > minor > > > > > > > > > > >> version > > > > > > > > > > >>>>> bump, one probably does not expect an exception at > > all. > > > > > > Another > > > > > > > > > > >> example > > > > > > > > > > >>>> is > > > > > > > > > > >>>>> that in Flink we still have so many usage of > > deprecated > > > > > > methods > > > > > > > > all > > > > > > > > > > >> over > > > > > > > > > > >>>>> the place, and I strongly doubt everyone knows when > > the > > > > > > source > > > > > > > > code > > > > > > > > > > >> of > > > > > > > > > > >>>>> these methods are deprecated and when they should > be > > > > > removed. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> So, the most important principle of API is simple > and > > > > > > > intuitive. > > > > > > > > > The > > > > > > > > > > >>>>> versioning semantic is a simple and universally > > > accepted > > > > > API > > > > > > > > > > >> stability > > > > > > > > > > >>>>> standard. If we ourselves as Flink developers are > > > relying > > > > > on > > > > > > > this > > > > > > > > > > >> for our > > > > > > > > > > >>>>> own dependencies. I don't think we can expect more > > from > > > > our > > > > > > > > users. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Another possible alternative: whenever there's a > > > > deprecated > > > > > > > > Public > > > > > > > > > > >> API > > > > > > > > > > >>>> that > > > > > > > > > > >>>>>> reaches a major version bump before the migration > > > > period, > > > > > > and > > > > > > > we > > > > > > > > > > >> also > > > > > > > > > > >>>>> don't > > > > > > > > > > >>>>>> want to carry it for all the next major release > > > series, > > > > we > > > > > > may > > > > > > > > > > >> consider > > > > > > > > > > >>>>>> releasing more minor releases for the previous > major > > > > > version > > > > > > > > after > > > > > > > > > > >> the > > > > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, > and > > > > then > > > > > we > > > > > > > > bump > > > > > > > > > > >> to > > > > > > > > > > >>>> 2.0, > > > > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That > should > > > > > provide > > > > > > > > users > > > > > > > > > > >>>> another > > > > > > > > > > >>>>>> choice rather than upgrading to 2.0, while > > satisfying > > > > the > > > > > > > > > > >>>> 2-minor-release > > > > > > > > > > >>>>>> migration period. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> This is another option, but I think it is very > likely > > > > more > > > > > > > > > expensive > > > > > > > > > > >> than > > > > > > > > > > >>>>> simply bumping the major version. And I imagine > there > > > > will > > > > > be > > > > > > > > > > >> questions > > > > > > > > > > >>>>> like "why this feature is in 1.20, but does not > exist > > > in > > > > > > 2.0"? > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> I think my major point is, we should not carry APIs > > > > > > deprecated > > > > > > > > in a > > > > > > > > > > >>>>>> previous major version along all the next major > > > version > > > > > > > series. > > > > > > > > > I'd > > > > > > > > > > >>>> like > > > > > > > > > > >>>>> to > > > > > > > > > > >>>>>> try giving users more commitments, i.e. the > > migration > > > > > > period, > > > > > > > as > > > > > > > > > > >> long > > > > > > > > > > >>>> as > > > > > > > > > > >>>>> it > > > > > > > > > > >>>>>> does not prevent us from making breaking changes. > If > > > it > > > > > > > doesn't > > > > > > > > > > >> work, > > > > > > > > > > >>>> I'd > > > > > > > > > > >>>>>> be in favor of not providing the migration period, > > but > > > > > > > fallback > > > > > > > > to > > > > > > > > > > >> only > > > > > > > > > > >>>>>> guarantee the compatibility within the major > > version. > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> The part I don't understand is if we are willing to > > > have > > > > a > > > > > > > > > migration > > > > > > > > > > >>>>> period, and do a minor version bump to remove an > API, > > > > what > > > > > do > > > > > > > we > > > > > > > > > > >> lose to > > > > > > > > > > >>>> do > > > > > > > > > > >>>>> a major version bump instead, so we don't break the > > > > common > > > > > > > > > versioning > > > > > > > > > > >>>>> semantic? > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Thanks, > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> Jiangjie (Becket) Qin > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> > > > > > > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song < > > > > > > > > > tonysong...@gmail.com > > > > > > > > > > <mailto:tonysong...@gmail.com>> > > > > > > > > > > >>>>> wrote: > > > > > > > > > > >>>>> > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> As an end user who only uses Public APIs, if I > > don't > > > > > change > > > > > > > my > > > > > > > > > > >> code > > > > > > > > > > >>>> at > > > > > > > > > > >>>>>>> all, my expectation is the following: > > > > > > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > > > > > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an > > > issue, I > > > > > am > > > > > > > fine > > > > > > > > > > >> with > > > > > > > > > > >>>>> all > > > > > > > > > > >>>>>>> the 2.x versions. > > > > > > > > > > >>>>>>> Actually I think there are some dependency > version > > > > > > resolution > > > > > > > > > > >>>> policies > > > > > > > > > > >>>>>> out > > > > > > > > > > >>>>>>> there which picks the highest minor version when > > the > > > > > > > > dependencies > > > > > > > > > > >>>> pull > > > > > > > > > > >>>>> in > > > > > > > > > > >>>>>>> multiple minor versions of the same jar, which > may > > be > > > > > > broken > > > > > > > if > > > > > > > > > > >> we > > > > > > > > > > >>>>> remove > > > > > > > > > > >>>>>>> the API in minor releases. > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> I see your point, that users would feel surprised > if > > > > they > > > > > > find > > > > > > > > > > >> things > > > > > > > > > > >>>> no > > > > > > > > > > >>>>>> longer work when upgrading to another 2.x minor > > > release. > > > > > > > > However, > > > > > > > > > > >> I'd > > > > > > > > > > >>>>> like > > > > > > > > > > >>>>>> to point out that PublicEvolving APIs would have > the > > > > > similar > > > > > > > > > > >> problem > > > > > > > > > > >>>>>> anyway. So the question is, how do we catch users' > > > > > attention > > > > > > > and > > > > > > > > > > >> make > > > > > > > > > > >>>>> sure > > > > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no > > > longer > > > > > be > > > > > > > > Public > > > > > > > > > > >> in > > > > > > > > > > >>>>> 2.0. > > > > > > > > > > >>>>>> There're various ways to do that, e.g., release > > notes, > > > > > > > warnings > > > > > > > > in > > > > > > > > > > >>>> logs, > > > > > > > > > > >>>>>> etc. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Another possible alternative: whenever there's a > > > > > deprecated > > > > > > > > Public > > > > > > > > > > >> API > > > > > > > > > > >>>>> that > > > > > > > > > > >>>>>> reaches a major version bump before the migration > > > > period, > > > > > > and > > > > > > > we > > > > > > > > > > >> also > > > > > > > > > > >>>>> don't > > > > > > > > > > >>>>>> want to carry it for all the next major release > > > series, > > > > we > > > > > > may > > > > > > > > > > >> consider > > > > > > > > > > >>>>>> releasing more minor releases for the previous > major > > > > > version > > > > > > > > after > > > > > > > > > > >> the > > > > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, > and > > > > then > > > > > we > > > > > > > > bump > > > > > > > > > > >> to > > > > > > > > > > >>>> 2.0, > > > > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That > should > > > > > provide > > > > > > > > users > > > > > > > > > > >>>> another > > > > > > > > > > >>>>>> choice rather than upgrading to 2.0, while > > satisfying > > > > the > > > > > > > > > > >>>> 2-minor-release > > > > > > > > > > >>>>>> migration period. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> I think my major point is, we should not carry > APIs > > > > > > deprecated > > > > > > > > in > > > > > > > > > a > > > > > > > > > > >>>>>> previous major version along all the next major > > > version > > > > > > > series. > > > > > > > > > I'd > > > > > > > > > > >>>> like > > > > > > > > > > >>>>> to > > > > > > > > > > >>>>>> try giving users more commitments, i.e. the > > migration > > > > > > period, > > > > > > > as > > > > > > > > > > >> long > > > > > > > > > > >>>> as > > > > > > > > > > >>>>> it > > > > > > > > > > >>>>>> does not prevent us from making breaking changes. > If > > > it > > > > > > > doesn't > > > > > > > > > > >> work, > > > > > > > > > > >>>> I'd > > > > > > > > > > >>>>>> be in favor of not providing the migration period, > > but > > > > > > > fallback > > > > > > > > to > > > > > > > > > > >> only > > > > > > > > > > >>>>>> guarantee the compatibility within the major > > version. > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Best, > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> Xintong > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler < > > > > > > > > > vvcep...@apache.org > > > > > > > > > > <mailto:vvcep...@apache.org> > > > > > > > > > > >>> > > > > > > > > > > >>>>> wrote: > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>>> Hi Becket, > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> Thanks for the reply! I’d like to continue the > > > > > conversation > > > > > > > > about > > > > > > > > > > >>>>>>> compatibility outside of this FLIP thread, but > for > > > > now, I > > > > > > can > > > > > > > > > > >> accept > > > > > > > > > > >>>>> your > > > > > > > > > > >>>>>>> decision. It’s certainly an improvement. > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> Thanks again, > > > > > > > > > > >>>>>>> John > > > > > > > > > > >>>>>>> > > > > > > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote: > > > > > > > > > > >>>>>>>> Hi John, > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Completely agree with all you said. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in > > > major > > > > > > > > releases > > > > > > > > > > >>>>> across > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>> board? I understand that Experimental and > > > > > PublicEvolving > > > > > > > APIs > > > > > > > > > > >> are > > > > > > > > > > >>>> by > > > > > > > > > > >>>>>>>>> definition less stable, but it seems like this > > > should > > > > > be > > > > > > > > > > >> reflected > > > > > > > > > > >>>>> in > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we > > > must > > > > > keep > > > > > > > > them > > > > > > > > > > >>>>> around > > > > > > > > > > >>>>>>> for > > > > > > > > > > >>>>>>>>> at least zero or one minor release, not that we > > can > > > > > drop > > > > > > > them > > > > > > > > > > >> in a > > > > > > > > > > >>>>>>> minor or > > > > > > > > > > >>>>>>>>> patch release. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Personally speaking, I would love to do this, > for > > > > > exactly > > > > > > > the > > > > > > > > > > >>>> reason > > > > > > > > > > >>>>>> you > > > > > > > > > > >>>>>>>> mentioned. However, I did not propose this due > to > > > the > > > > > > > > following > > > > > > > > > > >>>>>> reasons: > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> 1. I am hesitating a little bit about changing > the > > > > > > accepted > > > > > > > > > > >> FLIPs > > > > > > > > > > >>>> too > > > > > > > > > > >>>>>>> soon. > > > > > > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our > > > > > > development. > > > > > > > At > > > > > > > > > > >> this > > > > > > > > > > >>>>>>> point, > > > > > > > > > > >>>>>>>> Flink still lacks some design / routines to > > support > > > > good > > > > > > API > > > > > > > > > > >>>>>>> evolvability / > > > > > > > > > > >>>>>>>> extensibility. Just like you said, it takes some > > > time > > > > to > > > > > > be > > > > > > > > > > >> good at > > > > > > > > > > >>>>>> this. > > > > > > > > > > >>>>>>>> In this case, my concern is that only removing > > > > > > Experimental > > > > > > > / > > > > > > > > > > >>>>>>>> PublicEvolving APIs in major version changes may > > > > result > > > > > in > > > > > > > too > > > > > > > > > > >> much > > > > > > > > > > >>>>>>>> overhead and dramatically slow down the > > development > > > of > > > > > > > Flink. > > > > > > > > > > >> So, I > > > > > > > > > > >>>>> was > > > > > > > > > > >>>>>>>> thinking that we can start with the current > > status. > > > > > > > Hopefully > > > > > > > > > > >> after > > > > > > > > > > >>>>> we > > > > > > > > > > >>>>>>> are > > > > > > > > > > >>>>>>>> more comfortable with the maintenance overhead > of > > > > > > deprecated > > > > > > > > > > >> APIs, > > > > > > > > > > >>>> we > > > > > > > > > > >>>>>> can > > > > > > > > > > >>>>>>>> then have a stronger guarantee for Experimental > / > > > > > > > > > > >> PublicEvolving > > > > > > > > > > >>>>> APIs. > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Thanks, > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> Jiangjie (Becket) Qin > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler < > > > > > > > > > > >> vvcep...@apache.org <mailto:vvcep...@apache.org>> > > > > > > > > > > >>>>>>> wrote: > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>>> Hi Becket, > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation > > process > > > is > > > > > > > really > > > > > > > > > > >>>>>> important. > > > > > > > > > > >>>>>>> I > > > > > > > > > > >>>>>>>>> understand some people’s concerns about the > > > > additional > > > > > > > burden > > > > > > > > > > >> for > > > > > > > > > > >>>>>>> project > > > > > > > > > > >>>>>>>>> maintainers, but my personal experience with > > Kafka > > > > has > > > > > > been > > > > > > > > > > >> that > > > > > > > > > > >>>>> it’s > > > > > > > > > > >>>>>>> very > > > > > > > > > > >>>>>>>>> liveable and that it’s well worth the benefit > to > > > > users. > > > > > > In > > > > > > > > > > >> fact, > > > > > > > > > > >>>>> users > > > > > > > > > > >>>>>>>>> being able to confidently upgrade is also a > > benefit > > > > to > > > > > > > > > > >>>> maintainers, > > > > > > > > > > >>>>> as > > > > > > > > > > >>>>>>> we > > > > > > > > > > >>>>>>>>> will get fewer questions from people stuck on > > very > > > > old > > > > > > > > > > >> versions. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> One question: > > > > > > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs > in > > > > major > > > > > > > > > > >> releases > > > > > > > > > > >>>>> across > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>> board? I understand that Experimental and > > > > > PublicEvolving > > > > > > > APIs > > > > > > > > > > >> are > > > > > > > > > > >>>> by > > > > > > > > > > >>>>>>>>> definition less stable, but it seems like this > > > should > > > > > be > > > > > > > > > > >> reflected > > > > > > > > > > >>>>> in > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we > > > must > > > > > keep > > > > > > > > them > > > > > > > > > > >>>>> around > > > > > > > > > > >>>>>>> for > > > > > > > > > > >>>>>>>>> at least zero or one minor release, not that we > > can > > > > > drop > > > > > > > them > > > > > > > > > > >> in a > > > > > > > > > > >>>>>>> minor or > > > > > > > > > > >>>>>>>>> patch release. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> The advantage of forbidding the removal of any > > API > > > in > > > > > > minor > > > > > > > > or > > > > > > > > > > >>>> patch > > > > > > > > > > >>>>>>>>> releases is that users will get a strong > > guarantee > > > > that > > > > > > > they > > > > > > > > > > >> can > > > > > > > > > > >>>>> bump > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>> minor or patch version and still be able to > > > compile, > > > > or > > > > > > > even > > > > > > > > > > >> just > > > > > > > > > > >>>>>>> re-link > > > > > > > > > > >>>>>>>>> and know that they won’t face “MethodDef” > > > exceptions > > > > at > > > > > > run > > > > > > > > > > >> time. > > > > > > > > > > >>>>> This > > > > > > > > > > >>>>>>> is a > > > > > > > > > > >>>>>>>>> binary guarantee: if we allow removing even > > > > > Experimental > > > > > > > > APIs > > > > > > > > > > >>>>> outside > > > > > > > > > > >>>>>>> of > > > > > > > > > > >>>>>>>>> major releases, users can no longer confidently > > > > > upgrade. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a > couple > > > of > > > > > > > points: > > > > > > > > > > >>>>>>>>> * I’d use the official Deprecated annotation > > > instead > > > > of > > > > > > > > > > >>>> introducing > > > > > > > > > > >>>>>> our > > > > > > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is > > well > > > > > > > > integrated > > > > > > > > > > >>>> into > > > > > > > > > > >>>>>>> build > > > > > > > > > > >>>>>>>>> tools and IDEs. > > > > > > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in > > this > > > > > FLIP; > > > > > > > it > > > > > > > > > > >> seems > > > > > > > > > > >>>>>>>>> orthogonal, and something that should probably > be > > > > taken > > > > > > > > > > >>>> case-by-case > > > > > > > > > > >>>>>>>>> anyway. > > > > > > > > > > >>>>>>>>> * Aside from deprecation and removal, there > have > > > been > > > > > > some > > > > > > > > > > >>>>> discussions > > > > > > > > > > >>>>>>>>> about how to evolve APIs and behavior in > > compatible > > > > > ways. > > > > > > > > > > >> This is > > > > > > > > > > >>>>>>> somewhat > > > > > > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with > it > > > > > before, > > > > > > > > it’ll > > > > > > > > > > >>>> take > > > > > > > > > > >>>>>> some > > > > > > > > > > >>>>>>>>> time to become good at it. I feel like this > topic > > > > > should > > > > > > > also > > > > > > > > > > >> be > > > > > > > > > > >>>>>>> orthogonal > > > > > > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be > to > > > > > adopt a > > > > > > > > > > >> simple > > > > > > > > > > >>>>>> policy > > > > > > > > > > >>>>>>> not > > > > > > > > > > >>>>>>>>> to break existing user programs, and leave the > > > “how” > > > > up > > > > > > to > > > > > > > > > > >>>>>> implementers > > > > > > > > > > >>>>>>> and > > > > > > > > > > >>>>>>>>> reviewers. > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> Thanks again, > > > > > > > > > > >>>>>>>>> John > > > > > > > > > > >>>>>>>>> > > > > > > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote: > > > > > > > > > > >>>>>>>>>> Hi All, > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by > > Xintong > > > > is > > > > > a > > > > > > > > > > >> great > > > > > > > > > > >>>>> idea. > > > > > > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, > > i.e. > > > > > > > > > > >>>> @PublicEvolving > > > > > > > > > > >>>>>> --(2 > > > > > > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1 > > > major > > > > > > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a > > lot > > > of > > > > > > > > > > >> flexibility > > > > > > > > > > >>>>>>> without > > > > > > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are > > > allowed > > > > to > > > > > > > > > > >> change > > > > > > > > > > >>>>>> between > > > > > > > > > > >>>>>>>>> major > > > > > > > > > > >>>>>>>>>> releases. Changing annotations is acceptable > and > > > > > > provides > > > > > > > > > > >>>>> additional > > > > > > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the > APIs > > > > > > themself > > > > > > > > > > >> are > > > > > > > > > > >>>> not > > > > > > > > > > >>>>>>>>> changed. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> I had similar thoughts when I was facing those > > > > > issues. I > > > > > > > > > > >> want to > > > > > > > > > > >>>>>> move > > > > > > > > > > >>>>>>> one > > > > > > > > > > >>>>>>>>>> step further and suggest introducing one more > > > > > annotation > > > > > > > > > > >>>> @Retired. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Not like the @PublicRetired which is a > > compromise > > > of > > > > > > > > > > >> downgrading > > > > > > > > > > >>>>>>> @Public > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my > > > reply, > > > > > > Java > > > > > > > > > > >>>> standard > > > > > > > > > > >>>>>>>>>> @deprecated should be used in the early stage > of > > > the > > > > > > > > > > >> deprecation > > > > > > > > > > >>>>>>> process > > > > > > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since > > > Java > > > > > does > > > > > > > not > > > > > > > > > > >>>> allow > > > > > > > > > > >>>>>> us > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>> extend annotation, I think it would be > feasible > > to > > > > > have > > > > > > > the > > > > > > > > > > >> new > > > > > > > > > > >>>>>>> @Retired > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > >>>>>>>>>> help us monitor and manage the deprecation > > > process, > > > > > > house > > > > > > > > > > >>>>> cleaning, > > > > > > > > > > >>>>>>> etc. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Some ideas could be(open for discussion): > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> @Retired: > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> 1. There must be a replacement with > > functionality > > > > > > > > > > >> compatibility > > > > > > > > > > >>>>>> before > > > > > > > > > > >>>>>>>>> APIs > > > > > > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and > VOTE > > > > > > processes > > > > > > > > > > >> on > > > > > > > > > > >>>> the > > > > > > > > > > >>>>> ML > > > > > > > > > > >>>>>>> are > > > > > > > > > > >>>>>>>>>> mandatory (a FLIP is recommended). > > > > > > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed > > after 1 > > > > > minor > > > > > > > > > > >> release > > > > > > > > > > >>>>>>> sharply > > > > > > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further > check > > > > > whether > > > > > > > it > > > > > > > > > > >> is > > > > > > > > > > >>>>>>> possible). > > > > > > > > > > >>>>>>>>>> Devs who marked them as @Retired are > responsible > > > to > > > > > > remove > > > > > > > > > > >> them. > > > > > > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and > @PublicEvolving > > -> > > > > > > > @Retired > > > > > > > > > > >> are > > > > > > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and > > > @Internal > > > > > -> > > > > > > > > > > >> @Retired > > > > > > > > > > >>>>>> could > > > > > > > > > > >>>>>>>>> also > > > > > > > > > > >>>>>>>>>> be used if it can increase user-friendliness > or > > > > > > > > > > >>>> dev-friendliness, > > > > > > > > > > >>>>>> but > > > > > > > > > > >>>>>>> not > > > > > > > > > > >>>>>>>>>> mandatory. > > > > > > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired > to > > > > > support > > > > > > > the > > > > > > > > > > >>>>>>> deprecation > > > > > > > > > > >>>>>>>>>> process management. Further extension is > > possible, > > > > > since > > > > > > > the > > > > > > > > > > >>>>>>> annotation > > > > > > > > > > >>>>>>>>> is > > > > > > > > > > >>>>>>>>>> built by us. > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> Best regards, > > > > > > > > > > >>>>>>>>>> Jing > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin < > > > > > > > > > > >>>> becket....@gmail.com <mailto:becket....@gmail.com> > > > > > > > > > > >>>>>> > > > > > > > > > > >>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Hi Xintong, > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the > > > replies > > > > > > inline > > > > > > > > > > >>>> below. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting > a > > > > Public > > > > > > API > > > > > > > > > > >> is > > > > > > > > > > >>>>> also a > > > > > > > > > > >>>>>>>>> kind of > > > > > > > > > > >>>>>>>>>>>> such change, just like removing one, which > can > > > > only > > > > > > > > > > >> happen > > > > > > > > > > >>>> with > > > > > > > > > > >>>>>>> major > > > > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow > > > demoting > > > > > > > Public > > > > > > > > > > >>>> APIs > > > > > > > > > > >>>>>>>>> anytime, > > > > > > > > > > >>>>>>>>>>> but > > > > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen > > > before > > > > > > > > > > >> reaching > > > > > > > > > > >>>> the > > > > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, > > > > demoting > > > > > > > > > > >> would > > > > > > > > > > >>>> be a > > > > > > > > > > >>>>>>> weaker > > > > > > > > > > >>>>>>>>>>>> change compared to removing the API > > immediately > > > > upon > > > > > > > > > > >> major > > > > > > > > > > >>>>>> version > > > > > > > > > > >>>>>>>>> bumps, > > > > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the > > > > > > > 2-minor-release > > > > > > > > > > >>>>>> migration > > > > > > > > > > >>>>>>>>>>> period. > > > > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` -> > > > > > `@PublicEvolving` > > > > > > > > > > >> sounds > > > > > > > > > > >>>>>>> against > > > > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new > annotation > > > if > > > > > > > > > > >> necessary, > > > > > > > > > > >>>>>> e.g., > > > > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if > I > > > > don't > > > > > > > > > > >> change my > > > > > > > > > > >>>>> code > > > > > > > > > > >>>>>>> at > > > > > > > > > > >>>>>>>>> all, > > > > > > > > > > >>>>>>>>>>> my expectation is the following: > > > > > > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues. > > > > > > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without > an > > > > > issue, I > > > > > > > am > > > > > > > > > > >> fine > > > > > > > > > > >>>>>> with > > > > > > > > > > >>>>>>> all > > > > > > > > > > >>>>>>>>>>> the 2.x versions. > > > > > > > > > > >>>>>>>>>>> Actually I think there are some dependency > > > version > > > > > > > > > > >> resolution > > > > > > > > > > >>>>>>> policies > > > > > > > > > > >>>>>>>>> out > > > > > > > > > > >>>>>>>>>>> there which picks the highest minor version > > when > > > > the > > > > > > > > > > >>>> dependencies > > > > > > > > > > >>>>>>> pull > > > > > > > > > > >>>>>>>>> in > > > > > > > > > > >>>>>>>>>>> multiple minor versions of the same jar, > which > > > may > > > > be > > > > > > > > > > >> broken if > > > > > > > > > > >>>>> we > > > > > > > > > > >>>>>>>>> remove > > > > > > > > > > >>>>>>>>>>> the API in minor releases. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely > > > > "legal" > > > > > > > that > > > > > > > > > > >> we > > > > > > > > > > >>>>> bump > > > > > > > > > > >>>>>> up > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is > > > > needed. > > > > > > > > > > >> However, > > > > > > > > > > >>>>> this > > > > > > > > > > >>>>>>> also > > > > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that > > public > > > > APIs > > > > > > > will > > > > > > > > > > >>>> stay > > > > > > > > > > >>>>>>> stable > > > > > > > > > > >>>>>>>>>>>> within the major release series, as the > series > > > can > > > > > end > > > > > > > > > > >>>> anytime. > > > > > > > > > > >>>>>>> IMHO, > > > > > > > > > > >>>>>>>>>>> short > > > > > > > > > > >>>>>>>>>>>> major release series are not something "make > > the > > > > end > > > > > > > > > > >> users > > > > > > > > > > >>>>>> happy", > > > > > > > > > > >>>>>>> but > > > > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to > > > make > > > > > > > > > > >> frequent > > > > > > > > > > >>>>>> breaking > > > > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting > > > > > approach, > > > > > > we > > > > > > > > > > >> can > > > > > > > > > > >>>>>> still > > > > > > > > > > >>>>>>>>> have > > > > > > > > > > >>>>>>>>>>>> longer major release series, while only > > allowing > > > > > > Public > > > > > > > > > > >> APIs > > > > > > > > > > >>>>>>>>> deprecated > > > > > > > > > > >>>>>>>>>>> at > > > > > > > > > > >>>>>>>>>>>> the end of the previous major version to be > > > > removed > > > > > in > > > > > > > > > > >> the > > > > > > > > > > >>>> next > > > > > > > > > > >>>>>>> major > > > > > > > > > > >>>>>>>>>>>> version. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> I totally agree that frequent major version > > bumps > > > > are > > > > > > not > > > > > > > > > > >>>> ideal, > > > > > > > > > > >>>>>> but > > > > > > > > > > >>>>>>>>> here > > > > > > > > > > >>>>>>>>>>> we are comparing it with a minor version bump > > > which > > > > > > > > > > >> removes a > > > > > > > > > > >>>>>> Public > > > > > > > > > > >>>>>>>>> API. > > > > > > > > > > >>>>>>>>>>> So the context is that we have already > decided > > to > > > > > > remove > > > > > > > > > > >> this > > > > > > > > > > >>>>>> Public > > > > > > > > > > >>>>>>> API > > > > > > > > > > >>>>>>>>>>> while keeping everything else backwards > > > > compatible. I > > > > > > > > > > >> think a > > > > > > > > > > >>>>> major > > > > > > > > > > >>>>>>>>> version > > > > > > > > > > >>>>>>>>>>> bump is a commonly understood signal here, > > > compared > > > > > > with > > > > > > > a > > > > > > > > > > >>>> minor > > > > > > > > > > >>>>>>> version > > > > > > > > > > >>>>>>>>>>> change. From end users' perspective, for > those > > > who > > > > > are > > > > > > > not > > > > > > > > > > >>>>>> impacted, > > > > > > > > > > >>>>>>> in > > > > > > > > > > >>>>>>>>>>> this case upgrading a major version is not > > > > > necessarily > > > > > > > more > > > > > > > > > > >>>>>> involved > > > > > > > > > > >>>>>>>>> than > > > > > > > > > > >>>>>>>>>>> upgrading a minor version - both should be as > > > > smooth > > > > > > as a > > > > > > > > > > >>>>>> dependency > > > > > > > > > > >>>>>>>>>>> version change. For those who are impacted, > > they > > > > will > > > > > > > lose > > > > > > > > > > >> the > > > > > > > > > > >>>>>> Public > > > > > > > > > > >>>>>>>>> API > > > > > > > > > > >>>>>>>>>>> anyways and a major version bump ensures > there > > is > > > > no > > > > > > > > > > >> surprise. > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Thanks, > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong > Song < > > > > > > > > > > >>>>>> tonysong...@gmail.com <mailto: > tonysong...@gmail.com > > > > > > > > > > > > > >>>>>>>> > > > > > > > > > > >>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> Public API is a well defined common concept, > > and > > > > one > > > > > > of > > > > > > > > > > >> its > > > > > > > > > > >>>>>>>>>>>>> convention is that it only changes with a > > major > > > > > > version > > > > > > > > > > >>>>> change. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> I agree. And from my understanding, > demoting a > > > > > Public > > > > > > > > > > >> API is > > > > > > > > > > >>>>>> also a > > > > > > > > > > >>>>>>>>> kind > > > > > > > > > > >>>>>>>>>>>> of such change, just like removing one, > which > > > can > > > > > only > > > > > > > > > > >> happen > > > > > > > > > > >>>>>> with > > > > > > > > > > >>>>>>>>> major > > > > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow > > > demoting > > > > > > > Public > > > > > > > > > > >>>> APIs > > > > > > > > > > >>>>>>>>> anytime, > > > > > > > > > > >>>>>>>>>>> but > > > > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen > > > before > > > > > > > > > > >> reaching > > > > > > > > > > >>>> the > > > > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually, > > > > demoting > > > > > > > > > > >> would > > > > > > > > > > >>>> be a > > > > > > > > > > >>>>>>> weaker > > > > > > > > > > >>>>>>>>>>>> change compared to removing the API > > immediately > > > > upon > > > > > > > > > > >> major > > > > > > > > > > >>>>>> version > > > > > > > > > > >>>>>>>>> bumps, > > > > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the > > > > > > > 2-minor-release > > > > > > > > > > >>>>>> migration > > > > > > > > > > >>>>>>>>>>> period. > > > > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` -> > > > > > `@PublicEvolving` > > > > > > > > > > >> sounds > > > > > > > > > > >>>>>>> against > > > > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new > annotation > > > if > > > > > > > > > > >> necessary, > > > > > > > > > > >>>>>> e.g., > > > > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions. > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> But it should be > > > > > > > > > > >>>>>>>>>>>>> completely OK to bump up the major version > if > > > we > > > > > > really > > > > > > > > > > >> want > > > > > > > > > > >>>>> to > > > > > > > > > > >>>>>>> get > > > > > > > > > > >>>>>>>>> rid > > > > > > > > > > >>>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>> a public API, right? > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's > completely > > > > > "legal" > > > > > > > > > > >> that we > > > > > > > > > > >>>>>> bump > > > > > > > > > > >>>>>>> up > > > > > > > > > > >>>>>>>>> the > > > > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is > > > > needed. > > > > > > > > > > >> However, > > > > > > > > > > >>>>> this > > > > > > > > > > >>>>>>> also > > > > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that > > public > > > > APIs > > > > > > > will > > > > > > > > > > >>>> stay > > > > > > > > > > >>>>>>> stable > > > > > > > > > > >>>>>>>>>>>> within the major release series, as the > series > > > can > > > > > end > > > > > > > > > > >>>> anytime. > > > > > > > > > > >>>>>>> IMHO, > > > > > > > > > > >>>>>>>>>>> short > > > > > > > > > > >>>>>>>>>>>> major release series are not something "make > > the > > > > end > > > > > > > > > > >> users > > > > > > > > > > >>>>>> happy", > > > > > > > > > > >>>>>>> but > > > > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to > > > make > > > > > > > > > > >> frequent > > > > > > > > > > >>>>>> breaking > > > > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting > > > > > approach, > > > > > > we > > > > > > > > > > >> can > > > > > > > > > > >>>>>> still > > > > > > > > > > >>>>>>>>> have > > > > > > > > > > >>>>>>>>>>>> longer major release series, while only > > allowing > > > > > > Public > > > > > > > > > > >> APIs > > > > > > > > > > >>>>>>>>> deprecated > > > > > > > > > > >>>>>>>>>>> at > > > > > > > > > > >>>>>>>>>>>> the end of the previous major version to be > > > > removed > > > > > in > > > > > > > > > > >> the > > > > > > > > > > >>>> next > > > > > > > > > > >>>>>>> major > > > > > > > > > > >>>>>>>>>>>> version. > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> Given our track record I would prefer a > > regular > > > > > cycle > > > > > > > > > > >> (1-2 > > > > > > > > > > >>>>> years) > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, > and > > > not > > > > > put > > > > > > > it > > > > > > > > > > >>>> again > > > > > > > > > > >>>>>> to > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear > > > > > expectation > > > > > > > on > > > > > > > > > > >>>> when > > > > > > > > > > >>>>>>>>> breaking > > > > > > > > > > >>>>>>>>>>>>> changes can be made. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a > > good > > > > > time > > > > > > > > > > >> for new > > > > > > > > > > >>>>>> major > > > > > > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking > > > changes > > > > > > > > > > >> needed. > > > > > > > > > > >>>> That > > > > > > > > > > >>>>>>> makes > > > > > > > > > > >>>>>>>>> 1-2 > > > > > > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, > > while > > > > > > leaving > > > > > > > > > > >> us > > > > > > > > > > >>>> more > > > > > > > > > > >>>>>>> time > > > > > > > > > > >>>>>>>>> to > > > > > > > > > > >>>>>>>>>>>> prepare the major release if needed. > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> Best, > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> Xintong > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay > > > Schepler > > > > < > > > > > > > > > > >>>>>>> ches...@apache.org <mailto:ches...@apache.org> > > > > > > > > > > >>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>> wrote: > > > > > > > > > > >>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > > > > > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid > > releasing > > > > > minor > > > > > > > > > > >>>>> versions > > > > > > > > > > >>>>>>> for > > > > > > > > > > >>>>>>>>>>>>> previous > > > > > > > > > > >>>>>>>>>>>>>> major versions. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not > > sure > > > > how > > > > > > > > > > >> viable > > > > > > > > > > >>>> that > > > > > > > > > > >>>>>> is > > > > > > > > > > >>>>>>> in > > > > > > > > > > >>>>>>>>>>>>> practice. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially > > > dropping > > > > > > > > > > >> support > > > > > > > > > > >>>> for > > > > > > > > > > >>>>>> Java > > > > > > > > > > >>>>>>>>> 8/11, > > > > > > > > > > >>>>>>>>>>>>> which may very well be a problem for our > > > current > > > > > > users. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote: > > > > > > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the > > > > > confusion > > > > > > > > > > >> about > > > > > > > > > > >>>>>> Public > > > > > > > > > > >>>>>>>>> API > > > > > > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there > was a > > > > > mistake > > > > > > > > > > >> in > > > > > > > > > > >>>> the > > > > > > > > > > >>>>>>> NOTES > > > > > > > > > > >>>>>>>>> part > > > > > > > > > > >>>>>>>>>>>>> for > > > > > > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I > > just > > > > > fixed > > > > > > > > > > >> it. > > > > > > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to > hear > > > that > > > > > we > > > > > > > > > > >> are on > > > > > > > > > > >>>>> the > > > > > > > > > > >>>>>>> same > > > > > > > > > > >>>>>>>>>>>>> page on that note. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote: > > > > > > > > > > >>>>>>>>>>>>>> But it should be > > > > > > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version > > if > > > we > > > > > > > > > > >> really > > > > > > > > > > >>>> want > > > > > > > > > > >>>>>> to > > > > > > > > > > >>>>>>> get > > > > > > > > > > >>>>>>>>>>> rid > > > > > > > > > > >>>>>>>>>>>>> of > > > > > > > > > > >>>>>>>>>>>>>> a public API, right? > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it > took > > > to > > > > > get > > > > > > us > > > > > > > > > > >> to > > > > > > > > > > >>>>> 2.0. > > > > > > > > > > >>>>>> ;) > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on > > the > > > > > > cadence > > > > > > > > > > >> of > > > > > > > > > > >>>>> major > > > > > > > > > > >>>>>>>>> releases > > > > > > > > > > >>>>>>>>>>>>> going forward, and there seem to be > different > > > > > > opinions > > > > > > > > > > >> on > > > > > > > > > > >>>>> that. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor > > > releases > > > > > > > between > > > > > > > > > > >>>> major > > > > > > > > > > >>>>>>> ones, > > > > > > > > > > >>>>>>>>>>>>> that for us means that users have to > > > potentially > > > > > deal > > > > > > > > > > >> with > > > > > > > > > > >>>>>>> breaking > > > > > > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a > > lot. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> Given our track record I would prefer a > > regular > > > > > cycle > > > > > > > > > > >> (1-2 > > > > > > > > > > >>>>>> years) > > > > > > > > > > >>>>>>> to > > > > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, > and > > > not > > > > > put > > > > > > > it > > > > > > > > > > >>>> again > > > > > > > > > > >>>>>> to > > > > > > > > > > >>>>>>> the > > > > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear > > > > > expectation > > > > > > > on > > > > > > > > > > >>>> when > > > > > > > > > > >>>>>>>>> breaking > > > > > > > > > > >>>>>>>>>>>>> changes can be made. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> But again, maybe this should be in a > separate > > > > > thread. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote: > > > > > > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral > change > > in > > > > > mind? > > > > > > > > > > >> Not > > > > > > > > > > >>>>> sure > > > > > > > > > > >>>>>> I > > > > > > > > > > >>>>>>>>> fully > > > > > > > > > > >>>>>>>>>>>>>> understand the concern for behavioral > change > > > > here. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be > > > > > performance > > > > > > in > > > > > > > > > > >>>>> certain > > > > > > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe > > > > > unknowingly) > > > > > > > > > > >> relied > > > > > > > > > > >>>>> upon > > > > > > > > > > >>>>>>>>>>>>> ( > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I > > > > > > > > > ), > > > > > > > > > > a semantic change to some > > > > > > > > > > >> API. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> For a concrete example, consider the job > > > > > submission. > > > > > > A > > > > > > > > > > >> few > > > > > > > > > > >>>>>>> releases > > > > > > > > > > >>>>>>>>> back > > > > > > > > > > >>>>>>>>>>>>> we made changes such that the > initialization > > of > > > > the > > > > > > job > > > > > > > > > > >>>> master > > > > > > > > > > >>>>>>>>> happens > > > > > > > > > > >>>>>>>>>>>>> asynchronously. > > > > > > > > > > >>>>>>>>>>>>> This meant the job submission call returns > > > > sooner, > > > > > > and > > > > > > > > > > >> the > > > > > > > > > > >>>> job > > > > > > > > > > >>>>>>> state > > > > > > > > > > >>>>>>>>>>>>> enum was extended to cover this state. > > > > > > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible > > change, > > > > but > > > > > > the > > > > > > > > > > >>>>> observed > > > > > > > > > > >>>>>>>>> behavior > > > > > > > > > > >>>>>>>>>>>>> may be different. > > > > > > > > > > >>>>>>>>>>>>> > > > > > > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over > > > time > > > > we > > > > > > > > > > >> changed > > > > > > > > > > >>>>> what > > > > > > > > > > >>>>>>> some > > > > > > > > > > >>>>>>>>>>>>> metrics returned a few times. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
