Re: [DISCUSS] FLIP-321: Introduce an API deprecation process

[Becket Qin]

Hi Xintong, Jark and Jing,

Thanks for the reply. Yes, we can only mark the DataStream API as
@Deprecated after the ProcessFunction API is fully functional and mature.

It is a fair point that the condition of marking a @Public API as
deprecated should also be a part of this FLIP. I just added that to the
FLIP wiki. This is probably more of a clarification on the existing
convention, rather than a change.

It looks like we are on the same page now for this FLIP. If so, I'll start
a VOTE thread in two days.

Thanks,

Jiangjie (Becket) Qin

On Mon, Jun 26, 2023 at 8:09 PM Xintong Song <tonysong...@gmail.com> wrote:

> >
> > Considering DataStream API is the most fundamental and complex API of
> > Flink, I think it is worth a longer time than the general process for the
> > deprecation period to wait for the new API be mature.
> >
>
> This inspires me. In this specific case, compared to how long should
> DataStream API be removed after deprecation, it's probably more important
> to answer the question how long would ProcessFunction API become mature and
> stable after being introduced. According to FLIP-197[1], it requires 4
> minor releases by default to promote an @Experimental API to @Public. And
> for ProcessFunction API, which aims to replace DataStream API as one of the
> most fundamental API of Flink, I'd expect this to take at least the default
> time, or even longer. And we probably should wait until we believe
> ProcessFunction API is stable to mark DataStream API as deprecated, rather
> than as soon as it's introduced. Assuming we introduce the ProcessFunction
> API in 2.0, that means we would need to wait for 6 minor releases (4 for
> the new API to become stable, and 2 for the migration period) to remove
> DataStream API, which is ~2.5 year (assuming 5 months / minor release),
> which sounds acceptable for another major version bump.
>
> To wrap things up, it seems to me, sadly, that anyway we cannot avoid the
> overhead for maintaining both DataStream & ProcessFunction APIs for at
> least 6 minor releases.
>
> Best,
>
> Xintong
>
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process
>
>
>
> On Mon, Jun 26, 2023 at 5:41 PM Jing Ge <j...@ververica.com.invalid>
> wrote:
>
> > Hi all,
> >
> > Just want to make sure we are on the same page. There is another
> example[1]
> > I was aware of recently that shows why more factors need to be taken care
> > of than just the migration period. Thanks Galen for your hint.
> >
> > To put it simply, the concern about API deprecation is not that
> deprecated
> > APIs have been removed too early (min migration period is required). The
> > major concern is that APIs are marked as deprecated for a (too) long
> time,
> > much longer than the migration period discussed in this thread, afaik.
> > Since there is no clear picture/definition, no one knows when to do the
> > migration for users(after the migration period has expired) and when to
> > remove deprecated APIs for Flink developers.
> >
> > Based on all the information I knew, there are two kinds of obstacles
> that
> > will and should block the deprecation process:
> >
> > 1. Lack of functionalities in new APIs. It happens e.g. with the
> > SourceFunction to FLIP-27 Source migration. Users who rely on those
> > functions can not migrate to new APIs.
> > 2. new APIs have critical bugs. An example could be found at [1]. Users
> > have to stick to the deprecated APIs.
> >
> > Since FLIP-321 is focusing on the API deprecation process, those blocking
> > issues deserve attention and should be put into the FLIP. The current
> FLIP
> > seems to only focus on migration periods. If we consider those blocking
> > issues as orthogonal issues that are beyond the scope of this discussion,
> > does it make sense to change the FLIP title to something like "Introduce
> > minimum migration periods of API deprecation process"?
> >
> > Best regards,
> > Jing
> >
> > [1] https://lists.apache.org/thread/wxoo7py5pqqlz37l4w8jrq6qdvsdq5wc
> >
> > On Sun, Jun 25, 2023 at 2:01 PM Jark Wu <imj...@gmail.com> wrote:
> >
> > > I agree with Jingsong and Becket.
> > >
> > > Look at the legacy SourceFunction (a small part of DataStream API),
> > > the SourceFunction is still not and can't be marked deprecated[1] until
> > > now after the new Source was released 2 years ago, because the new
> Source
> > > still can't fully consume the abilities of legacy API. Considering
> > > DataStream
> > > API is the most fundamental and complex API of Flink, I think it is
> worth
> > > a longer time than the general process for the deprecation period to
> > > wait for the new API be mature. The above 2 options sound a bit of rush
> > > for such a widely used API.
> > >
> > > I fully understand the concern of maintenance overhead, but it's a bit
> > hard
> > > for others to estimate maintenance costs without a concrete design and
> > code
> > > of the new ProcessFunction API. I agree with Becket that maybe we can
> > > re-evaluate the API deprecation process once we have the new
> > > ProcessFunction
> > > API. If the maintenance is indeed huge, I think it is reasonable to
> have
> > a
> > > special rule for this case at that time.
> > >
> > > Best,
> > > Jark
> > >
> > >
> > > [1]: https://issues.apache.org/jira/browse/FLINK-28045
> > >
> > > On Sun, 25 Jun 2023 at 16:22, Becket Qin <becket....@gmail.com> wrote:
> > >
> > > > Hi Jingsong,
> > > >
> > > > Thanks for the reply. I completely agree with you.
> > > >
> > > > The above 2 options are based on the assumption that the community
> > cannot
> > > > afford to maintain the deprecated DataStream API for long. I'd say we
> > > > should try everything we can to maintain it for as much time as
> > possible.
> > > > DataStream API is actually the most used API in Flink by so many
> users
> > at
> > > > this point. Removing it any time soon will dramatically hurt our
> users.
> > > So
> > > > ideally we should keep it for at least 2 years after deprecation, if
> > not
> > > > more.
> > > >
> > > > The prohibitively high maintenance overhead is just an assumption.
> > > > Personally speaking, I don't feel this assumption is necessarily
> true.
> > We
> > > > should re-evaluate once we have the new ProcessFunction API in place.
> > > > Without the code it is hard to tell for sure. I am actually kind of
> > > > optimistic about the maintenance cost.
> > > >
> > > > Thanks,
> > > >
> > > > Jiangjie (Becket) Qin
> > > >
> > > >
> > > >
> > > > On Sun, Jun 25, 2023 at 11:30 AM Jingsong Li <jingsongl...@gmail.com
> >
> > > > wrote:
> > > >
> > > > > Thanks Becket and all for your discussion.
> > > > >
> > > > > > 1. We say this FLIP is enforced starting release 2.0. For current
> > 1.x
> > > > > APIs,
> > > > > we provide a migration period with best effort, while allowing
> > > exceptions
> > > > > for immediate removal in 2.0. That means we will still try with
> best
> > > > effort
> > > > > to get the ProcessFuncion API ready and deprecate the DataStream
> API
> > in
> > > > > 1.x, but will also be allowed to remove DataStream API in 2.0 if
> it's
> > > not
> > > > > deprecated 2 minor releases before the major version bump.
> > > > >
> > > > > > 2. We strictly follow the process in this FLIP, and will quickly
> > bump
> > > > the
> > > > > major version from 2.x to 3.0 once the migration period for
> > DataStream
> > > > API
> > > > > is reached.
> > > > >
> > > > > Sorry, I didn't read the previous detailed discussion because the
> > > > > discussion list was so long.
> > > > >
> > > > > I don't really like either of these options.
> > > > >
> > > > > Considering that DataStream is such an important API, can we offer
> a
> > > > third
> > > > > option:
> > > > >
> > > > > 3. Maintain the DataStream API throughout 2.X and remove it until
> > 3.x.
> > > > But
> > > > > there's no need to assume that 2.X is a short version, it's still a
> > > > normal
> > > > > major version.
> > > > >
> > > > > Best,
> > > > > Jingsong
> > > > >
> > > > > Becket Qin <becket....@gmail.com>于2023年6月22日 周四16:02写道：
> > > > >
> > > > > > Thanks much for the input, John, Stefan and Jing.
> > > > > >
> > > > > > I think Xingtong has well summarized the pros and cons of the two
> > > > > options.
> > > > > > Let's collect a few more opinions here and we can move forward
> with
> > > the
> > > > > one
> > > > > > more people prefer.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Jiangjie (Becket) Qin
> > > > > >
> > > > > > On Wed, Jun 21, 2023 at 3:20 AM Jing Ge
> <j...@ververica.com.invalid
> > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Thanks Xingtong for the summary. If I could only choose one of
> > the
> > > > > given
> > > > > > > two options, I would go with option 1. I understood that
> option 2
> > > > > worked
> > > > > > > great with Kafka. But the bridge release will still confuse
> users
> > > and
> > > > > my
> > > > > > > gut feeling is that many users will skip 2.0 and be waiting for
> > 3.0
> > > > or
> > > > > > even
> > > > > > > 3.x. And since fewer users will use Flink 2.x, the development
> > > focus
> > > > > will
> > > > > > > be on Flink 3.0 with the fact that the current Flink release is
> > > 1.17
> > > > > and
> > > > > > we
> > > > > > > are preparing 2.0 release. That is weird for me.
> > > > > > >
> > > > > > > THB, I would not name the change from @Public to @Retired as a
> > > > > demotion.
> > > > > > > The purpose of @Retire is to extend the API lifecycle with one
> > more
> > > > > > stage,
> > > > > > > like in the real world, people born, studied, graduated,
> worked,
> > > and
> > > > > > > retired. Afaiu from the previous discussion, there are two
> rules
> > > we'd
> > > > > > like
> > > > > > > to follow simultaneously:
> > > > > > >
> > > > > > > 1. Public APIs can only be changed between major releases.
> > > > > > > 2. A smooth migration phase should be offered to users, i.e. at
> > > > least 2
> > > > > > > minor releases after APIs are marked as @deprecated. There
> should
> > > be
> > > > > new
> > > > > > > APIs as the replacement.
> > > > > > >
> > > > > > > Agree, those rules are good to improve the user friendliness.
> > > Issues
> > > > we
> > > > > > > discussed are rising because we want to fulfill both of them.
> If
> > we
> > > > > take
> > > > > > > care of deprecation very seriously, APIs can be marked as
> > > > @Deprecated,
> > > > > > only
> > > > > > > when the new APIs as the replacement provide all
> functionalities
> > > the
> > > > > > > deprecated APIs have. In an ideal case without critical bugs
> that
> > > > might
> > > > > > > stop users adopting the new APIs. Otherwise the expected
> > > > "replacement"
> > > > > > will
> > > > > > > not happen. Users will still stick to the deprecated APIs,
> > because
> > > > the
> > > > > > new
> > > > > > > APIs can not be used. For big features, it will need at least 4
> > > minor
> > > > > > > releases(ideal case), i.e. 2+ years to remove deprecated APIs:
> > > > > > >
> > > > > > > - 1st minor release to build the new APIs as the replacement
> and
> > > > > waiting
> > > > > > > for feedback. It might be difficult to mark the old API as
> > > deprecated
> > > > > in
> > > > > > > this release, because we are not sure if the new APIs could
> cover
> > > > 100%
> > > > > > > functionalities.
> > > > > > > -  In the lucky case,  mark all old APIs as deprecated in the
> 2nd
> > > > minor
> > > > > > > release. (I would even suggest having the new APIs released at
> > > least
> > > > > for
> > > > > > > two minor releases before marking it as deprecated to make sure
> > > they
> > > > > can
> > > > > > > really replace the old APIs, in case we care more about smooth
> > > > > migration)
> > > > > > > - 3rd minor release for the migration period
> > > > > > > -  In another lucky case, the 4th release is a major release,
> the
> > > > > > > deprecated APIs could be removed.
> > > > > > >
> > > > > > > The above described scenario works only in an ideal case. In
> > > reality,
> > > > > it
> > > > > > > might take longer to get the new APIs ready and mark the old
> API
> > > > > > > deprecated. Furthermore, if the 4th release is not a major
> > release,
> > > > we
> > > > > > will
> > > > > > > have to maintain both APIs for many further minor releases. The
> > > > > question
> > > > > > is
> > > > > > > how to know the next major release in advance, especially 4
> minor
> > > > > > releases'
> > > > > > > period, i.e. more than 2 years in advance? Given that Flink
> > > contains
> > > > > many
> > > > > > > modules, it is difficult to ask devs to create a 2-3 years
> > > > deprecation
> > > > > > plan
> > > > > > > for each case. In case we want to build major releases at a
> fast
> > > > pace,
> > > > > > > let's say every two years, it means devs must plan any API
> > > > deprecation
> > > > > > > right after each major release. Afaiac, it is quite difficult.
> > > > > > >
> > > > > > > The major issue is, afaiu, if we follow rule 2, we have to keep
> > all
> > > > > > @Public
> > > > > > > APIs, e.g. DataStream, that are not marked as deprecated yet,
> to
> > > 2.0.
> > > > > > Then
> > > > > > > we have to follow rule 1 to keep it unchanged until we have
> 3.0.
> > > That
> > > > > is
> > > > > > > why @Retired is useful to give devs more flexibility and still
> > > > fulfill
> > > > > > both
> > > > > > > rules. Let's check it with examples:
> > > > > > >
> > > > > > > - we have @Public DataStream API in 1.18. It will not be marked
> > > > > > > as @Deprecated, because the new APIs as the replacement are not
> > > > ready.
> > > > > > > - we keep the DataStream API itself unchanged in 2.0, but
> change
> > > the
> > > > > > > annotation from @Public to @Retire. New APIs will be introduced
> > > too.
> > > > In
> > > > > > > this case, Rule 1 is ok, since public API is allowed to change
> > > > between
> > > > > > > major releases. Only changing annotation is the minimal change
> we
> > > > could
> > > > > > do
> > > > > > > and it does not break rule 1. Rule 2 is ok too, since the
> > > DataStream
> > > > > APIs
> > > > > > > work exactly the same.  Attention: the change of @Public ->
> > > @Retired
> > > > > can
> > > > > > > only be done between major releases, because @Public APIs can
> > only
> > > be
> > > > > > > changed between major releases.
> > > > > > > - in 2.1, DataStream API will be marked as deprecated.
> > > > > > > - in 2.2, DataStream will be kept for the migration period.
> > > > > > > - in 2.3, DataStream will be removed.
> > > > > > >
> > > > > > > Becket mentioned previously (please correct me if I didn't
> > > understand
> > > > > it
> > > > > > > correctly) that users might not check the changes of annotation
> > and
> > > > the
> > > > > > > upgrade process might not be smooth. I was wondering, if users
> > > don't
> > > > > pay
> > > > > > > attention to that, they will also not pay attention to
> > @deprecated.
> > > > The
> > > > > > > migration will not be smooth too even when we stick to the Java
> > > > > > > standard @deprecated. Let's go throw it with examples:
> > > > > > >
> > > > > > > Scenario 1: with @Retired
> > > > > > >
> > > > > > > Case 1: users upgrade from 1.18 to 2.0, 2.1, 2.2 - Nothing
> should
> > > be
> > > > > done
> > > > > > > for users who still want to use DataStream. The migration is
> > > smooth.
> > > > > But
> > > > > > > they should be aware of the annotation changes, i.e. @Public ->
> > > > > @Retired
> > > > > > > Case 2: users upgrade from 1.18 to 2.3 - DataStream is removed.
> > > Since
> > > > > it
> > > > > > is
> > > > > > > a major release upgrade, breaking changes are expected.
> > > > > > > Case 3: users upgrade from 2.0 to 2.3 -  DataStream is removed.
> > > Since
> > > > > > APIs
> > > > > > > are marked as @Retired, breaking changes are expected too.
> > > > > > >
> > > > > > > In addition to fulfilling both rules, there are two more
> benefits
> > > of
> > > > > this
> > > > > > > option:
> > > > > > > 1. users can upgrade to 2.0 with less/no effort, i.e Case 1.
> > > > > > > 2. no bridge release is required. DataStream API could be
> > > deprecated
> > > > in
> > > > > > 2.x
> > > > > > > without breaking any of the two rules. @Retired provides enough
> > > > > > flexibility
> > > > > > > for future deprecations and upgrades.
> > > > > > >
> > > > > > > The only issue I can find is that users might be surprised in
> > case
> > > 3,
> > > > > if
> > > > > > > they ignore the previous @Public -> @Retired change. But for
> such
> > > > > users,
> > > > > > > they will ignore @deprecated annotation too and will have
> issues
> > > with
> > > > > > > pure @deprecated
> > > > > > > annotation too:
> > > > > > >
> > > > > > > Scenario 2: with @deprecated
> > > > > > >
> > > > > > > The example will be slightly different from scenario 1, because
> > > > @Public
> > > > > > > APIs can only be removed between major releases. As an example,
> > > > > > DataStream
> > > > > > > API will be marked as deprecated in 1.19, kept in 1.20, and
> then
> > > > > removed
> > > > > > in
> > > > > > > 2.0
> > > > > > >
> > > > > > > Case 1: users upgrade from 1.18 to 2.0 - APIs that are not
> marked
> > > as
> > > > > > > deprecated are removed between major releases. Rule 1 is ok.
> The
> > 2
> > > > > minor
> > > > > > > releases period defined in rule 2 sounds like a smooth
> migration
> > > > plan,
> > > > > > but
> > > > > > > for users in this case, it is still not smooth.
> > > > > > > Case 2: users upgrade from 1.20 to 2.0 - Standard upgrade
> process
> > > > with
> > > > > > > deprecated APIs. But for those users who ignore annotation
> > changes,
> > > > > will
> > > > > > > have the same issue mentioned in case 1.
> > > > > > >
> > > > > > > I understood that, compared to scenario 1, users, who ignore
> > > @Public
> > > > > > > -> @Retired
> > > > > > > changes, will not face any breaking changes, after they
> migrated
> > to
> > > > > 2.0.
> > > > > > > But the cost of it is bridge releases. There might be many
> bridge
> > > > > > releases
> > > > > > > in the future. As I mentioned previously, it is not easy to
> find
> > > the
> > > > > > right
> > > > > > > timing to kick off the deprecation process of APIs before the
> > next
> > > > > > expected
> > > > > > > major release and 2-3 years in advance.
> > > > > > >
> > > > > > > Scenario 3: @PublicEvolving deprecation for users who ignore
> > > > > @deprecated
> > > > > > > annotation changes
> > > > > > >
> > > > > > > Sample API marked as deprecated in 1.19, kept in 1.20, removed
> in
> > > > 1.21
> > > > > > >
> > > > > > > Case 1: users upgrade from 1.18 to 1.21 - breaking change, no
> > > smooth
> > > > > > > migration.
> > > > > > > Case 2: users upgrade from 1.20 to 1.21 - same breaking change,
> > > > > > > since @deprecated is ignored
> > > > > > >
> > > > > > > In summary, for users who care about the annotation change,
> they
> > > will
> > > > > get
> > > > > > > more benefits with @Retired. For users who ignore annotation
> > > changes,
> > > > > > they
> > > > > > > will be always facing non-smooth upgrades even using pure
> > standard
> > > > > > > @deprecated
> > > > > > > annotation/process.
> > > > > > >
> > > > > > > For the case that IDEs support @deprecated, that is true. There
> > are
> > > > no
> > > > > > such
> > > > > > > supports for homebuilt annotation. But the intention is to let
> > > users
> > > > be
> > > > > > > aware of it. There will be other ways to do it for users who
> care
> > > > about
> > > > > > > annotation changes. For users who do not care, there should be
> no
> > > > > > > difference between with tool support or without. Using @Retired
> > > with
> > > > > > > @deprecation
> > > > > > > together is at least not worse than only using @deprecation
> > alone.
> > > > And
> > > > > > > there are additional benefits with @Retired.
> > > > > > >
> > > > > > > Just my 2 cents and looking forward to your feedback,
> especially
> > > > > > different
> > > > > > > opinions that will help me understand the issue better. Thanks!
> > > > > > >
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jing
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 20, 2023 at 4:50 PM Stefan Richter
> > > > > > > <srich...@confluent.io.invalid> wrote:
> > > > > > >
> > > > > > > > Hi Xintong,
> > > > > > > >
> > > > > > > > Thanks for the summary, most of the points that you agreed
> upon
> > > > also
> > > > > > make
> > > > > > > > sense to me.
> > > > > > > >
> > > > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > > > demoting
> > > > > > > APIs
> > > > > > > > >   from Public to PublicEvolving / Experimental / Retired in
> > any
> > > > > > version
> > > > > > > > bump,
> > > > > > > > >   are not good practices.
> > > > > > > >
> > > > > > > > I hope we can can go beyond calling it “not a good practice"
> > and
> > > > > reach
> > > > > > > > consensus that we will not demote or remove public APIs in
> > minor
> > > > > > releases
> > > > > > > > and (technically) enforce this rule for all contributions. I
> > > think
> > > > > > it’s a
> > > > > > > > reasonable expectation for stability from a project as mature
> > as
> > > > > Flink.
> > > > > > > >
> > > > > > > > > I'm personally in favor of option 1. As the
> migration-period
> > > rule
> > > > > is
> > > > > > > > newly
> > > > > > > > > proposed, I think it's fair to make exceptions for cases
> > where
> > > we
> > > > > > > already
> > > > > > > > > missed the best chance for planning the deprecation.
> > Moreover,
> > > I
> > > > do
> > > > > > > > believe
> > > > > > > > > having a quick major version bump does confuse users. Yes,
> we
> > > can
> > > > > > > explain
> > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> anything
> > > > other
> > > > > > than
> > > > > > > > the
> > > > > > > > > removal of an deprecated API. But having to explain this
> > itself
> > > > is
> > > > > an
> > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > understanding,
> > > > > > his
> > > > > > > > > major point is that a quick major version bump causes
> barely
> > > any
> > > > > > actual
> > > > > > > > > lose on users, while in option 1 not providing the
> migration
> > > > period
> > > > > > or
> > > > > > > > > providing a shorter on is an actual lose on users.
> (@Becket,
> > > > please
> > > > > > > > correct
> > > > > > > > > me if I'm mistaken.)
> > > > > > > > >
> > > > > > > >
> > > > > > > > I would be open to both options and maybe postpone this
> > decision
> > > > > until
> > > > > > we
> > > > > > > > have the complete roadmap for Flink 2.0. We would have a
> better
> > > > > > overview
> > > > > > > > about the technical consequences, e.g. how hard it would be
> to
> > > > offer
> > > > > > and
> > > > > > > > maintain both APIs side by side, how well we get both APIs
> > > > separated,
> > > > > > and
> > > > > > > > whether or not we will be able to use the full potential of
> our
> > > > > > breaking
> > > > > > > > changes before the DataStream API is completely removed. One
> > > > example
> > > > > > for
> > > > > > > > the last point I can think of would be moving from lazy to
> > eager
> > > > > state
> > > > > > > > declaration. If we believe that it is reasonable to support
> all
> > > > > planned
> > > > > > > > changes while maintaining both APIs, I'm leaning towards
> option
> > > 2.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Stefan
> > > > > > > >
> > > > > > > >
> > > > > > > > > On 20. Jun 2023, at 14:43, Xintong Song <
> > tonysong...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Becket and I had an offline voice call earlier today. We
> have
> > > > > reached
> > > > > > > > > consensus on some of the arguments, but still hold
> different
> > > > > opinions
> > > > > > > on
> > > > > > > > > some others. I'd like to post the outcome here for
> > > transparency.
> > > > > > > > >
> > > > > > > > > We both agree that:
> > > > > > > > >
> > > > > > > > >   1. Providing a migration period would be beneficial for
> our
> > > > > users,
> > > > > > > and
> > > > > > > > >   we should do that
> > > > > > > > >   2. Dropping deprecated Public APIs in minor releases, or
> > > > demoting
> > > > > > > APIs
> > > > > > > > >   from Public to PublicEvolving / Experimental / Retired in
> > any
> > > > > > version
> > > > > > > > bump,
> > > > > > > > >   are not good practices.
> > > > > > > > >   3. Ideally, with this FLIP, developers should be more
> > careful
> > > > and
> > > > > > > plan
> > > > > > > > >   API changes ahead. That means：
> > > > > > > > >      1. Be more careful with designing APIs and promoting
> > them
> > > to
> > > > > > > Public,
> > > > > > > > >      so that they won't be changed / removed very soon, and
> > > also
> > > > > the
> > > > > > > > >      maintenance overhead for keeping them after
> deprecation
> > > > should
> > > > > > be
> > > > > > > > >      affordable. (I believe this also aligns with John's
> > > > opinion.)
> > > > > > > > >      2. Plan the deprecation / removal of old APIs that we
> > > don't
> > > > > want
> > > > > > > to
> > > > > > > > >      carry to the next major version earlier, so that
> they'll
> > > > have
> > > > > > the
> > > > > > > > >      2-minor-release migration period before the next major
> > > > version
> > > > > > > bump.
> > > > > > > > >   4. A practical situation is that, we are not ready for
> > > > > deprecating
> > > > > > > the
> > > > > > > > >   DataStream API, nor afford to carry it for another couple
> > of
> > > > > years.
> > > > > > > > >      1. We cannot deprecate it now, because the new
> > > > ProcessFunction
> > > > > > API
> > > > > > > > >      (the planned replacement for DataStream API) is not
> yet
> > > > ready.
> > > > > > We
> > > > > > > > haven't
> > > > > > > > >      planned the ProcessFunction API earlier because the
> > > original
> > > > > > > > > plan was to do
> > > > > > > > >      in-place changes directly on the DataStream API in
> > release
> > > > > 2.0,
> > > > > > > > until the
> > > > > > > > >      smooth migration period is proposed. Before this, the
> > only
> > > > > > > > guarantees we
> > > > > > > > >      provide for Public APIs is that they'll stay
> compatible
> > > > until
> > > > > > the
> > > > > > > > next
> > > > > > > > >      major version, which led to the understanding the
> > in-place
> > > > > > changes
> > > > > > > > on
> > > > > > > > >      Public APIs should be fine.
> > > > > > > > >      2. We also cannot afford carrying the DataStream API
> for
> > > > > another
> > > > > > > > >      couple of years. One of the reasons that we want to
> > > > refactor /
> > > > > > > > replace
> > > > > > > > >      DataStream API is that it exposes and depends on too
> > many
> > > > > > > > > Flink's internal
> > > > > > > > >      runtime implementations, which has already limited /
> > > blocked
> > > > > > plans
> > > > > > > > for
> > > > > > > > >      Flink's internal improvements for many times. Also due
> > to
> > > > the
> > > > > > wide
> > > > > > > > and
> > > > > > > > >      transitive dependencies on the internals, maintaining
> > two
> > > > sets
> > > > > > of
> > > > > > > > lower
> > > > > > > > >      APIs itself is expensive and would be better to keep
> it
> > > > short.
> > > > > > > This
> > > > > > > > is
> > > > > > > > >      admittedly a historical design flaw which should be
> > > avoided
> > > > in
> > > > > > > > future.
> > > > > > > > >
> > > > > > > > > What we haven't reached consensus is about how to deal with
> > the
> > > > > > current
> > > > > > > > > situation around the DataStream API. There are two options.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >   1. We say this FLIP is enforced starting release 2.0. For
> > > > current
> > > > > > 1.x
> > > > > > > > >   APIs, we provide a migration period with best effort,
> while
> > > > > > allowing
> > > > > > > > >   exceptions for immediate removal in 2.0. That means we
> will
> > > > still
> > > > > > try
> > > > > > > > with
> > > > > > > > >   best effort to get the ProcessFuncion API ready and
> > deprecate
> > > > the
> > > > > > > > >   DataStream API in 1.x, but will also be allowed to remove
> > > > > > DataStream
> > > > > > > > API in
> > > > > > > > >   2.0 if it's not deprecated 2 minor releases before the
> > major
> > > > > > version
> > > > > > > > bump.
> > > > > > > > >   2. We strictly follow the process in this FLIP, and will
> > > > quickly
> > > > > > bump
> > > > > > > > >   the major version from 2.x to 3.0 once the migration
> period
> > > for
> > > > > > > > DataStream
> > > > > > > > >   API is reached.
> > > > > > > > >
> > > > > > > > > I'm personally in favor of option 1. As the
> migration-period
> > > rule
> > > > > is
> > > > > > > > newly
> > > > > > > > > proposed, I think it's fair to make exceptions for cases
> > where
> > > we
> > > > > > > already
> > > > > > > > > missed the best chance for planning the deprecation.
> > Moreover,
> > > I
> > > > do
> > > > > > > > believe
> > > > > > > > > having a quick major version bump does confuse users. Yes,
> we
> > > can
> > > > > > > explain
> > > > > > > > > to users that bumping from 2.x to 3.0 does not cost
> anything
> > > > other
> > > > > > than
> > > > > > > > the
> > > > > > > > > removal of an deprecated API. But having to explain this
> > itself
> > > > is
> > > > > an
> > > > > > > > > indicator that it might be confusing for users.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Becket, on the other hand, prefers option 2. From my
> > > > understanding,
> > > > > > his
> > > > > > > > > major point is that a quick major version bump causes
> barely
> > > any
> > > > > > actual
> > > > > > > > > lose on users, while in option 1 not providing the
> migration
> > > > period
> > > > > > or
> > > > > > > > > providing a shorter on is an actual lose on users.
> (@Becket,
> > > > please
> > > > > > > > correct
> > > > > > > > > me if I'm mistaken.)
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > And we'd like to hear more feedback from the community.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > >
> > > > > > > > > Xintong
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Jun 20, 2023 at 2:55 AM John Roesler <
> > > > vvcep...@apache.org
> > > > > > > > <mailto:vvcep...@apache.org>> wrote:
> > > > > > > > >
> > > > > > > > >> Hi Becket and Xintong,
> > > > > > > > >>
> > > > > > > > >> I hope you don't mind if I chime in a little.
> > > > > > > > >>
> > > > > > > > >> Once an API is marked Public, we're committing to support
> it
> > > > until
> > > > > > > it's
> > > > > > > > >> deprecated, and once it's deprecated, to leave it in place
> > for
> > > > at
> > > > > > > least
> > > > > > > > two
> > > > > > > > >> minor releases and then only remove it in the following
> > major
> > > > > > release.
> > > > > > > > >>
> > > > > > > > >> As a user, I would be dismayed to discover that the
> project
> > > > > > > maintainers
> > > > > > > > >> demoted a Public API to PublicEvolving or Experimental in
> > > order
> > > > to
> > > > > > > > violate
> > > > > > > > >> the spirit of the deprecation period for Public APIs.
> > > Likewise,
> > > > > I'd
> > > > > > > > view a
> > > > > > > > >> rapid sequence of minor releases for the purpose of
> dropping
> > > > > > > deprecated
> > > > > > > > >> APIs in the next major release as another violation of the
> > > > spirit
> > > > > of
> > > > > > > our
> > > > > > > > >> guarantees. I'm reminded of this xkcd:
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1494/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw3jFBaXa07_RvXzSLu66f6_
> > > > > > > > >>
> > > > > > > > >> I'm glad we're gaming out these situations before we
> > institute
> > > > > this
> > > > > > > > FLIP,
> > > > > > > > >> and I hope we all converge on a clear understanding of
> what
> > > > we're
> > > > > > > > >> guaranteeing. From where I'm sitting, the essence of this
> > FLIP
> > > > is
> > > > > > > > >> specifically to prevent us from doing whatever we want and
> > > > instead
> > > > > > to
> > > > > > > > >> commit to providing a suitable notice period to users who
> > rely
> > > > on
> > > > > > > Public
> > > > > > > > >> APIs.
> > > > > > > > >>
> > > > > > > > >> It honestly shouldn't matter what the maintenance overhead
> > is
> > > > (or,
> > > > > > > > rather,
> > > > > > > > >> evaluating the maintenance overhead should be a
> > consideration
> > > > > before
> > > > > > > we
> > > > > > > > >> promote an API to Public to begin with). My experience in
> > > other
> > > > > > > > projects is
> > > > > > > > >> that regretting a feature so much you want to drop it asap
> > is
> > > > > > > extremely
> > > > > > > > >> rare, and that living with the pain for a little while
> until
> > > the
> > > > > > > > >> deprecation period expires will probably make us better
> > > > engineers.
> > > > > > > > >>
> > > > > > > > >> With that all said, one thing that is not a violation is
> to
> > > > > propose
> > > > > > a
> > > > > > > > new
> > > > > > > > >> major release in order to drop especially burdensome
> > > deprecated
> > > > > APIs
> > > > > > > > that
> > > > > > > > >> have already enjoyed their full deprecation period. Those
> > > really
> > > > > bad
> > > > > > > > "case
> > > > > > > > >> A" features will be very rare.
> > > > > > > > >>
> > > > > > > > >> Looking over the FLIP again, I see a reference to the
> > release
> > > > > > > > >> documentation, but not to the build tooling support for
> the
> > > > > > Deprecated
> > > > > > > > >> annotation, so it might help to share this scenario: The
> way
> > > > this
> > > > > > will
> > > > > > > > all
> > > > > > > > >> come together in practice is that, by giving good
> stability
> > > > > > > guarantees,
> > > > > > > > we
> > > > > > > > >> will encourage our users to bump minor versions when they
> > are
> > > > > > > released,
> > > > > > > > >> which means that they will see compiler warnings
> immediately
> > > if
> > > > > any
> > > > > > of
> > > > > > > > the
> > > > > > > > >> APIs they've used are deprecated. In fact, I'd recommend
> for
> > > > them
> > > > > to
> > > > > > > use
> > > > > > > > >> the `-Werror` javac flag to be sure that they don't miss
> > those
> > > > > > > warnings.
> > > > > > > > >> Given our release cadence, this means that they will have
> > > about
> > > > > six
> > > > > > > > months
> > > > > > > > >> to migrate off of any deprecated APIs without being forced
> > to
> > > > > avoid
> > > > > > > > >> updates. And they really need to be able to adopt further
> > > minor
> > > > > > > updates
> > > > > > > > in
> > > > > > > > >> the series, since they may contain further improvements
> that
> > > > > > actually
> > > > > > > > >> facilitate migrating off of the deprecated APIs.
> > > > > > > > >>
> > > > > > > > >> The compiler warnings are a far more reliable mechanism to
> > > > > advertise
> > > > > > > > >> deprecation than release notes, since users' own builds
> will
> > > > > notify
> > > > > > > them
> > > > > > > > >> right away about only the deprecations that are relevant
> to
> > > > them,
> > > > > > but
> > > > > > > it
> > > > > > > > >> does rely on users feeling ok to bump up minor versions
> in a
> > > > > timely
> > > > > > > > >> fashion. If we consistently burn them by not observing the
> > > > > stability
> > > > > > > > >> guarantees, they'll try to avoid updates instead, and the
> > > whole
> > > > > > thing
> > > > > > > > falls
> > > > > > > > >> apart.
> > > > > > > > >>
> > > > > > > > >> Thanks again,
> > > > > > > > >> -John
> > > > > > > > >>
> > > > > > > > >> On 2023/06/19 10:43:05 Becket Qin wrote:
> > > > > > > > >>> Hi Xintong,
> > > > > > > > >>>
> > > > > > > > >>> Let's compare the following cases:
> > > > > > > > >>>
> > > > > > > > >>> A. If the maintenance overhead of the deprecated API is
> > high,
> > > > and
> > > > > > we
> > > > > > > > want
> > > > > > > > >>> to remove it after two minor releases. Then there are two
> > > > > options:
> > > > > > > > >>>    A1: Demote the API in the major version bump and
> remove
> > > the
> > > > > code
> > > > > > > > >> with a
> > > > > > > > >>> minor version bump.
> > > > > > > > >>>    A2: Do exactly the same as 1-A, except that when
> > removing
> > > > the
> > > > > > > code,
> > > > > > > > >>> bump up the major version. So this might be a short major
> > > > > version.
> > > > > > > > >>>
> > > > > > > > >>> B. If the maintenance overhead of the deprecated API is
> not
> > > > high,
> > > > > > > > >> therefore
> > > > > > > > >>> keeping it for a long time is affordable. There are also
> > two
> > > > > > options:
> > > > > > > > >>>    B1: Same as A-1, demote the API in the major version
> > bump
> > > > and
> > > > > > > remove
> > > > > > > > >>> the code with a minor version bump.
> > > > > > > > >>>    B2: Keep the API for all the minor versions in the
> major
> > > > > > version,
> > > > > > > > and
> > > > > > > > >>> only remove the code in the next major version.
> > > > > > > > >>>
> > > > > > > > >>> For case B, do we agree that B2 is the way to go?
> > > > > > > > >>>
> > > > > > > > >>> For case A:
> > > > > > > > >>> The following stays the same for A1 and A2:
> > > > > > > > >>>  - users will lose the API after two minor leases. So the
> > > > > migration
> > > > > > > > >> period
> > > > > > > > >>> is the same for A-1 and A-2.
> > > > > > > > >>>  - A1 and A2 will have exactly the same code after the
> > > removal
> > > > of
> > > > > > > > >>> deprecated API, the only difference is versioning.
> > > > > > > > >>>  - To move forward, users need to move to the next minor
> > > > version
> > > > > in
> > > > > > > A1,
> > > > > > > > >> or
> > > > > > > > >>> to the next major version in A2. Because the code are the
> > > same,
> > > > > the
> > > > > > > > >> actual
> > > > > > > > >>> effort to upgrade is the same for A1 and A2.
> > > > > > > > >>>
> > > > > > > > >>> The differences between A-1 and A-2 are:
> > > > > > > > >>>  - A1 allows keeping the major version release cadence.
> A-2
> > > > will
> > > > > > > have a
> > > > > > > > >>> short major version release.
> > > > > > > > >>>  - A1 breaks the well understood API semantic, while A-2
> > does
> > > > > not.
> > > > > > > > >>>
> > > > > > > > >>> From what I see, since there is no well establish
> standard
> > > > > > regarding
> > > > > > > > how
> > > > > > > > >>> long a major version should be released, A short major
> > > version
> > > > > > > release
> > > > > > > > is
> > > > > > > > >>> potential and emotional. It is not ideal but does not
> have
> > > > > material
> > > > > > > > >>> downsides compared with A1. I did not hear anyone
> > complaining
> > > > > about
> > > > > > > > Kafka
> > > > > > > > >>> only has two 1.x release. However, A1 actually breaks the
> > > well
> > > > > > > > understood
> > > > > > > > >>> API semantic, which has more material impact.
> > > > > > > > >>>
> > > > > > > > >>> Also, I'd imagine 90% or more of the Public APIs should
> > fall
> > > > into
> > > > > > > case
> > > > > > > > B.
> > > > > > > > >>> So, short major versions should be very occasional. I'd
> be
> > > very
> > > > > > > > concerned
> > > > > > > > >>> if the reason we choose A1 is simply because we cannot
> > afford
> > > > > > > > >> maintaining a
> > > > > > > > >>> bunch of deprecated APIs until the next major version.
> This
> > > > > > indicates
> > > > > > > > >> that
> > > > > > > > >>> the actual problem we need to solve is to lower the
> > > maintenance
> > > > > > > > overhead
> > > > > > > > >> of
> > > > > > > > >>> deprecated APIs, so that we are comfortable to keep them
> > > > longer.
> > > > > As
> > > > > > > > John
> > > > > > > > >>> and I mentioned earlier, there are ways to achieve this
> and
> > > we
> > > > > need
> > > > > > > to
> > > > > > > > >>> learn how to do it in Flink. Otherwise, our discussion
> > about
> > > > > > > versioning
> > > > > > > > >>> here does not bring much value, because we will end up
> > with a
> > > > > bunch
> > > > > > > of
> > > > > > > > >>> short-lived APIs which upset our users, no matter how we
> > > > version
> > > > > > the
> > > > > > > > >>> releases.
> > > > > > > > >>>
> > > > > > > > >>> So, if there are concrete examples that you think will
> > block
> > > us
> > > > > > from
> > > > > > > > >>> keeping API stability with affordable cost, let's take a
> > look
> > > > > > > together
> > > > > > > > >> and
> > > > > > > > >>> see if that can be improved.
> > > > > > > > >>>
> > > > > > > > >>> Thanks,
> > > > > > > > >>>
> > > > > > > > >>> Jiangjie (Becket) Qin
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> From what I see,
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>> On Mon, Jun 19, 2023 at 4:45 PM Xintong Song <
> > > > > > tonysong...@gmail.com
> > > > > > > > <mailto:tonysong...@gmail.com>>
> > > > > > > > >> wrote:
> > > > > > > > >>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> The part I don't understand is if we are willing to
> have
> > a
> > > > > > > migration
> > > > > > > > >>>>> period, and do a minor version bump to remove an API,
> > what
> > > do
> > > > > we
> > > > > > > > >> lose to
> > > > > > > > >>>> do
> > > > > > > > >>>>> a major version bump instead, so we don't break the
> > common
> > > > > > > versioning
> > > > > > > > >>>>> semantic?
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>> I think we are talking about the cases where a major
> > version
> > > > > bump
> > > > > > > > >> happens
> > > > > > > > >>>> before a deprecated Public API reaches its migration
> > period.
> > > > So
> > > > > > > > >> removing
> > > > > > > > >>>> the API with another major version bump means we have
> two
> > > > > > > consecutive
> > > > > > > > >> major
> > > > > > > > >>>> versions in a very short time. And as previously
> > mentioned,
> > > > > having
> > > > > > > > >> frequent
> > > > > > > > >>>> major version bumps would weaken the value of the
> > commitment
> > > > > > "Public
> > > > > > > > >> API
> > > > > > > > >>>> stay compatible within a major version". I think users
> > also
> > > > have
> > > > > > > > >>>> expectations about how long a major version should live,
> > > which
> > > > > > > should
> > > > > > > > >> be at
> > > > > > > > >>>> least a couple of years rather than 1-2 minor releases.
> > > > > > > > >>>>
> > > > > > > > >>>> This is another option, but I think it is very likely
> more
> > > > > > expensive
> > > > > > > > >> than
> > > > > > > > >>>>> simply bumping the major version. And I imagine there
> > will
> > > be
> > > > > > > > >> questions
> > > > > > > > >>>>> like "why this feature is in 1.20, but does not exist
> in
> > > > 2.0"?
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Yes, it's more expensive than simply bumping the major
> > > > version,
> > > > > > but
> > > > > > > > >> IMHO
> > > > > > > > >>>> it's probably cheaper than carrying the API until the
> next
> > > > major
> > > > > > > > >> version if
> > > > > > > > >>>> we don't bump the major version very soon. And see my
> > reply
> > > > > above
> > > > > > on
> > > > > > > > >> why
> > > > > > > > >>>> not bumping immediately.
> > > > > > > > >>>>
> > > > > > > > >>>> Best,
> > > > > > > > >>>>
> > > > > > > > >>>> Xintong
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> On Mon, Jun 19, 2023 at 4:22 PM Becket Qin <
> > > > > becket....@gmail.com
> > > > > > > > <mailto:becket....@gmail.com>>
> > > > > > > > >> wrote:
> > > > > > > > >>>>
> > > > > > > > >>>>> Hi Xintong,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Please see the replies below.
> > > > > > > > >>>>>
> > > > > > > > >>>>> I see your point, that users would feel surprised if
> they
> > > > find
> > > > > > > > >> things no
> > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> release.
> > > > > > However,
> > > > > > > > >> I'd
> > > > > > > > >>>>> like
> > > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > > similar
> > > > > > > > >> problem
> > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > attention
> > > > > and
> > > > > > > > >> make
> > > > > > > > >>>>> sure
> > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> longer
> > > be
> > > > > > Public
> > > > > > > > >> in
> > > > > > > > >>>>> 2.0.
> > > > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > > > warnings
> > > > > > in
> > > > > > > > >>>> logs,
> > > > > > > > >>>>>> etc.
> > > > > > > > >>>>>
> > > > > > > > >>>>> First of all, I am not a fan of removing PublicEvolving
> > > APIs
> > > > in
> > > > > > > minor
> > > > > > > > >>>>> version changes. Personally speaking, I want them to be
> > > also
> > > > > > > removed
> > > > > > > > >> in
> > > > > > > > >>>>> major version changes.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Empirically, I rarely see projects with complex API
> > > semantic
> > > > > work
> > > > > > > > >> well.
> > > > > > > > >>>>> Many, if not most, users don't read docs / release
> notes
> > /
> > > > > > warnings
> > > > > > > > >> in
> > > > > > > > >>>>> logs. I don't think that is their fault. Typically an
> > > > > application
> > > > > > > > >>>> developer
> > > > > > > > >>>>> will have to deal with dozens of libraries maintained
> by
> > > > > various
> > > > > > > > >>>>> communities / groups. It is just too difficult for them
> > to
> > > > keep
> > > > > > > > >> track of
> > > > > > > > >>>>> all the specific semantics each project puts there. If
> > you
> > > > > think
> > > > > > > > >> about
> > > > > > > > >>>> it,
> > > > > > > > >>>>> how many of Flink developers read all the release notes
> > of
> > > > > Guava,
> > > > > > > > >> Apache
> > > > > > > > >>>>> Commons, Apache Avro, ProtoBuf, etc, when you upgrade a
> > > > version
> > > > > > of
> > > > > > > > >> them?
> > > > > > > > >>>>> One would probably try bumping the dependency version
> and
> > > see
> > > > > if
> > > > > > > > >> there is
> > > > > > > > >>>>> an exception and solve them case by case. And if it is
> a
> > > > minor
> > > > > > > > >> version
> > > > > > > > >>>>> bump, one probably does not expect an exception at all.
> > > > Another
> > > > > > > > >> example
> > > > > > > > >>>> is
> > > > > > > > >>>>> that in Flink we still have so many usage of deprecated
> > > > methods
> > > > > > all
> > > > > > > > >> over
> > > > > > > > >>>>> the place, and I strongly doubt everyone knows when the
> > > > source
> > > > > > code
> > > > > > > > >> of
> > > > > > > > >>>>> these methods are deprecated and when they should be
> > > removed.
> > > > > > > > >>>>>
> > > > > > > > >>>>> So, the most important principle of API is simple and
> > > > > intuitive.
> > > > > > > The
> > > > > > > > >>>>> versioning semantic is a simple and universally
> accepted
> > > API
> > > > > > > > >> stability
> > > > > > > > >>>>> standard. If we ourselves as Flink developers are
> relying
> > > on
> > > > > this
> > > > > > > > >> for our
> > > > > > > > >>>>> own dependencies. I don't think we can expect more from
> > our
> > > > > > users.
> > > > > > > > >>>>>
> > > > > > > > >>>>> Another possible alternative: whenever there's a
> > deprecated
> > > > > > Public
> > > > > > > > >> API
> > > > > > > > >>>> that
> > > > > > > > >>>>>> reaches a major version bump before the migration
> > period,
> > > > and
> > > > > we
> > > > > > > > >> also
> > > > > > > > >>>>> don't
> > > > > > > > >>>>>> want to carry it for all the next major release
> series,
> > we
> > > > may
> > > > > > > > >> consider
> > > > > > > > >>>>>> releasing more minor releases for the previous major
> > > version
> > > > > > after
> > > > > > > > >> the
> > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> > then
> > > we
> > > > > > bump
> > > > > > > > >> to
> > > > > > > > >>>> 2.0,
> > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > > provide
> > > > > > users
> > > > > > > > >>>> another
> > > > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying
> > the
> > > > > > > > >>>> 2-minor-release
> > > > > > > > >>>>>> migration period.
> > > > > > > > >>>>>
> > > > > > > > >>>>> This is another option, but I think it is very likely
> > more
> > > > > > > expensive
> > > > > > > > >> than
> > > > > > > > >>>>> simply bumping the major version. And I imagine there
> > will
> > > be
> > > > > > > > >> questions
> > > > > > > > >>>>> like "why this feature is in 1.20, but does not exist
> in
> > > > 2.0"?
> > > > > > > > >>>>>
> > > > > > > > >>>>> I think my major point is, we should not carry APIs
> > > > deprecated
> > > > > > in a
> > > > > > > > >>>>>> previous major version along all the next major
> version
> > > > > series.
> > > > > > > I'd
> > > > > > > > >>>> like
> > > > > > > > >>>>> to
> > > > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > > > period,
> > > > > as
> > > > > > > > >> long
> > > > > > > > >>>> as
> > > > > > > > >>>>> it
> > > > > > > > >>>>>> does not prevent us from making breaking changes. If
> it
> > > > > doesn't
> > > > > > > > >> work,
> > > > > > > > >>>> I'd
> > > > > > > > >>>>>> be in favor of not providing the migration period, but
> > > > > fallback
> > > > > > to
> > > > > > > > >> only
> > > > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > > > >>>>>
> > > > > > > > >>>>> The part I don't understand is if we are willing to
> have
> > a
> > > > > > > migration
> > > > > > > > >>>>> period, and do a minor version bump to remove an API,
> > what
> > > do
> > > > > we
> > > > > > > > >> lose to
> > > > > > > > >>>> do
> > > > > > > > >>>>> a major version bump instead, so we don't break the
> > common
> > > > > > > versioning
> > > > > > > > >>>>> semantic?
> > > > > > > > >>>>>
> > > > > > > > >>>>> Thanks,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Jiangjie (Becket) Qin
> > > > > > > > >>>>>
> > > > > > > > >>>>>
> > > > > > > > >>>>> On Mon, Jun 19, 2023 at 3:20 PM Xintong Song <
> > > > > > > tonysong...@gmail.com
> > > > > > > > <mailto:tonysong...@gmail.com>>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> As an end user who only uses Public APIs, if I don't
> > > change
> > > > > my
> > > > > > > > >> code
> > > > > > > > >>>> at
> > > > > > > > >>>>>>> all, my expectation is the following:
> > > > > > > > >>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > >>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> issue, I
> > > am
> > > > > fine
> > > > > > > > >> with
> > > > > > > > >>>>> all
> > > > > > > > >>>>>>> the 2.x versions.
> > > > > > > > >>>>>>> Actually I think there are some dependency version
> > > > resolution
> > > > > > > > >>>> policies
> > > > > > > > >>>>>> out
> > > > > > > > >>>>>>> there which picks the highest minor version when the
> > > > > > dependencies
> > > > > > > > >>>> pull
> > > > > > > > >>>>> in
> > > > > > > > >>>>>>> multiple minor versions of the same jar, which may be
> > > > broken
> > > > > if
> > > > > > > > >> we
> > > > > > > > >>>>> remove
> > > > > > > > >>>>>>> the API in minor releases.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I see your point, that users would feel surprised if
> > they
> > > > find
> > > > > > > > >> things
> > > > > > > > >>>> no
> > > > > > > > >>>>>> longer work when upgrading to another 2.x minor
> release.
> > > > > > However,
> > > > > > > > >> I'd
> > > > > > > > >>>>> like
> > > > > > > > >>>>>> to point out that PublicEvolving APIs would have the
> > > similar
> > > > > > > > >> problem
> > > > > > > > >>>>>> anyway. So the question is, how do we catch users'
> > > attention
> > > > > and
> > > > > > > > >> make
> > > > > > > > >>>>> sure
> > > > > > > > >>>>>> they are aware that the Public APIs in 1.x may no
> longer
> > > be
> > > > > > Public
> > > > > > > > >> in
> > > > > > > > >>>>> 2.0.
> > > > > > > > >>>>>> There're various ways to do that, e.g., release notes,
> > > > > warnings
> > > > > > in
> > > > > > > > >>>> logs,
> > > > > > > > >>>>>> etc.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Another possible alternative: whenever there's a
> > > deprecated
> > > > > > Public
> > > > > > > > >> API
> > > > > > > > >>>>> that
> > > > > > > > >>>>>> reaches a major version bump before the migration
> > period,
> > > > and
> > > > > we
> > > > > > > > >> also
> > > > > > > > >>>>> don't
> > > > > > > > >>>>>> want to carry it for all the next major release
> series,
> > we
> > > > may
> > > > > > > > >> consider
> > > > > > > > >>>>>> releasing more minor releases for the previous major
> > > version
> > > > > > after
> > > > > > > > >> the
> > > > > > > > >>>>>> bump. E.g., an Public API is deprecated in 1.19, and
> > then
> > > we
> > > > > > bump
> > > > > > > > >> to
> > > > > > > > >>>> 2.0,
> > > > > > > > >>>>>> we can release one more 1.20 after 2.0. That should
> > > provide
> > > > > > users
> > > > > > > > >>>> another
> > > > > > > > >>>>>> choice rather than upgrading to 2.0, while satisfying
> > the
> > > > > > > > >>>> 2-minor-release
> > > > > > > > >>>>>> migration period.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> I think my major point is, we should not carry APIs
> > > > deprecated
> > > > > > in
> > > > > > > a
> > > > > > > > >>>>>> previous major version along all the next major
> version
> > > > > series.
> > > > > > > I'd
> > > > > > > > >>>> like
> > > > > > > > >>>>> to
> > > > > > > > >>>>>> try giving users more commitments, i.e. the migration
> > > > period,
> > > > > as
> > > > > > > > >> long
> > > > > > > > >>>> as
> > > > > > > > >>>>> it
> > > > > > > > >>>>>> does not prevent us from making breaking changes. If
> it
> > > > > doesn't
> > > > > > > > >> work,
> > > > > > > > >>>> I'd
> > > > > > > > >>>>>> be in favor of not providing the migration period, but
> > > > > fallback
> > > > > > to
> > > > > > > > >> only
> > > > > > > > >>>>>> guarantee the compatibility within the major version.
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Best,
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> Xintong
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>> On Mon, Jun 19, 2023 at 10:48 AM John Roesler <
> > > > > > > vvcep...@apache.org
> > > > > > > > <mailto:vvcep...@apache.org>
> > > > > > > > >>>
> > > > > > > > >>>>> wrote:
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>> Hi Becket,
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thanks for the reply! I’d like to continue the
> > > conversation
> > > > > > about
> > > > > > > > >>>>>>> compatibility outside of this FLIP thread, but for
> > now, I
> > > > can
> > > > > > > > >> accept
> > > > > > > > >>>>> your
> > > > > > > > >>>>>>> decision. It’s certainly an improvement.
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> Thanks again,
> > > > > > > > >>>>>>> John
> > > > > > > > >>>>>>>
> > > > > > > > >>>>>>> On Sun, Jun 18, 2023, at 21:42, Becket Qin wrote:
> > > > > > > > >>>>>>>> Hi John,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Completely agree with all you said.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Can we consider only dropping deprecated APIs in
> major
> > > > > > releases
> > > > > > > > >>>>> across
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > PublicEvolving
> > > > > APIs
> > > > > > > > >> are
> > > > > > > > >>>> by
> > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> should
> > > be
> > > > > > > > >> reflected
> > > > > > > > >>>>> in
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> must
> > > keep
> > > > > > them
> > > > > > > > >>>>> around
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> > > drop
> > > > > them
> > > > > > > > >> in a
> > > > > > > > >>>>>>> minor or
> > > > > > > > >>>>>>>>> patch release.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Personally speaking, I would love to do this, for
> > > exactly
> > > > > the
> > > > > > > > >>>> reason
> > > > > > > > >>>>>> you
> > > > > > > > >>>>>>>> mentioned. However, I did not propose this due to
> the
> > > > > > following
> > > > > > > > >>>>>> reasons:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> 1. I am hesitating a little bit about changing the
> > > > accepted
> > > > > > > > >> FLIPs
> > > > > > > > >>>> too
> > > > > > > > >>>>>>> soon.
> > > > > > > > >>>>>>>> 2. More importantly, to avoid slowing down our
> > > > development.
> > > > > At
> > > > > > > > >> this
> > > > > > > > >>>>>>> point,
> > > > > > > > >>>>>>>> Flink still lacks some design / routines to support
> > good
> > > > API
> > > > > > > > >>>>>>> evolvability /
> > > > > > > > >>>>>>>> extensibility. Just like you said, it takes some
> time
> > to
> > > > be
> > > > > > > > >> good at
> > > > > > > > >>>>>> this.
> > > > > > > > >>>>>>>> In this case, my concern is that only removing
> > > > Experimental
> > > > > /
> > > > > > > > >>>>>>>> PublicEvolving APIs in major version changes may
> > result
> > > in
> > > > > too
> > > > > > > > >> much
> > > > > > > > >>>>>>>> overhead and dramatically slow down the development
> of
> > > > > Flink.
> > > > > > > > >> So, I
> > > > > > > > >>>>> was
> > > > > > > > >>>>>>>> thinking that we can start with the current status.
> > > > > Hopefully
> > > > > > > > >> after
> > > > > > > > >>>>> we
> > > > > > > > >>>>>>> are
> > > > > > > > >>>>>>>> more comfortable with the maintenance overhead of
> > > > deprecated
> > > > > > > > >> APIs,
> > > > > > > > >>>> we
> > > > > > > > >>>>>> can
> > > > > > > > >>>>>>>> then have a stronger guarantee for Experimental /
> > > > > > > > >> PublicEvolving
> > > > > > > > >>>>> APIs.
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Thanks,
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>> On Sun, Jun 18, 2023 at 6:44 AM John Roesler <
> > > > > > > > >> vvcep...@apache.org <mailto:vvcep...@apache.org>>
> > > > > > > > >>>>>>> wrote:
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>> Hi Becket,
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks for this FLIP! Having a deprecation process
> is
> > > > > really
> > > > > > > > >>>>>> important.
> > > > > > > > >>>>>>> I
> > > > > > > > >>>>>>>>> understand some people’s concerns about the
> > additional
> > > > > burden
> > > > > > > > >> for
> > > > > > > > >>>>>>> project
> > > > > > > > >>>>>>>>> maintainers, but my personal experience with Kafka
> > has
> > > > been
> > > > > > > > >> that
> > > > > > > > >>>>> it’s
> > > > > > > > >>>>>>> very
> > > > > > > > >>>>>>>>> liveable and that it’s well worth the benefit to
> > users.
> > > > In
> > > > > > > > >> fact,
> > > > > > > > >>>>> users
> > > > > > > > >>>>>>>>> being able to confidently upgrade is also a benefit
> > to
> > > > > > > > >>>> maintainers,
> > > > > > > > >>>>> as
> > > > > > > > >>>>>>> we
> > > > > > > > >>>>>>>>> will get fewer questions from people stuck on very
> > old
> > > > > > > > >> versions.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> One question:
> > > > > > > > >>>>>>>>> Can we consider only dropping deprecated APIs in
> > major
> > > > > > > > >> releases
> > > > > > > > >>>>> across
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> board? I understand that Experimental and
> > > PublicEvolving
> > > > > APIs
> > > > > > > > >> are
> > > > > > > > >>>> by
> > > > > > > > >>>>>>>>> definition less stable, but it seems like this
> should
> > > be
> > > > > > > > >> reflected
> > > > > > > > >>>>> in
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> required deprecation period alone. I.e. that we
> must
> > > keep
> > > > > > them
> > > > > > > > >>>>> around
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>> at least zero or one minor release, not that we can
> > > drop
> > > > > them
> > > > > > > > >> in a
> > > > > > > > >>>>>>> minor or
> > > > > > > > >>>>>>>>> patch release.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> The advantage of forbidding the removal of any API
> in
> > > > minor
> > > > > > or
> > > > > > > > >>>> patch
> > > > > > > > >>>>>>>>> releases is that users will get a strong guarantee
> > that
> > > > > they
> > > > > > > > >> can
> > > > > > > > >>>>> bump
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>> minor or patch version and still be able to
> compile,
> > or
> > > > > even
> > > > > > > > >> just
> > > > > > > > >>>>>>> re-link
> > > > > > > > >>>>>>>>> and know that they won’t face “MethodDef”
> exceptions
> > at
> > > > run
> > > > > > > > >> time.
> > > > > > > > >>>>> This
> > > > > > > > >>>>>>> is a
> > > > > > > > >>>>>>>>> binary guarantee: if we allow removing  even
> > > Experimental
> > > > > > APIs
> > > > > > > > >>>>> outside
> > > > > > > > >>>>>>> of
> > > > > > > > >>>>>>>>> major releases, users can no longer confidently
> > > upgrade.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Aside from that, I’d share my 2 cents on a couple
> of
> > > > > points:
> > > > > > > > >>>>>>>>> * I’d use the official Deprecated annotation
> instead
> > of
> > > > > > > > >>>> introducing
> > > > > > > > >>>>>> our
> > > > > > > > >>>>>>>>> own flavor (Retired, etc), since Deprecated is well
> > > > > > integrated
> > > > > > > > >>>> into
> > > > > > > > >>>>>>> build
> > > > > > > > >>>>>>>>> tools and IDEs.
> > > > > > > > >>>>>>>>> * I wouldn’t worry about a demotion process in this
> > > FLIP;
> > > > > it
> > > > > > > > >> seems
> > > > > > > > >>>>>>>>> orthogonal, and something that should probably be
> > taken
> > > > > > > > >>>> case-by-case
> > > > > > > > >>>>>>>>> anyway.
> > > > > > > > >>>>>>>>> * Aside from deprecation and removal, there have
> been
> > > > some
> > > > > > > > >>>>> discussions
> > > > > > > > >>>>>>>>> about how to evolve APIs and behavior in compatible
> > > ways.
> > > > > > > > >> This is
> > > > > > > > >>>>>>> somewhat
> > > > > > > > >>>>>>>>> of an art, and if folks haven’t wrestled with it
> > > before,
> > > > > > it’ll
> > > > > > > > >>>> take
> > > > > > > > >>>>>> some
> > > > > > > > >>>>>>>>> time to become good at it. I feel like this topic
> > > should
> > > > > also
> > > > > > > > >> be
> > > > > > > > >>>>>>> orthogonal
> > > > > > > > >>>>>>>>> to this FLIP, but FWIW, my suggestion would be to
> > > adopt a
> > > > > > > > >> simple
> > > > > > > > >>>>>> policy
> > > > > > > > >>>>>>> not
> > > > > > > > >>>>>>>>> to break existing user programs, and leave the
> “how”
> > up
> > > > to
> > > > > > > > >>>>>> implementers
> > > > > > > > >>>>>>> and
> > > > > > > > >>>>>>>>> reviewers.
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> Thanks again,
> > > > > > > > >>>>>>>>> John
> > > > > > > > >>>>>>>>>
> > > > > > > > >>>>>>>>> On Sat, Jun 17, 2023, at 11:03, Jing Ge wrote:
> > > > > > > > >>>>>>>>>> Hi All,
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> The @Public -> @PublicEvolving proposed by Xintong
> > is
> > > a
> > > > > > > > >> great
> > > > > > > > >>>>> idea.
> > > > > > > > >>>>>>>>>> Especially, after he suggest @PublicRetired, i.e.
> > > > > > > > >>>> @PublicEvolving
> > > > > > > > >>>>>> --(2
> > > > > > > > >>>>>>>>>> minor release)--> @Public --> @deprecated --(1
> major
> > > > > > > > >>>>>>>>>> release)--> @PublicRetired. It will provide a lot
> of
> > > > > > > > >> flexibility
> > > > > > > > >>>>>>> without
> > > > > > > > >>>>>>>>>> breaking any rules we had. @Public APIs are
> allowed
> > to
> > > > > > > > >> change
> > > > > > > > >>>>>> between
> > > > > > > > >>>>>>>>> major
> > > > > > > > >>>>>>>>>> releases. Changing annotations is acceptable and
> > > > provides
> > > > > > > > >>>>> additional
> > > > > > > > >>>>>>>>>> tolerance i.e. user-friendliness, since the APIs
> > > > themself
> > > > > > > > >> are
> > > > > > > > >>>> not
> > > > > > > > >>>>>>>>> changed.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> I had similar thoughts when I was facing those
> > > issues. I
> > > > > > > > >> want to
> > > > > > > > >>>>>> move
> > > > > > > > >>>>>>> one
> > > > > > > > >>>>>>>>>> step further and suggest introducing one more
> > > annotation
> > > > > > > > >>>> @Retired.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Not like the @PublicRetired which is a compromise
> of
> > > > > > > > >> downgrading
> > > > > > > > >>>>>>> @Public
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>> @PublicEvolving. As I mentioned earlier in my
> reply,
> > > > Java
> > > > > > > > >>>> standard
> > > > > > > > >>>>>>>>>> @deprecated should be used in the early stage of
> the
> > > > > > > > >> deprecation
> > > > > > > > >>>>>>> process
> > > > > > > > >>>>>>>>>> and doesn't really meet our requirement. Since
> Java
> > > does
> > > > > not
> > > > > > > > >>>> allow
> > > > > > > > >>>>>> us
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>> extend annotation, I think it would be feasible to
> > > have
> > > > > the
> > > > > > > > >> new
> > > > > > > > >>>>>>> @Retired
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>> help us monitor and manage the deprecation
> process,
> > > > house
> > > > > > > > >>>>> cleaning,
> > > > > > > > >>>>>>> etc.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Some ideas could be(open for discussion):
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> @Retired:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> 1. There must be a replacement with functionality
> > > > > > > > >> compatibility
> > > > > > > > >>>>>> before
> > > > > > > > >>>>>>>>> APIs
> > > > > > > > >>>>>>>>>> can be marked as @Retired, i.e. DISCUSS and VOTE
> > > > processes
> > > > > > > > >> on
> > > > > > > > >>>> the
> > > > > > > > >>>>> ML
> > > > > > > > >>>>>>> are
> > > > > > > > >>>>>>>>>> mandatory (a FLIP is recommended).
> > > > > > > > >>>>>>>>>> 2. APIs marked as @Retired will be removed after 1
> > > minor
> > > > > > > > >> release
> > > > > > > > >>>>>>> sharply
> > > > > > > > >>>>>>>>>> (using ArchUnit to force it, needs further check
> > > whether
> > > > > it
> > > > > > > > >> is
> > > > > > > > >>>>>>> possible).
> > > > > > > > >>>>>>>>>> Devs who marked them as @Retired are responsible
> to
> > > > remove
> > > > > > > > >> them.
> > > > > > > > >>>>>>>>>> 3. Both @Public -> @Retired and @PublicEvolving ->
> > > > > @Retired
> > > > > > > > >> are
> > > > > > > > >>>>>>>>>> recommended. @Experimental -> @Retired and
> @Internal
> > > ->
> > > > > > > > >> @Retired
> > > > > > > > >>>>>> could
> > > > > > > > >>>>>>>>> also
> > > > > > > > >>>>>>>>>> be used if it can increase user-friendliness or
> > > > > > > > >>>> dev-friendliness,
> > > > > > > > >>>>>> but
> > > > > > > > >>>>>>> not
> > > > > > > > >>>>>>>>>> mandatory.
> > > > > > > > >>>>>>>>>> 4. Some variables will be defined in @Retired to
> > > support
> > > > > the
> > > > > > > > >>>>>>> deprecation
> > > > > > > > >>>>>>>>>> process management. Further extension is possible,
> > > since
> > > > > the
> > > > > > > > >>>>>>> annotation
> > > > > > > > >>>>>>>>> is
> > > > > > > > >>>>>>>>>> built by us.
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> Best regards,
> > > > > > > > >>>>>>>>>> Jing
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>> On Fri, Jun 16, 2023 at 10:31 AM Becket Qin <
> > > > > > > > >>>> becket....@gmail.com <mailto:becket....@gmail.com>
> > > > > > > > >>>>>>
> > > > > > > > >>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Hi Xintong,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks for the explanation. Please see the
> replies
> > > > inline
> > > > > > > > >>>> below.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I agree. And from my understanding, demoting a
> > Public
> > > > API
> > > > > > > > >> is
> > > > > > > > >>>>> also a
> > > > > > > > >>>>>>>>> kind of
> > > > > > > > >>>>>>>>>>>> such change, just like removing one, which can
> > only
> > > > > > > > >> happen
> > > > > > > > >>>> with
> > > > > > > > >>>>>>> major
> > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> demoting
> > > > > Public
> > > > > > > > >>>> APIs
> > > > > > > > >>>>>>>>> anytime,
> > > > > > > > >>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> before
> > > > > > > > >> reaching
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > demoting
> > > > > > > > >> would
> > > > > > > > >>>> be a
> > > > > > > > >>>>>>> weaker
> > > > > > > > >>>>>>>>>>>> change compared to removing the API immediately
> > upon
> > > > > > > > >> major
> > > > > > > > >>>>>> version
> > > > > > > > >>>>>>>>> bumps,
> > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > 2-minor-release
> > > > > > > > >>>>>> migration
> > > > > > > > >>>>>>>>>>> period.
> > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > `@PublicEvolving`
> > > > > > > > >> sounds
> > > > > > > > >>>>>>> against
> > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation
> if
> > > > > > > > >> necessary,
> > > > > > > > >>>>>> e.g.,
> > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> As an end user who only uses Public APIs, if I
> > don't
> > > > > > > > >> change my
> > > > > > > > >>>>> code
> > > > > > > > >>>>>>> at
> > > > > > > > >>>>>>>>> all,
> > > > > > > > >>>>>>>>>>> my expectation is the following:
> > > > > > > > >>>>>>>>>>> 1. Upgrading from 1.x to 2.x may have issues.
> > > > > > > > >>>>>>>>>>> 2. If I can upgrade from 1.x to 2.x without an
> > > issue, I
> > > > > am
> > > > > > > > >> fine
> > > > > > > > >>>>>> with
> > > > > > > > >>>>>>> all
> > > > > > > > >>>>>>>>>>> the 2.x versions.
> > > > > > > > >>>>>>>>>>> Actually I think there are some dependency
> version
> > > > > > > > >> resolution
> > > > > > > > >>>>>>> policies
> > > > > > > > >>>>>>>>> out
> > > > > > > > >>>>>>>>>>> there which picks the highest minor version when
> > the
> > > > > > > > >>>> dependencies
> > > > > > > > >>>>>>> pull
> > > > > > > > >>>>>>>>> in
> > > > > > > > >>>>>>>>>>> multiple minor versions of the same jar, which
> may
> > be
> > > > > > > > >> broken if
> > > > > > > > >>>>> we
> > > > > > > > >>>>>>>>> remove
> > > > > > > > >>>>>>>>>>> the API in minor releases.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > "legal"
> > > > > that
> > > > > > > > >> we
> > > > > > > > >>>>> bump
> > > > > > > > >>>>>> up
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > needed.
> > > > > > > > >> However,
> > > > > > > > >>>>> this
> > > > > > > > >>>>>>> also
> > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that public
> > APIs
> > > > > will
> > > > > > > > >>>> stay
> > > > > > > > >>>>>>> stable
> > > > > > > > >>>>>>>>>>>> within the major release series, as the series
> can
> > > end
> > > > > > > > >>>> anytime.
> > > > > > > > >>>>>>> IMHO,
> > > > > > > > >>>>>>>>>>> short
> > > > > > > > >>>>>>>>>>>> major release series are not something "make the
> > end
> > > > > > > > >> users
> > > > > > > > >>>>>> happy",
> > > > > > > > >>>>>>> but
> > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> make
> > > > > > > > >> frequent
> > > > > > > > >>>>>> breaking
> > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > approach,
> > > > we
> > > > > > > > >> can
> > > > > > > > >>>>>> still
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > > > Public
> > > > > > > > >> APIs
> > > > > > > > >>>>>>>>> deprecated
> > > > > > > > >>>>>>>>>>> at
> > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > removed
> > > in
> > > > > > > > >> the
> > > > > > > > >>>> next
> > > > > > > > >>>>>>> major
> > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> I totally agree that frequent major version bumps
> > are
> > > > not
> > > > > > > > >>>> ideal,
> > > > > > > > >>>>>> but
> > > > > > > > >>>>>>>>> here
> > > > > > > > >>>>>>>>>>> we are comparing it with a minor version bump
> which
> > > > > > > > >> removes a
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>>>> API.
> > > > > > > > >>>>>>>>>>> So the context is that we have already decided to
> > > > remove
> > > > > > > > >> this
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>> API
> > > > > > > > >>>>>>>>>>> while keeping everything else backwards
> > compatible. I
> > > > > > > > >> think a
> > > > > > > > >>>>> major
> > > > > > > > >>>>>>>>> version
> > > > > > > > >>>>>>>>>>> bump is a commonly understood signal here,
> compared
> > > > with
> > > > > a
> > > > > > > > >>>> minor
> > > > > > > > >>>>>>> version
> > > > > > > > >>>>>>>>>>> change. From end users' perspective, for those
> who
> > > are
> > > > > not
> > > > > > > > >>>>>> impacted,
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>>>> this case upgrading a major version is not
> > > necessarily
> > > > > more
> > > > > > > > >>>>>> involved
> > > > > > > > >>>>>>>>> than
> > > > > > > > >>>>>>>>>>> upgrading a minor version - both should be as
> > smooth
> > > > as a
> > > > > > > > >>>>>> dependency
> > > > > > > > >>>>>>>>>>> version change. For those who are impacted, they
> > will
> > > > > lose
> > > > > > > > >> the
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>>>> API
> > > > > > > > >>>>>>>>>>> anyways and a major version bump ensures there is
> > no
> > > > > > > > >> surprise.
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Thanks,
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> Jiangjie (Becket) Qin
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>> On Fri, Jun 16, 2023 at 10:13 AM Xintong Song <
> > > > > > > > >>>>>> tonysong...@gmail.com <mailto:tonysong...@gmail.com>
> > > > > > > > >>>>>>>>
> > > > > > > > >>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Public API is a well defined common concept, and
> > one
> > > > of
> > > > > > > > >> its
> > > > > > > > >>>>>>>>>>>>> convention is that it only changes with a major
> > > > version
> > > > > > > > >>>>> change.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I agree. And from my understanding, demoting a
> > > Public
> > > > > > > > >> API is
> > > > > > > > >>>>>> also a
> > > > > > > > >>>>>>>>> kind
> > > > > > > > >>>>>>>>>>>> of such change, just like removing one, which
> can
> > > only
> > > > > > > > >> happen
> > > > > > > > >>>>>> with
> > > > > > > > >>>>>>>>> major
> > > > > > > > >>>>>>>>>>>> version bumps. I'm not proposing to allow
> demoting
> > > > > Public
> > > > > > > > >>>> APIs
> > > > > > > > >>>>>>>>> anytime,
> > > > > > > > >>>>>>>>>>> but
> > > > > > > > >>>>>>>>>>>> only in the case major version bumps happen
> before
> > > > > > > > >> reaching
> > > > > > > > >>>> the
> > > > > > > > >>>>>>>>>>>> 2-minor-release migration period. Actually,
> > demoting
> > > > > > > > >> would
> > > > > > > > >>>> be a
> > > > > > > > >>>>>>> weaker
> > > > > > > > >>>>>>>>>>>> change compared to removing the API immediately
> > upon
> > > > > > > > >> major
> > > > > > > > >>>>>> version
> > > > > > > > >>>>>>>>> bumps,
> > > > > > > > >>>>>>>>>>>> in order to keep the commitment about the
> > > > > 2-minor-release
> > > > > > > > >>>>>> migration
> > > > > > > > >>>>>>>>>>> period.
> > > > > > > > >>>>>>>>>>>> If the concern is that `@Public` ->
> > > `@PublicEvolving`
> > > > > > > > >> sounds
> > > > > > > > >>>>>>> against
> > > > > > > > >>>>>>>>>>>> conventions, we may introduce a new annotation
> if
> > > > > > > > >> necessary,
> > > > > > > > >>>>>> e.g.,
> > > > > > > > >>>>>>>>>>>> `@PublicRetiring`, to avoid confusions.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> But it should be
> > > > > > > > >>>>>>>>>>>>> completely OK to bump up the major version if
> we
> > > > really
> > > > > > > > >> want
> > > > > > > > >>>>> to
> > > > > > > > >>>>>>> get
> > > > > > > > >>>>>>>>> rid
> > > > > > > > >>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>> a public API, right?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> I'm not sure about this. Yes, it's completely
> > > "legal"
> > > > > > > > >> that we
> > > > > > > > >>>>>> bump
> > > > > > > > >>>>>>> up
> > > > > > > > >>>>>>>>> the
> > > > > > > > >>>>>>>>>>>> major version whenever a breaking change is
> > needed.
> > > > > > > > >> However,
> > > > > > > > >>>>> this
> > > > > > > > >>>>>>> also
> > > > > > > > >>>>>>>>>>>> weakens the value of the commitment that public
> > APIs
> > > > > will
> > > > > > > > >>>> stay
> > > > > > > > >>>>>>> stable
> > > > > > > > >>>>>>>>>>>> within the major release series, as the series
> can
> > > end
> > > > > > > > >>>> anytime.
> > > > > > > > >>>>>>> IMHO,
> > > > > > > > >>>>>>>>>>> short
> > > > > > > > >>>>>>>>>>>> major release series are not something "make the
> > end
> > > > > > > > >> users
> > > > > > > > >>>>>> happy",
> > > > > > > > >>>>>>> but
> > > > > > > > >>>>>>>>>>>> backdoors that allow us as the developers to
> make
> > > > > > > > >> frequent
> > > > > > > > >>>>>> breaking
> > > > > > > > >>>>>>>>>>>> changes. On the contrary, with the demoting
> > > approach,
> > > > we
> > > > > > > > >> can
> > > > > > > > >>>>>> still
> > > > > > > > >>>>>>>>> have
> > > > > > > > >>>>>>>>>>>> longer major release series, while only allowing
> > > > Public
> > > > > > > > >> APIs
> > > > > > > > >>>>>>>>> deprecated
> > > > > > > > >>>>>>>>>>> at
> > > > > > > > >>>>>>>>>>>> the end of the previous major version to be
> > removed
> > > in
> > > > > > > > >> the
> > > > > > > > >>>> next
> > > > > > > > >>>>>>> major
> > > > > > > > >>>>>>>>>>>> version.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Given our track record I would prefer a regular
> > > cycle
> > > > > > > > >> (1-2
> > > > > > > > >>>>> years)
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and
> not
> > > put
> > > > > it
> > > > > > > > >>>> again
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > expectation
> > > > > on
> > > > > > > > >>>> when
> > > > > > > > >>>>>>>>> breaking
> > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> +1. I personally think 2-3 years would be a good
> > > time
> > > > > > > > >> for new
> > > > > > > > >>>>>> major
> > > > > > > > >>>>>>>>>>>> versions, or longer if there's no breaking
> changes
> > > > > > > > >> needed.
> > > > > > > > >>>> That
> > > > > > > > >>>>>>> makes
> > > > > > > > >>>>>>>>> 1-2
> > > > > > > > >>>>>>>>>>>> year a perfect time to revisit the topic, while
> > > > leaving
> > > > > > > > >> us
> > > > > > > > >>>> more
> > > > > > > > >>>>>>> time
> > > > > > > > >>>>>>>>> to
> > > > > > > > >>>>>>>>>>>> prepare the major release if needed.
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Best,
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> Xintong
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> On Thu, Jun 15, 2023 at 10:09 PM Chesnay
> Schepler
> > <
> > > > > > > > >>>>>>> ches...@apache.org <mailto:ches...@apache.org>
> > > > > > > > >>>>>>>>>>
> > > > > > > > >>>>>>>>>>>> wrote:
> > > > > > > > >>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> It would be valuable if we can avoid releasing
> > > minor
> > > > > > > > >>>>> versions
> > > > > > > > >>>>>>> for
> > > > > > > > >>>>>>>>>>>>> previous
> > > > > > > > >>>>>>>>>>>>>> major versions.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On paper, /absolutely /agree, but I'm not sure
> > how
> > > > > > > > >> viable
> > > > > > > > >>>> that
> > > > > > > > >>>>>> is
> > > > > > > > >>>>>>> in
> > > > > > > > >>>>>>>>>>>>> practice.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On the current 2.0 agenda is potentially
> dropping
> > > > > > > > >> support
> > > > > > > > >>>> for
> > > > > > > > >>>>>> Java
> > > > > > > > >>>>>>>>> 8/11,
> > > > > > > > >>>>>>>>>>>>> which may very well be a problem for our
> current
> > > > users.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 13/06/2023 17:26, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> Thanks for the feedback and sorry for the
> > > confusion
> > > > > > > > >> about
> > > > > > > > >>>>>> Public
> > > > > > > > >>>>>>>>> API
> > > > > > > > >>>>>>>>>>>>>> deprecation. I just noticed that there was a
> > > mistake
> > > > > > > > >> in
> > > > > > > > >>>> the
> > > > > > > > >>>>>>> NOTES
> > > > > > > > >>>>>>>>> part
> > > > > > > > >>>>>>>>>>>>> for
> > > > > > > > >>>>>>>>>>>>>> Public API due to a copy-paste error... I just
> > > fixed
> > > > > > > > >> it.
> > > > > > > > >>>>>>>>>>>>> I'm very relieved to hear that. Glad to hear
> that
> > > we
> > > > > > > > >> are on
> > > > > > > > >>>>> the
> > > > > > > > >>>>>>> same
> > > > > > > > >>>>>>>>>>>>> page on that note.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 15/06/2023 15:20, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> But it should be
> > > > > > > > >>>>>>>>>>>>>> completely OK to bump up the major version if
> we
> > > > > > > > >> really
> > > > > > > > >>>> want
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> get
> > > > > > > > >>>>>>>>>>> rid
> > > > > > > > >>>>>>>>>>>>> of
> > > > > > > > >>>>>>>>>>>>>> a public API, right?
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Technically yes, but look at how long it took
> to
> > > get
> > > > us
> > > > > > > > >> to
> > > > > > > > >>>>> 2.0.
> > > > > > > > >>>>>> ;)
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> There's a separate discussion to be had on the
> > > > cadence
> > > > > > > > >> of
> > > > > > > > >>>>> major
> > > > > > > > >>>>>>>>> releases
> > > > > > > > >>>>>>>>>>>>> going forward, and there seem to be different
> > > > opinions
> > > > > > > > >> on
> > > > > > > > >>>>> that.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> If we take the Kafka example of 2 minor
> releases
> > > > > between
> > > > > > > > >>>> major
> > > > > > > > >>>>>>> ones,
> > > > > > > > >>>>>>>>>>>>> that for us means that users have to
> potentially
> > > deal
> > > > > > > > >> with
> > > > > > > > >>>>>>> breaking
> > > > > > > > >>>>>>>>>>>>> changes every 6 months, which seems like a lot.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Given our track record I would prefer a regular
> > > cycle
> > > > > > > > >> (1-2
> > > > > > > > >>>>>> years)
> > > > > > > > >>>>>>> to
> > > > > > > > >>>>>>>>>>>>> force us to think about this whole topic, and
> not
> > > put
> > > > > it
> > > > > > > > >>>> again
> > > > > > > > >>>>>> to
> > > > > > > > >>>>>>> the
> > > > > > > > >>>>>>>>>>>>> wayside and giving us (and users) a clear
> > > expectation
> > > > > on
> > > > > > > > >>>> when
> > > > > > > > >>>>>>>>> breaking
> > > > > > > > >>>>>>>>>>>>> changes can be made.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> But again, maybe this should be in a separate
> > > thread.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> On 14/06/2023 11:37, Becket Qin wrote:
> > > > > > > > >>>>>>>>>>>>>> Do you have an example of behavioral change in
> > > mind?
> > > > > > > > >> Not
> > > > > > > > >>>>> sure
> > > > > > > > >>>>>> I
> > > > > > > > >>>>>>>>> fully
> > > > > > > > >>>>>>>>>>>>>> understand the concern for behavioral change
> > here.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> This could be a lot of things. It can be
> > > performance
> > > > in
> > > > > > > > >>>>> certain
> > > > > > > > >>>>>>>>>>>>> edge-cases, a bug fix that users (maybe
> > > unknowingly)
> > > > > > > > >> relied
> > > > > > > > >>>>> upon
> > > > > > > > >>>>>>>>>>>>> (
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://www.google.com/url?q=https://xkcd.com/1172/&source=gmail-imap&ust=1687869831000000&usg=AOvVaw2QkbaBd4VKyCsxxFcv1S4I
> > > > > > > ),
> > > > > > > > a semantic change to some
> > > > > > > > >> API.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> For a concrete example, consider the job
> > > submission.
> > > > A
> > > > > > > > >> few
> > > > > > > > >>>>>>> releases
> > > > > > > > >>>>>>>>> back
> > > > > > > > >>>>>>>>>>>>> we made changes such that the initialization of
> > the
> > > > job
> > > > > > > > >>>> master
> > > > > > > > >>>>>>>>> happens
> > > > > > > > >>>>>>>>>>>>> asynchronously.
> > > > > > > > >>>>>>>>>>>>> This meant the job submission call returns
> > sooner,
> > > > and
> > > > > > > > >> the
> > > > > > > > >>>> job
> > > > > > > > >>>>>>> state
> > > > > > > > >>>>>>>>>>>>> enum was extended to cover this state.
> > > > > > > > >>>>>>>>>>>>> API-wise we consider this a compatible change,
> > but
> > > > the
> > > > > > > > >>>>> observed
> > > > > > > > >>>>>>>>> behavior
> > > > > > > > >>>>>>>>>>>>> may be different.
> > > > > > > > >>>>>>>>>>>>>
> > > > > > > > >>>>>>>>>>>>> Metrics are another example; I believe over
> time
> > we
> > > > > > > > >> changed
> > > > > > > > >>>>> what
> > > > > > > > >>>>>>> some
> > > > > > > > >>>>>>>>>>>>> metrics returned a few times.
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>