Hi Xingtong,
Option 1, secure plan would be:
1. graduate kafka, File, JDBC connectors to @Public
2. graduate SinkV2 to @Public
3. remove SinkFunction.
Option 2, risky plan but at a fast pace:
1. graduate SinkV2 to @Public and expecting more maintenance effort since
there are many known and unsolved issues.
2. remove SinkFunction.
3. It depends on the connectors' contributors whether connectors can
upgrade to Flink 2.0, since we moved forward with SinkV2 API without
taking
care of implementations in external connectors.
I am ok with both of them and personally prefer option 1.
Best regards,
Jing
On Fri, Jun 30, 2023 at 3:41 AM Xintong Song <tonysong...@gmail.com>
wrote:
I see. Thanks for the explanation. I may have not looked into this
deeply
enough, and would trust the decision from you and the community members
who
participated in the discussion & vote.
Best,
Xintong
On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:
However, I'm not sure about 2.
I am not aware of a bylaw that states the specific requirements in
order
to
mark something as @Deprecated. My understanding from the discussion
and
the
vote was that the community recognizes the necessity to make it
explicit
that
the usage of the SourceFunction API is discouraged. This can actually
stimulate
authors of connectors that rely on this very specific and non-baseline
functionality to contribute extensions to the new Source API
themselves
in
order to
close the gap. ExternallyInducedSource, for instance, was driven by
Pravega
to
begin with, since it was only needed for their purposes [1]. We are
not
removing
anything - until 2.0 everything will continue to work and we can work
on
resolving the limitations until then, I personally don't see a big
issue
here.
Do you think it is feasible to resolve them by the feature freeze
date
of
1.18?
No, these are rather complex additions that would probably require
FLIP(s).
[1]
https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration
On Thu, 29 Jun 2023 at 14:25, Xintong Song <tonysong...@gmail.com>
wrote:
Thanks for the explanation, Alex.
Not blocking the deprecation on 1 & 3 makes sense to me. However,
I'm
not
sure about 2.
It sounds to me that, without FLINK-28051 & FLINK-28054, some of the
connectors cannot migrate to the new Source API, or at least further
investigation is needed to understand the situation. If this is the
case,
we probably should not deprecate the API until these issues are
resolved.
Do you think it is feasible to resolve them by the feature freeze
date
of
1.18?
Best,
Xintong
On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:
@Xintong
The original discussion [1] and vote [2] converged on the idea
that
it
is
better
to make it clear to the users that they should stop using
SourceFunction
since it
is going away. The longer we do not have this indication, the more
user
implementations will be based on it and the more pain will be
induced
when
we
finally drop it. Users now have an alternative API that they
should
use
and
which
is fully functional, from that perspective nothing blocks marking
it
@Deprecated.
As for the remaining work items - there are primarily three kinds:
1. Where Flink internally uses SourceFunction, without exposing
this
fact
to the
outside world:
- FLINK-28050 [3]
- FLINK-28229 [4]
- FLINK-28048 [5]
2. Very specific edge cases that might not be covered by the
Source
API
as
is:
- FLINK-28054 [6]
- FLINK-28051 [7]
3. Usability improvements - something that was easily doable with
SourceFunction,
but requires deep knowledge of the new, significantly more
complex,
Source API
to achieve:
- FLINK-28056 [8]
In my mind, none of those are blockers for proceeding with adding
the
@Deprecated
annotation:
(1) is a simple case of encapsulation, internals should not
concern
the
API
users
(2) is really only relevant for "exotic" use cases. Does not mean
we
should
not
consider those, but since it is irrelevant for 99.9% of the
users, I
do
not
think
we should get stuck here.
(3) is purely a nice to have. Formally speaking, all of the tools
are
there, it is
just that due to the complexity of the new Source API some
"simple"
things
become
non-trivial and ideally we want to do better here.
[1]
https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9
[2]
https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v
[3] https://issues.apache.org/jira/browse/FLINK-28050
[4] https://issues.apache.org/jira/browse/FLINK-28229
[5] https://issues.apache.org/jira/browse/FLINK-28048
[6] https://issues.apache.org/jira/browse/FLINK-28054
[7] https://issues.apache.org/jira/browse/FLINK-28051
[8] https://issues.apache.org/jira/browse/FLINK-28056
On Thu, 29 Jun 2023 at 13:13, Xintong Song <tonysong...@gmail.com
wrote:
Thanks for the inputs, Martijn, Jing and Alex.
@Martijn,
Regarding the Scala supports, I personally don't think "a fully
striked
through experience in the IDE" is something we want to avoid,
especially
given that we are planning to remove the deprecated APIs soon,
unlike
when
FLINK-29740 was resolved we didn't really know when they would
be
removed.
Moreover, the even entry point for DataStream Scala
(`StreamExecutionEnvironment`) is not annotated.
@Jing and @Alex,
IIUC, you mean SourceFunction can be annotated as `@Deprecated`
in
1.18,
and there's already a PR doing so. However, after the
deprecation,
there
are still issues that need to be addressed before removing it in
2.0?
This
sounds a bit weird. If the API cannot be dropped, which means
without
this
API some of functions cannot be supported, then how could it be
deprecated?
How would we expect users to migrate away from it?
@Jing,
Sounds like it's impractical to deprecate SinkFunction in 1.18.
Any
expectation / plan on when / how it can be deprecated / removed?
Best,
Xintong
On Thu, Jun 29, 2023 at 6:12 PM Alexander Fedulov <
alexander.fedu...@gmail.com> wrote:
Hi Xintong,
Thanks for bringing up this topic. I can provide some details
regarding
the SourceFunction deprecation efforts. Marking
SourceFunction as
deprecated was not possible until now since we have stringent
compiler
checks in flink-examples against using any deprecated APIs. We
actually
merged the migration of all examples to the new FLIP-27-based
DataGeneratorSource [1] just two days ago [2]. Now the PR
marking
it @Deprecated is finally unblocked [3] (I would be grateful
if
you
could
merge it).
With regards to the Flink 2.0 scope, I compiled a list of
items
required
to
be able to drop the SourceFunction API [4] a while ago and as
you
can
see,
there is still quite some work to be done. Some items [5]
might
even
require additions to the new Source API. Overall, I am happy
to
take
ownership of completing this work package.
Best,
Alex
[1] https://cwiki.apache.org/confluence/x/9Av1D
[2] https://github.com/apache/flink/pull/21774
[3] https://github.com/apache/flink/pull/20049
[4] https://issues.apache.org/jira/browse/FLINK-28045
[5] https://issues.apache.org/jira/browse/FLINK-28054
On Thu, 29 Jun 2023 at 10:45, Martijn Visser <
martijnvis...@apache.org
wrote:
Hi Xintong,
With regards to the deprecation of the Scala APIs, during
the
PR
review it was requested to not mark all APIs as deprecated
but
only
the entry point [1], to avoid a fully striked through
experience
in
the IDE. I think the same idea was applicable on the DataSet
API. I
think it depends on how formal we want to treat this: if
really
formal, then we should deprecate them in 1.18. I think in
both
cases,
it's quite well known that they are deprecated. I'm +0 for
either
way,
as long as we're all agreeing that they can be removed in
2.0.
With regards to Queryable State and Source/SinkFunction, +1
to
mark
these as deprecated.
Best regards,
Martijn
[1]
https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808
On Thu, Jun 29, 2023 at 10:23 AM Xintong Song <
tonysong...@gmail.com
wrote:
Hi devs,
Looking at the release 2.0 proposals [1], I noticed that
many
APIs
that
are
proposed to be removed in 2.0 are not (fully) deprecated
yet.
We
might
want
to properly mark them as `@Deprecated` in 1.18 if we agree
they
should
be
removed in 2.0. Moreover, according to FLIP-321 [2] (not
voted
yet
but
IMO
is close to consensus IMO), a migration period is required
after
APIs
are
deprecated and before they can be removed.
I might not be familiar with the status of all the APIs
below.
So
I'd
like
to bring them up and see if there's any concern regarding
deprecating
them
in 1.18. If there's concern for deprecating API, we can
start a
separate
discussion thread for it. For those with no objections,
I'd
create
JIRA
tickets and try to properly deprecate them in 1.18.
1. DataSet API
It's described as "legacy", "soft deprecated" in user
documentation
[3].
However, it's not annotated with `@Deprecated` in codes.
According
to
FLIP-131 [4], DataSet API should be deprecated when
DataStream
API
and
Table API / SQL meet certain requirements. AFAICS, all the
requirements
mentioned in the FLIP are already fulfilled. We should
annotate
it
as
`@Deprecated` now.
2. SourceFunction / SinkFunction
They are described as deprecated in the roadmap[5], and I
don't
find
anything regarding them in user documentation. But they
are
also
not
annotated with `@Deprecated` in codes. TBH, I'm not aware
of
any
formal
decision to deprecate these. AFAICS, the replacement for
SourceFunction
(Source) has already been promoted to `@Public`, while the
replacement
for
SinkFunction (SinkV2) is still `@PublicEvolving`. I found
a
discussion[6]
regarding promoting SinkV2 to `@Public`, but it's unclear
to
me
what
the
conclusion is.
3. Queryable State
It's described as approaching end-of-life in the roadmap
[5],
but
is
neither deprecated in codes nor in user documentation
[7]. I
also
found a
discussion [8] about rescuing it from deprecation, and it
seems
to
me
there
are more negative opinions than positive ones.
4. All Scala APIs
I think we agreed to drop Scala API support in FLIP-265
[9],
and
have
tried
to deprecate them in FLINK-29740 [10]. Also, both user
documentation
and
roadmap[5] shows that scala API supports are deprecated.
However,
AFAICS,
none of the APIs in `flink-streaming-scala` are annotated
with
`@Deprecated`, and only `ExecutionEnvironment` and
`package`
are
marked
`@Deprecated` in `flink-scala`.
Looking forward to your feedback.
Best,
Xintong
[1]
https://cwiki.apache.org/confluence/display/FLINK/2.0+Release
[2]
https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9
[3]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/
[4]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741
[5] https://flink.apache.org/roadmap/
[6]
https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx
[7]
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/
[8]
https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m
[9]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support
[10] https://issues.apache.org/jira/browse/FLINK-29740
