Hi Xingtong, Option 1, secure plan would be:
1. graduate kafka, File, JDBC connectors to @Public 2. graduate SinkV2 to @Public 3. remove SinkFunction. Option 2, risky plan but at a fast pace: 1. graduate SinkV2 to @Public and expecting more maintenance effort since there are many known and unsolved issues. 2. remove SinkFunction. 3. It depends on the connectors' contributors whether connectors can upgrade to Flink 2.0, since we moved forward with SinkV2 API without taking care of implementations in external connectors. I am ok with both of them and personally prefer option 1. Best regards, Jing On Fri, Jun 30, 2023 at 3:41 AM Xintong Song <tonysong...@gmail.com> wrote: > I see. Thanks for the explanation. I may have not looked into this deeply > enough, and would trust the decision from you and the community members who > participated in the discussion & vote. > > Best, > > Xintong > > > > On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov < > alexander.fedu...@gmail.com> wrote: > > > > However, I'm not sure about 2. > > > > I am not aware of a bylaw that states the specific requirements in order > to > > mark something as @Deprecated. My understanding from the discussion and > the > > vote was that the community recognizes the necessity to make it explicit > > that > > the usage of the SourceFunction API is discouraged. This can actually > > stimulate > > authors of connectors that rely on this very specific and non-baseline > > functionality to contribute extensions to the new Source API themselves > in > > order to > > close the gap. ExternallyInducedSource, for instance, was driven by > Pravega > > to > > begin with, since it was only needed for their purposes [1]. We are not > > removing > > anything - until 2.0 everything will continue to work and we can work on > > resolving the limitations until then, I personally don't see a big issue > > here. > > > > >Do you think it is feasible to resolve them by the feature freeze date > of > > 1.18? > > No, these are rather complex additions that would probably require > FLIP(s). > > > > [1] > > > > > https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration > > > > On Thu, 29 Jun 2023 at 14:25, Xintong Song <tonysong...@gmail.com> > wrote: > > > > > Thanks for the explanation, Alex. > > > > > > Not blocking the deprecation on 1 & 3 makes sense to me. However, I'm > not > > > sure about 2. > > > > > > It sounds to me that, without FLINK-28051 & FLINK-28054, some of the > > > connectors cannot migrate to the new Source API, or at least further > > > investigation is needed to understand the situation. If this is the > case, > > > we probably should not deprecate the API until these issues are > resolved. > > > Do you think it is feasible to resolve them by the feature freeze date > of > > > 1.18? > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov < > > > alexander.fedu...@gmail.com> wrote: > > > > > > > @Xintong > > > > The original discussion [1] and vote [2] converged on the idea that > it > > is > > > > better > > > > to make it clear to the users that they should stop using > > SourceFunction > > > > since it > > > > is going away. The longer we do not have this indication, the more > user > > > > implementations will be based on it and the more pain will be induced > > > when > > > > we > > > > finally drop it. Users now have an alternative API that they should > use > > > and > > > > which > > > > is fully functional, from that perspective nothing blocks marking it > > > > @Deprecated. > > > > As for the remaining work items - there are primarily three kinds: > > > > > > > > 1. Where Flink internally uses SourceFunction, without exposing this > > fact > > > > to the > > > > outside world: > > > > - FLINK-28050 [3] > > > > - FLINK-28229 [4] > > > > - FLINK-28048 [5] > > > > > > > > 2. Very specific edge cases that might not be covered by the Source > API > > > as > > > > is: > > > > - FLINK-28054 [6] > > > > - FLINK-28051 [7] > > > > > > > > 3. Usability improvements - something that was easily doable with > > > > SourceFunction, > > > > but requires deep knowledge of the new, significantly more > complex, > > > > Source API > > > > to achieve: > > > > - FLINK-28056 [8] > > > > > > > > In my mind, none of those are blockers for proceeding with adding the > > > > @Deprecated > > > > annotation: > > > > (1) is a simple case of encapsulation, internals should not concern > the > > > API > > > > users > > > > (2) is really only relevant for "exotic" use cases. Does not mean we > > > should > > > > not > > > > consider those, but since it is irrelevant for 99.9% of the users, I > do > > > not > > > > think > > > > we should get stuck here. > > > > (3) is purely a nice to have. Formally speaking, all of the tools are > > > > there, it is > > > > just that due to the complexity of the new Source API some "simple" > > > things > > > > become > > > > non-trivial and ideally we want to do better here. > > > > > > > > [1] https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > > > [2] https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > > > [3] https://issues.apache.org/jira/browse/FLINK-28050 > > > > [4] https://issues.apache.org/jira/browse/FLINK-28229 > > > > [5] https://issues.apache.org/jira/browse/FLINK-28048 > > > > [6] https://issues.apache.org/jira/browse/FLINK-28054 > > > > [7] https://issues.apache.org/jira/browse/FLINK-28051 > > > > [8] https://issues.apache.org/jira/browse/FLINK-28056 > > > > > > > > On Thu, 29 Jun 2023 at 13:13, Xintong Song <tonysong...@gmail.com> > > > wrote: > > > > > > > > > Thanks for the inputs, Martijn, Jing and Alex. > > > > > > > > > > @Martijn, > > > > > Regarding the Scala supports, I personally don't think "a fully > > striked > > > > > through experience in the IDE" is something we want to avoid, > > > especially > > > > > given that we are planning to remove the deprecated APIs soon, > unlike > > > > when > > > > > FLINK-29740 was resolved we didn't really know when they would be > > > > removed. > > > > > Moreover, the even entry point for DataStream Scala > > > > > (`StreamExecutionEnvironment`) is not annotated. > > > > > > > > > > > > > > > @Jing and @Alex, > > > > > > > > > > IIUC, you mean SourceFunction can be annotated as `@Deprecated` in > > > 1.18, > > > > > and there's already a PR doing so. However, after the deprecation, > > > there > > > > > are still issues that need to be addressed before removing it in > 2.0? > > > > This > > > > > sounds a bit weird. If the API cannot be dropped, which means > without > > > > this > > > > > API some of functions cannot be supported, then how could it be > > > > deprecated? > > > > > How would we expect users to migrate away from it? > > > > > > > > > > > > > > > @Jing, > > > > > > > > > > Sounds like it's impractical to deprecate SinkFunction in 1.18. Any > > > > > expectation / plan on when / how it can be deprecated / removed? > > > > > > > > > > > > > > > Best, > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > On Thu, Jun 29, 2023 at 6:12 PM Alexander Fedulov < > > > > > alexander.fedu...@gmail.com> wrote: > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > Thanks for bringing up this topic. I can provide some details > > > regarding > > > > > > the SourceFunction deprecation efforts. Marking SourceFunction as > > > > > > deprecated was not possible until now since we have stringent > > > compiler > > > > > > checks in flink-examples against using any deprecated APIs. We > > > actually > > > > > > merged the migration of all examples to the new FLIP-27-based > > > > > > DataGeneratorSource [1] just two days ago [2]. Now the PR marking > > > > > > it @Deprecated is finally unblocked [3] (I would be grateful if > you > > > > could > > > > > > merge it). > > > > > > > > > > > > With regards to the Flink 2.0 scope, I compiled a list of items > > > > required > > > > > to > > > > > > be able to drop the SourceFunction API [4] a while ago and as you > > can > > > > > see, > > > > > > there is still quite some work to be done. Some items [5] might > > even > > > > > > require additions to the new Source API. Overall, I am happy to > > take > > > > > > ownership of completing this work package. > > > > > > > > > > > > Best, > > > > > > Alex > > > > > > > > > > > > > > > > > > [1] https://cwiki.apache.org/confluence/x/9Av1D > > > > > > [2] https://github.com/apache/flink/pull/21774 > > > > > > [3] https://github.com/apache/flink/pull/20049 > > > > > > [4] https://issues.apache.org/jira/browse/FLINK-28045 > > > > > > [5] https://issues.apache.org/jira/browse/FLINK-28054 > > > > > > > > > > > > On Thu, 29 Jun 2023 at 10:45, Martijn Visser < > > > martijnvis...@apache.org > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Xintong, > > > > > > > > > > > > > > With regards to the deprecation of the Scala APIs, during the > PR > > > > > > > review it was requested to not mark all APIs as deprecated but > > only > > > > > > > the entry point [1], to avoid a fully striked through > experience > > in > > > > > > > the IDE. I think the same idea was applicable on the DataSet > > API. I > > > > > > > think it depends on how formal we want to treat this: if really > > > > > > > formal, then we should deprecate them in 1.18. I think in both > > > cases, > > > > > > > it's quite well known that they are deprecated. I'm +0 for > either > > > > way, > > > > > > > as long as we're all agreeing that they can be removed in 2.0. > > > > > > > > > > > > > > With regards to Queryable State and Source/SinkFunction, +1 to > > mark > > > > > > > these as deprecated. > > > > > > > > > > > > > > Best regards, > > > > > > > > > > > > > > Martijn > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808 > > > > > > > > > > > > > > On Thu, Jun 29, 2023 at 10:23 AM Xintong Song < > > > tonysong...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi devs, > > > > > > > > > > > > > > > > Looking at the release 2.0 proposals [1], I noticed that many > > > APIs > > > > > that > > > > > > > are > > > > > > > > proposed to be removed in 2.0 are not (fully) deprecated yet. > > We > > > > > might > > > > > > > want > > > > > > > > to properly mark them as `@Deprecated` in 1.18 if we agree > they > > > > > should > > > > > > be > > > > > > > > removed in 2.0. Moreover, according to FLIP-321 [2] (not > voted > > > yet > > > > > but > > > > > > > IMO > > > > > > > > is close to consensus IMO), a migration period is required > > after > > > > APIs > > > > > > are > > > > > > > > deprecated and before they can be removed. > > > > > > > > > > > > > > > > I might not be familiar with the status of all the APIs > below. > > So > > > > I'd > > > > > > > like > > > > > > > > to bring them up and see if there's any concern regarding > > > > deprecating > > > > > > > them > > > > > > > > in 1.18. If there's concern for deprecating API, we can > start a > > > > > > separate > > > > > > > > discussion thread for it. For those with no objections, I'd > > > create > > > > > JIRA > > > > > > > > tickets and try to properly deprecate them in 1.18. > > > > > > > > > > > > > > > > 1. DataSet API > > > > > > > > It's described as "legacy", "soft deprecated" in user > > > documentation > > > > > > [3]. > > > > > > > > However, it's not annotated with `@Deprecated` in codes. > > > According > > > > to > > > > > > > > FLIP-131 [4], DataSet API should be deprecated when > DataStream > > > API > > > > > and > > > > > > > > Table API / SQL meet certain requirements. AFAICS, all the > > > > > requirements > > > > > > > > mentioned in the FLIP are already fulfilled. We should > annotate > > > it > > > > as > > > > > > > > `@Deprecated` now. > > > > > > > > > > > > > > > > 2. SourceFunction / SinkFunction > > > > > > > > They are described as deprecated in the roadmap[5], and I > don't > > > > find > > > > > > > > anything regarding them in user documentation. But they are > > also > > > > not > > > > > > > > annotated with `@Deprecated` in codes. TBH, I'm not aware of > > any > > > > > formal > > > > > > > > decision to deprecate these. AFAICS, the replacement for > > > > > SourceFunction > > > > > > > > (Source) has already been promoted to `@Public`, while the > > > > > replacement > > > > > > > for > > > > > > > > SinkFunction (SinkV2) is still `@PublicEvolving`. I found a > > > > > > discussion[6] > > > > > > > > regarding promoting SinkV2 to `@Public`, but it's unclear to > me > > > > what > > > > > > the > > > > > > > > conclusion is. > > > > > > > > > > > > > > > > 3. Queryable State > > > > > > > > It's described as approaching end-of-life in the roadmap [5], > > but > > > > is > > > > > > > > neither deprecated in codes nor in user documentation [7]. I > > also > > > > > > found a > > > > > > > > discussion [8] about rescuing it from deprecation, and it > seems > > > to > > > > me > > > > > > > there > > > > > > > > are more negative opinions than positive ones. > > > > > > > > > > > > > > > > 4. All Scala APIs > > > > > > > > I think we agreed to drop Scala API support in FLIP-265 [9], > > and > > > > have > > > > > > > tried > > > > > > > > to deprecate them in FLINK-29740 [10]. Also, both user > > > > documentation > > > > > > and > > > > > > > > roadmap[5] shows that scala API supports are deprecated. > > However, > > > > > > AFAICS, > > > > > > > > none of the APIs in `flink-streaming-scala` are annotated > with > > > > > > > > `@Deprecated`, and only `ExecutionEnvironment` and `package` > > are > > > > > marked > > > > > > > > `@Deprecated` in `flink-scala`. > > > > > > > > > > > > > > > > Looking forward to your feedback. > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > Xintong > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > > > > > > > > > > > > > > > [2] > > > > https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > > > > > > > > > > > > > > > [3] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/ > > > > > > > > > > > > > > > > [4] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > > > > > > > > > > > > > > > > [5] https://flink.apache.org/roadmap/ > > > > > > > > > > > > > > > > [6] > > > > https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx > > > > > > > > > > > > > > > > [7] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/ > > > > > > > > > > > > > > > > [8] > > > > https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m > > > > > > > > > > > > > > > > [9] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support > > > > > > > > > > > > > > > > [10] https://issues.apache.org/jira/browse/FLINK-29740 > > > > > > > > > > > > > > > > > > > > > > > > > > > >
