Hi Alex,
> > 3. remove SinkFunction. > Which steps do you imply for the 1.18 release and for the 2.0 release? > for 2.0 release. 1.18 will be released soon. Best regards, Jing On Wed, Jul 5, 2023 at 1:08 PM Alexander Fedulov < alexander.fedu...@gmail.com> wrote: > @Jing > Just to clarify, when you say: > > 3. remove SinkFunction. > Which steps do you imply for the 1.18 release and for the 2.0 release? @Xintong > A side note - with the new Source API we lose the ability to control > checkpointing from the source since there is no lock anymore. This > functionality > is currently used in a variety of tests for the Sinks - the tests that rely > on tight > synchronization between specific elements passed from the source to the > sink before > allowing a checkpoint to complete (see FiniteTestSource [1]). Since FLIP-27 > Sources rely > on decoupling via the mailbox, without exposing the lock, it is not > immediately clear > if it is possible to achieve the same functionality without major > extensions in the > runtime for such testing purposes. My hope initially was that only the > legacy Sinks > relied on this - this would have made it possible to drop > SourceFunction+SinkFunction > together, but, in fact, it also already became part of the new SinkV2 > testing IT suits > [2]. Moreover, I know of at least one major connector that also relies on > it for > verifying committed sink metadata for a specific set of records (Iceberg) > [3]. In my > estimation this currently presents a major blocker for the SourceFunction > removal. > > [1] > > https://github.com/apache/flink/blob/master/flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/streaming/util/FiniteTestSource.java > [2] > > https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/sink/StreamingExecutionFileSinkITCase.java#L132 > [3] > > https://github.com/apache/iceberg/blob/master/flink/v1.17/flink/src/test/java/org/apache/iceberg/flink/source/BoundedTestSource.java#L75C1-L85C2 > > Best, > Alex > > On Wed, 5 Jul 2023 at 10:47, Chesnay Schepler <ches...@apache.org> wrote: > > > There's a whole bunch of metric APIs that would need to be deprecated. > > That is of course if the metric FLIPs are being accepted. > > > > Which makes me wonder if we aren't doing things the wrong way around; > > shouldn't the decision to deprecate an API be part of the FLIP > discussion? > > > > On 05/07/2023 07:39, Xintong Song wrote: > > > Thanks all for the discussion. > > > > > > It seems to me there's a consensus on marking the following as > deprecated > > > in 1.18: > > > - DataSet API > > > - SourceFunction > > > - Queryable State > > > - All Scala APIs > > > > > > More time is needed for deprecating SinkFunction. > > > > > > I'll leave this discussion open for a few more days. And if there's no > > > objections, I'll create JIRA tickets accordingly. > > > > > > Best, > > > > > > Xintong > > > > > > > > > > > > On Wed, Jul 5, 2023 at 1:34 PM Xintong Song <tonysong...@gmail.com> > > wrote: > > > > > >> Thanks for the input, Jing. I'd also be +1 for option 1. > > >> > > >> Best, > > >> > > >> Xintong > > >> > > >> > > >> > > >> On Mon, Jul 3, 2023 at 7:20 PM Jing Ge <j...@ververica.com.invalid> > > wrote: > > >> > > >>> Hi Xingtong, > > >>> > > >>> Option 1, secure plan would be: > > >>> > > >>> 1. graduate kafka, File, JDBC connectors to @Public > > >>> 2. graduate SinkV2 to @Public > > >>> 3. remove SinkFunction. > > >>> > > >>> Option 2, risky plan but at a fast pace: > > >>> > > >>> 1. graduate SinkV2 to @Public and expecting more maintenance effort > > since > > >>> there are many known and unsolved issues. > > >>> 2. remove SinkFunction. > > >>> 3. It depends on the connectors' contributors whether connectors can > > >>> upgrade to Flink 2.0, since we moved forward with SinkV2 API without > > >>> taking > > >>> care of implementations in external connectors. > > >>> > > >>> I am ok with both of them and personally prefer option 1. > > >>> > > >>> Best regards, > > >>> Jing > > >>> > > >>> > > >>> On Fri, Jun 30, 2023 at 3:41 AM Xintong Song <tonysong...@gmail.com> > > >>> wrote: > > >>> > > >>>> I see. Thanks for the explanation. I may have not looked into this > > >>> deeply > > >>>> enough, and would trust the decision from you and the community > > members > > >>> who > > >>>> participated in the discussion & vote. > > >>>> > > >>>> Best, > > >>>> > > >>>> Xintong > > >>>> > > >>>> > > >>>> > > >>>> On Thu, Jun 29, 2023 at 10:28 PM Alexander Fedulov < > > >>>> alexander.fedu...@gmail.com> wrote: > > >>>> > > >>>>>> However, I'm not sure about 2. > > >>>>> I am not aware of a bylaw that states the specific requirements in > > >>> order > > >>>> to > > >>>>> mark something as @Deprecated. My understanding from the discussion > > >>> and > > >>>> the > > >>>>> vote was that the community recognizes the necessity to make it > > >>> explicit > > >>>>> that > > >>>>> the usage of the SourceFunction API is discouraged. This can > actually > > >>>>> stimulate > > >>>>> authors of connectors that rely on this very specific and > > non-baseline > > >>>>> functionality to contribute extensions to the new Source API > > >>> themselves > > >>>> in > > >>>>> order to > > >>>>> close the gap. ExternallyInducedSource, for instance, was driven by > > >>>> Pravega > > >>>>> to > > >>>>> begin with, since it was only needed for their purposes [1]. We are > > >>> not > > >>>>> removing > > >>>>> anything - until 2.0 everything will continue to work and we can > work > > >>> on > > >>>>> resolving the limitations until then, I personally don't see a big > > >>> issue > > >>>>> here. > > >>>>> > > >>>>>> Do you think it is feasible to resolve them by the feature freeze > > >>> date > > >>>> of > > >>>>> 1.18? > > >>>>> No, these are rather complex additions that would probably require > > >>>> FLIP(s). > > >>>>> [1] > > >>>>> > > >>>>> > > >>> > > > https://flink.apache.org/2022/01/20/pravega-flink-connector-101/#checkpoint-integration > > >>>>> On Thu, 29 Jun 2023 at 14:25, Xintong Song <tonysong...@gmail.com> > > >>>> wrote: > > >>>>>> Thanks for the explanation, Alex. > > >>>>>> > > >>>>>> Not blocking the deprecation on 1 & 3 makes sense to me. However, > > >>> I'm > > >>>> not > > >>>>>> sure about 2. > > >>>>>> > > >>>>>> It sounds to me that, without FLINK-28051 & FLINK-28054, some of > the > > >>>>>> connectors cannot migrate to the new Source API, or at least > further > > >>>>>> investigation is needed to understand the situation. If this is > the > > >>>> case, > > >>>>>> we probably should not deprecate the API until these issues are > > >>>> resolved. > > >>>>>> Do you think it is feasible to resolve them by the feature freeze > > >>> date > > >>>> of > > >>>>>> 1.18? > > >>>>>> > > >>>>>> Best, > > >>>>>> > > >>>>>> Xintong > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Thu, Jun 29, 2023 at 8:02 PM Alexander Fedulov < > > >>>>>> alexander.fedu...@gmail.com> wrote: > > >>>>>> > > >>>>>>> @Xintong > > >>>>>>> The original discussion [1] and vote [2] converged on the idea > > >>> that > > >>>> it > > >>>>> is > > >>>>>>> better > > >>>>>>> to make it clear to the users that they should stop using > > >>>>> SourceFunction > > >>>>>>> since it > > >>>>>>> is going away. The longer we do not have this indication, the > more > > >>>> user > > >>>>>>> implementations will be based on it and the more pain will be > > >>> induced > > >>>>>> when > > >>>>>>> we > > >>>>>>> finally drop it. Users now have an alternative API that they > > >>> should > > >>>> use > > >>>>>> and > > >>>>>>> which > > >>>>>>> is fully functional, from that perspective nothing blocks marking > > >>> it > > >>>>>>> @Deprecated. > > >>>>>>> As for the remaining work items - there are primarily three > kinds: > > >>>>>>> > > >>>>>>> 1. Where Flink internally uses SourceFunction, without exposing > > >>> this > > >>>>> fact > > >>>>>>> to the > > >>>>>>> outside world: > > >>>>>>> - FLINK-28050 [3] > > >>>>>>> - FLINK-28229 [4] > > >>>>>>> - FLINK-28048 [5] > > >>>>>>> > > >>>>>>> 2. Very specific edge cases that might not be covered by the > > >>> Source > > >>>> API > > >>>>>> as > > >>>>>>> is: > > >>>>>>> - FLINK-28054 [6] > > >>>>>>> - FLINK-28051 [7] > > >>>>>>> > > >>>>>>> 3. Usability improvements - something that was easily doable with > > >>>>>>> SourceFunction, > > >>>>>>> but requires deep knowledge of the new, significantly more > > >>>> complex, > > >>>>>>> Source API > > >>>>>>> to achieve: > > >>>>>>> - FLINK-28056 [8] > > >>>>>>> > > >>>>>>> In my mind, none of those are blockers for proceeding with adding > > >>> the > > >>>>>>> @Deprecated > > >>>>>>> annotation: > > >>>>>>> (1) is a simple case of encapsulation, internals should not > > >>> concern > > >>>> the > > >>>>>> API > > >>>>>>> users > > >>>>>>> (2) is really only relevant for "exotic" use cases. Does not mean > > >>> we > > >>>>>> should > > >>>>>>> not > > >>>>>>> consider those, but since it is irrelevant for 99.9% of the > > >>> users, I > > >>>> do > > >>>>>> not > > >>>>>>> think > > >>>>>>> we should get stuck here. > > >>>>>>> (3) is purely a nice to have. Formally speaking, all of the tools > > >>> are > > >>>>>>> there, it is > > >>>>>>> just that due to the complexity of the new Source API some > > >>> "simple" > > >>>>>> things > > >>>>>>> become > > >>>>>>> non-trivial and ideally we want to do better here. > > >>>>>>> > > >>>>>>> [1] > > >>> https://lists.apache.org/thread/d6cwqw9b3105wcpdkwq7rr4s7x4ywqr9 > > >>>>>>> [2] > > >>> https://lists.apache.org/thread/kv9rj3w2rmkb8jtss5bqffhw57or7v8v > > >>>>>>> [3] https://issues.apache.org/jira/browse/FLINK-28050 > > >>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-28229 > > >>>>>>> [5] https://issues.apache.org/jira/browse/FLINK-28048 > > >>>>>>> [6] https://issues.apache.org/jira/browse/FLINK-28054 > > >>>>>>> [7] https://issues.apache.org/jira/browse/FLINK-28051 > > >>>>>>> [8] https://issues.apache.org/jira/browse/FLINK-28056 > > >>>>>>> > > >>>>>>> On Thu, 29 Jun 2023 at 13:13, Xintong Song < > tonysong...@gmail.com > > >>>>>> wrote: > > >>>>>>>> Thanks for the inputs, Martijn, Jing and Alex. > > >>>>>>>> > > >>>>>>>> @Martijn, > > >>>>>>>> Regarding the Scala supports, I personally don't think "a fully > > >>>>> striked > > >>>>>>>> through experience in the IDE" is something we want to avoid, > > >>>>>> especially > > >>>>>>>> given that we are planning to remove the deprecated APIs soon, > > >>>> unlike > > >>>>>>> when > > >>>>>>>> FLINK-29740 was resolved we didn't really know when they would > > >>> be > > >>>>>>> removed. > > >>>>>>>> Moreover, the even entry point for DataStream Scala > > >>>>>>>> (`StreamExecutionEnvironment`) is not annotated. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> @Jing and @Alex, > > >>>>>>>> > > >>>>>>>> IIUC, you mean SourceFunction can be annotated as `@Deprecated` > > >>> in > > >>>>>> 1.18, > > >>>>>>>> and there's already a PR doing so. However, after the > > >>> deprecation, > > >>>>>> there > > >>>>>>>> are still issues that need to be addressed before removing it in > > >>>> 2.0? > > >>>>>>> This > > >>>>>>>> sounds a bit weird. If the API cannot be dropped, which means > > >>>> without > > >>>>>>> this > > >>>>>>>> API some of functions cannot be supported, then how could it be > > >>>>>>> deprecated? > > >>>>>>>> How would we expect users to migrate away from it? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> @Jing, > > >>>>>>>> > > >>>>>>>> Sounds like it's impractical to deprecate SinkFunction in 1.18. > > >>> Any > > >>>>>>>> expectation / plan on when / how it can be deprecated / removed? > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> Best, > > >>>>>>>> > > >>>>>>>> Xintong > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Thu, Jun 29, 2023 at 6:12 PM Alexander Fedulov < > > >>>>>>>> alexander.fedu...@gmail.com> wrote: > > >>>>>>>> > > >>>>>>>>> Hi Xintong, > > >>>>>>>>> > > >>>>>>>>> Thanks for bringing up this topic. I can provide some details > > >>>>>> regarding > > >>>>>>>>> the SourceFunction deprecation efforts. Marking > > >>> SourceFunction as > > >>>>>>>>> deprecated was not possible until now since we have stringent > > >>>>>> compiler > > >>>>>>>>> checks in flink-examples against using any deprecated APIs. We > > >>>>>> actually > > >>>>>>>>> merged the migration of all examples to the new FLIP-27-based > > >>>>>>>>> DataGeneratorSource [1] just two days ago [2]. Now the PR > > >>> marking > > >>>>>>>>> it @Deprecated is finally unblocked [3] (I would be grateful > > >>> if > > >>>> you > > >>>>>>> could > > >>>>>>>>> merge it). > > >>>>>>>>> > > >>>>>>>>> With regards to the Flink 2.0 scope, I compiled a list of > > >>> items > > >>>>>>> required > > >>>>>>>> to > > >>>>>>>>> be able to drop the SourceFunction API [4] a while ago and as > > >>> you > > >>>>> can > > >>>>>>>> see, > > >>>>>>>>> there is still quite some work to be done. Some items [5] > > >>> might > > >>>>> even > > >>>>>>>>> require additions to the new Source API. Overall, I am happy > > >>> to > > >>>>> take > > >>>>>>>>> ownership of completing this work package. > > >>>>>>>>> > > >>>>>>>>> Best, > > >>>>>>>>> Alex > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> [1] https://cwiki.apache.org/confluence/x/9Av1D > > >>>>>>>>> [2] https://github.com/apache/flink/pull/21774 > > >>>>>>>>> [3] https://github.com/apache/flink/pull/20049 > > >>>>>>>>> [4] https://issues.apache.org/jira/browse/FLINK-28045 > > >>>>>>>>> [5] https://issues.apache.org/jira/browse/FLINK-28054 > > >>>>>>>>> > > >>>>>>>>> On Thu, 29 Jun 2023 at 10:45, Martijn Visser < > > >>>>>> martijnvis...@apache.org > > >>>>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hi Xintong, > > >>>>>>>>>> > > >>>>>>>>>> With regards to the deprecation of the Scala APIs, during > > >>> the > > >>>> PR > > >>>>>>>>>> review it was requested to not mark all APIs as deprecated > > >>> but > > >>>>> only > > >>>>>>>>>> the entry point [1], to avoid a fully striked through > > >>>> experience > > >>>>> in > > >>>>>>>>>> the IDE. I think the same idea was applicable on the DataSet > > >>>>> API. I > > >>>>>>>>>> think it depends on how formal we want to treat this: if > > >>> really > > >>>>>>>>>> formal, then we should deprecate them in 1.18. I think in > > >>> both > > >>>>>> cases, > > >>>>>>>>>> it's quite well known that they are deprecated. I'm +0 for > > >>>> either > > >>>>>>> way, > > >>>>>>>>>> as long as we're all agreeing that they can be removed in > > >>> 2.0. > > >>>>>>>>>> With regards to Queryable State and Source/SinkFunction, +1 > > >>> to > > >>>>> mark > > >>>>>>>>>> these as deprecated. > > >>>>>>>>>> > > >>>>>>>>>> Best regards, > > >>>>>>>>>> > > >>>>>>>>>> Martijn > > >>>>>>>>>> > > >>>>>>>>>> [1] > > >>>>>>>>>> > > >>>> > > https://github.com/apache/flink/pull/21176#pullrequestreview-1159706808 > > >>>>>>>>>> On Thu, Jun 29, 2023 at 10:23 AM Xintong Song < > > >>>>>> tonysong...@gmail.com > > >>>>>>>>>> wrote: > > >>>>>>>>>>> Hi devs, > > >>>>>>>>>>> > > >>>>>>>>>>> Looking at the release 2.0 proposals [1], I noticed that > > >>> many > > >>>>>> APIs > > >>>>>>>> that > > >>>>>>>>>> are > > >>>>>>>>>>> proposed to be removed in 2.0 are not (fully) deprecated > > >>> yet. > > >>>>> We > > >>>>>>>> might > > >>>>>>>>>> want > > >>>>>>>>>>> to properly mark them as `@Deprecated` in 1.18 if we agree > > >>>> they > > >>>>>>>> should > > >>>>>>>>> be > > >>>>>>>>>>> removed in 2.0. Moreover, according to FLIP-321 [2] (not > > >>>> voted > > >>>>>> yet > > >>>>>>>> but > > >>>>>>>>>> IMO > > >>>>>>>>>>> is close to consensus IMO), a migration period is required > > >>>>> after > > >>>>>>> APIs > > >>>>>>>>> are > > >>>>>>>>>>> deprecated and before they can be removed. > > >>>>>>>>>>> > > >>>>>>>>>>> I might not be familiar with the status of all the APIs > > >>>> below. > > >>>>> So > > >>>>>>> I'd > > >>>>>>>>>> like > > >>>>>>>>>>> to bring them up and see if there's any concern regarding > > >>>>>>> deprecating > > >>>>>>>>>> them > > >>>>>>>>>>> in 1.18. If there's concern for deprecating API, we can > > >>>> start a > > >>>>>>>>> separate > > >>>>>>>>>>> discussion thread for it. For those with no objections, > > >>> I'd > > >>>>>> create > > >>>>>>>> JIRA > > >>>>>>>>>>> tickets and try to properly deprecate them in 1.18. > > >>>>>>>>>>> > > >>>>>>>>>>> 1. DataSet API > > >>>>>>>>>>> It's described as "legacy", "soft deprecated" in user > > >>>>>> documentation > > >>>>>>>>> [3]. > > >>>>>>>>>>> However, it's not annotated with `@Deprecated` in codes. > > >>>>>> According > > >>>>>>> to > > >>>>>>>>>>> FLIP-131 [4], DataSet API should be deprecated when > > >>>> DataStream > > >>>>>> API > > >>>>>>>> and > > >>>>>>>>>>> Table API / SQL meet certain requirements. AFAICS, all the > > >>>>>>>> requirements > > >>>>>>>>>>> mentioned in the FLIP are already fulfilled. We should > > >>>> annotate > > >>>>>> it > > >>>>>>> as > > >>>>>>>>>>> `@Deprecated` now. > > >>>>>>>>>>> > > >>>>>>>>>>> 2. SourceFunction / SinkFunction > > >>>>>>>>>>> They are described as deprecated in the roadmap[5], and I > > >>>> don't > > >>>>>>> find > > >>>>>>>>>>> anything regarding them in user documentation. But they > > >>> are > > >>>>> also > > >>>>>>> not > > >>>>>>>>>>> annotated with `@Deprecated` in codes. TBH, I'm not aware > > >>> of > > >>>>> any > > >>>>>>>> formal > > >>>>>>>>>>> decision to deprecate these. AFAICS, the replacement for > > >>>>>>>> SourceFunction > > >>>>>>>>>>> (Source) has already been promoted to `@Public`, while the > > >>>>>>>> replacement > > >>>>>>>>>> for > > >>>>>>>>>>> SinkFunction (SinkV2) is still `@PublicEvolving`. I found > > >>> a > > >>>>>>>>> discussion[6] > > >>>>>>>>>>> regarding promoting SinkV2 to `@Public`, but it's unclear > > >>> to > > >>>> me > > >>>>>>> what > > >>>>>>>>> the > > >>>>>>>>>>> conclusion is. > > >>>>>>>>>>> > > >>>>>>>>>>> 3. Queryable State > > >>>>>>>>>>> It's described as approaching end-of-life in the roadmap > > >>> [5], > > >>>>> but > > >>>>>>> is > > >>>>>>>>>>> neither deprecated in codes nor in user documentation > > >>> [7]. I > > >>>>> also > > >>>>>>>>> found a > > >>>>>>>>>>> discussion [8] about rescuing it from deprecation, and it > > >>>> seems > > >>>>>> to > > >>>>>>> me > > >>>>>>>>>> there > > >>>>>>>>>>> are more negative opinions than positive ones. > > >>>>>>>>>>> > > >>>>>>>>>>> 4. All Scala APIs > > >>>>>>>>>>> I think we agreed to drop Scala API support in FLIP-265 > > >>> [9], > > >>>>> and > > >>>>>>> have > > >>>>>>>>>> tried > > >>>>>>>>>>> to deprecate them in FLINK-29740 [10]. Also, both user > > >>>>>>> documentation > > >>>>>>>>> and > > >>>>>>>>>>> roadmap[5] shows that scala API supports are deprecated. > > >>>>> However, > > >>>>>>>>> AFAICS, > > >>>>>>>>>>> none of the APIs in `flink-streaming-scala` are annotated > > >>>> with > > >>>>>>>>>>> `@Deprecated`, and only `ExecutionEnvironment` and > > >>> `package` > > >>>>> are > > >>>>>>>> marked > > >>>>>>>>>>> `@Deprecated` in `flink-scala`. > > >>>>>>>>>>> > > >>>>>>>>>>> Looking forward to your feedback. > > >>>>>>>>>>> > > >>>>>>>>>>> Best, > > >>>>>>>>>>> > > >>>>>>>>>>> Xintong > > >>>>>>>>>>> > > >>>>>>>>>>> > > >>>>>>>>>>> [1] > > >>>>>> https://cwiki.apache.org/confluence/display/FLINK/2.0+Release > > >>>>>>>>>>> [2] > > >>>>>>> https://lists.apache.org/thread/vmhzv8fcw2b33pqxp43486owrxbkd5x9 > > >>>>>>>>>>> [3] > > >>>>>>>>>>> > > >>> > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/dataset/overview/ > > >>>>>>>>>>> [4] > > >>>>>>>>>>> > > >>> > > > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741 > > >>>>>>>>>>> [5] https://flink.apache.org/roadmap/ > > >>>>>>>>>>> > > >>>>>>>>>>> [6] > > >>>>>>> https://lists.apache.org/thread/q62nj89rrz0t5xtggy5n65on95f2rmmx > > >>>>>>>>>>> [7] > > >>>>>>>>>>> > > >>> > > > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/queryable_state/ > > >>>>>>>>>>> [8] > > >>>>>>> https://lists.apache.org/thread/9hmwcjb3q5c24pk3qshjvybfqk62v17m > > >>>>>>>>>>> [9] > > >>>>>>>>>>> > > >>> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-265+Deprecate+and+remove+Scala+API+support > > >>>>>>>>>>> [10] https://issues.apache.org/jira/browse/FLINK-29740 > > > > > > >
