Hi, Thanks Martijn for bringing this to our attention. This discussion deserves a much wider range of attention. because graduation of the Sink API will have an impact on all connectors.
For me, briefly speaking, like I already pointed out in another thread [1], it is too hasty and risky to do it in 1.17 and even in 1.18. Here are some reasons I knew and there might be more: 1. commonly, it is fine to follow FLIP-179 [2], since most API has only one implementation. Two release cycles for @PublicEvolving should be good enough to test the API design and implementation and get feedback and make the API stable. But, Connector API (including the Sink API) is a special case. There will be more than 10 different implementations, Some of them are very important for Flink users' daily work. Rules defined in FLIP-179 should be adapted in this case. 2. Currently, afaik, there are only two Sink API implementations and none of them is graduated. KafkaSink is @PublicEvloving and FileSink is @Experimental. It is very risky to graduate the fundamental Sink API before none of its implementation has been graduated. 3. We have an awkward situation with one of the most important connectors, the Kafka connector. The KafakSink is @PublicEvolving but the FlinkKafkaConsumer is already deprecated. Users are confused and some of them are still sticking to FlinkKafkaConsumer, since it is normally not recommended to use @PublicEvolving in production. I tried to graduate the KafkaSink and remove the FlinkKafkaProducer to push users using KafkaSink in this thread [3]. But there are some tasks to finish as a prerequisite. Things got more complicated as expected and some new rules need to be defined and discussed. I will write a new FLIP to address this issue later. 4. Even if we could graduate the Kafka connector, it is still risky to graduate the SinkV2 API, which will make the Kafka connector special comparing to others, i.e. kafka connector as the first class vs. others as the second class. This is not what we want to see. 5. Afaik, there are still some fresh construction wrt the SinkV2 @Yun Gao <yungao...@aliyun.com> 6. Flink wants to be the unified batch and stream processing framework. It might be rational, if a more batch oriented Sink implementation could be done and graduated before we graduate the Sink API. The JDBC connector might be a feasible choice. Therefore, a solid solution should be: 1. Develop three SinkV2 implementations. Considering batch, stream, popularity, etc. I personally would choose Kafka, File, and JDBC. But the list deserves further discussion. 2. graduate all three SinkV2 implementations and remove old implementations. If there are any rejections in the community for removing the old implementation. It means the related SinkV2 implementation is not ready to graduate. We will need more time. 3. 1 - 2 release cycles to get feedback from users wrt those SinkV2 implementations. 3. graduate the SinkV2 API. Now we could see, if we could find developers focusing on them and deliver them on time, we would still need at least 4 release cycles before we could graduate the Sink API. If we want to pace up and could live with some risk, we could go with the following process: 1. Remove FlinkKafkaProucer and graduate KafkaSink in Flink 1.18 in the ideal case. 2. Get feedback and stabilize the API while we continue working on Flink and release 1.19 3. granduate the Sink API in Flink 1.20 4. be aware that we might have to develop SinkV3 if any unknown requirements become known after new Sink implementation is developed. Please pay attention, this process is still hasty and risky. Because it only satisfied Kafka and might have a big impact on other connectors, i.e. we are treating the Kafka connector as the king. Please don't get me wrong. I am not trying to block this change. On the contrary, I am keen to graduate SinkV2 API and remove old API and implementations. But I think it is important to be aware of the risk and make sure everyone is on the same page. There might be some other information I don't know and I am happy to hear any opposite opinions that can push the SinkV2 API to be graduated asap. Last but not least, this discussion starts during the holiday season in China. @Qingsheng Ren <renqs...@gmail.com> @Jark Wu <imj...@gmail.com> I would like to have your attention please. Best regards, Jing [1] https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1 [2] https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process [3] https://lists.apache.org/thread/m3o48c2d8j9g5t9s89hqs6qvr924s71o On Thu, Jan 19, 2023 at 3:26 AM Lijie Wang <wangdachui9...@gmail.com> wrote: > Hi Martijn, > > Thanks for driving this. I have a only concern about the Sink.InitContext. > > Does the Sink.InitContext will also be changed to @Public ? As described in > FLIP-287, currently the Sink.InitContext still lacks some necessary > information to migrate existing connectors to new sinks. If it is marked as > public/stable, we can no longer modify it in the future(since most > connectors are not migrated to SinkV2 currently, we may find we need more > information via InitContext in the future migrations). > > Best, > Lijie > > Yun Tang <myas...@live.com> 于2023年1月18日周三 21:13写道： > > > SinkV2 was introduced in Flink-1.15 and annotated as @PublicEvolving from > > the 1st day [1]. From FLIP-197, we can promote it to @Public since it > > already existed with two releases. > > And I didn't find a FLIP to discuss the process to deprecate APIs, > > considering the SinkFunction has actually been stale for some time, I > think > > we can deprecate it with the @Public SinkV2. > > > > Thus, +1 (binding) for this proposal. > > > > [1] https://issues.apache.org/jira/browse/FLINK-25555 > > > > Best > > Yun Tang > > > > ________________________________ > > From: Martijn Visser <martijnvis...@apache.org> > > Sent: Wednesday, January 18, 2023 18:50 > > To: dev <dev@flink.apache.org>; Jing Ge <j...@ververica.com>; Yun Tang < > > myas...@live.com> > > Subject: [DISCUSS] Promote SinkV2 to @Public and deprecate SinkFunction > > > > Hi all, > > > > While discussing FLIP-281 [1] the discussion also turned to the > > SinkFunction and the SinkV2 API. For a broader discussion I'm opening up > a > > separate discussion thread. > > > > As Yun Tang has mentioned in that discussion thread, it would be a good > > time to deprecate the SinkFunction to avoid the need to introduce new > > functions towards (to be) deprecated APIs. Jing rightfully mentioned that > > it would be confusing to deprecate the SinkFunction if its successor is > not > > yet marked as @Public (it's currently @PublicEvolving). > > > > My proposal would be to promote the SinkV2 API to @public in Flink 1.17 > > and mark the SinkFunction as @deprecated in Flink 1.17 > > > > The original Sink interface was introduced in Flink 1.12 with FLIP-143 > [2] > > and extended with FLIP-177 in Flink 1.14 [3] and has been improved on > > further as Sink V2 via FLIP-191 in Flink 1.15 [4]. > > > > Looking at the API stability graduation process [5], the fact that Sink > V2 > > was introduced in Flink 1.15 would mean that we could warrant a promotion > > to @public already (given that there have been two releases with 1.15 and > > 1.16 where it was introduced). Combined with the fact that SinkV2 has > been > > the result of iteration over the introduction of the original Sink API > > since Flink 1.12, I would argue that the promotion is overdue. > > > > If we promote the Sink API to @public, I think we should also immediately > > mark the SinkFunction as @deprecated. > > > > Looking forward to your thoughts. > > > > Best regards, > > > > Martijn > > > > > > [1] https://lists.apache.org/thread/l05m6cf8fwkkbpnjtzbg9l2lo40oxzd1 > > [2] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-143%3A+Unified+Sink+API > > [3] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-177%3A+Extend+Sink+API > > [4] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-191%3A+Extend+unified+Sink+interface+to+support+small+file+compaction > > [5] > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-197%3A+API+stability+graduation+process > > > > >
