[ https://issues.apache.org/jira/browse/FLINK-31483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17703064#comment-17703064 ]
Hang Ruan edited comment on FLINK-31483 at 3/21/23 7:05 AM: ------------------------------------------------------------ Hi, [~ruibin] , I found the mails about this part. Hope it is helpful. [1] [https://lists.apache.org/thread/7r4h7v5k281w9cnbfw9lb8tp56r30lwt] [2] [https://lists.apache.org/thread/4h3xl25zlys8wzhtlrrmlcpht62oglrm] [3] [https://lists.apache.org/thread/6cv801kp3r4y6tytf82p45zvoxfo3p07] was (Author: ruanhang1993): Hi, [~ruibin] , I have found the mails about this part. Hope it is helpful. [1] [https://lists.apache.org/thread/7r4h7v5k281w9cnbfw9lb8tp56r30lwt] [2] [https://lists.apache.org/thread/4h3xl25zlys8wzhtlrrmlcpht62oglrm] [3] [https://lists.apache.org/thread/6cv801kp3r4y6tytf82p45zvoxfo3p07] > Implement Split Deletion Support in Flink Kafka Connector > --------------------------------------------------------- > > Key: FLINK-31483 > URL: https://issues.apache.org/jira/browse/FLINK-31483 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka, Connectors / Parent > Reporter: Ruibin Xing > Priority: Major > > Currently, the Flink Kafka Connector does not support split deletion and is > left as a > [TODO|[https://github.com/apache/flink-connector-kafka/blob/9f72be91f8abdfc9b5e8fa46d15dee3f83e71332/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/enumerator/KafkaSourceEnumerator.java#L305]]. > I want to add this feature by doing these steps: > 1. Add SplitsDeletion event to flink-connector-base, which currently only has > SplitsAddition. > 2. Add a `deleteSplits` method in SplitEnumeratorContext, so it can send a > SplitsDeletion event to the source operator. To maintain compatibility, a > default empty implementation for this method will be added. > 3. Make SourceOperator handle the SplitsDeletion event, notifiying the > SourceReader to delete splits. > 4. Create a deleteSplits method in SourceReader to remove splits, including > remove them from Split state and stopping SourceReader from reading the > deleted splits. > As an alternative, without modifying the flink-connector-base, > KafkaSplitsEnumerator could send a custom SourceEvent to SourceOperator for > splits deletion and deal with it in the kafka-connector-specific code. But I > think it's better to have SplitsDeletion in flink-connector-base, so other > connectors can use it too. > Let me know if you have any thoughts or ideas. Thanks! > Related Issues: FLINK-30490 -- This message was sent by Atlassian Jira (v8.20.10#820010)