[
https://issues.apache.org/jira/browse/FLINK-25509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17708741#comment-17708741
]
Dong Lin commented on FLINK-25509:
----------------------------------
[~martijnvisser] Hmm... I am not sure there is consensus in the community that
externalized connectors should only use public APIs. Was this documented or
discussed on the mailing list?
I agree it is preferred to have externalized connectors using only public APIs.
But it is also important that we only expose a minimum set of well-designed
APIs rather than simply making the existing APIs public. This is non-trivial
effort and someone in the community needs to voluntary to do this. Before this
is done, I guess we have to stick with the fact that externalized repos will be
using internal APIs in the near future.
I guess we probably need to update the given externalized repo to only use
public APIs before having nightly build to verify that it can always compile
with the master head.
In this particular case, we have added an abstract method `removeSplits()` to
SplitFetcherManager. This is a backward incompatible change for externalized
repos like flink-connector-pulsar. This is because PulsarSourceFetcherManager
should override removeSplits() to compile with flink 1.18-snapshot. Then it
will fail to compile with flink-1.17 as there is no such method to override in
flink 1.17.
The simplest way to address this issue is to update Flink master branch to make
`removeSplits()` non-abstract. But IMO this approach makes the code hard to
read and is very unconventional.
I would prefer to build externalized repo in the same way as what most users
would do. That is, the master branch should only be compiled against one
version of Flink. And it can have a separate branch for each extra Flink
version. What do you think?
> FLIP-208: Add RecordEvaluator to dynamically stop source based on
> de-serialized records
> ---------------------------------------------------------------------------------------
>
> Key: FLINK-25509
> URL: https://issues.apache.org/jira/browse/FLINK-25509
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / Common, Connectors / Kafka
> Reporter: Dong Lin
> Assignee: Hang Ruan
> Priority: Major
> Labels: pull-request-available
>
> This feature is needed to migrate applications which uses
> KafkaDeserializationSchema::isEndOfStream() from using FlinkKafkaConsumer to
> using KafkaSource.
> Please checkout
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-208%3A+Add+RecordEvaluator+to+dynamically+stop+source+based+on+de-serialized+records
> for the motivation and the proposed changes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
