[
https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519852#comment-17519852
]
dengziming edited comment on FLINK-27137 at 4/9/22 12:50 AM:
-------------------------------------------------------------
[~arvid] In fact, Spark also use Adminclient to get offset in
[https://github.com/apache/spark/pull/29729] .
[~gyfora] Thank you for reporting this, here is some my opinions.
> Since Flink will absolutely need the KafkaConsumer API
Event though we are doomed to use KafkaConsumer, however KafkaConsumer is
created in TaskManager whereas we seem to get offsets in JobManager? so maybe
we will not create duplicated component here. And KafkaConsumer contains extra
component, for example `Fetcher` and `ConsumerCoordinator` which may send extra
rpc requests.
> regular kafka applications to increase compatibility
AdminClient is also supported by Kafka community and thus is compatible and
robust as KafkaConsumer, it's designed just like the Flink public API.
> In some environments using the AdminClient can be problematic
If AdminClient has bugs we should report it to Kafka community to help improve
it.
> Looking at the OffsetRetriever there are 4 requirements here
In fact there maybe new type of OffsetSpec, for example max-timestamp in
KIP-734, even though it's trivial but indicate AdminClient is better concerning
offset management.
was (Author: dengziming):
[~arvid] In fact, Spark also use Adminclient to get offset in
[https://github.com/apache/spark/pull/29729] .
[~gyfora]
> Since Flink will absolutely need the KafkaConsumer API
Event though we are doomed to use KafkaConsumer, however KafkaConsumer is
created in TaskManager whereas we seem to get offsets in JobManager? so maybe
we will not create duplicated component here. And KafkaConsumer contains extra
component, for example `Fetcher` and `ConsumerCoordinator` which may send extra
rpc requests.
> regular kafka applications to increase compatibility
AdminClient is also supported by Kafka community and thus is compatible and
robust as KafkaConsumer, it's designed just like the Flink public API.
> In some environments using the AdminClient can be problematic
If AdminClient has bugs we should report it to Kafka community to help improve
it.
> Looking at the OffsetRetriever there are 4 requirements here
In fact there maybe new type of OffsetSpec, for example max-timestamp in
KIP-734, even though it's trivial but indicate AdminClient is better concerning
offset management.
> Remove usage of AdminClient from KafkaSource logic
> --------------------------------------------------
>
> Key: FLINK-27137
> URL: https://issues.apache.org/jira/browse/FLINK-27137
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
> Affects Versions: 1.15.0, 1.14.4
> Reporter: Gyula Fora
> Priority: Major
>
> Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses
> the Kafka AdminClient instead of the KafkaConsumer.
> It seems that the KafkaConsumer already provides all the necessary
> information that the enumerator needs so there is no reason for introducing
> the AdminClient.
> In some environments using the AdminClient can be problematic even if we are
> not using certaing features.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
