[ https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17519591#comment-17519591 ]
Gyula Fora commented on FLINK-27137: ------------------------------------ [~arvid] "so I'd be inclined to close this ticket with "Won't Do" : let's try to keep a friendly attitude, I never suggested that you would need to do anything, I simply want to discuss this :) As for the context of this ticket. Some managed Kafka environments (including ours) use custom Kafka client implementations. This might restrict the use of certain APIs / functions such as the AdminClient. From an architectural standpoint we should generally try to limit what external APIs we use to increase compatibility between services. Since Flink will absolutely need the KafkaConsumer API for the main purpose of consuming / producing data there is clearly no way around that, but if we can limit the usage of new client interfaces such as AdminClient that can be beneficial. Regular kafka consumers and producer applications do not usually use the AdminClient (as the name also suggests) and one could argue that Flink should try to behave as much as regular kafka applications to increase compatibility. Looking at the OffsetRetriever there are 4 requirements here: getting committed group offsets, getting begin/end offsets and gettings offsets for timestamp. This could all be implemented using the KafkaConsumer (at least looking at the newer API [https://kafka.apache.org/30/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html)] So if we find that everything that we are trying to achieve can be done by the APIs that we already use (KafkaConsumer) then it would best to avoid using the AdminClient which is basically just an extra API dependency at that point without practical gain. If we see that there are some future features that cannot be covered by the KafkaConsumer we should at least make this logic pluggable / optional. > Remove usage of AdminClient from KafkaSource logic > -------------------------------------------------- > > Key: FLINK-27137 > URL: https://issues.apache.org/jira/browse/FLINK-27137 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka > Affects Versions: 1.15.0, 1.14.4 > Reporter: Gyula Fora > Priority: Major > > Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses > the Kafka AdminClient instead of the KafkaConsumer. > It seems that the KafkaConsumer already provides all the necessary > information that the enumerator needs so there is no reason for introducing > the AdminClient. > In some environments using the AdminClient can be problematic even if we are > not using certaing features. -- This message was sent by Atlassian Jira (v8.20.1#820001)