Re: Discuss making KafkaSubscriber Public

2022-04-20 Thread Mason Chen
Hi all, Just following up on this thread. The subject header may have been misleading--I was proposing to make KafkaSubscriber @PublicEvolving and expose a setter to pass a custom implementation. It seems logical since the KafkaSource is also @PublicEvolving and this lets the user know that the in

Re: Discuss making KafkaSubscriber Public

2022-04-13 Thread Mason Chen
Hi Chesnay, Typically, users want to plug in a KafkaSubscriber that depends on an external system [1][2]. We could also provide a higher level interface that doesn’t depend on the Kafka Admin Client, but I think it would be more flexible to be able to re-use the one created by the enumerator if ne

Re: Discuss making KafkaSubscriber Public

2022-04-13 Thread Chesnay Schepler
Could you expand a bit on possible alternative implementations that require this interface to become public, opposed to providing more built-in ways to subscribe? On 13/04/2022 11:26, Qingsheng Ren wrote: Thanks for the proposal Mason! I think exposing `KafkaSubscriber` as public API is helpf

Re: Discuss making KafkaSubscriber Public

2022-04-13 Thread Qingsheng Ren
Thanks for the proposal Mason! I think exposing `KafkaSubscriber` as public API is helpful for users to implement more complex subscription logics. +1 (non-binding) Cheers, Qingsheng > On Apr 12, 2022, at 11:46, Mason Chen wrote: > > Hi Flink Devs, > > I was looking to contribute to > ht

Discuss making KafkaSubscriber Public

2022-04-11 Thread Mason Chen
Hi Flink Devs, I was looking to contribute to https://issues.apache.org/jira/browse/FLINK-24660, which is a ticket to track changing the KafkaSubscriber from Internal to PublicEvolving. In the PR, it seems a few of us have agreement on making the subscriber pluggable in the KafkaSource, but I'd l