[jira] [Commented] (FLINK-27872) Allow KafkaBuilder to set arbitrary subscribers

2022-06-06 Thread Qingsheng Ren (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17550788#comment-17550788
 ] 

Qingsheng Ren commented on FLINK-27872:
---

I think this duplicates FLINK-24660.

> Allow KafkaBuilder to set arbitrary subscribers
> ---
>
> Key: FLINK-27872
> URL: https://issues.apache.org/jira/browse/FLINK-27872
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors / Kafka
>Reporter: Salva
>Priority: Major
>
> Currently, 
> [KafkaSourceBuilder|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java]
>  has two setters:
>  * 
> [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
>  * 
> [setTopicPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
> which under the hood instantiate the corresponding (concrete) subscribers. 
> This covers the most common needs, I agree, but it might fall short in some 
> cases. Why not add a more generic setter:
>  * {{setSubscriber (???)}}
> Otherwise, how can users read from kafka in combination with custom 
> subscribing logic? Without looking much into it, it seems that they basically 
> cannot, at least without having to replicate some parts of the connector, 
> which seems rather inconvenient.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (FLINK-27872) Allow KafkaBuilder to set arbitrary subscribers

2022-06-02 Thread Martijn Visser (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545583#comment-17545583
 ] 

Martijn Visser commented on FLINK-27872:


[~renqs] What do you think?

> Allow KafkaBuilder to set arbitrary subscribers
> ---
>
> Key: FLINK-27872
> URL: https://issues.apache.org/jira/browse/FLINK-27872
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors / Kafka
>Reporter: Salva
>Priority: Major
>
> Currently, 
> [KafkaSourceBuilder|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java]
>  has two setters:
>  * 
> [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
>  * 
> [setTopicPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
> which under the hood instantiate the corresponding (concrete) subscribers. 
> This covers the most common needs, I agree, but it might fall short in some 
> cases. Why not add a more generic setter:
>  * {{setSubscriber (???)}}
> Otherwise, how can users read from kafka in combination with custom 
> subscribing logic? Without looking much into it, it seems that they basically 
> cannot, at least without having to replicate some parts of the connector, 
> which seems rather inconvenient.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Commented] (FLINK-27872) Allow KafkaBuilder to set arbitrary subscribers

2022-06-02 Thread Salva (Jira) 


[ 
https://issues.apache.org/jira/browse/FLINK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17545390#comment-17545390
 ] 

Salva commented on FLINK-27872:
---

Based on a quick experiment, it seems that using a custom subscriber requires 
to create your own fork for these two:
- `KafkaSourceBuilder`

-`KafkaSource`

The rest of the connector seems reusable, but that is still far from ideal 
(although better than what I originally thought).

> Allow KafkaBuilder to set arbitrary subscribers
> ---
>
> Key: FLINK-27872
> URL: https://issues.apache.org/jira/browse/FLINK-27872
> Project: Flink
>  Issue Type: New Feature
>  Components: Connectors / Kafka
>Reporter: Salva
>Priority: Major
>
> Currently, 
> [KafkaBuilder|https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java]
>  has two setters:
>  * 
> [setTopics|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L157]
>  * 
> [setTopicPattern|https://github.com/apache/flink/blob/586715f23ef49939ab74e4736c58d71c643a64ba/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/KafkaSourceBuilder.java#L168]
> which under the hood instantiate the corresponding (concrete) subscribers. 
> This covers the most common needs, I agree, but it might fall short in some 
> cases. Why not add a more generic setter:
>  * {{setSubscriber (???)}}
> Otherwise, how can users read from kafka in combination with custom 
> subscribing logic? Without looking much into it, it seems that they basically 
> cannot, at least without having to replicate some parts of the connector, 
> which seems rather inconvenient.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)