[ https://issues.apache.org/jira/browse/KAFKA-5635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100399#comment-16100399 ]
Ewen Cheslack-Postava commented on KAFKA-5635: ---------------------------------------------- bq. Kafka connect currently uses kafka clients which directly connect to kafka brokers. bq. In a use case, wherein I have many kafka connect [producers] running remotely its a challenge to configure broker information on every connect agent. It doesn't seem like using a REST proxy would help here, you still need to configure *that* for all workers. The common solution to simplify this is to use a load balancer with a single, fixed address as your bootstrap servers. Then you have a single location where you need to maintain a list of broker addresses. bq. Also, in case of IP change [upgrade or cluster re-creation], we need to update every remote connect configuration. See the above solution, which also addresses this concern. bq. If kafka connect source connectors talk to ReST endpoint then client is unaware of broker details. This way we can transparently upgrade / re-create kafka cluster as long as ReST endpoint remains same. This would be unlikely to be implemented as part of the core Kafka project since REST proxies are 3rd party. There is no standard REST proxy API for Kafka that we could rely on the protocol for. > KIP-181 Kafka-Connect integrate with kafka ReST Proxy > ----------------------------------------------------- > > Key: KAFKA-5635 > URL: https://issues.apache.org/jira/browse/KAFKA-5635 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect > Reporter: Dhananjay Patkar > Labels: features, newbie > > Kafka connect currently uses kafka clients which directly connect to kafka > brokers. > In a use case, wherein I have many kafka connect [producers] running remotely > its a challenge to configure broker information on every connect agent. > Also, in case of IP change [upgrade or cluster re-creation], we need to > update every remote connect configuration. > If kafka connect source connectors talk to ReST endpoint then client is > unaware of broker details. This way we can transparently upgrade / re-create > kafka cluster as long as ReST endpoint remains same. -- This message was sent by Atlassian JIRA (v6.4.14#64029)