Hello all, We’ve been testing out kafka-connect as a possible way of streaming data and have run into some issues. I am a kafka/kafka connect newbie so hopefully these aren’t too basic. I have listed a couple questions below.
Thank you 1. Currently our connected-distributed.properties for kafka-connect looks like: dp-kafka-01:9092, dp-kafka-02:9092, dp-kafka-03:9092. I noticed that when any kafka node goes down, curl localhost/connectors and curl localhost/topics on kafka connect hangs. Because of this we have been considering putting kafka behind a load balancer. Although from reading this seems somewhat out of practice. I’m wondering if you all use a load balancer for kafka and if not how you deal with this situation. 2. In creating our HTTP sink connector we’ve run into some issues with the SSL configuration. Currently I have outlined the basic configuration below. Could (c) or (d) be our problem? * Our kafka brokers have their own keystore and truststore with the int and root cert * Our kafka connect nodes have their own keystore and truststore * Our connect distributed.properties lists: i. Worker.keystore=kconnect.keystore ii. Producer.keystore=kconnect.keystore iii. Consumer.keystore=kconnect.keystore (we tried kafka.keystore too) * Kconnect communicating with kafka on port 9092 but in the kafka broker listener configuration we have PLAINTEXT:9092, SSL:9093, SASL_SSL:9094., I tried switching to 9093 but it has trouble communicating even though I am able to telnet on that port 1. Newly spun up cluster has trouble seeing all kafka nodes as available even though the kafka service is running on all 3 * When I restart the kafka manager service: it sees node(2) as active broker. If I restart the kafka manager service again it sees (1) and (3) as active broker, if I restart it again I see (1) and (2) as active brokers. What might be causing this behavior and not allowing the manager service to “see” all 3 nodes at a time? 2. We are considering paying for some services through Confluent (training/guidance). Wondering if any of you all have used them and found them helpful ******************* PLEASE NOTE ******************* This E-Mail/telefax message and any documents accompanying this transmission may contain information that is privileged, confidential, and/or exempt from disclosure under applicable law and is intended solely for the addressee(s) named above. If you are not the intended addressee/recipient, you are hereby notified that any use of, disclosure, copying, distribution, or reliance on the contents of this E-Mail/telefax information is strictly prohibited and may result in legal action against you. Please reply to the sender advising of the error in transmission and immediately delete/destroy the message and any accompanying documents. Thank you.