Hello all,
We’ve been testing out kafka-connect as a possible way of streaming data and
have run into some issues. I am a kafka/kafka connect newbie so hopefully these
aren’t too basic.
I have listed a couple questions below.
Thank you
1. Currently our connected-distributed.properties for kafka-connect looks
like: dp-kafka-01:9092, dp-kafka-02:9092, dp-kafka-03:9092. I noticed that when
any kafka node goes down, curl localhost/connectors and curl localhost/topics
on kafka connect hangs. Because of this we have been considering putting kafka
behind a load balancer. Although from reading this seems somewhat out of
practice. I’m wondering if you all use a load balancer for kafka and if not how
you deal with this situation.
2. In creating our HTTP sink connector we’ve run into some issues with the
SSL configuration. Currently I have outlined the basic configuration below.
Could (c) or (d) be our problem?
* Our kafka brokers have their own keystore and truststore with the int
and root cert
* Our kafka connect nodes have their own keystore and truststore
* Our connect distributed.properties lists:
i.
Worker.keystore=kconnect.keystore
ii.
Producer.keystore=kconnect.keystore
iii.
Consumer.keystore=kconnect.keystore (we tried kafka.keystore too)
* Kconnect communicating with kafka on port 9092 but in the kafka broker
listener configuration we have PLAINTEXT:9092, SSL:9093, SASL_SSL:9094., I
tried switching to 9093 but it has trouble communicating even though I am able
to telnet on that port
1. Newly spun up cluster has trouble seeing all kafka nodes as available
even though the kafka service is running on all 3
* When I restart the kafka manager service: it sees node(2) as active
broker. If I restart the kafka manager service again it sees (1) and (3) as
active broker, if I restart it again I see (1) and (2) as active brokers. What
might be causing this behavior and not allowing the manager service to “see”
all 3 nodes at a time?
2. We are considering paying for some services through Confluent
(training/guidance). Wondering if any of you all have used them and found them
helpful
******************* PLEASE NOTE ******************* This E-Mail/telefax message
and any documents accompanying this transmission may contain information that
is privileged, confidential, and/or exempt from disclosure under applicable law
and is intended solely for the addressee(s) named above. If you are not the
intended addressee/recipient, you are hereby notified that any use of,
disclosure, copying, distribution, or reliance on the contents of this
E-Mail/telefax information is strictly prohibited and may result in legal
action against you. Please reply to the sender advising of the error in
transmission and immediately delete/destroy the message and any accompanying
documents. Thank you.
