Sönke Liebau created KAFKA-4567:
-----------------------------------
Summary: Connect Producer and Consumer ignore ssl parameters
configured for worker
Key: KAFKA-4567
URL: https://issues.apache.org/jira/browse/KAFKA-4567
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 0.10.1.1
Reporter: Sönke Liebau
Assignee: Ewen Cheslack-Postava
Priority: Minor
When using Connect with a SSL enabled Kafka cluster, the configuration options
are either documented a bit misleading, or handled in an incorrect way.
The documentation states the usual available SSL options
(ssl.keystore.location, ssl.truststore.location, ...) and these are picked up
and used for the producers and consumers that are used to communicate with the
status, offset and configs topics.
For the producers and consumers that are used for the actual data, these
parameters are ignored as can be seen
[here|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/Worker.java#L98],
which results in plaintext communication on an SSL port, leading to an OOM
exception ([KAFKA-4493|https://issues.apache.org/jira/browse/KAFKA-4493]).
So in order to get Connect to communicate with a secured cluster you need to
override all SSL configs with the prefixes _consumer._ and _producer._ and
duplicate the values already set at a global level.
The documentation states:
bq. The most critical site-specific options, such as the Kafka bootstrap
servers, are already exposed via the standard worker configuration.
Since the address for the cluster is exposed here, I would propose that there
is no reason not to also pass the SSL parameters through to the consumers and
producers, as it is clearly intended that communication happens with the same
cluster.
In fringe cases, these can still be overridden manually to achieve different
behavior.
I am happy to create a pull request to address this or clarify the docs, after
we decide which one is the appropriate course of action.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
