[
https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117125#comment-16117125
]
ASF GitHub Bot commented on KAFKA-5704:
---------------------------------------
GitHub user rhauch opened a pull request:
https://github.com/apache/kafka/pull/3641
KAFKA-5704 Corrected Connect distributed startup behavior to allow older
brokers to auto-create topics
When a Connect distributed worker starts up talking with broker versions
0.10.1.0 and later, it will use the AdminClient to look for the internal topics
and attempt to create them if they are missing. Although the AdminClient was
added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in
0.10.1.0 and later. This feature works as expected when Connect uses a broker
version 0.10.1.0 or later.
However, when a Connect distributed worker starts up using a broker older
than 0.10.1.0, the AdminClient is not able to find the required APIs and thus
will throw an UnsupportedVersionException. Unfortunately, this exception is not
caught and instead causes the Connect worker to fail even when the topics
already exist.
This change handles the UnsupportedVersionException by logging a debug
message and doing nothing. The existing producer logic will get information
about the topics, which will cause the broker to create them if they don’t
exist and broker auto-creation of topics is enabled. This is the same behavior
that existed prior to 0.11.0.0, and so this change restores that behavior for
brokers older than 0.10.1.0.
This change also adds a system test that verifies Connect works with a
variety of brokers and is able to run source and sink connectors. The test
verifies that Connect can read from the internal topics when the connectors are
restarted.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rhauch/kafka kafka-5704
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/3641.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #3641
----
commit 0d45fd113eeaff3844742181191db2cc508353fd
Author: Randall Hauch <rha...@gmail.com>
Date: 2017-08-07T19:32:29Z
KAFKA-5704 Corrected Connect distributed startup behavior to allow older
brokers to auto-create topics
When a Connect distributed worker starts up talking with broker versions
0.10.1.0 and later, it will use the AdminClient to look for the internal topics
and attempt to create them if they are missing. Although the AdminClient was
added in 0.11.0.0, the AdminClient uses APIs to create topics that existed in
0.10.1.0 and later. This feature works as expected when Connect uses a broker
version 0.10.1.0 or later.
However, when a Connect distributed worker starts up using a broker older
than 0.10.1.0, the AdminClient is not able to find the required APIs and thus
will throw an UnsupportedVersionException. Unfortunately, this exception is not
caught and instead causes the Connect worker to fail even when the topics
already exist.
This change handles the UnsupportedVersionException by logging a debug
message and doing nothing. The existing producer logic will get information
about the topics, which will cause the broker to create them if they don’t
exist and broker auto-creation of topics is enabled. This is the same behavior
that existed prior to 0.11.0.0, and so this change restores that behavior for
brokers older than 0.10.1.0.
This change also adds a system test that verifies Connect works with a
variety of brokers and is able to run source and sink connectors. The test
verifies that Connect can read from the internal topics when the connectors are
restarted.
----
> Auto topic creation causes failure with older clusters
> ------------------------------------------------------
>
> Key: KAFKA-5704
> URL: https://issues.apache.org/jira/browse/KAFKA-5704
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 0.11.0.0
> Reporter: Ewen Cheslack-Postava
> Assignee: Randall Hauch
>
> The new automatic internal topic creation always tries to check the topic and
> create it if missing. However, older brokers that we should still be
> compatible with don't support some requests that are used. This results in an
> UnsupportedVersionException which some of the TopicAdmin code notes that it
> can throw but then isn't caught in the initializers, causing the entire
> process to fail.
> We should probably just catch it, log a message, and allow things to proceed
> hoping that the user has already created the topics correctly (as we used to
> do).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
