[
https://issues.apache.org/jira/browse/KAFKA-10340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17292993#comment-17292993
]
Randall Hauch commented on KAFKA-10340:
---------------------------------------
[~xakassi] wrote:
{quote}
And also I cannot understand, why we even have the property
"topic.creation.enable" and do not create all topics by AdminClient? Isn't it a
better approach to create topics always explicitly using AdminClient than to
rely on auto creation?
{quote}
When we introduced KIP-158 to allow Connect to source connectors to specify the
rules for creating topics, it was important that we maintained backward
compatibility. Many people upgrade Kafka and Connect simply to get fixes, and
do not want new features they don't know about nor wish to use enabled by
default. This is an essential aspect of all KIPs and AK features, even when by
default most users *probably* would want to use the feature.
{quote}
According to KIP-158 we have an option to set replication.factor and partitions
to -1 to use the broker's default value. So why we cannot always create topics
by AdminClient and use these broker's default values in case connector does not
have any "topic.creation" properties?
As far as I understand this will completely solve the problem described in this
ticket.
{quote}
It actually does not solve the problem. If Connect were to have the behavior
you suggest, then we'd still run into the issue when the broker has
autocreation of topics disabled *and* the particular source connector did not
specify the topic creation rules.
The bottom line is that many things would be made easier *for some people* if
we could easily change defaults, but upgrading would become painfully difficult
for those that don't wish to use those new features or behaviors. By
maintaining backward compatibility, we dramatically reduce the overhead and
risk of upgrades to get fixes, which is an important characteristic that the AK
community cherishes.
We will have the option in 3.0 to *change* defaults and behaviors -- it is a
major release that may not be entirely backward compatible. Obviously, we'll
want to not break things unnecessarily, but changing the default
`topic.creation.enable = true` will be one such change we are very much looking
forward to making.
> Source connectors should report error when trying to produce records to
> non-existent topics instead of hanging forever
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-10340
> URL: https://issues.apache.org/jira/browse/KAFKA-10340
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
> Affects Versions: 2.5.1, 2.7.0, 2.6.1, 2.8.0
> Reporter: Arjun Satish
> Assignee: Chris Egerton
> Priority: Major
> Fix For: 3.0.0, 2.7.1
>
>
> Currently, a source connector will blindly attempt to write a record to a
> Kafka topic. When the topic does not exist, its creation is controlled by the
> {{auto.create.topics.enable}} config on the brokers. When auto.create is
> disabled, the producer.send() call on the Connect worker will hang
> indefinitely (due to the "infinite retries" configuration for said producer).
> In setups where this config is usually disabled, the source connector simply
> appears to hang and not produce any output.
> It is desirable to either log an info or an error message (or inform the user
> somehow) that the connector is simply stuck waiting for the destination topic
> to be created. When the worker has permissions to inspect the broker
> settings, it can use the {{listTopics}} and {{describeConfigs}} API in
> AdminClient to check if the topic exists, the broker can
> {{auto.create.topics.enable}} topics, and if these cases do not exist, either
> throw an error.
> With the recently merged
> [KIP-158|https://cwiki.apache.org/confluence/display/KAFKA/KIP-158%3A+Kafka+Connect+should+allow+source+connectors+to+set+topic-specific+settings+for+new+topics],
> this becomes even more specific a corner case: when topic creation settings
> are enabled, the worker should handle the corner case where topic creation is
> disabled, {{auto.create.topics.enable}} is set to false and topic does not
> exist.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
