[
https://issues.apache.org/jira/browse/KAFKA-10606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17214051#comment-17214051
]
Lincong Li commented on KAFKA-10606:
------------------------------------
[~huxi_2b] Thanks for your comment. However, clients set this to true
explicitly on all paths, and even if they didn't this breaks behavior when
getting MD for particular topics (where auto creation may be expected because
the topics were named.
For example,
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L37
Also changing default values for RPC objects changes the behavior for older
clients. Thus it is a breaking API change. So even if we want to change the
default value of that field (for hypothetical clients that don't explicitly set
it), it needs to be in a new RPC version.
A simple way to fix it on the server side is:
https://github.com/linkedin/kafka/pull/94/commits/95ad9add181db980914a13a6ffe1a88cd5636a6d
I prefer the change that does not require Kafka client version upgrade. Because
in some cases (e.g. LinkedIn), client version upgrade means hundreds or even
thousands of Kafka users will have to bump Kafka client version in their
project dependency and it makes this approach less feasible compared to the
server side fix.
There are definitely other options with more complex (or fancier) way to fix
this issue on the server side as well.
> Auto create non-existent topics when fetching metadata for all topics
> ---------------------------------------------------------------------
>
> Key: KAFKA-10606
> URL: https://issues.apache.org/jira/browse/KAFKA-10606
> Project: Kafka
> Issue Type: Bug
> Reporter: Lincong Li
> Assignee: Tom Bentley
> Priority: Major
>
> The "allow auto topic creation" flag is hardcoded to be true for the
> fetch-all-topic metadata request:
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L37
> In the below code, annotation claims that "*This never causes
> auto-creation*". It it NOT true and auto topic creation still gets triggered
> under some circumstances. So, this is a bug that needs to be fixed.
> https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java#L68
> For example, the bug could be manifested in the below situation:
> A topic T is being deleted and a request to fetch metadata for all topics
> gets sent to one broker. The broker reads names of all topics from its
> metadata cache (shown below).
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1196
> Then the broker authorizes all topics and makes sure that they are allowed to
> be described. Then the broker tries to get metadata for every authorized
> topic by reading the metadata cache again, once for every topic (show below).
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/server/KafkaApis.scala#L1240
> However, the metadata cache could have been updated while the broker was
> authorizing all topics and topic T and its metadata no longer exist in the
> cache since the topic got deleted and metadata update requests eventually got
> propagated from the controller to all brokers. So, at this point, when the
> broker tries to get metadata for topic T from its cache, it realizes that it
> does not exist and the broker tries to "auto create" topic T since the
> allow-auto-topic-creation flag was set to true in all the fetch-all-topic
> metadata requests.
> I think this bug exists since "*metadataRequest.allowAutoTopicCreation*" was
> introduced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
