[jira] [Commented] (KAFKA-15619) Deleted topics will come back again
[
https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17776006#comment-17776006
]
Deng Ziming commented on KAFKA-15619:
-
Hello [~showuon] , As [~ijuma] mentioned, auto.create.topics.enable only affect
`MetadataRequest`, we should firstly locate where are we sending this
`MetadataRequest` which creates the deleted topic automatically.
> Deleted topics will come back again
> ---
>
> Key: KAFKA-15619
> URL: https://issues.apache.org/jira/browse/KAFKA-15619
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 3.5.0, 3.5.1
>Reporter: Deng Ziming
>Priority: Major
>
> Deleted topics will come back again in Apache Spark structured streaming
> stress test after upgrade Kafka from 3.4.0 to 3.5.0, related ticket is:
> https://issues.apache.org/jira/browse/SPARK-45529 , the test randomly
> starts/stops/adds data/add partitions/delete topic/add topic/checks the
> result in a loop, I finally found that a deleted topic will come back again
> after some time.
> By constantly reseting the head of branch-3.5 and using {{gradlew install}}
> to repackage and rerunning of the stress test, I have basically inferred that
> this bug comes from [https://github.com/apache/kafka/pull/12590]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15619) Deleted topics will come back again
[
https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775996#comment-17775996
]
Luke Chen commented on KAFKA-15619:
---
So, in this case, I think it works as designed. For the test, it might need to
wait until consumer/producer closed, then delete the topic, otherwise, it's
possible the topic will be AUTO created. WDYT?
> Deleted topics will come back again
> ---
>
> Key: KAFKA-15619
> URL: https://issues.apache.org/jira/browse/KAFKA-15619
> Project: Kafka
> Issue Type: Improvement
>Affects Versions: 3.5.0, 3.5.1
>Reporter: Deng Ziming
>Priority: Major
>
> Deleted topics will come back again in Apache Spark structured streaming
> stress test after upgrade Kafka from 3.4.0 to 3.5.0, related ticket is:
> https://issues.apache.org/jira/browse/SPARK-45529 , the test randomly
> starts/stops/adds data/add partitions/delete topic/add topic/checks the
> result in a loop, I finally found that a deleted topic will come back again
> after some time.
> By constantly reseting the head of branch-3.5 and using {{gradlew install}}
> to repackage and rerunning of the stress test, I have basically inferred that
> this bug comes from [https://github.com/apache/kafka/pull/12590]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15619) Deleted topics will come back again
[ https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775991#comment-17775991 ] Ismael Juma commented on KAFKA-15619: - auto.create.topics.enable means that _metadata requests_ cause topics to be created. It looks like the stress test is likely to be flaky with auto topic creation independently of the change in 3.6 (but it's probably worse with the change in 3.6). > Deleted topics will come back again > --- > > Key: KAFKA-15619 > URL: https://issues.apache.org/jira/browse/KAFKA-15619 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.5.0, 3.5.1 >Reporter: Deng Ziming >Priority: Major > > Deleted topics will come back again in Spark structured streaming test after > upgrade Kafka from 3.4.0 to 3.5.0, related ticket is: > https://issues.apache.org/jira/browse/SPARK-45529 > > I have basically inferred that this bug comes from > https://github.com/apache/kafka/pull/12590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15619) Deleted topics will come back again
[ https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775988#comment-17775988 ] Deng Ziming commented on KAFKA-15619: - [~showuon] Yes, the stress test is using default config and 'auto.create.topics.enable' is true, after set this config to false the test no longer fails. I guess we will send a fetch request when closing and the ConsumerMetadata is not up to date, so the deleted topics are still in the fetch request. > Deleted topics will come back again > --- > > Key: KAFKA-15619 > URL: https://issues.apache.org/jira/browse/KAFKA-15619 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.5.0, 3.5.1 >Reporter: Deng Ziming >Priority: Major > > Deleted topics will come back again in Spark structured streaming test after > upgrade Kafka from 3.4.0 to 3.5.0, related ticket is: > https://issues.apache.org/jira/browse/SPARK-45529 > > I have basically inferred that this bug comes from > https://github.com/apache/kafka/pull/12590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15619) Deleted topics will come back again
[ https://issues.apache.org/jira/browse/KAFKA-15619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17775985#comment-17775985 ] Luke Chen commented on KAFKA-15619: --- [~dengziming], I'd like to get more info why the [PR|https://github.com/apache/kafka/pull/12590] will make the deleted topics back? Because we're trying to close the fetch session, and while it's closing, it found the topic is deleted, so AUTO create it? Does the broker enable "auto.create.topics.enable"? > Deleted topics will come back again > --- > > Key: KAFKA-15619 > URL: https://issues.apache.org/jira/browse/KAFKA-15619 > Project: Kafka > Issue Type: Improvement >Affects Versions: 3.5.0, 3.5.1 >Reporter: Deng Ziming >Priority: Major > > Deleted topics will come back again in Spark structured streaming test after > upgrade Kafka from 3.4.0 to 3.5.0, related ticket is: > https://issues.apache.org/jira/browse/SPARK-45529 > > I have basically inferred that this bug comes from > https://github.com/apache/kafka/pull/12590 -- This message was sent by Atlassian Jira (v8.20.10#820010)
