[
https://issues.apache.org/jira/browse/KAFKA-15834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787679#comment-17787679
]
A. Sophie Blee-Goldman commented on KAFKA-15834:
------------------------------------------------
Found the ticket: https://issues.apache.org/jira/browse/KAFKA-9398
And yes, it's still unresolved.
Given all the above, I think we can honestly just disable/remove the test, as
the named topologies feature was never made into a real public API. I do know
of a few people who are using it anyway but they're aware it was only an
experimental feature and not fully supported by Streams. So imo we don't need
to go out of our way to fix any flaky tests: provided we can demonstrate that
the issue is specific to named topologies and not potentially an issue with
Streams itself. Of course in this case it's actually the latter, but we've
recognized the root cause as a known issue, so I don't think there's anything
more this test can do for us besides be flaky and annoy everyone.
Thanks for digging into this!
> Subscribing to non-existent topic blocks StreamThread from stopping
> -------------------------------------------------------------------
>
> Key: KAFKA-15834
> URL: https://issues.apache.org/jira/browse/KAFKA-15834
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.6.0
> Reporter: Greg Harris
> Priority: Major
>
> In
> NamedTopologyIntegrationTest#shouldContinueProcessingOtherTopologiesWhenNewTopologyHasMissingInputTopics
> a topology is created which references an input topic which does not exist.
> The test as-written passes, but the KafkaStreams#close(Duration) at the end
> times out, and leaves StreamsThreads running.
> From some cursory investigation it appears that this is happening:
> 1. The consumer calls the StreamsPartitionAssignor, which calls
> TaskManager#handleRebalanceStart as a side-effect
> 2. handleRebalanceStart sets the rebalanceInProgress flag
> 3. This flag is checked by StreamThread.runLoop, and causes the loop to
> remain running.
> 4. The consumer never calls StreamsRebalanceListener#onPartitionsAssigned,
> because the topic does not exist
> 5. Because no partitions are ever assigned, the
> TaskManager#handleRebalanceComplete never clears the rebalanceInProgress flag
>
> This log message is printed in a tight loop while the close is ongoing and
> the consumer is being polled with zero duration:
> {noformat}
> [2023-11-15 11:42:43,661] WARN [Consumer
> clientId=NamedTopologyIntegrationTestshouldContinueProcessingOtherTopologiesWhenNewTopologyHasMissingInputTopics-942756f8-5213-4c44-bb6b-5f805884e026-StreamThread-1-consumer,
>
> groupId=NamedTopologyIntegrationTestshouldContinueProcessingOtherTopologiesWhenNewTopologyHasMissingInputTopics]
> Received unknown topic or partition error in fetch for partition
> unique_topic_prefix-topology-1-store-repartition-0
> (org.apache.kafka.clients.consumer.internals.FetchCollector:321)
> {noformat}
> Practically, this means that this test leaks two StreamsThreads and the
> associated clients and sockets, and delays the completion of the test until
> the KafkaStreams#close(Duration) call times out.
> Either we should change the rebalanceInProgress flag to avoid getting stuck
> in this rebalance state, or figure out a way to shut down a StreamsThread
> that is in an extended rebalance state during shutdown.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
