[
https://issues.apache.org/jira/browse/KAFKA-17946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18090587#comment-18090587
]
Eswarar Siva edited comment on KAFKA-17946 at 6/22/26 12:34 PM:
----------------------------------------------------------------
[~lucasbru] I looked at the fix you linked (PR #18253). It stabilizes the test
(it waits for the state updater to run the verified calls, or shuts the updater
down before verifying) and changes one log line in DefaultStateUpdater; it does
not touch resumeTask or transitToUpdateStandby. So I think the production path
behind this is still open.
resumeTask calls changelogReader.transitToUpdateStandby() with no guard on the
reader state (only if (updatingTasks.size() == 1)), and
transitToUpdateStandby() still throws IllegalStateException when the reader is
not in ACTIVE_RESTORING. Under pause/resume churn I can drive the reader into
STANDBY_UPDATING and then have resumeTask call transitToUpdateStandby again,
which trips that check and kills the StreamThread.
I reproduced it on 4.1.2 and 4.3.0 with a small standalone Streams app (same
reproducer as KAFKA-NNNNN, cold start churn plus pause/resume, with a Byteman
rule that widens the existing transition window so it is deterministic). The
error:
java.lang.IllegalStateException: The changelog reader is not restoring active
tasks (is STANDBY_UPDATING) while
trying to transit to update standby tasks: {...}
at
StoreChangelogReader.transitToUpdateStandby(StoreChangelogReader.java:326)
at
DefaultStateUpdater$StateUpdaterThread.resumeTask(DefaultStateUpdater.java:661)
at DefaultStateUpdater$StateUpdaterThread.resumeTasks(...)
at DefaultStateUpdater$StateUpdaterThread.runOnce(...)
The resumeTask and transitToUpdateStandby code is unchanged from when this was
resolved, so the test was stabilized but the production transition is still not
idempotent. Do you want me to reopen this, or file a separate production facing
ticket? I have a Byteman based reproduction I can attach.
was (Author: JIRAUSER313660):
[~lucasbru] I looked at the fix you linked (PR #18253). It stabilizes the test
(it waits for the state updater to run the
verified calls, or shuts the updater down before verifying) and changes one log
line in DefaultStateUpdater; it does
not touch resumeTask or transitToUpdateStandby. So I think the production path
behind this is still open.
resumeTask calls changelogReader.transitToUpdateStandby() with no guard on the
reader state (only
if (updatingTasks.size() == 1)), and transitToUpdateStandby() still throws
IllegalStateException when the reader is
not in ACTIVE_RESTORING. Under pause/resume churn I can drive the reader into
STANDBY_UPDATING and then have
resumeTask call transitToUpdateStandby again, which trips that check and kills
the StreamThread.
I reproduced it on 4.1.2 and 4.3.0 with a small standalone Streams app (same
reproducer as KAFKA-NNNNN, cold start
churn plus pause/resume, with a Byteman rule that widens the existing
transition window so it is deterministic). The
error:
java.lang.IllegalStateException: The changelog reader is not restoring active
tasks (is STANDBY_UPDATING) while
trying to transit to update standby tasks: {...}
at
StoreChangelogReader.transitToUpdateStandby(StoreChangelogReader.java:326)
at
DefaultStateUpdater$StateUpdaterThread.resumeTask(DefaultStateUpdater.java:661)
at DefaultStateUpdater$StateUpdaterThread.resumeTasks(...)
at DefaultStateUpdater$StateUpdaterThread.runOnce(...)
The resumeTask and transitToUpdateStandby code is unchanged from when this was
resolved, so the test was stabilized
but the production transition is still not idempotent. Do you want me to reopen
this, or file a separate production
facing ticket? I have a Byteman based reproduction I can attach.
> Flaky test DeafultStateUpdaterTest::shouldResumeStandbyTask due to
> concurrency issue
> ------------------------------------------------------------------------------------
>
> Key: KAFKA-17946
> URL: https://issues.apache.org/jira/browse/KAFKA-17946
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Reporter: Ao Li
> Priority: Major
>
> Function `transitToUpdateStandby` may be called after verify statement is
> executed.
> See branch: https://github.com/aoli-al/kafka/tree/KAFKA-111 to reproduce the
> failure.
> Test command: ./gradlew cleanTest :streams:test --tests
> org.apache.kafka.streams.processor.internals.DefaultStateUpdaterTest.shouldResumeStandbyTask
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
