lucasbru opened a new pull request, #15117:
URL: https://github.com/apache/kafka/pull/15117
Streams fails to close task after restoration when input partitions are
updated in a new assignment happening at the same time.
There is a race condition in the state updater that can cause the following:
1. We have an active task in the state updater
2. We get fenced. We recreate the producer, transactions now uninitialized.
We ask the state updater to give back the task, add a pending action to close
the task clean once it’s handed back
3. We get a new assignment with updated input partitions. The task is still
owned by the state updater, so we ask the state updater again to hand it back
and add a pending action to update its input partition
4. The task is handed back by the state updater. We update its input
partitions but forget to close it clean (pending action was overwritten)
5. Now the task is in an initialized state, but the underlying producer
does not have transactions initialized
This can cause an IllegalStateException: `Invalid transition attempted from
state UNINITIALIZED to state IN_TRANSACTION` when running in EOSv2.
To fix this, we introduce a new pending action
CloseReviveAndUpdateInputPartitions that is added when we handle a new
assignment with updated input partitions, but we still need to close the task
before reopening it.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
