Guozhang Wang created KAFKA-14641: ------------------------------------- Summary: Cleanup CommitNeeded after EOS-V1 is removed Key: KAFKA-14641 URL: https://issues.apache.org/jira/browse/KAFKA-14641 Project: Kafka Issue Type: Improvement Reporter: Guozhang Wang
This is a follow-up of KAFKA-14294. Today we have several flags to determine if KS need to execute a commit: 1) task-level "commitNeeded" which is set whenever process() or punctuator() is called, 2) if there are input topic offsets to commit, retrieved from the "task.prepareCommit()", 3) the "transactionInFlight" flag from producer as a fix of KAFKA-14294 (this subsumes the first "commitNeeded" functionality). Given that we are still having EOS-v1, cleanup this would be a bit complex. But after the deprecated EOS-V1 is removed, we can cleanup those controls since for any commit cases, we would need to commit all tasks anyways whereas in EOS-v1, we would commit probably a subset of tasks since they are done by different producers and hence different txns. A quick thought is the following: 1) We would not need the per-task "commitNeeded" anymore. 2) We would maintain a single "commitNeeded" flag on the task-executor, hence on the thread level. It is set whenever `process()` or `punctuator` is called. 3) Whenever we need to commit, either a) periodically, b) upon revocation, c) upon user request, we simply check that flag, and if necessary commit all tasks and reset the flag. -- This message was sent by Atlassian Jira (v8.20.10#820010)