nicktelford opened a new pull request #11610: URL: https://github.com/apache/kafka/pull/11610
Records are explicitly deleted once they have been fully consumed. Currently, this is done every time the Task is committed, resulting in "delete records" requests being sent every `commit.interval.ms` milliseconds. When `commit.interval.ms` is set very low, for example when `processing.guarantee` is set to `exactly_once_v2`, this causes delete records requests to be sent extremely frequently, potentially reducing throughput and causing a high volume of log messages to be logged by the brokers. Disconnecting delete records requests from the commit interval resolves this problem. We now only explicitly delete records for a repartition topic when we commit, if it's been at least `delete.interval.ms` milliseconds since the last time we deleted records. Because we still require a commit to trigger record deletion, the lower-bound of `delete.interval.ms` is effectively capped at the `commit.interval.ms`. For compatibility, the default `delete.interval.ms` is set to 30 seconds, the same as the default `commit.interval.ms`. Users who have configured a different `commit.interval.ms` may need to review and change `delete.interval.ms`. Unlike `commit.interval.ms`, we don't dynamically change the default for `delete.interval.ms` when EOS processing is enabled, as it's important not to flood brokers with the record deletions, and we want a sensible default. *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so use a separate comment to ping reviewers.* *Summary of testing strategy (including rationale) for the feature or bug fix. Unit and/or integration tests are expected for any behaviour change and system tests should be considered for larger changes.* This code is my own work and is licensed to the Apache Kafka project under the terms of the same license (ASL 2) as the project itself. ### Committer Checklist (excluded from commit message) - [X] Verify design and implementation - [X] Verify test coverage and CI build status - [X] Verify documentation (including upgrade notes) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: jira-unsubscr...@kafka.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
