Matthias J. Sax created KAFKA-13549:
---------------------------------------
Summary: Add "delete interval" config
Key: KAFKA-13549
URL: https://issues.apache.org/jira/browse/KAFKA-13549
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
Kafka Streams uses "delete record" requests to aggressively purge data from
repartition topics. Those request are sent each time we commit.
For at-least-once with a default commit interval of 30 seconds, this works
fine. However, for exactly-once with a default commit interval of 100ms, it's
very aggressive. The main issue is broker side, because the broker logs every
"delete record" request, and thus broker logs are spammed if EOS is enabled.
We should consider to add a new config (eg `delete.record.interval.ms` or
similar) to have a dedicated config for "delete record" requests, to decouple
it from the commit interval config and allow to purge data less aggressively,
even if the commit interval is small to avoid the broker side log spamming.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
