Hi PoAn, Thanks for the KIP! I agree the number of batches to retain should be configurable to improve the throughput.
Comments: 1. Could you add the issue: KAFKA-18905 <https://issues.apache.org/jira/browse/KAFKA-18905> into the motivation section? I think this is the issue we want to address, right? 2. > Introduce a new config on the broker, as the broker must know how much memory to allocate. Operators can set a limitation on the broker side to prevent malicious producers. This configuration only takes effect for idempotent producers. I think not only the idempotent producers, but also the transactional producers, as long as they have the PID. 3. About the producer response update, I'm wondering if it is necessary? Currently, when producer with `max.in.flight.requests.per.connection=10` and NUM_BATCHES_TO_RETAIN=5, we won't adjust the producer config to 5. Of course it is possible to the duplication cannot be detected, but that might be user's choice to improve the throughput (though it might be rare). So, I'd like to know why we have to adjust the `max.in.flight.requests.per.connection` value in the producer side? Using the example above, after this KIP, the `max.in.flight.requests.per.connection=10` cannot be retained unless NUM_BATCHES_TO_RETAIN is set to 10, right? 4. The default value of `max.idempotence.batches.to.retain` In the performance test you showed, it obviously shows larger `max.idempotence.batches.to.retain` will get better throughput. Also, the memory usage is small, do we have any reason we keep the default value for 5? Thank you, Luke On Sun, Feb 22, 2026 at 9:48 PM PoAn Yang <[email protected]> wrote: > Hi all, > > I would like to start a discussion thread on KIP-1269. In this KIP, we aim > to remove limitation of maximal number of batches to retain for a > idempotent producer. In our test, it can improve throughput and reduce > latency. > > https://cwiki.apache.org/confluence/x/loI8G > > Please take a look and feel free to share any thoughts. > > Thanks. > PoAn
