The reason for the lower commit interval for EOS is to provide low end-to-end latency.
If your application needs higher throughput and is not latency sensitive, it's totally ok to increase the commit interval. -Matthias On 11/16/20 10:04 AM, Alex Craig wrote: > Hi all, I've been experimenting a bit with ways of increasing the > throughput of some kafka streams applications that have "exactly_once" > enabled. (these applications consume from a topic, transform the data, and > do 2 different aggregations before finally writing the output to a sink > topic) Increasing the producer properties of linger.ms and batch.size > helps, but the thing that seems to make the biggest difference by far is > increasing commit.interval.ms. The tradeoff to increasing this, as I > understand it, is increased overall end-to-end latency. (if I'm wrong on > that please let me know) > > Here's my question: Is it bad practice to increase commit.interval.ms when > using EOS? The reason I ask is because the default with EOS enabled is > 100ms, yet without EOS the default is 30000. (30 seconds) That's a huge > difference, and I'm curious about the reasoning behind that. Thanks for any > guidance, > > Alex C >
