Hello 王美功, I am using 2.1.0. And, I think you nailed it on the head, because my application is low throughput and I am seeing UNKNOWN_PRODUCER_ID all the time with exactly once enabled. I've googled this before but couldn't identify the cause. Thank you!
Setting retry.backoff.ms to 5 brought the latency down from 1.3s to 750ms. That's tolerable for me, but I am wondering: when the KAFKA-7190 is fixed, will the latency drop further? Dmitry On Wed, Dec 19, 2018 at 8:38 PM meigong.wang <meigong.w...@okcoin.com> wrote: > Which version are you using? This bug( > https://issues.apache.org/jira/browse/KAFKA-7190) may increase the > latency of your application, try to reduce the retry.backoff.ms,the > default value is 100 ms. > > > 王美功 > > > 原始邮件 > 发件人:Dmitry minkovskydminkov...@gmail.com > 收件人:usersus...@kafka.apache.org > 发送时间:2018年12月20日(周四) 09:25 > 主题:High end-to-end latency with processing.guarantee=exactly_once > > > I have a process that spans several Kafka Streams applications. With the > streams commit interval and producer linger both set to 5ms, when exactly > once delivery is disabled, this process takes ~250ms. With exactly once > enabled, the same process takes anywhere from 800-1200ms. In Enabling > Exactly-Once in Kafka Streams ,"> > https://www.confluent.io/blog/enabling-exactly-kafka-streams/, Guozhang > writes In Kafka Streams, because a new transaction is created whenever > commit is called, the average transaction size is determined by the commit > interval: with the same incoming traffic, a shorter commit interval will > result in smaller transactions. In practice users should therefore tune the > commit.interval.ms setting when exactly-once is enabled to make a good > trade-off between throughput versus end-to-end processing latency. But I am > not seeing much of a difference when I tune commit.interval.ms with > exactly once enabled. `though()` and `.to()/.stream()` take 100-250ms even > with commit.interval.ms set to 5ms. Do these latency differences sound > right? Is something off? Thank you, Dmitry
