Re: Kafka Streams Hang While Committing

2017-10-24 Thread Matthias J. Sax
Well. When a commit a triggered, Streams need to flush all caches and flush all pending write of the producers. And as this happens on the same thread that does processing, there won't be any processing of new data until the commit is finished. So I guess, it is expected. -Matthias On 10/24/17

Kafka Streams Hang While Committing

2017-10-24 Thread Sicheng Liu
Hi Kafka Community, I have a question regarding Kafka Streams. Currently we set our commit interval to 5 mins and buffer size to be 1GB. We observe that every 5 minutes, there is a spike of network out, which is expected. However, we also found that the record process rate dropped to 1/4 during co