Questions about Producer per Task/Partition in Streams EoS Impl.

2019-10-15 Thread Sean Glover
Hi, I would like to understand better how a `KafkaProducer` is used within Kafka Streams EoS use cases. In the Streams Exactly Once Design [1] document it states that there is one producer per StreamThread: Each thread contains one producer client and two consumer clients (one for > normal fetch

Re: Questions about Producer per Task/Partition in Streams EoS Impl.

2019-10-17 Thread Guozhang Wang
Hello Sean, Yes atm we have one producer per task when EOS is turned on compared to one producer per thread without EOS. There's an ongoing KIP-447 aiming to bring this back to one producer per thread with EOS as well which involves broker-side changes. To answer your question: 1. Assuming each

Re: Questions about Producer per Task/Partition in Streams EoS Impl.

2019-10-17 Thread Sean Glover
Hi Guozhang, Thanks a lot for pointing me to this KIP and providing background information on the current performance implications. I'll track progress on the KIP. Regards, Sean On Thu, Oct 17, 2019 at 12:29 PM Guozhang Wang wrote: > Hello Sean, > > Yes atm we have one producer per task when