Hi Yan, My (uneducated) guess is that the performance gains come from batching. I don't know if the new producer ever batches by destination broker. If not and it only batches by (broker,topic,partition) then I doubt that one vs two producers will affect performance as they send to different topics.
Cheers, Roger On Tue, Aug 18, 2015 at 12:26 AM, Yan Fang <yanfang...@gmail.com> wrote: > Hi Tao, > > First, one kafka producer has an i/o thread. (correct me if I am wrong). > > Second, after Samza 0.10.0, we have a coordinator stream, which stores the > checkpoint, config and other locality information for auto-scaling, dynamic > configuration, etc purpose. (See Samza-348 > <https://issues.apache.org/jira/browse/SAMZA-348>). So we have a producer > for this coordinator stream. > > Therefore, each contains will have at least two producers, one is for the > coordinator stream, one is for the users system. > > My question is, can we use only one producer for both coordinator stream > and the users system to have better performance? (from the doc, it may > retrieve better performance.) > > Thanks, > > Fang, Yan > yanfang...@gmail.com > > On Mon, Aug 17, 2015 at 9:49 PM, Tao Feng <fengta...@gmail.com> wrote: > > > Hi Yan, > > > > Naive question: what do we need producer thread of coordinator stream > for? > > > > Thanks, > > -Tao > > > > On Mon, Aug 17, 2015 at 2:09 PM, Yan Fang <yanfang...@gmail.com> wrote: > > > > > Hi guys, > > > > > > I have this question because Kafka's doc > > > < > > > > > > http://kafka.apache.org/082/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html > > > > > > > seems recommending having one producer shared by all threads ("*The > > > producer is thread safe and should generally be shared among all > threads > > > for best performance.*"), while currently the coordinator stream is > > using a > > > separate producer (usually, there are two producers(two producer > threads) > > > in each container: one is for the coordinator stream , one is for the > > > "real" job) > > > > > > 1. Will having one producer shared by all thread really improve the > > > performance? (haven't done the perf test myself. Guess Kafka has some > > > proof). > > > > > > 2. if yes, should we go this way? > > > > > > Thanks, > > > > > > Fang, Yan > > > yanfang...@gmail.com > > > > > >
