When you say Threadsafe, I assume that the calls to the send method is Synchronized or? If it is the case, I see this as a bottleneck. We have a high frequency system and the frequency at which the calls are made to the send method is very high. This was the reason why I came up with multiple instances of the producer!
On Mon, May 23, 2016 at 2:22 PM, Tom Crayford <[email protected]> wrote: > That's accurate. Why are you creating so many producers? The Kafka producer > is thread safe and *should* be shared to take advantage of batching, so I'd > recommend just having a single producer. > > Thanks > > Tom Crayford > Heroku Kafka > > On Mon, May 23, 2016 at 10:41 AM, Joe San <[email protected]> wrote: > > > In one of our application, we have the following setting: > > > > # kafka configuration > > # ~~~~~ > > kafka { > > # comma seperated list of brokers > > # for e.g., "localhost:9092,localhost:9032" > > brokers = "localhost:9092,localhost:9032" > > topic = "asset-computed-telemetry" > > isEnabled = true > > # for a detailed list of configuration options see > > # under New Producer Configs > > # http://kafka.apache.org/082/documentation.html#producerconfigs > > requestRequiredAcks = 1 > > requestTimeout = 3.seconds > > bufferMemoryBytes = "33554432" > > blockOnBufferFull = false > > # setting this to 0 indicates that the producer will never > > # block and will just drop messages once the queue buffer to > > # kafka broker is full > > queueEnqueTimeoutMs = 0.seconds > > producerType = "async" > > messageSendMaxRetries = 1 > > } > > > > As you can see from the configuration is that, we have a buffer of 33 MB. > > Now I have one topic on my broker and this topic has 20 partitions. So > what > > I do in my producer application is that I create 10 instances of my > > Producer and I write to the topic. > > > > So each producer instance gets a copy of this configuration. Does this > mean > > that I will reserve 33 times 10 instances = 330 MB os space just for the > > buffer? What if I have more and more topics in the future? Will I use all > > the memory only for the buffer? > > >
