When you say Threadsafe, I assume that the calls to the send method is
Synchronized or? If it is the case, I see this as a bottleneck. We have a
high frequency system and the frequency at which the calls are made to the
send method is very high. This was the reason why I came up with multiple
instances of the producer!
On Mon, May 23, 2016 at 2:22 PM, Tom Crayford <tcrayf...@heroku.com> wrote:
> That's accurate. Why are you creating so many producers? The Kafka producer
> is thread safe and *should* be shared to take advantage of batching, so I'd
> recommend just having a single producer.
>
> Thanks
>
> Tom Crayford
> Heroku Kafka
>
> On Mon, May 23, 2016 at 10:41 AM, Joe San <codeintheo...@gmail.com> wrote:
>
> > In one of our application, we have the following setting:
> >
> > # kafka configuration
> > # ~~~~~
> > kafka {
> > # comma seperated list of brokers
> > # for e.g., "localhost:9092,localhost:9032"
> > brokers = "localhost:9092,localhost:9032"
> > topic = "asset-computed-telemetry"
> > isEnabled = true
> > # for a detailed list of configuration options see
> > # under New Producer Configs
> > # http://kafka.apache.org/082/documentation.html#producerconfigs
> > requestRequiredAcks = 1
> > requestTimeout = 3.seconds
> > bufferMemoryBytes = "33554432"
> > blockOnBufferFull = false
> > # setting this to 0 indicates that the producer will never
> > # block and will just drop messages once the queue buffer to
> > # kafka broker is full
> > queueEnqueTimeoutMs = 0.seconds
> > producerType = "async"
> > messageSendMaxRetries = 1
> > }
> >
> > As you can see from the configuration is that, we have a buffer of 33 MB.
> > Now I have one topic on my broker and this topic has 20 partitions. So
> what
> > I do in my producer application is that I create 10 instances of my
> > Producer and I write to the topic.
> >
> > So each producer instance gets a copy of this configuration. Does this
> mean
> > that I will reserve 33 times 10 instances = 330 MB os space just for the
> > buffer? What if I have more and more topics in the future? Will I use all
> > the memory only for the buffer?
> >
>
