Re: Kafka Producer and Buffer Issues

2016-05-23 Thread Tom Crayford
It puts things on several internal queues. I'd benchmark what kind of rates you're looking at - we handily do a few hundred thousand per second per process with a 2GB JVM heap. On Mon, May 23, 2016 at 1:31 PM, Joe San wrote: > When you say Threadsafe, I assume that the calls to the send method i

Re: Kafka Producer and Buffer Issues

2016-05-23 Thread Joe San
When you say Threadsafe, I assume that the calls to the send method is Synchronized or? If it is the case, I see this as a bottleneck. We have a high frequency system and the frequency at which the calls are made to the send method is very high. This was the reason why I came up with multiple insta

Re: Kafka Producer and Buffer Issues

2016-05-23 Thread Tom Crayford
That's accurate. Why are you creating so many producers? The Kafka producer is thread safe and *should* be shared to take advantage of batching, so I'd recommend just having a single producer. Thanks Tom Crayford Heroku Kafka On Mon, May 23, 2016 at 10:41 AM, Joe San wrote: > In one of our app

Kafka Producer and Buffer Issues

2016-05-23 Thread Joe San
In one of our application, we have the following setting: # kafka configuration # ~ kafka { # comma seperated list of brokers # for e.g., "localhost:9092,localhost:9032" brokers = "localhost:9092,localhost:9032" topic = "asset-computed-telemetry" isEnabled = true # for a detailed l