Hi,
I think this is the best reference on the actual trade-off (more/less
partitions):
http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/
As a guess, for a three-node Kafka cluster, I don't see how useful would be
to have 1024 consumers in a single consum
Hi again,
For the record I filed an issue about this here:
https://issues.apache.org/jira/browse/KAFKA-2986
Cheers,
Jens
–
Skickat från Mailbox
On Fri, Dec 11, 2015 at 7:56 PM, Jens Rantil wrote:
> Hi,
> We've been experimenting a little with running Kafka internally for better
> ha
Guozhang,
Correct. Even with linger.ms at 1000 ms, the metrics indicate no batching.
The KafkaProducer instance is recognizing the linger.ms setting since sending
100 records
with linger.ms=1000 then takes over 100 seconds.
Here is the test harness. There is an abstract parent class with met
Gary,
You are calling "kafkaProducer.send(record).get();" for each message, the
get() call block until the Future is initialized, which effectively
synchronize all message sent by asking for the ACK for each message before
sending the next message, hence no batching.
You can try using "send(recor
Guozhang,
Yes - you identified the problem!
We had inserted the .get() for debugging, but didn’t think of the (huge!)
side-effects.
Using the async callback works perfectly well.
We are now able to send 100,000 records in 14 sec from a laptop to the Bluemix
cloud - ~1000x faster,
Thank you v
