Re: More partitions => less throughput?

2019-12-01 Thread Craig Pastro
nd to broker contains higher message > count. > Eric > > -Original Message- > From: Craig Pastro > Sent: Thursday, November 28, 2019 9:10 PM > To: users@kafka.apache.org > Subject: More partitions => less throughput? > > External > > Hello there, > &

RE: More partitions => less throughput?

2019-11-30 Thread Eric Owhadi
, 2019 9:10 PM To: users@kafka.apache.org Subject: More partitions => less throughput? External Hello there, I was wondering if anyone here could help me with some insight into a conundrum that I am facing. Basically, the story is that I am running three Kafka brokers via docker on a single

Re: More partitions => less throughput?

2019-11-30 Thread Peter Bukowinski
Testing multiple brokers VMs on a single host won’t give you accurate performance numbers unless that is how you will be deploying kafka in production. (Don’t do this.) All your kafka networking is being handled by a single host, so instead of being spread out between machines to increase total

Re: More partitions => less throughput?

2019-11-30 Thread Tom Brown
I think the number of partitions needs to be tuned to the size of the cluster; 64 partitions on what is essentially a single box seems high. Do you know what hardware you will be deploying on in production? Can you run your benchmark on that instead of a vm? —Tom On Thursday, November 28, 2019,

More partitions => less throughput?

2019-11-28 Thread Craig Pastro
Hello there, I was wondering if anyone here could help me with some insight into a conundrum that I am facing. Basically, the story is that I am running three Kafka brokers via docker on a single vm with log.flush.interval.messages = 1 and min.insync.replicas = 2. Then I create two topics: both