Can you please share Kafka version you've used for the tests?
On Thu, May 23, 2013 at 8:57 PM, Jason Weiss <jason_we...@rapid7.com> wrote: > Folks, > > As I posted to the group here yesterday, my 3 server test in AWS produced > an average of 273,132 events per second with a fixed-size 2K message > payload. (Please see that thread for details.) In order to determine the > horizontal scalability, I added an additional server and MORE producer > clients into the mix. With 3 servers in the cluster, the average was 91,044 > eps per server (273,132). After adding a 4th server to the mix, and an > additional set of producers, I achieved an extended run of 361,320 events > per second with the same fixed-size 2K message payload. The new server > added an additional 88,188 events per second throughput capacity to the > cluster – near perfect horizontal scaling. > > While there may be a limit, achieving 500,00 or 750,000 or even 1 million > events per second with a Kafka cluster isn't beyond reach for the truly > motivated. > > Nice work committers, very nice work. > > Jason > > > > This electronic message contains information which may be confidential or > privileged. The information is intended for the use of the individual or > entity named above. If you are not the intended recipient, be aware that > any disclosure, copying, distribution or use of the contents of this > information is prohibited. If you have received this electronic > transmission in error, please notify us by e-mail at ( > postmas...@rapid7.com) immediately. >
