Folks, As I posted to the group here yesterday, my 3 server test in AWS produced an average of 273,132 events per second with a fixed-size 2K message payload. (Please see that thread for details.) In order to determine the horizontal scalability, I added an additional server and MORE producer clients into the mix. With 3 servers in the cluster, the average was 91,044 eps per server (273,132). After adding a 4th server to the mix, and an additional set of producers, I achieved an extended run of 361,320 events per second with the same fixed-size 2K message payload. The new server added an additional 88,188 events per second throughput capacity to the cluster – near perfect horizontal scaling.
While there may be a limit, achieving 500,00 or 750,000 or even 1 million events per second with a Kafka cluster isn't beyond reach for the truly motivated. Nice work committers, very nice work. Jason This electronic message contains information which may be confidential or privileged. The information is intended for the use of the individual or entity named above. If you are not the intended recipient, be aware that any disclosure, copying, distribution or use of the contents of this information is prohibited. If you have received this electronic transmission in error, please notify us by e-mail at (postmas...@rapid7.com) immediately.
