Hi,
I am running simple experiments to evaluate the scalability of Kafka
consumers with respect to the number of partitions. I assign every consumer
to a specific partition. Each consumer polls the records in its assigned
partition and print the first one, then polls again from the offset of the
printed record until all records are printed. Prior to running the test, I
produce 10 Million records evenly among partitions. After running the test,
I measure the time it took for the consumers to print all the records. I
was expecting Kafka to scale as I increase the number of
consumers/partitions. However, the scalability diminishes as I increase the
number of partitions/consumers, beyond certain number. Going from 1,2,4,8
the scalability is great as the duration of the test is reduced by the
factor increase of the number of partitions/consumers. However, beyond 8
consumers/partitions, the duration of the test reaches a steady state. I am
monitoring the resources of my server and didn't see any bottleneck. Am I
missing something here? Shouldn't Kafka consumers scale with the number of
partitions?
--
Best Regards,
Yazeed Alabdulkarim
