Hello,
We're looking into using Kafka for a improved version of a system and the
question of how to scale Kafka came up. Specifically, we want to try to make
the system scale as transparently as possible. The concern was that if we go
from N to N*2 consumers that we would have some that are
If you plan ahead of time with enough partitions then you won't fall into
an issue of backed up consumers when you scale them up.
If you have 100 partitions 20 consumers can read from them (each could read
from 5 partitions). You can scale up to 100 consumers (one for each
partition) as the upper
While it’s good to plan ahead for growth, Kafka will still let you add more
partitions to a topic
https://kafka.apache.org/081/ops.html#basic_ops_modify_topic. This will
rebalance the hashing if you are partitioning by your key, and consumers will
probably end up with different partitions, but
You need to be thoughtful about adding more partitions. This is paramount
if you are doing semantic partitioning in which case adding more partitions
could break things downstream.
If you average lets say 100,000 messages per second and at full tilt
consumer 1:1 for each partition you can process
Meant to say burst 1,000,000 messages per second on those X partitions for
10 minutes
On Fri, Nov 21, 2014 at 12:12 AM, Joe Stein joe.st...@stealth.ly wrote:
You need to be thoughtful about adding more partitions. This is paramount
if you are doing semantic partitioning in which case adding
