The kafka fast connector handles this differently than the standard kafka client (which requires one consumer per partition at most), by breaking offsets into consumable ranges which allows one partition to be read by multiple conumers where each consumer uniquely receives a different offset range.
See: https://github.com/gerritjvv/kafka-fast That said: There are two usage scenarios: Lots of topics (10 or more) where you'd want 1-2-4 partitions or less topics where you want more partitions, if yours is the latter you'd want more than a single partition 4-6 are better numbers ( numbers are from my experience with 5 nodes). On 24 Feb 2017 17:30, "Jakub Stransky" <stransky...@gmail.com> wrote: > Hello everyone, > > I was reading/checking kafka documentation regarding point-2-point and > publish subscribe communications patterns in kafka and I am wondering how > to scale up consumer side in point to point scenario when consuming from > single kafka topic. > > Let say I have a single topic with single partition and I have one node > where the kafka consumer is running. If I want to scale up my service I add > another node - which has the same configuration as the first one (topic, > partition and consumer group id). Those two nodes start competing for > messages from kafka topic. > > What I am not sure in this scenario and is actually subject of my question > is "*Whether they do get each node unique messages or there is still > possibility that some messages will be consumed by both nodes etc*". > Because I can see scenarios that both nodes are started at the same time - > they gets the same topic offset from zookeeper and started consuming > messages from that offset. OR am I thinking in a wrong direction? > > Thanks > Jakub >
