Re: Scaling up kafka consumers

2017-02-24 Thread Gerrit Jansen van Vuuren
The kafka fast connector handles this differently than the standard kafka client (which requires one consumer per partition at most), by breaking offsets into consumable ranges which allows one partition to be read by multiple conumers where each consumer uniquely receives a different offset

Re: Scaling up kafka consumers

2017-02-24 Thread Ian Wrigley
Hi If you have two consumers in your consumer group, but only one partition in the topic, then only one consumer will do any work. It’s not the case that “those two nodes start competing for messages” — one node will read from the partition, the other will have nothing to do. So to scale up by

Re: Scaling up kafka consumers

2017-02-24 Thread Pradeep Gollakota
A single partition can be consumed by at most a single consumer. Consumers compete to take ownership of a partition. So, in order to gain parallelism you need to add more partitions. There is a library that allows multiple consumers to consume from a single partition

Scaling up kafka consumers

2017-02-24 Thread Jakub Stransky
Hello everyone, I was reading/checking kafka documentation regarding point-2-point and publish subscribe communications patterns in kafka and I am wondering how to scale up consumer side in point to point scenario when consuming from single kafka topic. Let say I have a single topic with single