Hi there, I don't know about the consumer, but I'd *strongly* recommend not designing your application around this. Kafka has severe and notable stability concerns with large numbers of partitions, and requiring "one partition per customer" is going to be limiting, unless you only ever expect to have *very* small customer numbers (hundreds at most, ever). Instead, use a hash function and a key, as recommended to land customers on the same partition.
Thanks Tom Crayford Heroku Kafka On Tue, May 23, 2017 at 9:46 AM, David Espinosa <espi...@gmail.com> wrote: > Hi, > > In order to keep separated (physically) the data from different customers > in our application, we are using a custom partitioner to drive messages to > a concrete partition of a topic. We know that we are loosing parallelism > per topic this way, but our requirements regarding multitenancy are higher > than our throughput requirements. > > So, in order to increase the number of customers working on a cluster, we > are increasing the number of partitions dinamically per topic as the new > customer arrives using kafka AdminUtilities. > Our problem arrives when using the new kafka consumer and a new partition > is added into the topic, as this consumer doesn't get updated with the "new > partition" and therefore messages driven into that new partition never > arrives to this consumer unless we reload the consumer itself. What was > surprising was to check that using the old consumer (configured to deal > with Zookeeper), a consumer does get messages from a new added partition. > > Is there a way to emulate the old consumer behaviour when new partitions > are added in the new consumer? > > Thanks in advance, > David >
