That sounds a lot -- even if brokers can handle quite some load. How many brokers do you got?
Main question: why do you need a consumer per partition? If you processing super expensive such that one consumer cannot handle multiple partitions? -Matthias On 3/31/17 5:02 AM, Laxmi Narayan wrote: > Hi, > > Is there any performance downside of creating so many consumers ? > > I mean literally I am gonna create atleast 7k connections in that case , I > have nearly 7k partitions with a given topic. > > > Keep learning keep moving ..... > > On Fri, Mar 31, 2017 at 12:48 PM, Matthias J. Sax <[email protected]> > wrote: > >> You need to create a KafkaConsumer per thread. >> >> -Matthias >> >> On 3/30/17 10:51 PM, Laxmi Narayan wrote: >>> Hi , >>> >>> I was thinking to listen each partition with separate thread in Kafka. >>> But i get error saying : >>> >>> >>> >>> >>> *org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer >> is >>> not safe for multi-threaded accessSubscription to topics, partitions and >>> pattern are mutually exclusive.* >>> >>> Is there any way to achieve this ? >>> >>> >>> ** >>> Keep learning keep moving ..... >>> >> >> >
signature.asc
Description: OpenPGP digital signature
