Hi, am a also new to Kafka but let me explain my understanding which someone with more knowledge can confirm. There are actually 2 scenarios:
1. If all 5 of your consumers are in different "consumer groups" then this will behave like a JMS topic where all 5 of your consumers will each get a copy of all of the messages. You can add as many consumers as you want regardless of the fact that there is only 1 partition. 2. If all 5 of your consumers are in the same "consumer group" and there is only 1 partition, only 1 consumer will get a copy of the messages. You should increase your number of partitions so there are as many as there are consumers in this case so each will get 1/5 of the messages (assuming the messages are evenly distributed across the partitions). Is that what you were looking for? Can someone confirm that what I stated is accurate? Thanks Josh ________________________________ From: Vinicius Carvalho <[email protected]> To: [email protected] Sent: Tuesday, July 2, 2013 2:55 PM Subject: Partitions and highlevel consumers Hi guys, we are starting with kafka in our project. We are using version 0.8. I come from a traditional JMS MoM architecture, and some things are new to me. One thing that I'm not getting is the mapping between partitions and number of threads. On a single consumer I can see the relationship but what happens when you have multiple consumers? So say, I have a topic and I have 5 instances of services that acts as consumers. On a traditional pub/sub I would have all 5 instances to consume the message right? Even if this topic only has one partition and now we have 5 threads (one per jvm instance) consuming it. Does this proceed? Just want to make sure if that is the expected behavior, from the docs it wasn't clear if its the number of threads per consumer, or the total number of threads around all clients to the topic. Regards -- The intuitive mind is a sacred gift and the rational mind is a faithful servant. We have created a society that honors the servant and has forgotten the gift.
