Re: Scaling Apache Kafka Producers & Consumers
Very good description with pictures in the book Kafka: The Definitive Guide https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch04.html -hans > On Mar 26, 2020, at 12:00 PM, sunil chaudhari > wrote: > > Again > A consumer can have one or more consumer thread. > The analogy of 12 partitions and 4 consumer is true when each consumer has > 3 consumer threads. > Please don’t skip the important factor “consumer thread” in this matter. > > If you run each consumer with threads then you may need max 3 consumers for > that consumer group. > > If you have 12 partitions and you run 4 consumers with 4 consumer threads > then 4 threads will be idle at any time T1. > > I hope this is clear. > > Thanks, > Sunil. > > On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen wrote: > >>> As per my understanding, in Apache Kafka a single consumer from a >> consumer >>> group can consume messages from one partition only. >> >> Not correct. A single consumer from a consumer group can consume from many >> partitions. For example if you had a topic with 12 partitions and 4 >> consumers in a consumer group, each consumer in the group would consume >> from 3 partitions. >> >> -hans
Re: Scaling Apache Kafka Producers & Consumers
Again A consumer can have one or more consumer thread. The analogy of 12 partitions and 4 consumer is true when each consumer has 3 consumer threads. Please don’t skip the important factor “consumer thread” in this matter. If you run each consumer with threads then you may need max 3 consumers for that consumer group. If you have 12 partitions and you run 4 consumers with 4 consumer threads then 4 threads will be idle at any time T1. I hope this is clear. Thanks, Sunil. On Thu, 26 Mar 2020 at 7:52 PM, Hans Jespersen wrote: > > As per my understanding, in Apache Kafka a single consumer from a > consumer > > group can consume messages from one partition only. > > Not correct. A single consumer from a consumer group can consume from many > partitions. For example if you had a topic with 12 partitions and 4 > consumers in a consumer group, each consumer in the group would consume > from 3 partitions. > > -hans
Re: Scaling Apache Kafka Producers & Consumers
> As per my understanding, in Apache Kafka a single consumer from a consumer > group can consume messages from one partition only. Not correct. A single consumer from a consumer group can consume from many partitions. For example if you had a topic with 12 partitions and 4 consumers in a consumer group, each consumer in the group would consume from 3 partitions. -hans
Re: Scaling Apache Kafka Producers & Consumers
Hi Prasad, Want to correct a bit. Ots not one consumer per partitions. Its one consumer thread per partitions. On Thu, 26 Mar 2020 at 4:49 PM, Prasad Suhas Shembekar < ps00516...@techmahindra.com> wrote: > Hi, > > I am using Apache Kafka as a Message Broker in our application. The > producers and consumers are running as Docker containers in Kubernetes. > Right now, the producer publishes messages to a topic in single partition. > While the consumer consumes it from the topic. > As per my understanding, in Apache Kafka a single consumer from a consumer > group can consume messages from one partition only. Meaning, if there is > only a single partition and multiple consumers in a consumer group, only > one consumer will consume the message and the rest will remain idle, till > Apache Kafka does the partition rebalancing. > As mentioned earlier, we have a single topic and single partition and > multiple consumers in a single group. Thus we won't be able to achieve the > horizontal scaling for message consumption. > > Please let me know if the above understanding is correct. > > I am looking out on how to create partitions dynamically in the topic, as > and when a new consumer is added to consumer group (K8S auto scaling of > PODS). > Also, how to make the producer write to these different partitions created > dynamically, without overloading few partitions. > > Request you to provide some inputs / suggestions on how to achieve this. > > Thanks & Regards, > Prasad Shembekar > Blue Marble > WST-020, D Non-ODC, Mihan SEZ, > Nagpur > Extension: 6272148 > Direct: 0712-6672148 > > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > >
Re: Scaling Apache Kafka Producers & Consumers
Hey Prasad (#StayAtHomeSaveLives), On Thu, 26 Mar 2020 at 11:19, Prasad Suhas Shembekar < ps00516...@techmahindra.com> wrote: > Hi, > > I am using Apache Kafka as a Message Broker in our application. The > producers and consumers are running as Docker containers in Kubernetes. > Right now, the producer publishes messages to a topic in single partition. > While the consumer consumes it from the topic. > As per my understanding, in Apache Kafka a single consumer from a consumer > group can consume messages from one partition only. Meaning, if there is > only a single partition and multiple consumers in a consumer group, only > one consumer will consume the message and the rest will remain idle, till > Apache Kafka does the partition rebalancing. > Yes this is correct. > As mentioned earlier, we have a single topic and single partition and > multiple consumers in a single group. Thus we won't be able to achieve the > horizontal scaling for message consumption. > > Please let me know if the above understanding is correct. > Yes this is correct. > > I am looking out on how to create partitions dynamically in the topic, as > and when a new consumer is added to consumer group (K8S auto scaling of > PODS). > Also, how to make the producer write to these different partitions created > dynamically, without overloading few partitions. > > Request you to provide some inputs / suggestions on how to achieve this. > > Before anyone could answer any specific use case-related questions, perhaps you could read this https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ I believe this could serve as a great pointer and learning experience (it certainly did for myself) before you could tackle more precise cases. Feel free to follow up and share your concerns after this. > Thanks & Regards, > Prasad Shembekar > Blue Marble > WST-020, D Non-ODC, Mihan SEZ, > Nagpur > Extension: 6272148 > Direct: 0712-6672148 > > > Disclaimer: This message and the information contained herein is > proprietary and confidential and subject to the Tech Mahindra policy > statement, you may review the policy at > http://www.techmahindra.com/Disclaimer.html externally > http://tim.techmahindra.com/tim/disclaimer.html internally within > TechMahindra. > >
Scaling Apache Kafka Producers & Consumers
Hi, I am using Apache Kafka as a Message Broker in our application. The producers and consumers are running as Docker containers in Kubernetes. Right now, the producer publishes messages to a topic in single partition. While the consumer consumes it from the topic. As per my understanding, in Apache Kafka a single consumer from a consumer group can consume messages from one partition only. Meaning, if there is only a single partition and multiple consumers in a consumer group, only one consumer will consume the message and the rest will remain idle, till Apache Kafka does the partition rebalancing. As mentioned earlier, we have a single topic and single partition and multiple consumers in a single group. Thus we won't be able to achieve the horizontal scaling for message consumption. Please let me know if the above understanding is correct. I am looking out on how to create partitions dynamically in the topic, as and when a new consumer is added to consumer group (K8S auto scaling of PODS). Also, how to make the producer write to these different partitions created dynamically, without overloading few partitions. Request you to provide some inputs / suggestions on how to achieve this. Thanks & Regards, Prasad Shembekar Blue Marble WST-020, D Non-ODC, Mihan SEZ, Nagpur Extension: 6272148 Direct: 0712-6672148 Disclaimer: This message and the information contained herein is proprietary and confidential and subject to the Tech Mahindra policy statement, you may review the policy at http://www.techmahindra.com/Disclaimer.html externally http://tim.techmahindra.com/tim/disclaimer.html internally within TechMahindra.
