Re: Multiple consumers for a Single Partition in Kafka
Sounds like the parallel consumer is what you want: https://github.com/confluentinc/parallel-consumer Or at least decide why that won't work before writing something yourself. - alex On Tue, Oct 3, 2023 at 4:20 PM Sree Sanjeev Kandasamy Gokulrajan < sreesanjee...@gmail.com> wrote: > Hey Neeraj > I had a kafka assignment where I had to consume from a single partition. > When I was looking to scale the application, I was planning to consume the > partition with more than one consumer when I came across this issue. > > Regards, > Sree Sanjeev. > > On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya > wrote: > > > Hi Sree Sanjeev, > > Other than the info from Colt, why can't you just increase partitions and > > scale the number of consumers accordingly ? > > Or build your application as a KStreams application and take advantage of > > horizontal and vertical scalability ? > > > > Regards, > > Neeraj > > > > > > On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy < > > c...@littlehorse.io> wrote: > > > > Hello, > > > > Currently, you cannot have more than one consumer consuming from a > > partition in Kafka. However, there is active discussion about a proposal > to > > enable queueing semantics for Kafka, which is a superset of the feature > you > > are talking about. Feel free to check KIP-932: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > > > > Hope this helps, > > Colt McNealy > > > > *Founder, LittleHorse.dev* > > > > > > On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan < > > sreesanjee...@gmail.com> wrote: > > > > > I'm interested in knowing if we can achieve parallelism by allowing > > > multiple consumers to subscribe to a single partition of a topic. > > > > > > To explore potential solutions, I'm considering the following approach. > > > > > > 1. Implementing a locking mechanism to control access to the offsets > in a > > > partition to avoid concurrent processing issues. > > > > > > My primary concern is maintaining message ordering within the > partition, > > as > > > Kafka guarantees ordering within a partition. and message ordering is > not > > > the primary concern for many applications, so can we have a flag to > > enable > > > multiple consumers to a single partition ? > > > > > > I appreciate any insights, advice, or experiences that the Kafka > > community > > > can share on this topic. Thank you! > > > > > > Regards, > > > Sree Sanjeev. > > > > > >
Re: Multiple consumers for a Single Partition in Kafka
Hey Neeraj I had a kafka assignment where I had to consume from a single partition. When I was looking to scale the application, I was planning to consume the partition with more than one consumer when I came across this issue. Regards, Sree Sanjeev. On Tue, Oct 3, 2023 at 3:48 PM Neeraj Vaidya wrote: > Hi Sree Sanjeev, > Other than the info from Colt, why can't you just increase partitions and > scale the number of consumers accordingly ? > Or build your application as a KStreams application and take advantage of > horizontal and vertical scalability ? > > Regards, > Neeraj > > > On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy < > c...@littlehorse.io> wrote: > > Hello, > > Currently, you cannot have more than one consumer consuming from a > partition in Kafka. However, there is active discussion about a proposal to > enable queueing semantics for Kafka, which is a superset of the feature you > are talking about. Feel free to check KIP-932: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka > > Hope this helps, > Colt McNealy > > *Founder, LittleHorse.dev* > > > On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan < > sreesanjee...@gmail.com> wrote: > > > I'm interested in knowing if we can achieve parallelism by allowing > > multiple consumers to subscribe to a single partition of a topic. > > > > To explore potential solutions, I'm considering the following approach. > > > > 1. Implementing a locking mechanism to control access to the offsets in a > > partition to avoid concurrent processing issues. > > > > My primary concern is maintaining message ordering within the partition, > as > > Kafka guarantees ordering within a partition. and message ordering is not > > the primary concern for many applications, so can we have a flag to > enable > > multiple consumers to a single partition ? > > > > I appreciate any insights, advice, or experiences that the Kafka > community > > can share on this topic. Thank you! > > > > Regards, > > Sree Sanjeev. > > >
Re: Multiple consumers for a Single Partition in Kafka
Hi Sree Sanjeev, Other than the info from Colt, why can't you just increase partitions and scale the number of consumers accordingly ? Or build your application as a KStreams application and take advantage of horizontal and vertical scalability ? Regards, Neeraj On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy wrote: Hello, Currently, you cannot have more than one consumer consuming from a partition in Kafka. However, there is active discussion about a proposal to enable queueing semantics for Kafka, which is a superset of the feature you are talking about. Feel free to check KIP-932: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka Hope this helps, Colt McNealy *Founder, LittleHorse.dev* On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan < sreesanjee...@gmail.com> wrote: > I'm interested in knowing if we can achieve parallelism by allowing > multiple consumers to subscribe to a single partition of a topic. > > To explore potential solutions, I'm considering the following approach. > > 1. Implementing a locking mechanism to control access to the offsets in a > partition to avoid concurrent processing issues. > > My primary concern is maintaining message ordering within the partition, as > Kafka guarantees ordering within a partition. and message ordering is not > the primary concern for many applications, so can we have a flag to enable > multiple consumers to a single partition ? > > I appreciate any insights, advice, or experiences that the Kafka community > can share on this topic. Thank you! > > Regards, > Sree Sanjeev. >
Re: Multiple consumers for a Single Partition in Kafka
Hi Sree. Yeah, parallel-processing-per-partition requirement often arises particularly when the throughput is limited due to external I/O latency and there are some solutions: - https://github.com/line/decaton * provides per-key ordering guarantee (or can be unordered if no ordering is necessary) * also refs: https://speakerdeck.com/line_devday2020/decaton-high-performance-task-processing-with-kafka - https://github.com/confluentinc/parallel-consumer * also seems to provide per-key ordering guarantee or can be unordered 2023年10月3日(火) 23:11 Sree Sanjeev Kandasamy Gokulrajan < sreesanjee...@gmail.com>: > I'm interested in knowing if we can achieve parallelism by allowing > multiple consumers to subscribe to a single partition of a topic. > > To explore potential solutions, I'm considering the following approach. > > 1. Implementing a locking mechanism to control access to the offsets in a > partition to avoid concurrent processing issues. > > My primary concern is maintaining message ordering within the partition, as > Kafka guarantees ordering within a partition. and message ordering is not > the primary concern for many applications, so can we have a flag to enable > multiple consumers to a single partition ? > > I appreciate any insights, advice, or experiences that the Kafka community > can share on this topic. Thank you! > > Regards, > Sree Sanjeev. > -- Okada Haruki ocadar...@gmail.com
Re: Multiple consumers for a Single Partition in Kafka
Hello, Currently, you cannot have more than one consumer consuming from a partition in Kafka. However, there is active discussion about a proposal to enable queueing semantics for Kafka, which is a superset of the feature you are talking about. Feel free to check KIP-932: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka Hope this helps, Colt McNealy *Founder, LittleHorse.dev* On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan < sreesanjee...@gmail.com> wrote: > I'm interested in knowing if we can achieve parallelism by allowing > multiple consumers to subscribe to a single partition of a topic. > > To explore potential solutions, I'm considering the following approach. > > 1. Implementing a locking mechanism to control access to the offsets in a > partition to avoid concurrent processing issues. > > My primary concern is maintaining message ordering within the partition, as > Kafka guarantees ordering within a partition. and message ordering is not > the primary concern for many applications, so can we have a flag to enable > multiple consumers to a single partition ? > > I appreciate any insights, advice, or experiences that the Kafka community > can share on this topic. Thank you! > > Regards, > Sree Sanjeev. >
Multiple consumers for a Single Partition in Kafka
I'm interested in knowing if we can achieve parallelism by allowing multiple consumers to subscribe to a single partition of a topic. To explore potential solutions, I'm considering the following approach. 1. Implementing a locking mechanism to control access to the offsets in a partition to avoid concurrent processing issues. My primary concern is maintaining message ordering within the partition, as Kafka guarantees ordering within a partition. and message ordering is not the primary concern for many applications, so can we have a flag to enable multiple consumers to a single partition ? I appreciate any insights, advice, or experiences that the Kafka community can share on this topic. Thank you! Regards, Sree Sanjeev.
