Hi Sree Sanjeev,
Other than the info from Colt, why can't you just increase partitions and scale
the number of consumers accordingly ?
Or build your application as a KStreams application and take advantage of
horizontal and vertical scalability ?
Regards,
Neeraj
On Wednesday, 4 October, 2023 at 01:19:37 am GMT+11, Colt McNealy
<[email protected]> wrote:
Hello,
Currently, you cannot have more than one consumer consuming from a
partition in Kafka. However, there is active discussion about a proposal to
enable queueing semantics for Kafka, which is a superset of the feature you
are talking about. Feel free to check KIP-932:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A+Queues+for+Kafka
Hope this helps,
Colt McNealy
*Founder, LittleHorse.dev*
On Tue, Oct 3, 2023 at 9:11 AM Sree Sanjeev Kandasamy Gokulrajan <
[email protected]> wrote:
> I'm interested in knowing if we can achieve parallelism by allowing
> multiple consumers to subscribe to a single partition of a topic.
>
> To explore potential solutions, I'm considering the following approach.
>
> 1. Implementing a locking mechanism to control access to the offsets in a
> partition to avoid concurrent processing issues.
>
> My primary concern is maintaining message ordering within the partition, as
> Kafka guarantees ordering within a partition. and message ordering is not
> the primary concern for many applications, so can we have a flag to enable
> multiple consumers to a single partition ?
>
> I appreciate any insights, advice, or experiences that the Kafka community
> can share on this topic. Thank you!
>
> Regards,
> Sree Sanjeev.
>
