Hi Sree. Yeah, parallel-processing-per-partition requirement often arises particularly when the throughput is limited due to external I/O latency and there are some solutions:
- https://github.com/line/decaton * provides per-key ordering guarantee (or can be unordered if no ordering is necessary) * also refs: https://speakerdeck.com/line_devday2020/decaton-high-performance-task-processing-with-kafka - https://github.com/confluentinc/parallel-consumer * also seems to provide per-key ordering guarantee or can be unordered 2023年10月3日(火) 23:11 Sree Sanjeev Kandasamy Gokulrajan < sreesanjee...@gmail.com>: > I'm interested in knowing if we can achieve parallelism by allowing > multiple consumers to subscribe to a single partition of a topic. > > To explore potential solutions, I'm considering the following approach. > > 1. Implementing a locking mechanism to control access to the offsets in a > partition to avoid concurrent processing issues. > > My primary concern is maintaining message ordering within the partition, as > Kafka guarantees ordering within a partition. and message ordering is not > the primary concern for many applications, so can we have a flag to enable > multiple consumers to a single partition ? > > I appreciate any insights, advice, or experiences that the Kafka community > can share on this topic. Thank you! > > Regards, > Sree Sanjeev. > -- ======================== Okada Haruki ocadar...@gmail.com ========================
