Hi team, The initial draft of this feature is now available in the Google Doc: https://docs.google.com/document/d/1_wf3GwqWV3yiD-0oG2d6jwN8bfdnbk-Kz13btva_QAc/edit?usp=sharing . Kindly review the document and share your feedback or suggestions in the comments to help enhance the content further.
Regards, Shekhar Prasad Rajak, On Tuesday 2 September 2025 at 03:39:31 pm GMT+5:30, Gyula Fóra <[email protected]> wrote: Hello! I see there is a WIP PR that has a bit more architecture / technical information here: https://github.com/apache/flink-connector-kafka/pull/189 It would be nice to put this into a google doc proposal that follows the FLIP format for discussion. One question that came to me immediately is regarding the exactly-once semantics. The current Flink kafka consumer implementation provides exactly-once read semantics, but it's not clear if this new approach would only have at-least-once or exactly-once is also possible. Thanks Gyula On Sat, Aug 30, 2025 at 7:30 PM RShekhar Prasad <[email protected]> wrote: > Hello team, > KIP-932 introduces share groups as a new consumption model that provides > queue semantics. > This directly addresses use cases where: > > 1. Multiple consumers need to process items efficiently in parallel from a > single/multiple topic(s). > 2. Messages need explicit acknowledgment/release (to avoid reprocessing > or allow retries).Use cases where scaling Flink ML/LLM workload is critical > - Shifting Kafka coordination and assignment logic to the broker side would > simplify today’s complex Flink source management, making consumption more > efficient, scalable, and far less error-prone. > Operational Benefits > > - Higher Throughput: ShareGroupHeartbeat helps in Queue-like workloads, > maximum throughput scenarios. Share groups distribute messages at the > record level, not partition level, so multiple readers can consume from the > same topic with Kafka coordinating message distribution. > - Better Availability and Flexible Scaling: consumers assignment logic > is simpler in server side and rebalancing frequency is minimised. > > Let's have discussion over the design and how the checkpointing will work > when we use KafkaShareConsumer API from Kafka 4.1 . > Regards, > Shekhar Prasad Rajak, > Blog | Github | Twitter >
