[
https://issues.apache.org/jira/browse/KAFKA-20934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-20934:
------------------------------------
Component/s: group-coordinator
streams
> Considering Share Groups for decoupling stateless Kafka Streams processing
> parallelism from source partitions
> -------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20934
> URL: https://issues.apache.org/jira/browse/KAFKA-20934
> Project: Kafka
> Issue Type: Improvement
> Components: group-coordinator, streams
> Reporter: sanghyeok An
> Assignee: sanghyeok An
> Priority: Minor
> Labels: needs-kip
>
> I am creating this Jira ticket for ideation. If Kafka maintainers or the
> community think this direction is worth discussing, I would be happy to write
> a KIP and develop the discussion further.
> Currently, Kafka Streams processing parallelism is closely tied to the number
> of partitions in the source topic. In contrast, Share Groups allow multiple
> consumers to share the same partition, and the number of consumers can exceed
> the number of partitions. Therefore, using Share Groups may provide a way to
> decouple processing parallelism from the number of source partitions.
> Applying this model to existing stateful Kafka Streams topologies does not
> appear to be straightforward. Stateful processing in Kafka Streams is based
> on a model in which a task owns specific input partitions and local state
> stores. In addition, with Share Groups, records from the same partition may
> be processed by different consumers, and partition-level ordering is not
> guaranteed overall.
> However, there may be room to use Share Groups for stateless Kafka Streams
> topologies where record processing is order-independent. In such topologies,
> processing records from the same source partition across multiple Streams
> instances would not introduce conflicts in terms of state ownership,
> potentially allowing the number of processing instances to exceed the number
> of source partitions.
> Although proper performance evaluation would be necessary, this could
> potentially improve throughput for stateless topologies where
> application-side processing is the bottleneck. It could also reduce the need
> to over-partition topics solely to achieve higher processing parallelism,
> which may in turn reduce the operational overhead associated with maintaining
> a large number of partitions. KIP-932 also describes over-partitioning for
> parallel consumption as one of the problems that Share Groups are intended to
> address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
