sanghyeok An created KAFKA-20934:
------------------------------------
Summary: Considering Share Groups for decoupling stateless Kafka
Streams processing parallelism from source partitions
Key: KAFKA-20934
URL: https://issues.apache.org/jira/browse/KAFKA-20934
Project: Kafka
Issue Type: Improvement
Reporter: sanghyeok An
Assignee: sanghyeok An
I am creating this Jira ticket for ideation. If Kafka maintainers or the
community think this direction is worth discussing, I would be happy to write a
KIP and develop the discussion further.
Currently, Kafka Streams processing parallelism is closely tied to the number
of partitions in the source topic. In contrast, Share Groups allow multiple
consumers to share the same partition, and the number of consumers can exceed
the number of partitions. Therefore, using Share Groups may provide a way to
decouple processing parallelism from the number of source partitions.
Applying this model to existing stateful Kafka Streams topologies does not
appear to be straightforward. Stateful processing in Kafka Streams is based on
a model in which a task owns specific input partitions and local state stores.
In addition, with Share Groups, records from the same partition may be
processed by different consumers, and partition-level ordering is not
guaranteed overall.
However, there may be room to use Share Groups for stateless Kafka Streams
topologies where record processing is order-independent. In such topologies,
processing records from the same source partition across multiple Streams
instances would not introduce conflicts in terms of state ownership,
potentially allowing the number of processing instances to exceed the number of
source partitions.
Although proper performance evaluation would be necessary, this could
potentially improve throughput for stateless topologies where application-side
processing is the bottleneck. It could also reduce the need to over-partition
topics solely to achieve higher processing parallelism, which may in turn
reduce the operational overhead associated with maintaining a large number of
partitions. KIP-932 also describes over-partitioning for parallel consumption
as one of the problems that Share Groups are intended to address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
