Sophie Blee-Goldman created KAFKA-9282:
------------------------------------------
Summary: Consider more flexible node grouping for Pattern
subscription
Key: KAFKA-9282
URL: https://issues.apache.org/jira/browse/KAFKA-9282
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Sophie Blee-Goldman
The current grouping for input topics using pattern subscription creates a
single node group for all matching topics, meaning the number of tasks scales
with the maximum partition count across all topics. This reduces overhead and
is suitable for some scenarios, but limits the ability to scale out and
prevents easily parallelized processing of completely independent partitions.
We should consider making it possible for the number of tasks to instead scale
with the total number of partitions summed across all matching input topics.
Ideally Streams could just autoscale based on some heuristic and the currently
available resources, but we would have to be careful if those things change.
Alternatively we could just leave this up to the user to decide, potentially by
augmenting the Pattern-based source KStream method with a new overload allowing
this grouping to be specified. For example
{code:java}
StreamsBuilder {
public KStream stream(topicPattern, numTasks);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
