Sophie Blee-Goldman created KAFKA-9281:
------------------------------------------
Summary: Consider more flexible node grouping for Pattern
subscription
Key: KAFKA-9281
URL: https://issues.apache.org/jira/browse/KAFKA-9281
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Sophie Blee-Goldman
Input topics subscribed to using pattern subscription will currently all be
grouped into the same node group, meaning the number of tasks is determined by
the maximum partition count of any matching topic. This means less overhead per
partition and is suitable for some scenarios, but it limits the ability to
scale out by preventing further parallelization that is possible with
independent partitions. We should consider making it possible for pattern
subscription to create a task for every partition summed across all matching
topics.
We don't necessarily want to change the default (current) behavior, but we
could make this more flexible either by autoscaling based on some heuristic, or
making it customizable by the user. One possibility would be to augment the
Pattern based source KStream method with an optional parameter that to tell
Streams how to generate tasks for that pattern, for example
{code:java}
public synchronized KStream stream(pattern, numTasks);
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
