Matthias J. Sax created KAFKA-4969:
--------------------------------------
Summary: State-store workload-aware StreamsPartitionAssignor
Key: KAFKA-4969
URL: https://issues.apache.org/jira/browse/KAFKA-4969
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Matthias J. Sax
Currently, {{StreamPartitionsAssigner}} does not distinguish different "types"
of tasks. For example, task can be stateless of have one or multiple stores.
This can lead to an suboptimal task placement: assume there are 2 stateless and
2 stateful tasks and the app is running with 2 instances. To share the "store
load" it would be good to place one stateless and one stateful task per
instance. Right now, there is no guarantee about this, and it can happen, that
one instance processed both stateless tasks while the other processes both
stateful tasks.
We should improve {{StreamPartitionAssignor}} and introduce "task types"
including a cost model for task placement. We should consider the following
parameters:
- number of stores
- number of sources/sinks
- number of processors
- regular task vs standby task
This improvement should be backed by a design document in the project wiki (no
KIP required though) as it's a fairly complex change.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
