Matthias J. Sax created KAFKA-21060:
---------------------------------------
Summary: Make StickyAssignor aware of stateless vs stateful tasks
Key: KAFKA-21060
URL: https://issues.apache.org/jira/browse/KAFKA-21060
Project: Kafka
Issue Type: Improvement
Components: group-coordinator, streams
Reporter: Matthias J. Sax
Kafka Streams currently ships three implementation of sticky task assignor:
* LegacyStickyTaskAssignor (internal only – classic protocol)
*
o.a.k.streams.processor.assignment.assignors.StickyTaskAssignor (base on public
API – classic protocol)
*
o.a.k.coordinator.group.streams.assignor.StickyTaskAssignor (streams protocol)
All three implementation basically round-robin across all task, and they do not
distinguish between stateless and stateful tasks.
It might be a good improvement, to actually have a "task bucket" for each
flavor (stateless and stateful), and round-robin over each bucket
independently, to ensure that we don't end up with two stateless tasks assigned
to one member, and two stateful tasks assigned to a different member.
We should also double check how we compute load, and might want to have two
load metrics, one for stateless load and one for stateful load.
Not sure if we want to make this change in all three assignors, or maybe only
for the new streams protocol one, to support adoption of KIP-1071. Note, for
"classic" sticky assignor is not the default, while for "streams" it is...
As this is only a high level idea, we might also want to do some socialized
testing to see if such an change is really beneficial or not? Not sure. Maybe
we can also assess intellectually that it's a good idea, w/o specific
"benchmarking"?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
