Vinoth Chandar created KAFKA-8799:
-------------------------------------
Summary: Support ability to pass global user data to consumers
during Assignment
Key: KAFKA-8799
URL: https://issues.apache.org/jira/browse/KAFKA-8799
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Vinoth Chandar
This is a follow up from KAFKA-7149
*Background :*
Although we reduced the size of the AssignmentInfo object sent during each
rebalance from leader to all followers in KAFKA-7149, we still repeat the same
_partitionsByHost_ map for each host (all this when interactive queries are
enabled) and thus still end up sending redundant bytes to the broker and also
logging a large kafka message.
With 100s of streams instances, this overhead can grow into tens of megabytes
easily.
*Proposal :*
Extend the group assignment protocol to be able to support passing of an
additional byte[], which can now contain the HostInfo ->
partitions/partitionsByHost data just one time.
{code}
final class GroupAssignment {
private final Map<String, Assignment> assignments;
// bytes sent to each consumer from leader
private final byte[] globalUserData
...
}
{code}
This can generally be handy to any other application like Streams, that does
some stateful processing or lightweight cluster management
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
