Vinoth Chandar created KAFKA-8799: ------------------------------------- Summary: Support ability to pass global user data to consumers during Assignment Key: KAFKA-8799 URL: https://issues.apache.org/jira/browse/KAFKA-8799 Project: Kafka Issue Type: Improvement Components: streams Reporter: Vinoth Chandar
This is a follow up from KAFKA-7149 *Background :* Although we reduced the size of the AssignmentInfo object sent during each rebalance from leader to all followers in KAFKA-7149, we still repeat the same _partitionsByHost_ map for each host (all this when interactive queries are enabled) and thus still end up sending redundant bytes to the broker and also logging a large kafka message. With 100s of streams instances, this overhead can grow into tens of megabytes easily. *Proposal :* Extend the group assignment protocol to be able to support passing of an additional byte[], which can now contain the HostInfo -> partitions/partitionsByHost data just one time. {code} final class GroupAssignment { private final Map<String, Assignment> assignments; // bytes sent to each consumer from leader private final byte[] globalUserData ... } {code} This can generally be handy to any other application like Streams, that does some stateful processing or lightweight cluster management -- This message was sent by Atlassian JIRA (v7.6.14#76016)