suzhiking opened a new pull request, #23419: URL: https://github.com/apache/kafka/pull/23419
The streams rebalance protocol (KIP-1071) ships a single built-in task assignor, "sticky", which minimizes task movement between rebalances. The classic protocol also offers the HighAvailabilityTaskAssignor, whose placement spreads the tasks of each subtopology over the instances and evens out the per-thread task load, at the price of moving more tasks when the membership changes. This change adds that placement strategy to the streams protocol as a second built-in assignor named "balanced". The new BalancedTaskAssignor is a translation of the placement half of the classic HighAvailabilityTaskAssignor: stateful active tasks are dealt round-robin over the processes in sorted order and then moved between processes while a move reduces the skew of the per-member load; standby tasks go to the least loaded process that does not hold the task and are evened out the same way; stateless active tasks fill in the gaps by active task load. Since the broker assigns tasks to members rather than to processes, the assignor also spreads each process's tasks over its members, keeping a task on the member that currently owns it where the quota allows, which is what the classic client does when it distributes tasks over its stream threads. The cross-process placement does not depend on the previous assignment. The parts of the classic assignor that are not assignor concerns in the streams protocol are deliberately left out: warm-up tasks and probing rebalances are handled by the group coordinator's assignment refiner (KAFKA-20665), and rack-aware standby task assignment is tracked in KAFKA-20999 for both built-in assignors. Consequently the assignor does not read the members' warm-up tasks or reported offsets. The assignor is appended to the built-in list, so group.streams.assignors defaults to "sticky,balanced" and a group can select it with streams.assignor.name=balanced; "sticky" remains the default assignor. The KIP names this assignor "highly_available"; the name "balanced" reflects that high availability is delivered by the refiner rather than by the assignor, and the KIP is to be amended accordingly. Testing: BalancedTaskAssignorTest covers even distribution across processes and members, subtopology interleaving, the contrast with the sticky assignor on the same previous assignment, member stickiness within a process, standby placement and capacity shortfall, stateless task placement, independence from warm-up and offset inputs, determinism, and a randomized validity check. GroupCoordinatorConfigTest is updated for the new default, and StreamsGroupHeartbeatRequestTest selects the balanced assignor through the group config. StreamsBalancedAssignorBenchmark mirrors the sticky assignor benchmark; on the largest grid points the balanced assignor takes 1-5x the sticky assignor's time, except for groups of 1000 single-member processes where the all-pairs skew loop dominates (about 6-18 ms per assignment). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
