[ https://issues.apache.org/jira/browse/KAFKA-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084542#comment-17084542 ]
ASF GitHub Bot commented on KAFKA-6145: --------------------------------------- ableegoldman commented on pull request #8497: KAFKA-6145: KIP-441 Build state constrained assignment from balanced one URL: https://github.com/apache/kafka/pull/8497 John's awesome `TaskAssignorConvergenceTest` revealed some issues with the current assignor, which he nailed down as being due to the state constrained and balanced assignments not converging. One way to get an assignment that is as close to the balanced assignment as possible while still being state constrained is of course to start with the balanced assignment, and move tasks around as necessary to satisfy the state constraint. With this basic approach, the converge test is passing. This PR also includes some semi-orthogonal refactoring, most significantly the removal of the assignment maps; we now just immediately assign tasks to the `ClientState` rather than first sticking them in an intermediate map. Also moves `ValidClientsByTaskLoadQueue` to its own file ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Warm up new KS instances before migrating tasks - potentially a two phase > rebalance > ----------------------------------------------------------------------------------- > > Key: KAFKA-6145 > URL: https://issues.apache.org/jira/browse/KAFKA-6145 > Project: Kafka > Issue Type: New Feature > Components: streams > Reporter: Antony Stubbs > Assignee: Sophie Blee-Goldman > Priority: Major > Labels: needs-kip > > Currently when expanding the KS cluster, the new node's partitions will be > unavailable during the rebalance, which for large states can take a very long > time, or for small state stores even more than a few ms can be a deal breaker > for micro service use cases. > One workaround would be two execute the rebalance in two phases: > 1) start running state store building on the new node > 2) once the state store is fully populated on the new node, only then > rebalance the tasks - there will still be a rebalance pause, but would be > greatly reduced > Relates to: KAFKA-6144 - Allow state stores to serve stale reads during > rebalance -- This message was sent by Atlassian Jira (v8.3.4#803005)