[
https://issues.apache.org/jira/browse/KAFKA-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17084542#comment-17084542
]
ASF GitHub Bot commented on KAFKA-6145:
---------------------------------------
ableegoldman commented on pull request #8497: KAFKA-6145: KIP-441 Build state
constrained assignment from balanced one
URL: https://github.com/apache/kafka/pull/8497
John's awesome `TaskAssignorConvergenceTest` revealed some issues with the
current assignor, which he nailed down as being due to the state constrained
and balanced assignments not converging.
One way to get an assignment that is as close to the balanced assignment as
possible while still being state constrained is of course to start with the
balanced assignment, and move tasks around as necessary to satisfy the state
constraint. With this basic approach, the converge test is passing.
This PR also includes some semi-orthogonal refactoring, most significantly
the removal of the assignment maps; we now just immediately assign tasks to
the `ClientState` rather than first sticking them in an intermediate map.
Also moves `ValidClientsByTaskLoadQueue` to its own file
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Warm up new KS instances before migrating tasks - potentially a two phase
> rebalance
> -----------------------------------------------------------------------------------
>
> Key: KAFKA-6145
> URL: https://issues.apache.org/jira/browse/KAFKA-6145
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: Antony Stubbs
> Assignee: Sophie Blee-Goldman
> Priority: Major
> Labels: needs-kip
>
> Currently when expanding the KS cluster, the new node's partitions will be
> unavailable during the rebalance, which for large states can take a very long
> time, or for small state stores even more than a few ms can be a deal breaker
> for micro service use cases.
> One workaround would be two execute the rebalance in two phases:
> 1) start running state store building on the new node
> 2) once the state store is fully populated on the new node, only then
> rebalance the tasks - there will still be a rebalance pause, but would be
> greatly reduced
> Relates to: KAFKA-6144 - Allow state stores to serve stale reads during
> rebalance
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
