[ https://issues.apache.org/jira/browse/KAFKA-6145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Antony Stubbs updated KAFKA-6145: --------------------------------- Description: Currently when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal breaker for micro service use cases. One workaround would be two execute the rebalance in two phases: 1) start running state store building on the new node 2) once the state store is fully populated on the new node, only then rebalance the tasks - there will still be a rebalance pause, but would be greatly reduced Relates to: KAFKA-6144 - Allow state stores to serve stale reads during rebalance was: Currently when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal breaker for micro service use cases. One workaround would be two execute the rebalance in two phases: 1) start running state store building on the new node 2) once the state store is fully populated on the new node, only then rebalance the tasks - there will still be a rebalance pause, but would be greatly reduced > Warm up new KS instances before migrating tasks - potentially a two phase > rebalance > ----------------------------------------------------------------------------------- > > Key: KAFKA-6145 > URL: https://issues.apache.org/jira/browse/KAFKA-6145 > Project: Kafka > Issue Type: New Feature > Components: streams > Reporter: Antony Stubbs > > Currently when expanding the KS cluster, the new node's partitions will be > unavailable during the rebalance, which for large states can take a very long > time, or for small state stores even more than a few ms can be a deal breaker > for micro service use cases. > One workaround would be two execute the rebalance in two phases: > 1) start running state store building on the new node > 2) once the state store is fully populated on the new node, only then > rebalance the tasks - there will still be a rebalance pause, but would be > greatly reduced > Relates to: KAFKA-6144 - Allow state stores to serve stale reads during > rebalance -- This message was sent by Atlassian JIRA (v6.4.14#64029)