[ https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950326#comment-16950326 ]
Navinder Brar commented on KAFKA-6144: -------------------------------------- [~vinoth] started a KIP today: [https://cwiki.apache.org/confluence/display/KAFKA/KIP-535%3A+Allow+state+stores+to+serve+stale+reads+during+rebalance]. > Allow state stores to serve stale reads during rebalance > -------------------------------------------------------- > > Key: KAFKA-6144 > URL: https://issues.apache.org/jira/browse/KAFKA-6144 > Project: Kafka > Issue Type: New Feature > Components: streams > Reporter: Antony Stubbs > Priority: Major > Labels: needs-kip > Attachments: image-2019-10-09-20-33-37-423.png, > image-2019-10-09-20-47-38-096.png > > > Currently when expanding the KS cluster, the new node's partitions will be > unavailable during the rebalance, which for large states can take a very long > time, or for small state stores even more than a few ms can be a deal breaker > for micro service use cases. > One workaround is to allow stale data to be read from the state stores when > use case allows. > Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks - > potentially a two phase rebalance > This is the description from KAFKA-6031 (keeping this JIRA as the title is > more descriptive): > {quote} > Currently reads for a key are served by single replica, which has 2 drawbacks: > - if replica is down there is a down time in serving reads for keys it was > responsible for until a standby replica takes over > - in case of semantic partitioning some replicas might become hot and there > is no easy way to scale the read load > If standby replicas would have endpoints that are exposed in StreamsMetadata > it would enable serving reads from several replicas, which would mitigate the > above drawbacks. > Due to the lag between replicas reading from multiple replicas simultaneously > would have weaker (eventual) consistency comparing to reads from single > replica. This however should be acceptable tradeoff in many cases. > {quote} -- This message was sent by Atlassian Jira (v8.3.4#803005)