[jira] [Updated] (KAFKA-6144) Allow state stores to serve stale reads during rebalance
[
https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navinder Brar updated KAFKA-6144:
-
Summary: Allow state stores to serve stale reads during rebalance (was:
Allow serving interactive queries from in-sync Standbys)
> Allow state stores to serve stale reads during rebalance
>
>
> Key: KAFKA-6144
> URL: https://issues.apache.org/jira/browse/KAFKA-6144
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Antony Stubbs
>Assignee: Navinder Brar
>Priority: Major
> Labels: kip-535
> Attachments: image-2019-10-09-20-33-37-423.png,
> image-2019-10-09-20-47-38-096.png
>
>
> Currently when expanding the KS cluster, the new node's partitions will be
> unavailable during the rebalance, which for large states can take a very long
> time, or for small state stores even more than a few ms can be a deal breaker
> for micro service use cases.
> One workaround is to allow stale data to be read from the state stores when
> use case allows.
> Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
> potentially a two phase rebalance
> This is the description from KAFKA-6031 (keeping this JIRA as the title is
> more descriptive):
> {quote}
> Currently reads for a key are served by single replica, which has 2 drawbacks:
> - if replica is down there is a down time in serving reads for keys it was
> responsible for until a standby replica takes over
> - in case of semantic partitioning some replicas might become hot and there
> is no easy way to scale the read load
> If standby replicas would have endpoints that are exposed in StreamsMetadata
> it would enable serving reads from several replicas, which would mitigate the
> above drawbacks.
> Due to the lag between replicas reading from multiple replicas simultaneously
> would have weaker (eventual) consistency comparing to reads from single
> replica. This however should be acceptable tradeoff in many cases.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-6144) Allow state stores to serve stale reads during rebalance
[
https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Navinder Brar updated KAFKA-6144:
-
Attachment: image-2019-10-09-20-33-37-423.png
> Allow state stores to serve stale reads during rebalance
>
>
> Key: KAFKA-6144
> URL: https://issues.apache.org/jira/browse/KAFKA-6144
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Antony Stubbs
>Priority: Major
> Labels: needs-kip
> Attachments: image-2019-10-09-20-33-37-423.png
>
>
> Currently when expanding the KS cluster, the new node's partitions will be
> unavailable during the rebalance, which for large states can take a very long
> time, or for small state stores even more than a few ms can be a deal breaker
> for micro service use cases.
> One workaround is to allow stale data to be read from the state stores when
> use case allows.
> Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
> potentially a two phase rebalance
> This is the description from KAFKA-6031 (keeping this JIRA as the title is
> more descriptive):
> {quote}
> Currently reads for a key are served by single replica, which has 2 drawbacks:
> - if replica is down there is a down time in serving reads for keys it was
> responsible for until a standby replica takes over
> - in case of semantic partitioning some replicas might become hot and there
> is no easy way to scale the read load
> If standby replicas would have endpoints that are exposed in StreamsMetadata
> it would enable serving reads from several replicas, which would mitigate the
> above drawbacks.
> Due to the lag between replicas reading from multiple replicas simultaneously
> would have weaker (eventual) consistency comparing to reads from single
> replica. This however should be acceptable tradeoff in many cases.
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-6144) Allow state stores to serve stale reads during rebalance
[
https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-6144:
---
Labels: needs-kip (was: )
> Allow state stores to serve stale reads during rebalance
>
>
> Key: KAFKA-6144
> URL: https://issues.apache.org/jira/browse/KAFKA-6144
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Antony Stubbs
>Priority: Major
> Labels: needs-kip
>
> Currently when expanding the KS cluster, the new node's partitions will be
> unavailable during the rebalance, which for large states can take a very long
> time, or for small state stores even more than a few ms can be a deal breaker
> for micro service use cases.
> One workaround is to allow stale data to be read from the state stores when
> use case allows.
> Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
> potentially a two phase rebalance
> This is the description from KAFKA-6031 (keeping this JIRA as the title is
> more descriptive):
> {quote}
> Currently reads for a key are served by single replica, which has 2 drawbacks:
> - if replica is down there is a down time in serving reads for keys it was
> responsible for until a standby replica takes over
> - in case of semantic partitioning some replicas might become hot and there
> is no easy way to scale the read load
> If standby replicas would have endpoints that are exposed in StreamsMetadata
> it would enable serving reads from several replicas, which would mitigate the
> above drawbacks.
> Due to the lag between replicas reading from multiple replicas simultaneously
> would have weaker (eventual) consistency comparing to reads from single
> replica. This however should be acceptable tradeoff in many cases.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6144) Allow state stores to serve stale reads during rebalance
[
https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-6144:
---
Description:
Currently when expanding the KS cluster, the new node's partitions will be
unavailable during the rebalance, which for large states can take a very long
time, or for small state stores even more than a few ms can be a deal breaker
for micro service use cases.
One workaround is to allow stale data to be read from the state stores when use
case allows.
Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
potentially a two phase rebalance
This is the description from KAFKA-6031 (keeping this JIRA as the title is more
descriptive):
{quote}
Currently reads for a key are served by single replica, which has 2 drawbacks:
- if replica is down there is a down time in serving reads for keys it was
responsible for until a standby replica takes over
- in case of semantic partitioning some replicas might become hot and there is
no easy way to scale the read load
If standby replicas would have endpoints that are exposed in StreamsMetadata it
would enable serving reads from several replicas, which would mitigate the
above drawbacks.
Due to the lag between replicas reading from multiple replicas simultaneously
would have weaker (eventual) consistency comparing to reads from single
replica. This however should be acceptable tradeoff in many cases.
{quote}
was:
Currently when expanding the KS cluster, the new node's partitions will be
unavailable during the rebalance, which for large states can take a very long
time, or for small state stores even more than a few ms can be a deal breaker
for micro service use cases.
One workaround is to allow stale data to be read from the state stores when use
case allows.
Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
potentially a two phase rebalance
This is the description from KAFKA-6031 (keeping this JIRA as the title is more
descriptive):
{quote}
Currently reads for a key are served by single replica, which has 2 drawbacks:
- if replica is down there is a down time in serving reads for keys it was
responsible for until a standby replica takes over
in case of semantic partitioning some replicas might become hot and there is no
easy way to scale the read load
If standby replicas would have endpoints that are exposed in StreamsMetadata it
would enable serving reads from several replicas, which would mitigate the
above drawbacks.
Due to the lag between replicas reading from multiple replicas simultaneously
would have weaker (eventual) consistency comparing to reads from single
replica. This however should be acceptable tradeoff in many cases.
{quote}
> Allow state stores to serve stale reads during rebalance
>
>
> Key: KAFKA-6144
> URL: https://issues.apache.org/jira/browse/KAFKA-6144
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Antony Stubbs
>Priority: Major
>
> Currently when expanding the KS cluster, the new node's partitions will be
> unavailable during the rebalance, which for large states can take a very long
> time, or for small state stores even more than a few ms can be a deal breaker
> for micro service use cases.
> One workaround is to allow stale data to be read from the state stores when
> use case allows.
> Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
> potentially a two phase rebalance
> This is the description from KAFKA-6031 (keeping this JIRA as the title is
> more descriptive):
> {quote}
> Currently reads for a key are served by single replica, which has 2 drawbacks:
> - if replica is down there is a down time in serving reads for keys it was
> responsible for until a standby replica takes over
> - in case of semantic partitioning some replicas might become hot and there
> is no easy way to scale the read load
> If standby replicas would have endpoints that are exposed in StreamsMetadata
> it would enable serving reads from several replicas, which would mitigate the
> above drawbacks.
> Due to the lag between replicas reading from multiple replicas simultaneously
> would have weaker (eventual) consistency comparing to reads from single
> replica. This however should be acceptable tradeoff in many cases.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6144) Allow state stores to serve stale reads during rebalance
[
https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-6144:
---
Description:
Currently when expanding the KS cluster, the new node's partitions will be
unavailable during the rebalance, which for large states can take a very long
time, or for small state stores even more than a few ms can be a deal breaker
for micro service use cases.
One workaround is to allow stale data to be read from the state stores when use
case allows.
Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
potentially a two phase rebalance
This is the description from KAFKA-6031 (keeping this JIRA as the title is more
descriptive):
{quote}
Currently reads for a key are served by single replica, which has 2 drawbacks:
- if replica is down there is a down time in serving reads for keys it was
responsible for until a standby replica takes over
in case of semantic partitioning some replicas might become hot and there is no
easy way to scale the read load
If standby replicas would have endpoints that are exposed in StreamsMetadata it
would enable serving reads from several replicas, which would mitigate the
above drawbacks.
Due to the lag between replicas reading from multiple replicas simultaneously
would have weaker (eventual) consistency comparing to reads from single
replica. This however should be acceptable tradeoff in many cases.
{quote}
was:
Currently when expanding the KS cluster, the new node's partitions will be
unavailable during the rebalance, which for large states can take a very long
time, or for small state stores even more than a few ms can be a deal breaker
for micro service use cases.
One workaround is to allow stale data to be read from the state stores when use
case allows.
Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
potentially a two phase rebalance
> Allow state stores to serve stale reads during rebalance
>
>
> Key: KAFKA-6144
> URL: https://issues.apache.org/jira/browse/KAFKA-6144
> Project: Kafka
> Issue Type: New Feature
> Components: streams
>Reporter: Antony Stubbs
>Priority: Major
>
> Currently when expanding the KS cluster, the new node's partitions will be
> unavailable during the rebalance, which for large states can take a very long
> time, or for small state stores even more than a few ms can be a deal breaker
> for micro service use cases.
> One workaround is to allow stale data to be read from the state stores when
> use case allows.
> Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks -
> potentially a two phase rebalance
> This is the description from KAFKA-6031 (keeping this JIRA as the title is
> more descriptive):
> {quote}
> Currently reads for a key are served by single replica, which has 2 drawbacks:
> - if replica is down there is a down time in serving reads for keys it was
> responsible for until a standby replica takes over
> in case of semantic partitioning some replicas might become hot and there is
> no easy way to scale the read load
> If standby replicas would have endpoints that are exposed in StreamsMetadata
> it would enable serving reads from several replicas, which would mitigate the
> above drawbacks.
> Due to the lag between replicas reading from multiple replicas simultaneously
> would have weaker (eventual) consistency comparing to reads from single
> replica. This however should be acceptable tradeoff in many cases.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (KAFKA-6144) Allow state stores to serve stale reads during rebalance
[ https://issues.apache.org/jira/browse/KAFKA-6144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antony Stubbs updated KAFKA-6144: - Description: Currently when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal breaker for micro service use cases. One workaround is to allow stale data to be read from the state stores when use case allows. Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks - potentially a two phase rebalance was: Currently when expanding the KS cluster, the new node's partitions will be unavailable during the rebalance, which for large states can take a very long time, or for small state stores even more than a few ms can be a deal breaker for micro service use cases. One workaround is to allow stale data to be read from the state stores when use case allows. > Allow state stores to serve stale reads during rebalance > > > Key: KAFKA-6144 > URL: https://issues.apache.org/jira/browse/KAFKA-6144 > Project: Kafka > Issue Type: New Feature > Components: streams >Reporter: Antony Stubbs > > Currently when expanding the KS cluster, the new node's partitions will be > unavailable during the rebalance, which for large states can take a very long > time, or for small state stores even more than a few ms can be a deal breaker > for micro service use cases. > One workaround is to allow stale data to be read from the state stores when > use case allows. > Relates to KAFKA-6145 - Warm up new KS instances before migrating tasks - > potentially a two phase rebalance -- This message was sent by Atlassian JIRA (v6.4.14#64029)
