[
https://issues.apache.org/jira/browse/KAFKA-6645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Giridhar Addepalli updated KAFKA-6645:
--------------------------------------
Description:
Since Kafka Streams applications have lot of state in the stores in general, it
would be good to remember the assignment of partitions to machines. So that
when whole application is restarted for whatever reason, there is a way to use
past assignment of partitions to machines and there won't be need to build up
state by reading off of changelog kafka topic and would result in faster
start-up.
Samza has support for Host Affinity
([https://samza.apache.org/learn/documentation/0.14/yarn/yarn-host-affinity.html])
KIP-54
([https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy)]
, handles cases where some members of consumer group goes down / comes up, and
KIP-54 ensures there is minimal diff between assignments before and after
rebalance.
But to handle whole restart use case, we need to remember past assignment
somewhere, and use it after restart.
Please let us know if this is already solved problem / some cleaner way of
achieving this objective
was:
Since Kafka Streams applications have lot of state in the stores in the
general, it would be good to remember the assignment of partitions to machines.
So that when whole application is restarted for whatever reason, there is a way
to use past assignment of partitions to machines and there won't be need to
build up state by reading off of changelog kafka topic and would result in
faster start-up.
Samza has support for Host Affinity
(https://samza.apache.org/learn/documentation/0.14/yarn/yarn-host-affinity.html)
KIP-54
([https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy)]
, handles cases where some members of consumer group goes down / comes up, and
KIP-54 ensures there is minimal diff between assignments before and after
rebalance.
But to handle whole restart use case, we need to remember past assignment
somewhere, and use it after restart.
Please let us know if this is already solved problem / some cleaner way of
achieving this objective
> Sticky Partition Assignment to facilitate faster restarts of kafka streams
> applications
> ---------------------------------------------------------------------------------------
>
> Key: KAFKA-6645
> URL: https://issues.apache.org/jira/browse/KAFKA-6645
> Project: Kafka
> Issue Type: New Feature
> Components: streams
> Reporter: Giridhar Addepalli
> Priority: Major
>
> Since Kafka Streams applications have lot of state in the stores in general,
> it would be good to remember the assignment of partitions to machines. So
> that when whole application is restarted for whatever reason, there is a way
> to use past assignment of partitions to machines and there won't be need to
> build up state by reading off of changelog kafka topic and would result in
> faster start-up.
> Samza has support for Host Affinity
> ([https://samza.apache.org/learn/documentation/0.14/yarn/yarn-host-affinity.html])
> KIP-54
> ([https://cwiki.apache.org/confluence/display/KAFKA/KIP-54+-+Sticky+Partition+Assignment+Strategy)]
> , handles cases where some members of consumer group goes down / comes up,
> and KIP-54 ensures there is minimal diff between assignments before and after
> rebalance.
> But to handle whole restart use case, we need to remember past assignment
> somewhere, and use it after restart.
> Please let us know if this is already solved problem / some cleaner way of
> achieving this objective
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
