[
https://issues.apache.org/jira/browse/KAFKA-8023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17499831#comment-17499831
]
Guozhang Wang commented on KAFKA-8023:
--------------------------------------
One idea I'm having at the moment is that when we have separate restoration
thread (pool), we could remove the global thread with the restoration thread
pool. The challenge would though be if some standby / active tasks are in
place, how to schedule in order for smart resource allocation.
> Improve global state store restoration by using multiple update threads
> -----------------------------------------------------------------------
>
> Key: KAFKA-8023
> URL: https://issues.apache.org/jira/browse/KAFKA-8023
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Reporter: Patrik Kleindl
> Priority: Major
> Labels: new-streams-runtime-should-fix, performance
>
> Currently global state stores are restored sequentially and the partitions of
> each global state store are restored sequentially too.
> Loop over stores:
> https://github.com/apache/kafka/blob/b03e8c234a8aeecd10c2c96b683cfb39b24b548a/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java#L155
> Loop over partitions:
> https://github.com/apache/kafka/blob/b03e8c234a8aeecd10c2c96b683cfb39b24b548a/streams/src/main/java/org/apache/kafka/streams/processor/internals/GlobalStateManagerImpl.java#L256
> It would be a great improvement if one or both of those loops could be
> processed in parallel.
> Possible related task is https://issues.apache.org/jira/browse/KAFKA-6721
> Mail discussion:
> https://lists.apache.org/thread.html/6fc4772eb8635c04b0ee6682003a99a5ef37ebccffea6c89752e96b1@%3Cusers.kafka.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
