[ https://issues.apache.org/jira/browse/KAFKA-6106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16387289#comment-16387289 ]
ASF GitHub Bot commented on KAFKA-6106: --------------------------------------- kamalcph opened a new pull request #4651: KAFKA-6106: Postpone normal processing of tasks until restoration of all tasks completed URL: https://github.com/apache/kafka/pull/4651 Once all the state stores are restored, then the processing of tasks takes place. This approach will reduce the time taken to restore the state stores as single thread is used to restore the state and process the task. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Postpone normal processing of tasks within a thread until restoration of all > tasks have completed > ------------------------------------------------------------------------------------------------- > > Key: KAFKA-6106 > URL: https://issues.apache.org/jira/browse/KAFKA-6106 > Project: Kafka > Issue Type: Improvement > Components: streams > Affects Versions: 0.11.0.1, 1.0.0 > Reporter: Guozhang Wang > Assignee: Kamal Chandraprakash > Priority: Major > Labels: newbie++ > > Let's say a stream thread hosts multiple tasks, A and B. At the very > beginning when A and B are assigned to the thread, the thread state is > {{TASKS_ASSIGNED}}, and the thread start restoring these two tasks during > this state using the restore consumer while using normal consumer for > heartbeating. > If task A's restoration has completed earlier than task B, then the thread > will start processing A immediately even when it is still in the > {{TASKS_ASSIGNED}} phase. But processing task A will slow down restoration of > task B since it is single-thread. So the thread's transition to {{RUNNING}} > when all of its assigned tasks have completed restoring and now can be > processed will be delayed. > Note that the streams instance's state will only transit to {{RUNNING}} when > all of its threads have transit to {{RUNNING}}, so the instance's transition > will also be delayed by this scenario. > We'd better to not start processing ready tasks immediately, but instead > focus on restoration during the {{TASKS_ASSIGNED}} state to shorten the > overall time of the instance's state transition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)