Sophie Blee-Goldman created KAFKA-9177:
------------------------------------------
Summary: Pause completed partitions on restore consumer
Key: KAFKA-9177
URL: https://issues.apache.org/jira/browse/KAFKA-9177
Project: Kafka
Issue Type: Improvement
Components: streams
Reporter: Sophie Blee-Goldman
The StoreChangelogReader is responsible for tracking and restoring active
tasks, but once a store has finished restoring it will continue polling for
records on that partition.
Ordinarily this doesn't make a difference as a store is not completely restored
until its entire changelog has been read, so there are no more records for poll
to return anyway. But if the restoring state is actually an optimized source
KTable, the changelog is just the source topic and poll will keep returning
records for that partition until all stores have been restored.
Note that this isn't a correctness issue since it's just the restore consumer,
but it is wasteful to be polling for records and throwing them away. We should
pause completed partitions in StoreChangelogReader so we don't slow down the
restore consumer in reading from the unfinished changelog topics, and avoid
wasted network.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
