[
https://issues.apache.org/jira/browse/KAFKA-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicholas Telford updated KAFKA-20711:
-------------------------------------
Summary: Streams task restore-remaining-records metric never reaches 0
(was: Streams task restore-remaining-records metric invalid under EOS)
> Streams task restore-remaining-records metric never reaches 0
> -------------------------------------------------------------
>
> Key: KAFKA-20711
> URL: https://issues.apache.org/jira/browse/KAFKA-20711
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 4.3.0
> Reporter: Nicholas Telford
> Assignee: Nicholas Telford
> Priority: Minor
>
> The Kafka Streams Task-level metric {{restore-remaining-records}} is intended
> to track the total number of records that still need to be restored.
> When the application runs under EOS, this metric is inaccurate, never
> actually dropping to 0 for fully restored tasks, and always showing values
> substantially higher than reality.
> The root-cause is that the metric is initialized with a total number of
> records to restored derived as {{logEndOffset - committedOffset}}, using a
> READ_UNCOMMITTED consumer.
> This offset range naturally includes uncommitted records and transaction
> markers, in addition to the actual records to restore.
> When decrementing the metric during restore, we decrement by the actual
> number of (committed) records that were restored. Since this excludes
> uncommitted records and transaction markers, we will never decrement the
> metric by the total it was initialized with.
> I have a fix that I will raise a PR for.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
