nicktelford opened a new pull request, #22614: URL: https://github.com/apache/kafka/pull/22614
The `restore-remaining-records-total` task metric overshoots and never reaches zero under EOS (and, more mildly, on compacted changelogs). The metric is initialized in `prepareChangelogs` from the offset range `restoreEndOffset - startOffset`, which counts every offset slot — including those occupied by transaction markers and aborted/compacted-away records. It was then decremented in `restoreChangelog` by the number of records actually restored (`bufferedLimitIndex`). Because the `read_committed` restore consumer never returns markers or aborted records (though its position advances past them), the sum of the decrements is always smaller than the initial value by exactly the number of those offset slots, so the metric stays stuck at a positive value even after restoration completes. The fix makes the decrements consistent with the initialization by expressing restoration progress in changelog offset slots rather than record counts. Each batch advances the metric by the change in the store's offset (`storeMetadata.offset()`, which `restore()` sets to the last restored record's offset), and on completion any trailing slots between the last restored record and the end offset are accounted for, so the metric lands at exactly zero. A new `restoreStartOffset` field records the consumer position at which restoration began, so the first batch is measured from the true start (covering the from-the-beginning case and any leading marker offsets). Since `restoreSensor` (`restore-total`/`restore-rate`) is fed the same value, its rate now matches the rate at which the remaining-records metric declines, keeping the two consistent. The standby `update-total`/`update-rate` metrics are routed through the same offset-slot logic so they remain comparable with the active restore metrics. The per-batch listener callbacks (`onBatchRestored`/`onBatchLoaded`) and `totalRestored` continue to report true record counts, so the actual number of records restored is unchanged and still available. A unit test reproduces the gap (a changelog whose consumer position advances past offsets that are never returned as records) and asserts the metric decrements to exactly zero on completion. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
