nicktelford commented on PR #22614: URL: https://github.com/apache/kafka/pull/22614#issuecomment-4797719144
> Thanks for the fix. > > On thing I am wondering is about `restore-total` and `restore-rate` metric. My understanding is, that both would measure how many actual records we did restore. So if we add TX-markers and offset-gaps (from compaction), it seems these metrics would not become incorrect? > > There a point about keeping `restore-remaining` and `restore-total` "in sync", but in the end `restore-remaining` can only be an _estimation_ of how many actually records are remaining. Ideally `restore-remaining` would tell us how many records are remaining, but this is something we just don't know, and can only estimate via offsets -- but it seem to go a step too far to "mess" with the other two metrics which are correct now. @mjsax Yeah, I opted to change the semantics of `restore-total` and `restore-rate`, because I think it will be more problematic for users if they measure something different to `restore-remaining`. Ultimately, I doubt anyone is dependent on `restore-total`/`restore-rate` not including transaction markers and offset gaps. A concrete example here is that if we ensure these metrics all measure the same thing, you can derive an estimated time remaining for restore by calculating `restore-remaining / restore-rate`, which would be much less accurate if the two metrics tracked different things. It would of course be best for `restore-remaining` to report actual records, but since that's not possible, I think aligning the other metrics on offsets is the next-best option. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
