mjsax commented on PR #22614: URL: https://github.com/apache/kafka/pull/22614#issuecomment-4993759912
Well, consumers do already exposes `records-lag` on a per-partition basis. Won't this do the trick? But sure, we could also do a KIP to add a new offset base metric similar to `restore-total` / `restore-rate`. But it seems this would be a follow up PR? While your example with aborted TX and TX markers is "valid", the same problem exist for log-compaction with the main difference that many offsets don't exist any longer. The log compaction case is the more common one (I would only expect a low percentage of aborted TX), and if we would count non-existing offset as part of restore rate, it could be very off, and it might also vary a lot (ie, older segments with more compaction would have more missing offsets, while newer segment might not be compacted yet, appearing to restore slower, even if they are not, if we look at offsets). > This enables users to calculate things like lag, consumption rate, etc. from just the broker metrics, and these metrics are entirely offset-based. Maybe, but for compacted topic this approach seems to be broken? In the end, `restore-total` / `restore-rate` should be a metric about actually performed restore _work_, ie, data transferred over the network etc. Counting non-existing offsets from log-compacted topics would make these metrics "useless" to get a sense of actual restore "speed"? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
