Jackie-Jiang commented on PR #14142: URL: https://github.com/apache/pinot/pull/14142#issuecomment-2430348047
Before the PR, the logic is: - For every message batch, we make a separate metadata fetch request to get the latest offset This PR: - Move the latest offset fetching logic into `IngestionDelayTracker` and periodically refresh `IngestionInfo._latestOffset` - Still use `_latestOffset - _currentOffset` to compute the ingestion lag - The problem of this approach is that `_latestOffset` and `_currentOffset` are not updated together To solve the above problem: - Move the latest offset fetching logic into `IngestionDelayTracker` - Fetch latest offset within `updateIngestionMetrics()`, but with the given minimum interval (not fetch for every invocation) - Directly store `_offsetLag` instead of `_latestOffset` in the `IngestionInfo` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
