skywalker0618 commented on PR #20735: URL: https://github.com/apache/hudi/pull/20735#issuecomment-6992654095
> A high-level design question, is it possible to track lag and backlog in the Source V2 reader or enumerator without changing the core incremental read path, particularly IncrementalInputSplits and HoodieSourceSplit? Thanks for the suggestion! It was originally done that way because the information was already available during split generation. On second thought, it's indeed better not to touch the core components. In the revised commit, the enumerator reads the commit metadata of each new commit a second time to compute the lag info, so IncrementalInputSplits, HoodieSourceSplit and the serializers are unchanged. On restart, the enumerator refetches the lag info for the restored splits asynchronously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
