skywalker0618 commented on PR #20735:
URL: https://github.com/apache/hudi/pull/20735#issuecomment-6992654095

   > A high-level design question, is it possible to track lag and backlog in 
the Source V2 reader or enumerator without changing the core incremental read 
path, particularly IncrementalInputSplits and HoodieSourceSplit?
   
   Thanks for the suggestion! It was originally done that way because the 
information was already available during split generation. On second thought, 
it's indeed better not to touch the core components. In the revised commit, the 
enumerator reads the commit metadata of each new commit a second time to 
compute the lag info, so IncrementalInputSplits, HoodieSourceSplit and the 
serializers are unchanged. On restart, the enumerator refetches the lag info 
for the restored splits asynchronously.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to