cshuo commented on issue #19516: URL: https://github.com/apache/hudi/issues/19516#issuecomment-5186842564
Thanks for the detailed analysis. I checked the V1 batch source, and this proposal is consistent with its scheduling behavior. V1 bounded reads use a shared split pool: MOR/incremental/CDC use DefaultInputSplitAssigner, while COW uses the locality-aware LocatableInputSplitAssigner. Splits are pulled as readers finish rather than permanently pinned to subtasks. There is also historical precedent: before [PR #18082](https://github.com/apache/hudi/pull/18082), DefaultHoodieSplitProvider used a single shared queue. That PR introduced per-subtask assignment for streaming distribution parity, but the provider was also applied to the bounded branch. So +1 to using a shared provider by default for Source V2 bounded reads while keeping the existing affinity-based provider for streaming. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
