danny0405 commented on issue #19516: URL: https://github.com/apache/hudi/issues/19516#issuecomment-5186706945
Conceptually +1. For the static enumerator, a single pull-based pool is also consistent with Flink's `StaticFileSplitEnumerator` + `SimpleSplitAssigner` model. I would make this the default for the non-streaming branch; I could not find a Source V2 partitioning contract exposed to downstream operators that would make the current bucket/file-id affinity load-bearing for a bounded scan. A few points I think the implementation/PR should make explicit: > exactly one split per file group, no cross-commit continuation, no ordering relationship between splits The static branch is broader than the measured COW snapshot case: `createBatchHoodieSplits()` also covers MOR snapshot, read-optimized, bounded incremental, and incremental CDC. Please verify this independence claim for each of those modes (or narrow the selection if one is not safe). A parameterized source-routing test would make the intended scope clear. > No enumerator change is required Agreed, but restore is an easy place to accidentally regress because `HoodieSource.createEnumerator(..., enumeratorState)` currently constructs the provider before the streaming/static branch. Please test both fresh creation and restore, and assert that restored pending splits stay in the shared provider and can be claimed by a different subtask. The failure test should also cover `addSplitsBack` after some other readers have already received `NoMoreSplits`, not just a direct provider enqueue/dequeue. One measurement clarification: the table reports a pinned wall clock of 3.80 h (228 min), while the slowest reader is 203 min; the shared numbers (2.77 h / 166 min) line up. Could you state what interval each metric uses (job submission, source deployment, first record, etc.) so the before/after comparison is reproducible? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
