yihua opened a new issue, #830: URL: https://github.com/apache/hudi-rs/issues/830
Engines that plan splits by byte range (Spark with Hudi 1.3, where copy-on-write base files are reported as splittable, driving Velox through the C++ bridge) hand hudi-rs a file slice plus a split's start and length. The bridge's `read_file_slice_from_paths` reads the whole slice, so a base file cut into N splits comes back N times. The caller currently works around it by planning one split per file, which leaves a large base file on a single task. Proposed: a bridge entry point that takes the split's start and length and, for a slice without log files, reads only the row groups whose midpoint falls in `[start, start + length)`, as Spark's and Velox's parquet readers do, so the splits of a file together return exactly the whole file. A slice with log files merges log records across the whole base file, so a split that does not cover the whole file must fail loudly rather than return a subset. This builds on the row-group range reader (`BaseFileRange`, `read_file_slice_range_stream`), which also lets the DataFusion scan spread one merge-on-read slice over partitions when the log blocks carry record positions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
