voonhous opened a new pull request, #19913: URL: https://github.com/apache/hudi/pull/19913
### Describe the issue this Pull Request addresses Fixes #19911. `BatchedBlobReader` reads out-of-line blobs one Spark task at a time: it groups the task's rows by file, sorts them by offset, merges nearby ranges into one read, and slices each row's bytes out of the buffer. When two rows in the same task reference overlapping byte ranges of one file, `mergeRanges` throws `Overlapping blob ranges detected` and the query fails. Whether two rows share a task is a partitioning accident. The same table reads fine when the rows land in different tasks and fails when they land in the same one. #19906 hit this on Azure by changing the test harness from four shuffle partitions to two: `TestLanceDataSource#testBlobOutOfLine` writes two rows pointing at `[0, 1024)` and `[0, 512)` of one file, passed at four partitions, and failed all six cases at two ([Azure build 17015, log 98](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_apis/build/builds/17015/logs/98)). Overlapping references are legitimate: two records may point at nested or shared bytes of one blob file, and the merged read already covers them, because `currentEndOffset` is the maximum end seen and each row is sliced by its own offset and length. ### Summary and Changelog - `BatchedBlobReader.mergeRanges` merges an overlapping row into the current range instead of throwing. The merge condition becomes `gap <= maxGap`; a negative gap is an overlap. The read and slice code is unchanged. - `TestBatchedBlobReaderMerge#testOverlappingRangesMergeIntoOne` replaces the test that expected the exception: `[0,100)` and `[50,150)` merge into one range `[0,150)` carrying both rows. - `TestBatchedBlobReader#testOverlappingRangesAreServedFromOneRead` replaces the end-to-end test that expected the exception: both rows come back with their own 100 bytes, verified against the file content. ### Impact Reads with overlapping blob references no longer depend on task layout. No format or config change. ### Risk Level low The only behaviour change is on input that used to fail. Rows that already merged, adjacent or within the gap threshold, take the same path as before. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
