voonhous opened a new issue, #19911: URL: https://github.com/apache/hudi/issues/19911
`BatchedBlobReader.mergeRanges` throws "Overlapping blob ranges detected" when two rows in the same task reference overlapping byte ranges of one external file. Which rows share a task depends on partitioning, so the same table reads fine or fails depending on layout. **Reproduction**: `TestLanceDataSource#testBlobOutOfLine` writes rows pointing at `[0, 1024)` and `[0, 512)` of one file. It passes with the test harness's default four shuffle partitions and fails all six cases at two (#19906, [Azure build 17015 log 98](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_apis/build/builds/17015/logs/98); reproduced locally on JDK 11 / Spark 3.5). ``` java.lang.IllegalArgumentException: Overlapping blob ranges detected: previous range [0, 1024) and current row [0, 512) in file .../blob_file_2.bin at org.apache.spark.sql.hudi.blob.BatchedBlobReader.mergeRanges(BatchedBlobReader.scala:339) ``` Overlapping references are legitimate (nested or shared bytes of one file), and the merged read already covers them: the range end is the maximum end seen and each row is sliced by its own offset and length. The throw is the only thing preventing it. Introduced in #18098. Fix: merge overlapping rows instead of throwing, and flip the two tests that assert the exception (`TestBatchedBlobReader#testOverlappingRangesThrowsException`, `TestBatchedBlobReaderMerge#testOverlappingRangesThrow`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
