hudi-agent commented on code in PR #19913:
URL: https://github.com/apache/hudi/pull/19913#discussion_r3993281096
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/blob/BatchedBlobReader.scala:
##########
@@ -331,9 +339,16 @@ class BatchedBlobReader(
currentStartOffset = row.offset
currentEndOffset = row.offset + row.length
currentRows = ArrayBuffer(row)
+ } else if (row.offset == currentRows.last.offset && row.length ==
currentRows.last.length) {
+ // Same descriptor as the previous row: one blob referenced by more
than one row (join
+ // fan-out, duplicate records). It is served from the current read and
the range does
+ // not grow.
+ currentRows += row
} else {
val gap = row.offset - currentEndOffset
- // Check for overlap
+ // A blob is a distinct entity, so two blobs never share bytes. Rows
are sorted by
+ // (offset, length) and identical descriptors were handled above, so a
start inside the
+ // current range means two different blobs overlap, which indicates
corruption.
if (row.offset < currentEndOffset) {
Review Comment:
🤖 Restoring the throw brings back the failure from #19911, doesn't it?
`verifyBlobOutOfLine` in TestLanceDataSource still writes ids 3 and 4 as
`(filePath2, 0, 1024)` and `(filePath2, 0, 512)` — exactly the nested shape
`testNestedRangeThrows` now pins as an error — and that test data is unchanged
here. As soon as those two rows land in the same task (deterministically under
#19906's two shuffle partitions) the query fails again. Could you either adjust
that fixture or reconsider the throw?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag
quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
