voonhous commented on issue #19911: URL: https://github.com/apache/hudi/issues/19911#issuecomment-5638139760
Context on where the throw came from, since the description above calls overlapping references legitimate and that is not how it was settled in #18098. I raised overlapping ranges on the original reader in https://github.com/apache/hudi/pull/18098#discussion_r3021340022 and asked for a test in https://github.com/apache/hudi/pull/18098#discussion_r3021419273. @the-other-tim-brown's position was that a blob is a distinct entity, so overlapping ranges imply corruption and the reader should throw. Commit a00dcfff0237 added the `Overlapping blob ranges detected` check in response. The throw was deliberate, not a leftover. Two things argue for dropping it anyway: 1. The check fires on `row.offset < currentEndOffset`, which also catches two rows carrying the identical descriptor, not only partial overlaps. That case is ordinary: a join with fan-out (`SELECT ..., read_blob(b.payload) FROM blobs b JOIN events e ON b.id = e.blob_id`) lands the N matching rows for one blob in the same task, since the batched read wraps the join output with no shuffle of its own, and insert/bulk_insert do not dedupe by default. Neither is corruption under the #18098 model. 2. Whether the check fires depends on which rows share a task, so it cannot act as a corruption guard either way. The Lance test from #18575 has carried an overlap since April and only failed when #19906 changed the partition count. Tim, does the distinct-entity invariant still need enforcing? If so, could it live at write time, validating descriptors as they are written, rather than in the batched reader? #19913 removes the read-side check on the grounds above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
