wombatu-kun commented on code in PR #19236:
URL: https://github.com/apache/hudi/pull/19236#discussion_r3558244240
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkLanceWriter.java:
##########
@@ -446,12 +467,43 @@ protected void updateRecordMetadata(InternalRow row,
@AllArgsConstructor(staticName = "of")
private static class SparkArrowWriter implements ArrowWriter<InternalRow> {
private final LanceArrowWriter lanceArrowWriter;
+ // Parallel arrays of top-level BLOB column ordinals and names for the
per-row guard.
+ private final int[] blobFieldOrdinals;
+ private final String[] blobFieldNames;
@Override
public void write(InternalRow row) {
+ validateBlobRow(row);
lanceArrowWriter.write(row);
}
+ /**
+ * Reject descriptor-shaped BLOB rows before they are persisted. A row is
descriptor-shaped when
+ * a top-level BLOB struct is non-null, its type is INLINE, its data is
null, and its reference
+ * is non-null; this only arises when a DESCRIPTOR-mode read leaks onto a
write path, and writing
+ * it would silently drop the inline bytes. The legitimate empty-inline
shape {INLINE, null, null}
+ * stays writable.
+ */
+ private void validateBlobRow(InternalRow row) {
+ for (int i = 0; i < blobFieldOrdinals.length; i++) {
+ int ordinal = blobFieldOrdinals[i];
+ if (row.isNullAt(ordinal)) {
+ continue;
+ }
+ InternalRow blob = row.getStruct(ordinal, 3);
+ if (blob.isNullAt(0) ||
!INLINE_TYPE_TOKEN.equals(blob.getUTF8String(0))) {
+ continue;
+ }
+ if (blob.isNullAt(1) && !blob.isNullAt(2)) {
+ throw new HoodieException(
+ "BLOB column '" + blobFieldNames[i] + "' has an INLINE row with
null data but a "
+ + "populated reference: a DESCRIPTOR-mode read leaked into
the write path, and "
+ + "persisting it would silently drop the blob bytes.
Internal rewrites "
+ + "(compaction, clustering, merge) must read with
hoodie.read.blob.inline.mode=CONTENT.");
Review Comment:
The Impact section still reads "Fixes silent data loss on internal rewrites
of Lance INLINE blobs. No public API or config default changes." The behavior
change is not there: with the guard, `INSERT INTO t2 SELECT * FROM t1` over a
Lance blob table at the DESCRIPTOR default now throws instead of writing a
table whose INLINE rows carry a dangling reference. `read_blob()` already
refuses those rows (`BatchedBlobReader` throws on INLINE with null data), so
the guard is right, but it turns a silent-and-later-unreadable write into a
hard failure at write time, and that belongs under Impact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
