JingsongLi commented on code in PR #10643:
URL: https://github.com/apache/paimon/pull/10643#discussion_r4313576477
##########
paimon-core/src/main/java/org/apache/paimon/operation/DataEvolutionSplitRead.java:
##########
@@ -778,6 +808,31 @@ private static int findBlobFieldIndex(RowType rowType) {
return -1;
}
+ /**
+ * Emits the rows of blob files for a read type with no blob field,
without opening them: the
+ * blob format cannot be read without one.
+ */
+ private FileRecordReader<InternalRow> blobRowsWithoutBlobField(
+ Range range,
+ List<DataFileMeta> files,
+ List<Range> rowRanges,
+ RowType readRowType,
+ @Nullable DeletionVectorWithRange deletionVector) {
+ long sequenceNumber =
+
files.stream().mapToLong(DataFileMeta::maxSequenceNumber).max().getAsLong();
Review Comment:
[P2] Preserve the winning sequence number for each placeholder row
This maximum belongs to the whole file group, so it incorrectly updates rows
that are covered only by an older blob file. I reproduced this with actual
commits: snapshot 1 writes normal rows 0 and 1; snapshot 2 writes picture
`[a,b]` for row IDs `[0,1]`; snapshot 3 writes picture `c` only for row ID 1.
Reading `incremental-between=1,3` with projection `[_ROW_ID,_SEQUENCE_NUMBER]`
returns `(0,3),(1,3)`, while adding `picture` returns `(0,2,a),(1,3,c)`. Thus
dropping a BLOB column changes the version of the unchanged row, which can
misorder or misclassify downstream incremental updates. Please generate
sequence-aware placeholder ranges using each row's winning file version,
consistent with the blob fallback reader, and add a multi-snapshot regression
comparing tracking fields with and without the BLOB projection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
