JingsongLi commented on PR #10643: URL: https://github.com/apache/paimon/pull/10643#issuecomment-6339523134
I'm concerned that the scope is too broad relative to the current usage of this feature. BLOB updates and streaming/incremental reads of those updates are both uncommon. The bug is valid, but I would prefer to narrow this fix before merging it. In particular, routing every singleton BLOB group through the merge reader also changes BLOB compaction: placeholders in singleton row-range groups are now persisted as NULL. A compaction task can contain several files while still producing singleton groups during reading. This expands the compatibility and verification burden beyond the incremental-read problem. There is also a concrete regression at `bf9dd6693c`: with a schema `(f0 INT NOT NULL, f2 BLOB)`, write a normal row, update only `f2`, then read the full-schema stream delta. The new path throws `Field f0 INT NOT NULL is not null but can't find any file contains it` from `DataEvolutionReadPlanner`. The same real-commit reproduction passes on the PR base `6528cb5d3a`, where the omitted normal column is returned as NULL. Please consider restricting the new behavior to blob-only incremental groups and preserving the existing batch/compaction paths. The existing `DataSplit.isStreaming()` flag appears to provide a suitable boundary: it is set for streaming deltas and batch `incremental-between` splits, while compaction-created splits leave it false. Any compaction placeholder normalization could then be evaluated separately with its own justification and tests. Without a concrete user need for this combination, I would also be comfortable deferring the change and documenting the current limitation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
