JingsongLi commented on code in PR #1164:
URL: https://github.com/apache/paimon-rust/pull/1164#discussion_r4313451271
##########
crates/paimon/src/table/kv_file_reader.rs:
##########
@@ -676,6 +676,20 @@ impl KeyValueFileReader {
.map(|budget| Arc::new(budget.without_prefetch()))
};
let mut file_streams: Vec<ArrowRecordBatchStream> =
Vec::new();
+ // One sorted run of files without retract rows, under an
+ // engine whose files never repeat a key, has nothing to
+ // merge. Planning trusts the same invariant when it marks
a
+ // single-file split raw convertible.
+ let skip_merge = input_stream_count == 1
+ && matches!(
+ config.merge_engine,
+ MergeEngine::Deduplicate | MergeEngine::FirstRow
+ )
+ && merge_group.iter().all(|run| {
+ run.files
+ .iter()
+ .all(|file| file.file.delete_row_count ==
Some(0))
Review Comment:
[P2] Preserve retract filtering for historical native files
`delete_row_count == Some(0)` does not prove these files contain only add
rows. Before 13e370ae (#318, May 20), the Rust KV writer accepted `_VALUE_KIND`
but persisted `Some(0)` unconditionally; normal cross-partition writes also
generated DELETE batches. Those files have no writer-version guard here. I
reproduced a full table read with two disjoint files in one nonraw split: file
A has INSERT keys 1 and 2; file B has a DELETE for key 3 with that historical
metadata. Base 46e68710 returns `[1, 2]`, while this head returns `[1, 2, 3]`,
exposing the deleted row. The test asserts that the planner marks this two-file
split nonraw, so this is an expansion beyond the pre-existing single-file raw
path. Please preserve a cheap RowKind filter on the direct path, or restrict
the optimization to files whose add-only invariant can actually be verified,
and cover this legacy case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
