JingsongLi commented on code in PR #1164:
URL: https://github.com/apache/paimon-rust/pull/1164#discussion_r4345144510
##########
crates/paimon/src/table/kv_file_reader.rs:
##########
@@ -711,6 +761,18 @@ impl KeyValueFileReader {
.map(|budget| Arc::new(budget.without_prefetch()))
};
let mut file_streams: Vec<ArrowRecordBatchStream> =
Vec::new();
+ // One sorted run never repeats a key, so under these
engines
+ // every key is a lone version and needs no merge. Retract
+ // rows are still dropped per batch below: older Rust
writers
+ // recorded `delete_row_count = 0` for files holding them.
+ let skip_merge = skip_single_runs
Review Comment:
[P2] Preserve merging for historical files after an engine change
The current read engine does not establish that every file in this run has
unique keys. Before #839 (baac87ca), the Rust aggregation writer deliberately
retained multiple same-key versions in one file. The Rust filesystem catalog
currently accepts ALTER merge-engine=deduplicate on such a table. I reproduced
that real persisted ALTER/reload with a legacy file containing
(key=1,seq=0,value=10) and (1,1,11), plus a disjoint key=2 file. The normal
planner produces a two-file non-raw split, but this shortcut returns both key=1
rows. Exact base cea0caaf returns only (1,11), as Deduplicate requires; the
original aggregation control correctly sums to (1,21). Please retain merging
when the files' historical schema/producer does not establish key uniqueness,
or explicitly reject the transition before using this shortcut, and cover this
accepted Rust ALTER lifecycle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
