JingsongLi commented on code in PR #1164:
URL: https://github.com/apache/paimon-rust/pull/1164#discussion_r4365612252
##########
crates/paimon/src/table/kv_file_reader.rs:
##########
@@ -711,6 +761,42 @@ impl KeyValueFileReader {
.map(|budget| Arc::new(budget.without_prefetch()))
};
let mut file_streams: Vec<ArrowRecordBatchStream> =
Vec::new();
+ // One sorted run never repeats a key, so under these
engines
+ // every key is a lone version and needs no merge. Retract
+ // rows are still dropped per batch below: older Rust
writers
+ // recorded `delete_row_count = 0` for files holding them.
+ let mut skip_merge = skip_single_runs
+ && input_stream_count == 1
+ && matches!(
+ config.merge_engine,
+ MergeEngine::Deduplicate | MergeEngine::FirstRow
+ );
+ // A file written under another engine may hold several
+ // versions of a key (the aggregation writer kept them
before
+ // #839), and ALTER can switch the engine afterwards.
+ if skip_merge {
+ 'files: for run in &merge_group {
+ for file in &run.files {
+ let schema_id = file.file.schema_id;
+ if schema_id == table_schema_id {
Review Comment:
[P2] Check the stored engine when read options preserve the schema ID
Matching IDs do not establish the file's writer engine when the read handle
has dynamic options: Table::copy_with_options explicitly keeps the schema
version, and merge-engine is currently applied rather than retained as an
immutable option. Using the same valid legacy aggregation files as the
persisted-ALTER regression,
table.copy_with_options({"merge-engine":"deduplicate"}) leaves both file and
read schema at ID 0. This continue skips the stored aggregation schema, so the
two-file single-run read returns [(1,10),(1,11),(2,20)]. The identical public
write/commit/read probe returns only [(1,11),(2,20)] on base fae8acff; the
aggregation control and persisted-ALTER control pass on this head. Check the
file's stored engine even when its ID matches a dynamically overridden read
schema, or disable this shortcut for such overrides.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
