JeonDaehong commented on code in PR #1164:
URL: https://github.com/apache/paimon-rust/pull/1164#discussion_r4314953596
##########
crates/paimon/src/table/kv_file_reader.rs:
##########
@@ -676,6 +676,20 @@ impl KeyValueFileReader {
.map(|budget| Arc::new(budget.without_prefetch()))
};
let mut file_streams: Vec<ArrowRecordBatchStream> =
Vec::new();
+ // One sorted run of files without retract rows, under an
+ // engine whose files never repeat a key, has nothing to
+ // merge. Planning trusts the same invariant when it marks
a
+ // single-file split raw convertible.
+ let skip_merge = input_stream_count == 1
+ && matches!(
+ config.merge_engine,
+ MergeEngine::Deduplicate | MergeEngine::FirstRow
+ )
+ && merge_group.iter().all(|run| {
+ run.files
+ .iter()
+ .all(|file| file.file.delete_row_count ==
Some(0))
Review Comment:
Good catch, thanks. Fixed in 476312c: the shortcut no longer relies on
`delete_row_count`. Each unmerged batch drops retract rows the same way the
merge path handles a lone version of a key. First-row without `ignore-delete`
still errors on them.
`single_run_read_drops_retracts_hidden_by_legacy_metadata` covers your two-file
case for deduplicate, first-row and first-row + `ignore-delete`. Without the
filter it returns `[1, 2, 3]`.
While running CI I also found that `read_with_merge_function` took the
shortcut, which dropped `-D` rows from `$audit_log`
(`test_audit_log_system_table_keeps_row_kinds_and_sequence_numbers`). The
shortcut now applies only to `read`, so custom merge functions always merge.
Rebased onto current main and re-measured on one instance, alternating: main
157 ms vs this head 60 ms. Ranges don't overlap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
