JingsongLi opened a new pull request, #1173: URL: https://github.com/apache/paimon-rust/pull/1173
### Problem and change A selective file bitmap query on a Parquet data-evolution table still opened the primary Parquet file despite an aligned ROW sidecar. Projecting `_ROW_ID` also disabled that bitmap selection. This differs from Java's sidecar selection behavior and loses the sparse-read I/O benefit. Choose the sidecar from the intersection of bitmap matches and requested row ranges, before deletion vectors or LIMIT narrow the physical selection. Permit bitmap selection while projecting `_ROW_ID`, while keeping decoder row filtering separate so original row IDs remain correct. Normalize ranges supplied through external splits before intersecting them. ### Regression coverage - Remove the primary Parquet file and require file bitmap queries to read the ROW sidecar, including nonzero row-ID bases, `_ROW_ID` projection, full/sparse/disjoint requested ranges and directly constructed unsorted splits. - Keep negative controls proving that LIMIT or deletion vectors alone do not trigger a missing sidecar. - Exercise real global bitmap/btree indexes with ordinary and nested Variant values, SQL NULL versus JSON null, reordered fields, strict path extraction, unselected invalid casts, Variant-only updates, historical renames and deletion vectors. Remove the primary files of all providers to prove sidecar use. - Include a Parquet Variant-shredding case; its ROW sidecar preserves ordinary logical Variant values. ### Validation - `cargo test -p paimon --test data_evolution_row_sidecar_test --test data_evolution_variant_sidecar_test --test row_variant_test --test row_variant_interop_test --locked` — 18 passed. - `cargo fmt --all -- --check` — passed. - `cargo clippy -p paimon --all-targets --features fulltext,vortex --locked -- -D warnings -A clippy::nonminimal_bool` — passed. The allowance retains the existing Rust 1.95 baseline for unchanged code. - Read four Python-written data-evolution warehouses with real bitmap/btree indexes and all primary Parquet files removed. Verified five rows of updated/renamed ordinary and nested Variants, SQL NULL versus JSON null and row IDs 25 through 29 against an external oracle. Java read the same four warehouses successfully. Both targeted regressions were confirmed failing before their fixes: a bitmap match opened the deleted primary file, and an external unsorted split silently lost the matching row. ### Scope Global bitmap/btree row-ID selections work with column-merged files. File-bitmap-only pruning of column-merged groups retains its existing Parquet fallback; selecting an index there requires planning against each field's latest provider to avoid pruning live values using an obsolete version. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
