JingsongLi opened a new pull request, #10598: URL: https://github.com/apache/paimon/pull/10598
## Purpose Use Rust for the existing PyPaimon BTree/Bitmap and vindex global-index build capabilities on REST data-evolution tables with deletion vectors. Depends on https://github.com/apache/paimon-rust/pull/1136. This PR is draft until that change reaches Rust main. CI continues building apache/paimon-rust main; no fork or branch pin is introduced. ## Changes - Allow the existing Rust delegation on deletion-vector tables for BTree, Bitmap and all five supported vindex families. Keep full-text's existing rejection and the existing REST/Parquet/authorization/primary-key eligibility checks. - Align Python sorted-index build splits with Java `GlobalIndexBuilderUtils.splitByContiguousRowRange`: omit deletion files so indexes cover physical row IDs; query readers apply their snapshot's live-row filtering. This does not alter query splits. - Keep Python as a wrapper around the existing Rust builder, with no new APIs or dependencies. - Cover partial/all-row deletion, original row-ID coverage, Python/native reads, training-byte equivalence, both vector build modes, incremental builds, deletion between preparation and commit, snapshot queries, live NULL vectors in Native search, and invalid dimensions in deleted vectors. ## Validation Built the companion Rust branch locally with maturin. - Before the fix, the native routing regression fell back to Python; the pure-Python split regression retained deletion metadata. - Native REST sorted/vector/generic/parallel index-build suites: 291 passed. - Ordinary Python global-index build suite: 41 passed, 25 subtests passed. - flake8, Python 3.6 syntax checks for changed ordinary Python files, license-header check and git diff --check: passed. - Independent Rust/Java and Python integration review found no actionable P1/P2 findings. ## Existing dependency limitation The paimon-vindex 0.5.0 Python FFI can fail IVF-PQ searches when a candidate bitmap includes live NULL vectors and K exceeds the valid vector count (`native result length mismatch`). This is reproducible independently of build delegation. The Native Rust reader handles that case and is covered explicitly here. Cross-reader build tests also retain NULL vectors in physical training input, with those rows deleted in the query snapshot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
