JingsongLi opened a new pull request, #1088: URL: https://github.com/apache/paimon-rust/pull/1088
## Purpose PyPaimon already builds local BTree and Bitmap global indexes, but cannot use the core Rust builder while retaining its build-then-commit API. Expose that existing capability through a thin Python wrapper and make prepared indexes safe to publish with a regular TableCommit. ## Changes - Add SortedGlobalIndexBuildBuilder.build() returning unpublished CommitMessages; retain execute() with its latest-snapshot guard. - Support partition-only predicates using the shared partition validation and retain the table's REST/branch snapshot manager. - Use table-configured global-index external paths. - Align sorted-index.records-per-file, its fallback keys and the 25,000,000 default with Java SortedIndexOptions. - Write the Java DataEvolutionIndexSourceMeta (DEIX/version 1) source snapshot. On every commit attempt, validate current row coverage and target-column changes since that snapshot, including historical DELETEs and same-commit updates. Append-only changes and unrelated columns remain allowed. Returned files are retained on failure; explicit abort remains available. - Normalize nullable IN/NOT IN literals in core and accept them in the Python predicate bridge, including Java's empty-set NULL behavior. - Add the builder binding, Table factory and matching datafusion.pyi declarations. The PyPaimon dispatcher uses only its existing single-column BTree/Bitmap capability. ## Java references - SortedIndexOptions - SortedGlobalIndexWriter / DataEvolutionIndexSourceMeta - DataEvolutionConflictDetection global-index row-ID existence checks - In / NotIn predicate evaluators ## Validation - cargo test -p paimon --lib sorted: 91 passed, 1 ignored - cargo test -p paimon --lib table::table_commit: 118 passed - cargo test -p paimon --lib spec::predicate: 106 passed - cargo test -p paimon --lib data_evolution_index_source: 1 passed - cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets -- -D warnings - PyPaimon REST/native and existing regressions: 283 passed, 32 subtests; 104 native plans exercised - Final native index integration module: 87 passed Regression cases cover unpublished files, explicit abort, partition/incremental builds, REST snapshot selection/errors, external paths, scalar key types, both committers, uncertain publication outcomes, source changes/deletions, and equivalent empty partition encodings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
