JingsongLi opened a new pull request, #10488: URL: https://github.com/apache/paimon/pull/10488
### Purpose Delegate PyPaimon's existing local REST vindex builds to the common Rust global-index builder for `ivf-flat`, `ivf-pq`, `ivf-sq`, `ivf-rq` and `diskann`. Depends on https://github.com/apache/paimon-rust/pull/1093 for NULL vectors, sparse relative row IDs, all-NULL shards and Java-compatible training defaults. This PR remains draft until that Rust change is merged. CI continues to install Rust from `main`. ### Changes - Enable the existing native route for the five vindex families on eligible REST/Parquet data-evolution tables. - Reuse Python's existing Java-compatible option mapping and training-ratio parsing before forwarding options. Rust owns row scanning, snapshot/range planning, nullable-vector compaction, training and index file generation. - Keep prepare/commit ownership: build returns unpublished messages, native failures propagate, and submitted/returned files remain available after uncertain commits. - Replace the dense-vector fallback test with end-to-end sparse-vector native tests. ### Verification `PYPAIMON_TEST_NATIVE_PLAN=1 .venv/bin/python -m pytest pypaimon/tests/native_vector_index_build_test.py pypaimon/tests/native_generic_index_build_test.py pypaimon/tests/native_sorted_index_build_test.py pypaimon/tests/global_index_build_test.py -q --tb=short` **205 passed and 19 subtests passed**, with 55 native plans exercised, against the paired Rust implementation. Coverage includes: - Both Python and Rust readers return original sparse row IDs for all five families. - All-NULL shards create no index file or snapshot; later shards and incremental builds retain correct global IDs. - Sparse full-spill files match the existing Python writer byte for byte, including training ratios, automatic sizing, PQ defaults, field aliases, OPQ and tuning budgets. - Real valid-vector cardinality drives automatic PQ size-budget validation; all-NULL input skips native training. - Partition filtering, external index placement and both committers continue to work; source conflicts preserve returned files. - Invalid non-NULL vectors and invalid build parallelism fail without retrying through the Python row writer. Flake8, Python 3.6 syntax checks and `git diff --check` passed. ### Scope The public Python build/commit APIs remain unchanged. Local native builds use the existing eligibility checks; Ray execution retains its current implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
