JingsongLi opened a new pull request, #1093: URL: https://github.com/apache/paimon-rust/pull/1093
### Purpose Complete native vindex construction for behavior already supported by PyPaimon and Java, so PyPaimon can delegate local REST vector-index builds to Rust. Java `NativeVectorGlobalIndexWriter` skips NULL vectors, preserves their logical row coverage and adds the original shard-relative IDs for non-NULL vectors. The previous Rust pipeline required dense vectors, so the Python adapter excluded all five vindex families. ### Changes - Compact nullable Arrow List / FixedSizeList batches while validating every logical row ID. Retain Arrow storage for dense batches; ignore child payloads hidden behind NULL parents and reject invalid non-NULL vectors. - Spill original relative IDs separately from compacted float vectors. Train on evenly spaced samples of the non-NULL stream, with bounded read buffers; retain full logical `row_count` and row ranges in manifest metadata. - Produce no file or commit for all-NULL shards. Granule builds that encounter NULL vectors drain private worker tasks before preparing through full spill against the pinned snapshot. - Align automatic `nlist`, inferred PQ width/OPQ, option aliases, field/build precedence and tuning objectives with Java. Compile the native configuration when the real valid-vector count is known; avoid synthetic-count budget failures. - Preserve documented raw native aliases and strict foreign-family option rejection for explicit Rust/SQL builder options. Keep PK query dimension aliases working. - Accept and validate Python's build parallelism option; native shard preparation currently remains sequential. - Update SQL configuration and NULL-vector documentation. Reference implementations: `paimon-vector/.../NativeVectorGlobalIndexerFactory.java`, `NativeVectorGlobalIndexWriter.java`, and `paimon-python/pypaimon/globalindex/vindex/vindex_vector_index_writer.py` in Apache Paimon master. ### Verification - `cargo test -p paimon --lib vindex --no-fail-fast`: 223 passed, 1 previously ignored. - `cargo test -p paimon --features fulltext --lib index_build --no-fail-fast`: 150 passed, 2 previously ignored. - `cargo test -p paimon --features fulltext --lib pk_vector --no-fail-fast`: 160 passed. - `cargo test -p paimon-datafusion --features fulltext --test procedures index --no-fail-fast`: 27 passed. - `cargo +1.98.0 clippy -p paimon -p pypaimon_rust --all-targets --features paimon/fulltext -- -D warnings`: passed. - `cargo fmt --all -- --check` and `git diff --check`: passed. - Paired PyPaimon REST suites: 205 passed and 19 subtests passed, including all five vindex families, both committers, sparse original IDs queried by both readers, all-NULL/incremental shards, default/tuning options, and byte-for-byte comparisons with the existing Python writer. - A 262,144-row regression places the only NULL in the granule plan's later read, verifies private training is discarded without an index output, then verifies sparse preparation retains logical coverage. ### API and storage Uses the existing common global-index builder and commit-message APIs. Existing index wire formats and DEIX metadata remain unchanged. Returned/submitted messages retain the existing commit ownership rules; cleanup is limited to failed private preparation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
