Stephen0421 opened a new pull request, #10569: URL: https://github.com/apache/paimon/pull/10569
### Purpose LATERAL vector search always evaluated every query vector locally, even when `vector-search.distribute.enabled` is true. Single-vector search already ships index splits and primary-key buckets to the Spark cluster under that option. This change applies the same rule to batch search. When the option is enabled and there are at least `global-index.thread-num * 2` splits, data-evolution index groups, raw row ranges, and primary-key buckets are evaluated in Spark. Smaller inputs keep the local thread pool. The driver merges one result per query vector, in input order, and still runs refine on the merged candidates. `PrimaryKeyVectorRead.searchBuckets(List, float[][])` is now public, and `mergeBatchGroupResults` is added, so the Spark module can search and merge a bucket group for every query. `PrimaryKeyVectorRead` is not a `@Public` API. ### Tests - `mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -Pspark3 -DwildcardSuites=none -DfailIfNoTests=false -Dtest=SparkDataEvolutionBatchVectorReadTest,SparkDataEvolutionVectorReadTest test` - `SparkDataEvolutionBatchVectorReadTest` covers distributed and local index search, distributed raw search, builder selection, and primary-key bucket grouping. - `SparkDataEvolutionVectorReadTest` covers the shared split-grouping helpers used by single-vector search. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
