tzphh opened a new issue, #10658: URL: https://github.com/apache/paimon/issues/10658
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Motivation I would like to create independent vector indexes on multiple vector columns in the same primary-key table. A typical use case is a record containing both image_embedding and text_embedding, or embeddings produced by different models. Each column needs independent ANN search and may have a different dimension, distance metric, and index configuration. Currently, pk-vector.index.columns accepts exactly one column. As a result, multiple vector fields belonging to the same entity cannot all have primary-key vector indexes in one table. Splitting them into separate tables adds data duplication and coordination work for updates and deletes. ### Solution Allow `pk-vector.index.columns` to contain multiple columns, with each column owning an independent index configured through the existing `fields.<column>.pk-vector.*` options. For example, the following Flink SQL illustrates the desired configuration: ```sql CREATE TABLE items ( id BIGINT, image_embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;512', text_embedding ARRAY<FLOAT> COMMENT '__VECTOR_FIELD;768', PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'bucket' = '4', 'merge-engine' = 'deduplicate', 'deletion-vectors.enabled' = 'true', 'deletion-vectors.merge-on-read' = 'false', 'pk-vector.index.columns' = 'image_embedding,text_embedding', 'fields.image_embedding.pk-vector.index.type' = 'ivf-flat', 'fields.image_embedding.pk-vector.distance.metric' = 'cosine', 'fields.text_embedding.pk-vector.index.type' = 'ivf-flat', 'fields.text_embedding.pk-vector.distance.metric' = 'l2' ); ``` Expected behavior: - Both columns can be indexed, with validation of each column's type, dimension, and index settings. - Queries explicitly select a vector column and use that column's index and options. - Index maintenance, compaction, recovery, and deletion-vector filtering work correctly for every configured field under the existing snapshot and index-coverage semantics. - Existing single-column configurations continue to work unchanged. The initial scope is independent per-column indexing and querying. Combining results from multiple vector fields can be discussed separately. ### Anything else? The example above describes the requested behavior; it is currently rejected by the single-column schema check. The current limitation is documented here: https://paimon.apache.org/docs/master/primary-key-table/global-index/#limitations PR #8549 already describes the plural option name as preparation for independent indexes on multiple vector fields. This request follows that direction. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
