Gabriel39 opened a new issue, #66340: URL: https://github.com/apache/doris/issues/66340
## Goal Track the Doris 4.2 support matrix for Lance Catalog, Lance dataset access, and vector search. Related work: - Initial implementation: #65730 - Original tracking issue: #66322 - User documentation: apache/doris-website#4024 ## Tracking rules - Every item with a checkbox is a Doris 4.2 work item. - Check an item only after the implementation, tests, documentation, and verification are complete. - Items marked as follow-up are not adapted in #65730 but remain planned for Doris 4.2. - Items under "Explicitly not planned for Doris 4.2" have no checkbox because the current Lance Scanner/lance-c SDK path does not provide the required native capability. ## Doris 4.2 work items ### Catalog integration - [ ] **Filesystem Lance Catalog with a local path** — Implemented in #65730; pending merge and verification. - [ ] **Filesystem Lance Catalog with file://** — Implemented in #65730; pending merge and verification. - [ ] **Filesystem Lance Catalog with s3://** — Implemented in #65730; pending merge and verification. - [ ] **Lance REST Namespace Catalog** — Implemented in #65730; pending merge and verification. ### Authentication and credentials - [ ] **Bearer Token authentication** — Implemented in #65730; pending merge and verification. - [ ] **API Key authentication** — Implemented in #65730; pending merge and verification. - [ ] **Custom HTTP headers** — Implemented in #65730; pending merge and verification. - [ ] **Temporary storage credentials returned by the Namespace service** — Implemented in #65730; pending merge and verification. ### Namespace and metadata - [ ] **Map multilevel Lance Namespaces to Doris databases** — Implemented in #65730; pending merge and verification. - [ ] **SHOW DATABASES** — Implemented in #65730; pending merge and verification. - [ ] **SHOW TABLES** — Implemented in #65730; pending merge and verification. - [ ] **DESC** — Implemented in #65730; pending merge and verification. - [ ] **Create, alter, and drop Namespaces** — Follow-up; #65730 provides a read-only Catalog. ### Versioning and consistency - [ ] **Pin the dataset version during planning** — Implemented in #65730 to preserve snapshot consistency; pending merge and verification. - [ ] **Time travel and explicit version selection** — Follow-up; Lance supports versioned reads, but #65730 does not expose version selection. - [ ] **REST Namespace managed versioning in the BE reader** — Follow-up. ### Scan execution - [ ] **Lance reader in File Scanner V2** — Implemented in #65730; pending merge and verification. - [ ] **Parallel scan by Lance Fragment** — Implemented in #65730; pending merge and verification. - [ ] **Column pruning** — Implemented in #65730; pending merge and verification. - [ ] **Retain unsupported predicates for execution in Doris** — Implemented in #65730; pending merge and verification. ### Predicate pushdown - [ ] **Push semantically compatible scalar predicates to Lance** — Implemented in #65730; pending merge and verification. - [ ] **SQL filter for vector_search()** — Implemented in #65730; pending merge and verification. - [ ] **Substrait filter for regular Lance scans** — Implemented in #65730; pending merge and verification. ### Observability - [ ] **Show pushed predicates in EXPLAIN** — Implemented in #65730; pending merge and verification. ### Direct dataset access - [ ] **Read a Lance dataset through the s3() TVF** — Implemented in #65730; pending merge and verification. - [ ] **Read a Lance dataset through the local() TVF** — Implemented in #65730; pending merge and verification. ### Paths and splits - [ ] **Split one Lance dataset into multiple Fragment splits** — Implemented in #65730; pending merge and verification. - [ ] **Expand glob patterns to multiple Lance datasets** — Follow-up. - [ ] **Read multiple Lance datasets in one TVF invocation** — Follow-up. - [ ] **Support additional file TVF entry points and broader path handling** — Follow-up. ### Read and write operations - [ ] **Read Lance datasets** — Implemented in #65730; pending merge and verification. - [ ] **Create Lance datasets/tables** — Follow-up; the current Catalog is read-only. - [ ] **INSERT into Lance** — Follow-up. - [ ] **UPDATE Lance data** — Follow-up. - [ ] **DELETE Lance data** — Follow-up. ### Search modes - [ ] **ANN index search** — Implemented in #65730 with use_index=true; pending merge and verification. - [ ] **Flat exact search** — Implemented in #65730 with use_index=false; pending merge and verification. - [ ] **Radius/Range Search** — Follow-up; Lance core supports lower and upper distance bounds, but lance-c needs a binding. - [ ] **Full-text search with BM25** — Follow-up; lance-c has a standalone FTS API, but #65730 does not expose it. - [ ] **Multi-vector Search** — Follow-up; Lance core supports it, but the current lance-c nearest API cannot represent a two-dimensional query vector. ### Vector index algorithms - [ ] **IVF_FLAT query support** — Implemented through Lance automatic index selection; pending verification. - [ ] **IVF_SQ query support** — Implemented through Lance automatic index selection; pending verification. - [ ] **IVF_PQ query support** — Implemented through Lance automatic index selection; the current regression fixture uses this type. - [ ] **IVF_HNSW_FLAT query support** — Implemented through Lance automatic index selection; pending verification. - [ ] **IVF_HNSW_SQ query support** — Implemented through Lance automatic index selection; pending verification. - [ ] **IVF_HNSW_PQ query support** — Implemented through Lance automatic index selection; pending verification. ### Distance metrics - [ ] **L2** — Implemented in #65730; pending merge and verification. - [ ] **Cosine** — Implemented in #65730; pending merge and verification. - [ ] **Dot Product** — Implemented in #65730; pending merge and verification. - [ ] **Hamming** — Implemented in #65730 for compatible binary vectors and indexes; pending merge and verification. ### Data types - [ ] **FixedSizeList<Float16>** — Implemented in #65730; pending merge and verification. - [ ] **FixedSizeList<Float32>** — Implemented in #65730; pending merge and verification. - [ ] **FixedSizeList<Float64>** — Implemented in #65730; ANN compatibility still depends on the index and metric. - [ ] **FixedSizeList<UInt8>** — Implemented in #65730; Hamming requires a compatible binary index. - [ ] **FixedSizeList<Int8>** — Implemented in #65730; pending merge and verification. - [ ] **List<FixedSizeList<...>> for multi-vector columns** — Follow-up. - [ ] **Blob v2** — Follow-up; not mapped by #65730. - [ ] **Arrow JSON Extension** — Follow-up; not mapped by #65730. - [ ] **BFloat16 Extension** — Follow-up; not mapped by #65730. - [ ] **Arrow null** — Follow-up; no Doris column mapping in #65730. - [ ] **Arrow duration** — Follow-up; no Lance/Arrow-to-Doris mapping in #65730. ### Query input - [ ] **One query vector per search** — Implemented in #65730; pending merge and verification. - [ ] **Multiple sub-vectors in one logical query** — Follow-up for Multi-vector Search. ### Filtering - [ ] **Scalar pre-filter** — Implemented in #65730; supplying filter currently forces prefilter=true. - [ ] **Scalar post-filter** — Follow-up; Lance/lance-c supports prefilter=false, but #65730 does not expose the choice. ### Index lifecycle - [ ] **Create vector indexes** — Follow-up; lance-c provides lance_dataset_create_vector_index. - [ ] **Create scalar indexes** — Follow-up; lance-c provides BTree, Bitmap, LabelList, and Inverted index creation APIs. - [ ] **Show the number of indexes** — Follow-up; lance-c provides lance_dataset_index_count. - [ ] **Show index names, UUIDs, columns, types, and dataset versions** — Follow-up; lance-c provides index-list metadata. - [ ] **Replace or rebuild an index with the same name** — Follow-up; the lance-c create API provides replace. - [ ] **Drop indexes by name** — Follow-up; lance-c provides the drop-index API. ### Index selection - [ ] **Automatically select an index for the target vector column** — Implemented in #65730; pending merge and verification. ### ANN tuning - [ ] **nprobes** — Implemented in #65730; pending merge and verification. - [ ] **refine_factor** — Implemented in #65730; pending merge and verification. - [ ] **ef** — Implemented in #65730; pending merge and verification. - [ ] **Disable the ANN index with use_index=false** — Implemented in #65730; pending merge and verification. ### Result output - [ ] **Top-K** — Implemented in #65730; pending merge and verification. - [ ] **Offset** — Implemented in #65730; pending merge and verification. - [ ] **Return distance through the _distance virtual column** — Implemented in #65730; quantized ANN may return an approximate distance until refinement recomputes it from raw vectors. ### Tests and documentation - [ ] **FE unit tests** — Implemented in #65730; pending merge and verification. - [ ] **BE unit tests** — Implemented in #65730; pending merge and verification. - [ ] **Regression tests for Catalog queries, predicate pushdown, file TVFs, and vector search** — Implemented in #65730; pending merge and verification. - [ ] **Lance Catalog user documentation** — Submitted in apache/doris-website#4024; pending merge and verification. ## Explicitly not planned for Doris 4.2 The following capabilities are not natively supported by the current Lance Scanner/lance-c query path. Doris 4.2 will not implement alternative semantics for them. They may be reconsidered only after the SDK provides a stable native interface. | Category | Capability | Reason | |---|---|---| | Search modes | 🚫 **Hybrid Search (Vector + BM25)** | LanceDB can orchestrate two searches at a higher layer, but Lance Scanner/lance-c makes nearest and FTS mutually exclusive. | | Search modes | 🚫 **Batch Search** | The current Scanner/lance-c nearest API accepts only one independent query vector. | | Search modes | 🚫 **Joint search and scoring across multiple vector columns** | One Lance vector query accepts only one vector column. | | Distance metrics | 🚫 **Custom distance functions** | Lance ANN accepts only its built-in distance enum and cannot accept a callback, SQL UDF, or arbitrary expression. | | Data types | 🚫 **Vector element types outside the current nearest C ABI** | The current nearest C ABI defines Float16, Float32, Float64, UInt8, and Int8 only. | | Index selection | 🚫 **Select index_name at query time** | Lance Scanner Query and lance-c nearest do not expose an index-name parameter. | | Index selection | 🚫 **Select index_type at query time** | The current query path does not expose an index-type selector. | ## Completion criteria - The corresponding implementation is merged into Doris 4.2. - FE/BE unit tests and regression tests cover the capability. - User documentation describes the supported behavior, parameter semantics, and limitations. - The capability is verified with real Lance datasets and object-storage access. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
