chinahy opened a new issue, #69712: URL: https://github.com/apache/doris/issues/69712
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description ## Background Starting from Doris 4.0, Doris supports native ANN vector search based on Faiss. Currently, each ANN approximate query (inner_product_approximate / l2_distance_approximate) can only target **one single vector column**. For multi-channel fusion scenarios (multiple vector columns or vector + BM25 hybrid search), the official recommended approach is RRF (Reciprocal Rank Fusion). But RRF is only a manual SQL pattern instead of a built-in native function. It is rank-based fusion and discards the gap information of raw similarity scores. Many RAG / search business scenarios need **weighted linear fusion on normalized raw similarity**, which is similar to ES bool.should boost: final_score = w1 * norm(sim1) + w2 * norm(sim2) + ... Now users have to write long CTE with UNION ALL + GROUP BY manually to implement this workaround: 1. run separate ANN approximate query for each vector / text channel 2. union candidate results 3. group by doc id, fill null similarity scores 4. apply min-max / z-score normalization for each channel 5. weighted sum and order by fused score limit K This approach works but has many drawbacks: 1. verbose and error-prone SQL 2. every business re-implements normalization logic inconsistently 3. hard to tune candidate expand size uniformly 4. lack unified planner optimization for candidate pruning ## Request We hope Doris can provide native support for weighted score fusion, split into two phases: Phase 1 (Low-risk, high priority): Encapsulate this multi-recall + normalize + weighted fusion logic as a built-in TVF / function for **native Doris table ANN index** (not Lance catalog). It still executes separate ANN recall for each channel (no changes to Faiss index inner logic), automatically merge candidate set, do score normalization, weighted sum and return final TopK. Expose parameters: weights, candidate expand N, normalization type(min-max/z-score). Phase 2 (Long-term roadmap): Explore the possibility of streaming merge TopK inside retrieval layer, to reduce candidate expansion overhead (similar to ES bool.should). ## Comparison with RRF RRF is suitable for rank-focused scenario, while weighted score fusion is needed when the magnitude difference of similarity should affect final ranking. Both should coexist as two optional fusion strategies. ### Use case Multi-vector column RAG: title_emb + content_emb, title has higher weight. Vector + BM25 hybrid search with configurable score boost. ### Related issues _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
