zhangstar333 opened a new issue, #67435: URL: https://github.com/apache/doris/issues/67435
## Parent - Tracked by #66493 - Part of #66340 - Related upstream work: https://github.com/lance-format/lance/pull/8822 ## Motivation Doris executes Lance full-text search in a distributed architecture. The FE plans one query and assigns different Lance fragments or FTS index segments to multiple BEs for parallel execution. If each BE or scanner builds BM25 statistics only from its local segments, its document count, total token count, average document length, and query-term document frequencies can differ from other workers. The resulting BM25 scores are not directly comparable, and merging local Top-K results in the FE cannot restore globally consistent ranking. The FE Java layer therefore needs an API to collect query-aware global BM25 statistics for one pinned Lance dataset snapshot and broadcast the same scorer inputs to every BE. Document scores are still calculated by the BEs; the FE collects and distributes the global statistics used to calculate those scores. ```mermaid flowchart TB subgraph FE["Doris FE - Java query planning"] Q["FTS query"] --> P["Pin dataset version and resolve FTS segments"] C["Collect global BM25 statistics<br/>N, total tokens, term DF, prepared vocabulary"] B["Broadcast identical statistics<br/>with scan ranges"] M["Merge comparable local Top-K results"] P --> C --> B end subgraph LANCE["One Lance dataset snapshot"] S1["FTS segment 1"] S2["FTS segment 2"] S3["FTS segment 3"] end subgraph BES["Doris BE cluster - parallel FTS execution"] BE1["BE 1<br/>scan segment 1<br/>use global scorer"] BE2["BE 2<br/>scan segment 2<br/>use global scorer"] BE3["BE 3<br/>scan segment 3<br/>use global scorer"] end S1 --> C S2 --> C S3 --> C B --> BE1 B --> BE2 B --> BE3 BE1 --> S1 BE2 --> S2 BE3 --> S3 BE1 --> M BE2 --> M BE3 --> M ``` ## Requirements - Pin one Lance dataset version for the complete query lifecycle. - Resolve the exact committed FTS segment set used by the query. - Collect the corpus document count, total token count, and document frequencies for every required query term. - Preserve one globally prepared vocabulary for fuzzy or prefix queries, or reject these query modes until they can be transported correctly. - Bind the statistics to the dataset version, segment UUIDs, indexed column, and document granularity. - Propagate the same statistics or scorer inputs from the FE to every BE scan task. - Make every BE scanner calculate BM25 scores using the same corpus-wide statistics before the FE merges local Top-K results. - Explicitly define how unindexed fragments and mixed indexed/non-indexed execution are handled. ## Upstream Lance work The Lance changes will be submitted in two separate pull requests: 1. A format-contract PR containing the versioned protobuf schema, matching format documentation, and only compile-enabling changes. 2. A follow-up implementation PR exposing global-statistics collection through Rust, JNI, and the Java API. Doris integration will then use the Java API in the FE and propagate the returned statistics to BE/lance-c execution. ## Completion criteria - Distributed FTS ranking matches a single full-dataset execution over the same dataset version and FTS segment set. - Regression tests cover multiple fragments or segments where local and global BM25 statistics produce different rankings. - Fuzzy and prefix queries preserve the same globally prepared vocabulary, or return a clear unsupported error. - EXPLAIN or runtime profiles show whether global BM25 statistics are collected and used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
