Gabriel39 opened a new pull request, #68028:
URL: https://github.com/apache/doris/pull/68028
### What problem does this PR solve?
Issue Number: close #66494
`vector_search` now accepts one query matrix for a Lance
`List<FixedSizeList<T, D>>` column on branch-4.1, supporting Float16, Float32
and Float64. A matrix such as `[[1,0],[0,1]]` searches logical rows containing
variable numbers of subvectors. Ordinary single-vector requests retain their
existing wire format and behavior.
The row score is `sum(q in query, min(v in row, distance(q, v)))`; filtering
precedes candidate selection, and TopK/offset count rows. Multi-vector
candidates are refined to keep indexed and appended rows on the same score
scale. A Lance-C adapter restores distance ordering before applying the result
window because fragment-scoped payload reads can reorder batches. FE validation
rejects malformed matrices and unsupported types, and schema version 2 fences
incompatible BEs.
The vendored Lance-C 0.1.9 patch adds the matrix API without changing
existing API signatures. **Lance-C must be rebuilt with this patch; previously
installed third-party libraries do not contain the new symbol.** The pinned
Lance version supports cosine multi-vector indexes; L2/dot are covered through
exact search. Stored subvectors must be non-nullable and contain finite,
non-null elements. Outer null/empty rows do not rank.
### Validation
Completed local self-review before committing; fixed allocation validation,
request compatibility and fragment result-window correctness issues found
during review.
- 31 FE tests passed using a focused harness compiling the changed classes
and generated query-vector Thrift class against cached dependencies.
- 48 Lance BE reader tests passed using the compiled changed reader/test
sources and Thrift type linked with the patched Lance-C library and existing BE
test dependencies. The multi-vector indexed/filter/offset case also passed 10
consecutive runs.
- 3 native Lance-C integration tests passed, including repeated
fragment-scoped indexed searches and invalid request/window checks.
- FE Checkstyle, clang-format 16, Groovy syntax, fixture self-checks and
clean patch application passed.
- Added SQL regression coverage for three element types, three metrics,
one/two query subvectors, nested payloads, null/empty rows, partial index
coverage, prefilters, TopK/offset and invalid inputs. Indexed plans are
asserted with EXPLAIN; scoring uses an independent oracle. The reproducible
fixture contains three cosine IVF_FLAT indexes and a subsequent append.
The complete SQL regression suite has **not** been run locally against a
newly built Doris cluster; it remains for CI. The focused harness results above
are not a full FE/BE build.
### Release note
Support multi-vector Lance columns and query matrices in `vector_search`,
with documented row-level distance, filtering and TopK semantics.
### Check List (For Author)
- Test
- [x] Regression test added
- [x] Unit Test
- Behavior changed
- [x] Yes: multi-vector columns accept matrix queries; ordinary vector
queries are unchanged.
- Documentation
- [x] Added `docs/lance-multivector-search.md` with query format, scoring
and compatibility requirements.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
