Kristian Rickert created OPENNLP-1910:
-----------------------------------------
Summary: Introduce a bounded in-memory vector index API for
static embedding similarity search
Key: OPENNLP-1910
URL: https://issues.apache.org/jira/browse/OPENNLP-1910
Project: OpenNLP
Issue Type: New Feature
Reporter: Kristian Rickert
Assignee: Kristian Rickert
OpenNLP's static embedding module can produce dense vectors, while OPENNLP-1895
adds compact TurboQuant storage. Applications still need a small,
dependency-free way to search a bounded collection of vectors inside one JVM.
Add an experimental {{VectorIndex}} contract to {{{}opennlp-embeddings{}}}. The
index has a build-once, read-many lifecycle: callers add fixed-dimension
vectors under application-defined identifiers, freeze the index, and then issue
top-k cosine-similarity queries.
Provide two implementations:
- An exact full-precision flat scan suitable for small collections and use as
a correctness baseline.
- A TurboQuant implementation that searches the quantized representation
without reconstructing every original vector.
The contract should support safely published indexes being queried concurrently
after freezing. It should define common validation, lifecycle, result ordering,
dimensionality, zero-vector behavior, and cosine score semantics.
This is intended for document-local or bounded-corpus use inside an application
process. It is not intended to replace Lucene, Solr, OpenSearch, or another
general search system.
h3. Acceptance criteria
- Add an experimental {{VectorIndex}} API with:
** caller-provided string identifiers
** fixed vector dimensionality
** build, freeze, and query lifecycle
** top-k cosine-similarity results
** index size and dimension inspection
- Add an exact full-precision implementation.
- Add a TurboQuant-backed implementation using OPENNLP-1895.
- Reject null, blank, duplicate, or line-breaking identifiers.
- Reject null, incorrectly sized, or non-finite vectors.
- Reject queries before the index is frozen.
- Reject additions after the index is frozen.
- Define behavior for empty indexes, zero vectors, zero queries, and {{k}}
larger than the collection.
- Permit concurrent queries after a frozen index has been safely published.
- Apply one parameterized contract test suite to every shipped implementation.
- Document intended usage and lifecycle in the OpenNLP manual.
- Add no third-party runtime dependency.
h3. Out of scope
- Distributed or remote search
- Disk-backed mutable indexes
- Incremental updates after freezing
- Sharding, replication, failover, or clustering
- Boolean, lexical, faceted, or hybrid search
- Large-scale approximate nearest-neighbor indexing
- A general search server
- gRPC APIs
h3. Dependencies
- OPENNLP-1877, static embeddings
- OPENNLP-1895, quantized vector storage
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
