Kristian Rickert created OPENNLP-1911:
-----------------------------------------
Summary: Add reproducible evaluation tooling for bounded in-memory
embedding search
Key: OPENNLP-1911
URL: https://issues.apache.org/jira/browse/OPENNLP-1911
Project: OpenNLP
Issue Type: New Feature
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Add a reproducible evaluation harness for the bounded, single-JVM vector
indexes provided by OpenNLP.
The harness should compare exact full-precision search with TurboQuant search
over a documented corpus. It should measure quantization fidelity, proxy
retrieval quality, build time, single-thread query throughput, and serialized
storage footprint.
The evaluation uses tasks that can be reproduced without manually labeled
relevance judgments, including definition-to-headword retrieval and retrieval
of one passage segment from another segment of the same passage. These are
diagnostic proxy tasks, not claims of general search quality.
Include a Lucene HNSW comparison as a test-scope baseline. Lucene must not
become a runtime dependency or a shipped OpenNLP search provider (lucene
support would be better scoped for OpenNLP gRPC after it lands or during
development on another ticket). The comparison exists to distinguish exact
scan, quantized scan, and graph-index tradeoffs.
Results should be emitted in both human-readable Markdown and machine-readable
TSV forms. Corpus acquisition and normalization must have explicit provenance,
licensing documentation, and reproducible commands.
h3. Acceptance criteria
- Provide a command for evaluating an OpenNLP static embedding model over a
bounded passage corpus.
- Compare exact full-precision and TurboQuant indexes using the same vectors
and queries.
- Report:
** recall at k against the exact index
** rank-1 agreement
** definition-to-headword retrieval
** half-passage retrieval
** index build time
** single-thread query throughput
** serialized storage footprint per vector
** total and indexable input counts
- Emit equivalent Markdown and TSV reports.
- Make randomized operations deterministic through an explicit seed.
- Define and test handling of:
** empty inputs
** zero embeddings
** {{k}} larger than an index
** malformed corpus records
** missing or duplicate identifiers
** non-finite measurements
- Include reproducible corpus acquisition and normalization tooling.
- Record corpus provenance, source URLs, snapshots or checksums, and licensing.
- Provide small repository fixtures so tests do not require network access.
- Include a Lucene HNSW comparison only in test scope.
- Ensure the shipped OpenNLP module gains no Lucene runtime dependency.
- Document exact commands for reproducing the reports.
- Describe benchmark numbers as measurements from a particular environment,
not universal performance guarantees.
h3. Out of scope
- A production Lucene or HNSW provider
- General web, lexical, or hybrid search evaluation
- Distributed search evaluation
- Persistent search services
- Claims about production relevance without labeled judgments
- Comparing every available ANN implementation
- Treating serialized index size as JVM memory consumption
- gRPC service benchmarks
h3. Dependencies
- OPENNLP-1877, static embeddings
- OPENNLP-1895, quantized vector storage
- The new bounded in-memory {{VectorIndex}} ticket
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
