Kristian Rickert created OPENNLP-1895:
-----------------------------------------
Summary: Quantized static embedding tables (2-4 bits per dimension)
Key: OPENNLP-1895
URL: https://issues.apache.org/jira/browse/OPENNLP-1895
Project: OpenNLP
Issue Type: New Feature
Affects Versions: 3.0.0-M4
Reporter: Kristian Rickert
Assignee: Kristian Rickert
Follow-up to OPENNLP-1877 (opennlp-embeddings, static embedding engine).
Static embedding tables are large: a 500k-row, 300-dim float32 table is roughly
600 MB. Modern scalar quantization brings that to 75 MB at 4 bits per dimension
or 38 MB at 2 bits, with near-optimal distortion, which puts full-vocabulary
tables in range of memory-constrained deployments and speeds up embedding
throughput (the workload is memory-bandwidth-bound, so reading 4-16x fewer
bytes is a direct win).
This ticket adds quantized table storage to opennlp-embeddings using the
TurboQuant construction ([arXiv:2504.19874|https://arxiv.org/abs/2504.19874],
Zandieh, Daliri, Hadian, Mirrokni, "TurboQuant: Online Vector Quantization with
Near-optimal Distortion Rate"): a seeded randomized rotation to make
coordinates near-independent with a known distribution, followed by
per-coordinate optimal scalar quantizers, with an optional 1-bit quantized-JL
residual for unbiased inner product estimation.
h3. Scope
* A quantized table implementation behind the same lookup seam as the existing
float table, so components are unaware of the storage form.
* A converter (CLI) that quantizes an existing float table offline.
* Pure JVM, implemented from the paper (cited in javadoc); no ports of any
third-party implementation, no new dependencies. Hot loops use the
\{{jdk.incubator.vector}} patterns already accepted in the module.
* Out of scope: any approximate nearest-neighbor index. This is table storage
and lookup only; indexing belongs to search engines.
h3. Acceptance measurements (will be in the PR)
* Round-trip distortion and downstream parity: cosine similarity of decoded vs
original vectors, and doccat accuracy delta on a fixed fixture, at 2, 3, and 4
bits.
* Throughput and memory vs the float table.
* A cross-check against a reference Rust implementation of the same algorithm
on identical tables, so the JVM cost is stated honestly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
