[
https://issues.apache.org/jira/browse/OPENNLP-1895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kristian Rickert updated OPENNLP-1895:
--------------------------------------
Description:
Follow-up to OPENNLP-1877 (opennlp-embeddings, static embedding engine).
Static embedding tables are large: a 500k-row, 300-dim float32 table is roughly
600 MB. Modern scalar quantization brings that to 75 MB at 4 bits per dimension
or 38 MB at 2 bits, with near-optimal distortion, which puts full-vocabulary
tables in range of memory-constrained deployments and speeds up embedding
throughput (the workload is memory-bandwidth-bound, so reading 4-16x fewer
bytes is a direct win).
This ticket adds quantized table storage to opennlp-embeddings using the
TurboQuant construction ([arXiv:2504.19874|https://arxiv.org/abs/2504.19874],
Zandieh, Daliri, Hadian, Mirrokni, "TurboQuant: Online Vector Quantization with
Near-optimal Distortion Rate"): a seeded randomized rotation to make
coordinates near-independent with a known distribution, followed by
per-coordinate optimal scalar quantizers, with an optional 1-bit quantized-JL
residual for unbiased inner product estimation.
h3. Scope
* A quantized table implementation behind the same lookup seam as the existing
float table, so components are unaware of the storage form.
* A converter (CLI) that quantizes an existing float table offline.
* Pure JVM, implemented from the paper (cited in javadoc); no ports of any
third-party implementation, no new dependencies. Hot loops use the
{{jdk.incubator.vector}} patterns already accepted in the module.
* Out of scope: any approximate nearest-neighbor index. This is table storage
and lookup only; indexing belongs to search engines.
h3. Acceptance measurements (will be in the PR)
* Round-trip distortion and downstream parity: cosine similarity of decoded vs
original vectors, and doccat accuracy delta on a fixed fixture, at 2, 3, and 4
bits.
* Throughput and memory vs the float table.
was:
Follow-up to OPENNLP-1877 (opennlp-embeddings, static embedding engine).
Static embedding tables are large: a 500k-row, 300-dim float32 table is roughly
600 MB. Modern scalar quantization brings that to 75 MB at 4 bits per dimension
or 38 MB at 2 bits, with near-optimal distortion, which puts full-vocabulary
tables in range of memory-constrained deployments and speeds up embedding
throughput (the workload is memory-bandwidth-bound, so reading 4-16x fewer
bytes is a direct win).
This ticket adds quantized table storage to opennlp-embeddings using the
TurboQuant construction ([arXiv:2504.19874|https://arxiv.org/abs/2504.19874],
Zandieh, Daliri, Hadian, Mirrokni, "TurboQuant: Online Vector Quantization with
Near-optimal Distortion Rate"): a seeded randomized rotation to make
coordinates near-independent with a known distribution, followed by
per-coordinate optimal scalar quantizers, with an optional 1-bit quantized-JL
residual for unbiased inner product estimation.
h3. Scope
* A quantized table implementation behind the same lookup seam as the existing
float table, so components are unaware of the storage form.
* A converter (CLI) that quantizes an existing float table offline.
* Pure JVM, implemented from the paper (cited in javadoc); no ports of any
third-party implementation, no new dependencies. Hot loops use the
\{{jdk.incubator.vector}} patterns already accepted in the module.
* Out of scope: any approximate nearest-neighbor index. This is table storage
and lookup only; indexing belongs to search engines.
h3. Acceptance measurements (will be in the PR)
* Round-trip distortion and downstream parity: cosine similarity of decoded vs
original vectors, and doccat accuracy delta on a fixed fixture, at 2, 3, and 4
bits.
* Throughput and memory vs the float table.
* A cross-check against a reference Rust implementation of the same algorithm
on identical tables, so the JVM cost is stated honestly.
> Quantized vector storage and search for static embeddings
> ----------------------------------------------------------
>
> Key: OPENNLP-1895
> URL: https://issues.apache.org/jira/browse/OPENNLP-1895
> Project: OpenNLP
> Issue Type: New Feature
> Affects Versions: 3.0.0-M4
> Reporter: Kristian Rickert
> Assignee: Kristian Rickert
> Priority: Major
>
> Follow-up to OPENNLP-1877 (opennlp-embeddings, static embedding engine).
>
> Static embedding tables are large: a 500k-row, 300-dim float32 table is
> roughly 600 MB. Modern scalar quantization brings that to 75 MB at 4 bits per
> dimension or 38 MB at 2 bits, with near-optimal distortion, which puts
> full-vocabulary tables in range of memory-constrained deployments and speeds
> up embedding throughput (the workload is memory-bandwidth-bound, so reading
> 4-16x fewer bytes is a direct win).
>
> This ticket adds quantized table storage to opennlp-embeddings using the
> TurboQuant construction ([arXiv:2504.19874|https://arxiv.org/abs/2504.19874],
> Zandieh, Daliri, Hadian, Mirrokni, "TurboQuant: Online Vector Quantization
> with Near-optimal Distortion Rate"): a seeded randomized rotation to make
> coordinates near-independent with a known distribution, followed by
> per-coordinate optimal scalar quantizers, with an optional 1-bit quantized-JL
> residual for unbiased inner product estimation.
>
> h3. Scope
>
> * A quantized table implementation behind the same lookup seam as the
> existing float table, so components are unaware of the storage form.
> * A converter (CLI) that quantizes an existing float table offline.
> * Pure JVM, implemented from the paper (cited in javadoc); no ports of any
> third-party implementation, no new dependencies. Hot loops use the
> {{jdk.incubator.vector}} patterns already accepted in the module.
> * Out of scope: any approximate nearest-neighbor index. This is table
> storage and lookup only; indexing belongs to search engines.
>
> h3. Acceptance measurements (will be in the PR)
>
> * Round-trip distortion and downstream parity: cosine similarity of decoded
> vs original vectors, and doccat accuracy delta on a fixed fixture, at 2, 3,
> and 4 bits.
> * Throughput and memory vs the float table.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
