chrevanthreddy opened a new pull request, #19318:
URL: https://github.com/apache/hudi/pull/19318
### Describe the issue this Pull Request addresses
Resolves #19098 (umbrella: #19094, RFC-109).
Third RFC-109 slice: an engine-neutral RaBitQ encoder and metric-aware,
multibit scorer contract.
> **Stacked PR.** This branch includes #19317 (MDT schema/payload, #19097)
and #19310 (options, #19096). The top commit, `feat(index): add RaBitQ encoder
and multibit-aware scorer contract`, is the unique change for this PR.
### Summary and Changelog
- **`RaBitQEncoder` / `QuantizedVector`**: deterministic 1-8 bit encoding,
residual encoding, packed sign/extended codes, validated immutable encoded
payloads, and finite-input checks.
- **Metric scoring**: explicit `MetricQueryState` ranking contracts for L2
(squared ranking distance), cosine, and dot product; validated query/rotation
state; separate `RaBitQDistanceScorer` for packed-code reconstruction.
- **Posting scan primitives**: `RaBitQByteLutScorer`, `RaBitQPlaneKernel`,
and `VectorQueryPlanes`, including persisted-plane ordering and posting-layout
validation.
- **Neutral factors**: safe pass-1 bounds, normalized quality gating, exact
D=1 handling, and posting-block-owned factor-layout versioning.
- **Conflict resolution**: preserves #19310's `VectorQuantizer` option enum;
the old speculative runtime interface is replaced by concrete
encoder/query/value types.
- **Tests**: 67 encoder/scorer tests in this slice; 82 focused stacked tests
pass overall. Coverage includes deterministic encoding, all metrics, bits
1/2/4/8, byte-LUT vs scalar identity through persisted posting planes,
non-byte-aligned dimensions, buffer offsets, malformed layouts/inputs,
neutral-factor bounds, rotation, residual recall, and metadata/options
integration.
### Validation
Java 17 reactor validation:
```text
mvn -Punit-tests -Djava17 -Djava.version=17 -pl hudi-common -am \
-Dtest='TestVectorIndexOptions,TestRaBitQEncoder,TestRaBitQByteLutScorer,MetricEstimatorIdentityTest,TestMetricQueryStateRotation,TestRaBitQNeutralFactors,TestRaBitQPlaneKernel,TestRaBitQResidualHypothesis,TestRaBitQResidualRecall,TestVectorDistanceMetric,TestVectorIndexMetadataKey,TestVectorIndexMetadataPayload'
\
-Dsurefire.failIfNoSpecifiedTests=false test
```
Result: 82 tests, 0 failures/errors/skips; Checkstyle 0 violations; Apache
RAT passed; residual multibit Recall@10 = 0.918.
### Impact
Additive classes under `hudi-common`; no write/read runtime path is wired in
this PR. Bootstrap, lifecycle/publication, incremental maintenance, and query
integration remain follow-up slices.
### Risk Level
Medium: isolated from existing runtime paths, but this establishes
persisted-code and numeric contracts used by later slices.
### Documentation Update
Contracts and multibit decomposition are documented in Javadoc. The
user-facing RFC is tracked in #19309.
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact described above
- [x] Necessary doc changes done or have another open PR
- [x] Large work is split under umbrella issue #19094
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
