xiangfu0 opened a new pull request, #19305:
URL: https://github.com/apache/pinot/pull/19305
Stacked on #19285 (`xiangfu0/codex/codec-stack/02-runtime`). Part of the
split of #18229.
## Summary
Adds the first two transform codecs to the package-private codec runtime in
`org.apache.pinot.segment.local.io.codec`:
- `BaseDeltaCodecDefinition` — shared structure for the delta family. Both
codecs are
`CodecKind.TRANSFORM` stages restricted to `INT`/`LONG` stored types, and
both are
**typed-layout-preserving** (`preservesTypedValueLayout() == true`): they
map `count`
column-typed values to the same `count` same-width values with no
per-frame header, so they can
be chained ahead of another transform (or each other) and `maxEncodedSize`
is the identity.
Two's-complement wrap-around on encode/decode is an intentional,
test-locked contract:
overflow-adjacent sequences round-trip bit-for-bit through the wrapping
`-`/`+` arithmetic.
- `DeltaCodecDefinition` (`DELTA`) — first value verbatim, then successive
deltas.
- `DeltaDeltaCodecDefinition` (`DELTADELTA`) — first value, first delta,
then delta-of-deltas
(good for near-regular timestamp intervals).
Both are registered in `CodecRegistry.DEFAULT`, so specs like `DELTA,LZ4`,
`DELTADELTA,ZSTD(3)`,
and `DELTA,DELTADELTA,LZ4` now parse, validate, and execute through
`CodecPipelineExecutor`.
The names `DELTA`/`DELTADELTA` are frozen on-disk contracts (stored verbatim
in V7 segment
headers by a later PR in the stack) and must never be renamed.
## Tests
- `DeltaCodecRoundTripTest` (new) — executor-driven round trips for `DELTA`,
`DELTADELTA`,
`DELTA,LZ4`, `DELTADELTA,LZ4` over INT and LONG datasets: empty,
single-element, two-element
(DELTADELTA boundary), constant, monotonic, negative/sign-changing, and
overflow-adjacent
(`MIN_VALUE` next to `MAX_VALUE`) sequences; both the allocating decode
path and the bounded
decode-into (segment-reader) path. Also a fail-closed corrupt-input case:
a frame whose byte
length is not a whole number of elements is rejected on both encode and
bounded decode.
- `CodecPipelineValidatorTest` — real-transform scenarios against
`CodecRegistry.DEFAULT`:
valid delta pipelines, typed-layout chaining (including `DELTA,DELTA` and
mixed order),
transform-after-compression rejected, INT/LONG-only type check, argument
rejection, and the
typed-value-layout contract assertions for both codecs.
- `CodecRegistryTest` — DELTA/DELTADELTA now resolve from `DEFAULT`
(case-insensitive); the
parent PR's absence assertion for unregistered codec names is left
untouched.
- `CompressionCodecCorruptInputTest` — the multi-stage inner-frame-guard
test now uses
`DELTA,SNAPPY`, tightening the scratch bound to exactly the four-byte
declared chunk size.
- `TableConfigUtilsTest` — the codecSpec feature gate still rejects a
well-formed
`DELTADELTA,LZ4` config on a supported LONG column with
`"codecSpec is not supported yet for column: longCol"`, proving
registration of the runtime
codecs did not open the config surface.
## Deliberately excluded (carried by later PRs in the stack)
- **T64 and GORILLA packing transforms** — next slice (04). They are the
codecs with
`preservesTypedValueLayout() == false`; until then the packing-placement
rules remain covered
by the synthetic `PACKING` handler in `CodecPipelineValidatorTest`.
- **V7 raw forward index format** (writer/reader that persists the canonical
spec in segment
headers) — slice 05.
- **Opening the feature gate** (config/creator/reader/reload wiring, reload
change-detection via
`CodecPipelineExecutor.getCanonicalSpec()`) — final slice.
## Why master stays safe with only this merged
The codec runtime remains package-private and unreachable from production
paths: the only public
entry point is `CodecPipelineExecutor`, and every config surface still throws
`"codecSpec is not supported yet for column: %s"` (re-asserted in this PR's
`TableConfigUtilsTest`
addition). Registering DELTA/DELTADELTA in the closed registry adds
dead-until-enabled code with
full unit coverage; no segment format, wire protocol, or config behavior
changes.
## Verification
```
./mvnw -q -T 1C install -DskipTests -Ppinot-fastdev -pl pinot-segment-local
-am
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local \
-Dtest=DeltaCodecRoundTripTest,CodecPipelineValidatorTest,CodecRegistryTest,CodecPipelineExecutorTest,CompressionCodecCorruptInputTest
\
-Dsurefire.failIfNoSpecifiedTests=false
./mvnw -q test -Ppinot-fastdev -pl pinot-segment-local
-Dtest=TableConfigUtilsTest \
-Dsurefire.failIfNoSpecifiedTests=false
./mvnw spotless:apply license:format -pl pinot-segment-local
./mvnw checkstyle:check license:check -pl pinot-segment-local
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
