xiangfu0 opened a new pull request, #19223:
URL: https://github.com/apache/pinot/pull/19223
## What this PR does
Adds opt-in, experimental Apache Fory JSON support without changing the
behavior or implementation selected by existing Pinot JSON functions.
- Adds `jsonExtractScalarFory(json, path, resultType[, default])` for
query-time typed extraction.
- Adds `jsonPathStringFory`, `jsonPathLongFory`, and `jsonPathDoubleFory`
for ingestion transforms.
- Uses a per-thread Fory streaming codec for eligible query-side simple
paths.
- Uses a shared dynamic-tree parser plus direct path navigation for
ingestion functions.
- Falls back to the existing Jackson/Jayway implementation for unsupported
paths and values, precision-sensitive results, BYTES input, deeply nested or
malformed documents, runtime/linkage failures, and other unsupported cases.
- Registers the functions with the parser, transform factory, planner
fixtures, and UDF snapshot.
- Adds release LICENSE/NOTICE attribution for Fory and code embedded in its
published fat JAR.
- Extends the JSON JMH coverage with end-to-end ValueBlock benchmarks.
All four public names are documented as experimental and may change or be
removed while the integration is evaluated.
## Why
Fory's generated JSON parser benchmarks are promising, but Pinot's relevant
workload is dynamic JSON path extraction rather than fixed-schema POJO binding.
This PR provides an isolated surface for evaluating that workload in real Pinot
query and ingestion paths while preserving the current implementation as the
compatibility fallback.
## Local benchmark results
Exploratory late-field query throughput relative to `jsonExtractScalarFast`:
| JSON document | 1 thread | 8 threads |
| --- | ---: | ---: |
| 700 B | 1.11x | 1.60x |
| 8 KiB | 1.52x | 2.41x |
| 64 KiB | 3.53x | 4.72x |
These are local zero-fork JMH measurements and should be treated as
directional. Fory's current released string-skip path trades throughput for
allocation: approximately 1.65x, 12x, and 84x the allocation of Fast at 700 B,
8 KiB, and 64 KiB respectively. An upstream zero-allocation fast string-skip
API would materially improve this tradeoff.
## Validation
- Fory/common suite: 64 tests, 0 failures.
- Core registry, ingestion evaluator, and typed transform suites: 266 tests,
0 failures.
- `QueryCompilationTest`: 229 tests, 0 failures.
- `QueryRunnerTest`: 135 tests, 0 failures.
- Spotless, license formatting, checkstyle, and license validation passed
for:
- `pinot-common`
- `pinot-core`
- `pinot-integration-tests`
- `pinot-perf`
- `pinot-query-planner`
- `pinot-query-runtime`
- `git diff --check` is clean.
The real `JsonPathTest` integration cluster and full binary-distribution
build were not run locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
