xiangfu0 opened a new pull request, #18757:
URL: https://github.com/apache/pinot/pull/18757
## Description
Adds `FstVsTextSizePerfBenchmarkTest`, a **disabled-by-default**
(`@Test(enabled = false)`) micro-benchmark that empirically compares the
**FST** index against the **TEXT** (Lucene full-text) index — both per-row and
a dictionary-level variant — on a single `STRING` column.
It is gated out of CI the same way `VectorSearchBenchmark` is; flip `enabled
= true` (or run the method from an IDE) to reproduce the numbers, which are
printed to stdout.
### What it measures
Over a 1,000,000-row column across cardinalities `{10, 1k, 100k, 1M}`, for
each cardinality it builds baseline / FST / TEXT segments plus a standalone
Lucene text index over the **distinct values only** (dictionary-level), and
reports:
- **On-disk index size**, isolated as `(segment size − baseline size)` so
the shared dictionary/forward index cancels out.
- **End-to-end latency** = index lookup + `dictId → docId` forward-scan
resolution, with the real FST query-engine path included as a cross-check that
the manual resolution matches production behavior.
A built-in assertion validates that all paths (FST engine, FST manual,
TEXT/dict, TEXT/row) select the same rows, plus an exact-count guard so the
comparison can't pass vacuously on a zero-match regression.
### Findings (for a `%word%` contains query)
| | index size @1M card | e2e latency @1M card |
|---|---:|---:|
| FST | ~0.6 MB (smallest) | ~400 ms (slowest) |
| TEXT/dict | 1–13 MB (scales with cardinality) | ~10 ms |
| TEXT/row | ~13 MB (largest) | ~1 ms (no resolution) |
FST index size scales with cardinality; per-row TEXT scales with row count;
a dictionary-level TEXT index sits in between on size while staying flat and
~40× faster end-to-end than FST for leading-wildcard contains queries (the
FST's worst case).
## Upgrade Notes
None — test-only, disabled by default, does not run in CI.
## Testing
- `spotless:apply`, `license:check`, `checkstyle:check` clean on
`pinot-core`.
- Ran the benchmark locally (temporarily enabled) — all assertions pass.
## Labels
`testing`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
