gortiz opened a new pull request, #19409:
URL: https://github.com/apache/pinot/pull/19409
Contributes to #18740 (umbrella: cost-based optimization for the multi-stage
query engine).
Together with the two PRs stacked on it, this replaces #18741, which carried
all of Phase 1 as one
7.2k-line change that two reviewers independently asked to see split up.
**Nothing in this PR is reachable from a query.** It defines what a
statistic is and where one is
stored. Nothing produces or consumes a statistic until the next PR.
```text
┌──────────────────┐ ┌──────────────────┐ ┌──────────────────┐
┌──────────────┐ ┌──────────────┐
│ 1 — this PR │──▶│ 2 — collection │──▶│ 3 — selection │──▶│ 4 —
planner │──▶│ 5 — join │
│ contracts and │ │ read segment ZK │ │ store by name + │ │
wiring │ │ reordering │
│ the two stores │ │ metadata │ │ purge endpoint │ │
│ │ │
└──────────────────┘ └──────────────────┘ └──────────────────┘
└──────────────┘ └──────────────┘
```
## What a statistic is
`TableStatistics` and `ColumnStatistics` are the values; `StatConfidence` is
the trust tier carried
by each one. Confidence is attached **per statistic**, not per table, so a
source that can only
estimate some values does not devalue the rest. Consumers are expected to
treat a low-confidence
statistic as *absent* rather than trusting it — that is what will keep table
types with biased raw
counts (upsert, dedup, consuming segments) on today's behavior instead of
producing confidently
wrong plans once a consumer exists.
`StatsAggregations` holds the rollup semantics every store must share, so
two implementations
cannot disagree about what the same stored rows mean: time-overlap
interpolation, document-weighted
averages that exclude the "unknown" sentinel rather than averaging it in as
a measurement, and
min/max folded under the ordering the column actually has.
That ordering is **recorded per row** as a `ColumnValueType` rather than
guessed from the text,
because guessing is wrong in both directions: a string column holding `"9"`
and `"10"` orders
lexically in Pinot but would compare numerically, and a long beyond 2^53
loses digits as a double —
in the direction that *narrows* the range, which would exclude rows that
exist. When the ordering is
unknown, or segments disagree about it, both bounds are reported as absent:
a bound folded under two
different orderings is neither a true minimum nor a true maximum, and there
is no honest way to
describe it as merely untrusted.
## Where statistics are stored
The default is an embedded SQLite database — WAL, a small read-connection
pool, drop-and-rebuild
recovery if the file is unreadable. The reason is **heap, not durability**:
statistics for hundreds
of thousands of segments must not compete with query execution for JVM heap.
Surviving a restart is
a side benefit; the file also lets the broker skip re-collecting segments
whose crc has not changed.
There is no migration framework. SQLite's own `user_version` pragma carries
the schema version, and
a store whose version does not match is **discarded and rebuilt** rather
than migrated. Every row is
derived from ZooKeeper metadata the broker re-reads at startup, so a rebuild
costs nothing that was
not already being read — and it puts schema change, corruption and an
unreadable file on one
recovery path.
`InMemoryStatsStore` is for brokers that cannot or should not write a file.
Both serve table-level
reads from a rollup recomputed only after a write, since a planner asks for
these on every compile
while writes arrive at segment-push cadence. A version stamp is what makes
that cache safe without
holding the write lock: a rollup computed from rows that changed underneath
it is used for that call
but never published.
## Reviewer notes
- **One new dependency**, broker-only: `org.xerial:sqlite-jdbc` (13.5 MB,
bundling native libraries
for 17 platform/arch combinations). `LICENSE-binary` updated — it is
dual-licensed Apache-2.0 and
BSD-2-Clause, and both are recorded.
- **New SPI package** `org.apache.pinot.query.planner.spi.stats`, which
`PluginManager` already
exports to plugin realms.
- **Nothing calls this code yet.** That is the point of the split: the
vocabulary and the storage
land before anything depends on them, so the SPI shape can be argued about
on its own.
- No wire format, no serialization change, no mixed-version concern:
everything here is
broker-local.
## Testing
- A shared `StatsStoreContractTest` (26 cases) runs against **both** stores,
because the optimizer
must not behave differently depending on which store an operator
configured. Each implementation
adds only what is specific to it: durability and corruption recovery for
SQLite, starting-empty
for in-memory.
- `StatsAggregationsTest` and `ColumnValueTypeTest` cover the rollup
semantics in the module that
owns them: averages that exclude the unknown sentinel, bounds dropped when
the ordering is unknown
or segments disagree, and every ordering path including the
malformed-value fallbacks — which must
degrade rather than throw, since they will sit on the query-planning path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
