gortiz opened a new pull request, #19575:
URL: https://github.com/apache/pinot/pull/19575
Slice 5 of 5 of the cost-based-optimizer foundation, stacked on #19441.
Part of #18740. **Draft until its parents merge**: GitHub diffs a stacked PR
against master, so this shows its parents' commits too.
## What this adds
A **gated, default-off** join-reordering phase for the multi-stage planner,
plus a way to ask why it did or did not fire.
The phase runs after the logical Hep program and before the trait phase,
using Calcite's `JOIN_TO_MULTI_JOIN` → `MULTI_JOIN_OPTIMIZE` pairing so
orderings are compared on the statistics-backed row counts slice 4 made
available.
| Option | Default | Meaning |
|---|---|---|
| `useJoinReorder` | off | run the phase |
| `joinReorderMaxJoins` | 10 | bound planning time |
| `joinReorderFeedback` | off | attach a `joinReorder` entry to
`responseMetadata` |
Broker-side equivalents exist for the first two.
## Eligibility gates
The phase returns the plan untouched, with a reason, unless all of these
hold:
- at least two joins, and no more than `joinReorderMaxJoins`
- every join is INNER
- no join carries a Pinot `joinOptions` hint
- no join carries correlation state — `MultiJoin` has no `variablesSet`
component, so folding a correlated join in would silently discard it. Note a
`Correlate` itself does *not* disqualify: it is a `BiRel`, never matched by
`JoinToMultiJoinRule`, so it stays an opaque factor and its binder travels with
the input it binds. `CROSS JOIN UNNEST` keeps working.
- every **leaf** is a `TableScan` with a row count usable for costing, via
`PinotTable#getUsableRowCount()`. The gate is on leaves rather than on scans
because a non-scan leaf (`Uncollect`, `Values`, a table function) would be
costed from a Calcite default — the mixed known/guessed case this must not
reorder on.
`maybeReorder` never throws; any unexpected error logs a WARN and returns
the original plan.
## Observability
With `joinReorderFeedback=true` the response carries, for that query alone:
```json
{"joinReorder": {"outcome": "APPLIED", "numJoins": 2, "planChanged": true,
"estimatedCostBefore": 1000.0, "estimatedCostAfter": 250.0,
"reorderTimeMs": 3}}
```
or `{"outcome": "SKIPPED", "reason": "TOO_FEW_JOINS", "numJoins": 1}`. The
costs are cumulative over the subtree, deliberately not the root's row count —
reordering changes the size of the *intermediate* joins, which is exactly what
the root hides (a `COUNT(*)` root is 1 row whatever happens underneath).
The diagnostics are only computed when feedback is requested; each is a
full-tree metadata walk.
## Known limitations
1. **Hints veto the whole plan, not one join.** Any `joinOptions` hint
anywhere disables reordering entirely. A per-join veto needs a design decision
first: reordering *moves* joins, so a hint attached to a particular `Join` may
describe a join that does not exist in the reordered tree — and `MultiJoin` is
`final` and not `Hintable`, so hints cannot ride through it. Follow-up.
2. **`joinReorderFeedback` is silent when `useJoinReorder` is off** (the
default), because the entry is attached by the phase. An operator asking "why
was my query not reordered?" on a default-configured broker gets an empty
`responseMetadata`. Considering a `SKIPPED` / `DISABLED` entry as a follow-up.
## Testing
`JoinReorderOptimizerTest` (27 tests) covers each gate by its reported
`SkipReason` rather than by plan-string equality — an unchanged plan is also
what a phase that ran and picked the same order looks like. Default-off is
asserted through the feedback channel, since that is the only signal that
distinguishes "did not run" from "ran and changed nothing".
`pinot-query-planner` 1977, `pinot-common` 2302, `pinot-broker` 635 — all
green on this base.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
