yashmayya opened a new pull request, #18941: URL: https://github.com/apache/pinot/pull/18941
Follow-up to #18237 and #18924; closes out the rollout planned in https://github.com/apache/pinot/issues/18201. ## What Flips `pinot.broker.multistage.logical.planner.use.broker.pruning` (`DEFAULT_LOGICAL_PLANNER_USE_BROKER_PRUNING`) from `false` to `true`, so broker-side segment pruning is enabled by default on the MSE **logical planner** path — matching the physical optimizer path (`pinot.broker.multistage.use.broker.pruning`), which has defaulted to `true` since it was introduced. The flag itself is unchanged and can still be set to `false` cluster-wide, and `SET useBrokerPruning=false` still disables pruning per query. ## Why now The logical-planner flag was introduced in #18237 with a `false` default explicitly so the rollout could proceed independently, path by path. That rollout is complete: * Non-partitioned leaf path: #18237 * Partitioned leaf path + logical tables: #18924 * All-segments-pruned short-circuit: #18538 With every leaf path supported, the `false` default now just means MSE queries fan out to servers that provably hold no matching segments — the motivating problem of #18201 (a single unhealthy server degrading nearly all queries on a tenant). ## Scope / risk * **Pruning only activates on tables with segment pruners configured** (`routing.segmentPrunerTypes` etc.). Those tables already get identical pruner behavior on the single-stage engine today; this brings MSE routing in line with SSE routing for them. Tables without pruners get the same segments back from the filtered routing query — results and routing unchanged. * The pruning implementation is fail-open throughout (unsupported plan shapes, routing failures, all-pruned, pre-partitioned/colocated-join leaves), so the residual failure mode is unpruned-but-correct routing. * **Observable changes** on pruner-enabled tables: `numServersQueried`/`numSegmentsQueried` drop, `numSegmentsPrunedByBroker` is now populated, and filters matching zero segments short-circuit at the broker (#18538). Worth a release-note callout since dashboards keyed on these stats will shift. * Broker-local planning behavior only — no wire format or mixed-version concerns. * **Validation basis**: the physical-optimizer path has run with pruning default-enabled since its introduction; the segment pruners themselves are the same ones SSE has always used at the same spot in routing; and the logical-planner implementation is covered by unit tests for every fallback (disabled, unsupported shape, routing failure, all-pruned, unavailable segments, pre-partitioned gate) plus end-to-end integration tests on a real partitioned realtime table and logical tables. **Release-note callout**: MSE (default planner) now performs broker-side segment pruning by default on tables with segment pruners configured. Disable cluster-wide with `pinot.broker.multistage.logical.planner.use.broker.pruning=false` or per query with `SET useBrokerPruning=false`. ## Testing * `WorkerManagerTest`: default-on is now asserted directly (filtered routing query without any `SET`; partitioned leaf prunes without any `SET`), and the disabled path stays covered via explicit `SET useBrokerPruning=false` variants. * Integration tests added in #18924 that used a bare query as the "unpruned" baseline now disable pruning explicitly, and the partitioned-table test doubles as an end-to-end regression test of the new default (a plain query must report `numSegmentsPrunedByBroker > 0`). * Three `EXPLAIN IMPLEMENTATION PLAN` snapshots in `ExplainPhysicalPlans.json` are updated: in the planner-test fixture, the partitioned tables' segments all live in the first partitions, so with pruning on by default the *filtered, shuffle-feeding* leaf drops its segment-less partitions (4 workers → 1). The diffs are a nice demonstration of the safety gates: the unfiltered scan, the join-bearing leaf (unsupported shape → fail-open), and the `[PARTITIONED]` direct-exchange chain are all untouched; only shuffle/broadcast-feeding filtered leaves compact. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
