yashmayya opened a new pull request, #19771:
URL: https://github.com/apache/pinot/pull/19771
This PR depends on #19769, which depends on #19767 and #19766. It contains
their commits. Only the last commit is new.
## The bug
With `explainAskingServers` (or the broker config
`pinot.query.multistage.explain.include.segment.plan`), the broker gets the
plan of each leaf stage from the servers. Then `PlanNodeToRelConverter`
converts the plan nodes back to Calcite nodes.
`visitUnnest` converts an `UnnestNode` to an `Uncollect` over a `Project` of
the arrays. The output of this `Uncollect` has only the element (and
ordinality) columns. But for CROSS JOIN UNNEST, the `UnnestNode` output has the
input columns first, then the element columns. So a parent node in the leaf
stage cannot find its input fields, and EXPLAIN shows it as `UnknownAggregate`
or `UnknownProject`.
For example:
```sql
SELECT Carrier, u.a, COUNT(*)
FROM mytable CROSS JOIN UNNEST(DivAirports) AS u(a)
GROUP BY Carrier, u.a
```
EXPLAIN shows the leaf stage as:
```
PinotLogicalExchange(distribution=[hash[0, 1]])
UnknownAggregate
Uncollect
LogicalProject(DivAirports=[$1])
LeafStageCombineOperator(table=[mytable])
```
The broker logs `IllegalArgumentException: field ordinal [2] out of range;
input fields are: [DivAirports]`. These queries have the same problem: GROUP BY
only the element column, WITH ORDINALITY, ROLLUP, `SELECT Carrier, u.a ...
LIMIT 10`, and queries with `unnestColumnPruning=true`.
## The fix
If the `UnnestNode` output has input columns, `visitUnnest` now builds the
same shape as the logical plan. This shape is a `LogicalCorrelate` of the input
with an `Uncollect` that reads the arrays through the correlation variable. If
the planner pruned some input columns (`unnestColumnPruning`), a `Project`
keeps only the remaining input columns. A plain UNNEST without input columns
does not change.
Only the EXPLAIN output changes. Query execution does not use this
conversion.
## Testing
- `PlanNodeToRelConverterTest.testAggregateOverUnnest` converts an aggregate
over CROSS JOIN UNNEST. `testProjectOverPrunedUnnest` converts a project over a
pruned UNNEST WITH ORDINALITY. Without this change, they fail with
`UnknownAggregate` and `UnknownProject`.
- I ran EXPLAIN for 7 such queries in a temporary integration test with the
broker config above. Before this change, all of them showed an `Unknown` node.
After it, none do, and the leaf stages match the logical plan shape. For the
query above:
```
PinotLogicalExchange(distribution=[hash[0, 1]])
LogicalAggregate(group=[{0, 2}], agg#0=[COUNT()])
LogicalCorrelate(correlation=[$cor0], joinType=[inner],
requiredColumns=[{1}])
LeafStageCombineOperator(table=[mytable])
Uncollect
LogicalProject(DivAirports=[$cor0.DivAirports])
LogicalValues(tuples=[[{ 0 }]])
```
- All 2662 `pinot-query-planner` tests pass. `spotless`, `checkstyle` and
`license` are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
