gortiz opened a new pull request, #20092: URL: https://github.com/apache/pinot/pull/20092
## Summary A lookup join now reads the dimension table by the table column names. Before this change, it used the output names of the right input, and these names can be aliases. The fix follows each column of the dimension-side Project to the TableScan column that it references. ## Background The planner limits the right input of a lookup join to an `InputRef`-only Project over a TableScan (`RelToPlanNodeConverter.convertLogicalJoin`). Aliases are still legal in that Project. `LookupJoinOperator` took the column names from the right input `DataSchema`. It used them for two things: - `compileKeyPlan` matches the join keys against the primary key columns by name. - `DimensionTableDataManager.lookupValues(key, columns)` reads the row values by name. With a dimension subquery such as `JOIN (SELECT id AS k, a AS x FROM dim) d`, these names are not table columns. The results on master, for both the v1 and v2 planners: | Dimension subquery | Result on master | |---|---| | `SELECT id AS k, a` (alias on the primary key) | Error: `join key on column: k, which is not a primary key column` | | `SELECT id, a AS x` | `x` is null in every row | | `SELECT id, a, a AS a2` | `a2` is null in every row | | `SELECT id, a AS b, b AS a` | The values of `a` and `b` are swapped | The last three return wrong results with no error. ## Change - `LookupJoinOperator.resolveDimensionColumns` maps each column of the right input to its TableScan column. It accepts a bare TableScan, or a Project of `InputRef`s over a TableScan. Any other shape fails with an error that names the dimension table. - The constructor passes these names to `compileKeyPlan` and to `lookupValues`. - `compileKeyPlan` checks that the number of resolved columns equals the size of the right schema. ## Tests - `LookupJoin.json`: a new `lookup_join_aliased_dimension_columns` test case with the four queries above. It also has two queries that pin the planner errors for a filter and for a computed column in the dimension subquery. All four alias queries fail on master and pass with this change, on the v1 and v2 planners. - `LookupJoinOperatorTest`: unit tests for `resolveDimensionColumns` (TableScan, aliased and repeated columns, computed column, Project over Filter). Also unit tests for `compileKeyPlan` with an alias on a primary key column and with a column count that does not match. - `ResourceBasedQueriesTest` and `LookupJoinOperatorTest` pass (4222 tests). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
