LuciferYang opened a new issue, #12845:
URL: https://github.com/apache/gluten/issues/12845
### Description
`ClickHouseAdaptiveQueryExecSuite` carries a Gluten rewrite of `SPARK-37652:
optimize skewed join through union`. It was migrated from `gluten-ut/spark33`
to the 3.4 / 3.5 / 4.0 / 4.1 modules in #12837, and it is committed there as
`ignoreGluten` because it fails on ClickHouse 3.5.
The immediate failure is that `findTopLevelSortMergeJoinTransform` collects
zero joins where the case expects two:
```
- Gluten - SPARK-37652: optimize skewed join through union *** FAILED ***
List() had size 0 instead of expected size 2
(ClickHouseAdaptiveQueryExecSuite.scala:1597)
```
This is a type mismatch rather than a plan-shape difference. The helper
matches `org.apache.gluten.execution.SortMergeJoinExecTransformer`, which is
the concrete Velox case class (`SortMergeJoinExecTransformer.scala:230`).
ClickHouse plans these joins as `CHSortMergeJoinExecTransformer` or
`CHShuffledHashJoinExecTransformer`, which extend
`SortMergeJoinExecTransformerBase` (`SortMergeJoinExecTransformer.scala:37`)
and `ShuffledHashJoinExecTransformerBase` respectively, and neither is a
subtype of the Velox class. So on ClickHouse the helper returns an empty result
for any query.
A backend-neutral type is available: `ColumnarShuffledJoin`
(`JoinExecTransformer.scala:42`) declares `isSkewJoin` and is mixed into both
`MergeJoinLikeExecTransformer` and `ShuffledHashJoinExecTransformerBase`, so
`collect(plan) { case j: ColumnarShuffledJoin => j }` covers both backends and
both join shapes.
That alone may not be enough to re-enable the case. The expected counts
(2/2, 1/1, 3/0) are copied verbatim from spark33 and have never been checked
against ClickHouse 3.5, so someone with a ClickHouse build needs to confirm or
correct them.
Two related observations for whoever picks this up:
`findTopLevelSortMergeJoinTransform` has eight other call sites in the same
suite, and they are all subject to the same mismatch on ClickHouse.
The neighbouring skew-join case in that suite already has its
`checkSkewJoin(inner, 2, 1)` calls commented out
(`ClickHouseAdaptiveQueryExecSuite.scala:707`), which suggests this area was
parked earlier for what may be the same reason.
Fixing the helper touches shared test infrastructure and is out of scope for
#12837, which is a pure test migration.
This description was written with AI assistance (Claude Code).
### Gluten version
main branch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
