lgbo-ustc opened a new pull request, #13235:
URL: https://github.com/apache/gluten/pull/13235
CH window top-k aggregate optimization can fail when the window partition
keys and order keys are exactly the same expressions.
## What changes are proposed in this pull request?
Fixes #13234.
For queries like:
```sql
ROW_NUMBER() OVER (
PARTITION BY l_returnflag
ORDER BY l_returnflag DESC
)
```
the partition key and order key are identical. This shape is not suitable
for the CH window-to-aggregate-topk optimization. When the aggregate top-k path
is forced, native execution may cast the generated `rn` column from a wrong
source column and fail with errors such as:
```text
Cannot convert NULL value to non-Nullable type: while converting source
column `rn#...` to destination column `rn#...`
```
or with a string-to-Int32 cast error when the misplaced source column
contains a non-null partition key value.
This patch updates the CH window-to-aggregate-topk rule to skip the
aggregate top-k rewrite when the partition keys and order keys are exactly the
same expressions. The query will keep the regular window/filter execution path
instead.
A regression test is added to `GlutenClickHouseTPCHSaltNullParquetSuite`.
The test forces the aggregate-topk optimization to be considered and verifies
that this same-key case does not produce `CHAggregateGroupLimitExecTransformer`.
## How was this patch tested?
- Added `GlutenClickHouseTPCHSaltNullParquetSuite` coverage for `row_number`
with identical partition/order keys.
- Locally reproduced the failure by forcing:
- `spark.sql.optimizer.windowGroupLimitThreshold=-1`
-
`spark.gluten.sql.columnar.backend.ch.runtime_settings.enable_window_group_limit_to_aggregate=true`
-
`spark.gluten.sql.columnar.backend.ch.runtime_settings.window.aggregate_topk_high_cardinality_threshold=2.0`
- Verified after the fix that the plan no longer contains
`CHAggregateGroupLimitExecTransformer` for the same-key case and the query
executes successfully.
- Locally built with:
```bash
./build/mvn -pl backends-clickhouse -am \
-DskipTests \
-Pbackends-clickhouse \
-Pspark-3.5 \
-Pscala-2.13 \
-Pceleborn \
-Pdelta \
-Djava.io.tmpdir=/home/liangjiabiao/w21/tmp \
package
```
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: DeepSeek Harness gpt-5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
