WangGuangxin opened a new pull request, #58329:
URL: https://github.com/apache/spark/pull/58329
### What changes were proposed in this pull request?
This PR adds code generation support for `array_sort` when it uses the
default comparator.
The default comparator path is recognized by rebuilding the default
comparator with the analyzed
lambda variables and comparing the canonicalized lambda expression. When
the comparator is not the
default one, `array_sort` keeps using the existing interpreted fallback
path.
This PR also adds:
- A DataFrame API test that runs default `array_sort` under `CODEGEN_ONLY`.
- A benchmark case for default `array_sort` in
`HigherOrderFunctionsBenchmark`.
### Why are the changes needed?
`array_sort(array)` currently supports whole-stage codegen around the
query, but the expression
itself still falls back to interpreted evaluation. The default comparator
is a common case and has
fixed semantics, so it can be code-generated without changing behavior or
supporting arbitrary
custom comparator codegen.
This avoids per-row interpreted expression evaluation overhead for the
default comparator path.
Local benchmark result on Apple M3 Pro, JDK 17:
```
[info] Running benchmark: array_sort
[info] Running case: codegen
[info] Stopped after 10 iterations, 34935 ms
[info] Running case: interpreted
[info] Stopped after 10 iterations, 46261 ms
[info] OpenJDK 64-Bit Server VM 17.0.15+0 on Mac OS X 15.6.1
[info] Apple M3 Pro
[info] array_sort: Best Time(ms) Avg
Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
[info]
------------------------------------------------------------------------------------------------------------------------
[info] codegen 3433
3494 55 2.9 343.3 1.0X
[info] interpreted 4572
4626 63 2.2 457.2 0.8X
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added unit test coverage for the default `array_sort` DataFrame API path
under `CODEGEN_ONLY`.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: TraeCode (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
