surendralilhore opened a new pull request, #12095:
URL: https://github.com/apache/gluten/pull/12095
Detect Spark's default null-handling comparator in ArraySort and strip the
lambda so Velox uses its 1-arg array_sort (ascending, nulls-last) which has
identical semantics.
## What changes were proposed in this pull request?
When Spark's array_sort is called without an explicit comparator (the
default ascending sort), Spark internally
generates a LambdaFunction that wraps the simple ascending comparison in
null-handling If/IsNull logic. Velox's
rewriteArraySortCall uses a SimpleComparisonMatcher that cannot parse this
null-handling wrapper, causing the
expression to fall back to Vanilla Spark.
This PR detects whether the ArraySort expression uses Spark's default
comparator by reconstructing it via
ArraySort.comparator() and comparing semantically. When a match is found,
the lambda is stripped and only the array
argument is passed, so Velox uses its native 1-arg array_sort which already
provides ascending sort with nulls-last —
identical to Spark's default semantics.
Custom comparators (e.g., descending sort) continue to be passed through as
the 2-arg form.
## How was this patch tested?
Added "array_sort - default comparator offloading" test in MiscOperatorSuite
covering:
- array_sort(a) — default ascending sort
- sort_array(a) — which uses the default comparator internally
- sort_array(a, false) — descending sort
- array_sort(a, (l, r) -> ...) — custom comparator (2-arg form preserved)
All cases verify that ProjectExecTransformer is present in the Gluten plan.
Was this patch authored or co-authored using generative AI tooling?
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude claude-opus-4.6
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
