vladimirg-db opened a new pull request, #58710:
URL: https://github.com/apache/spark/pull/58710
### What changes were proposed in this pull request?
This change makes `ExtractGenerator` preserve SELECT-list order when it
extracts independent
generator expressions. A generator may still be extracted ahead of an
earlier generator when the
earlier generator has a direct or transitive lateral-column-alias dependency
on a generator to its
right.
The internal configuration `spark.sql.generator.preserveSelectListOrder`
defaults to `true` for the
new behavior. Setting it to `false` restores the previous first-ready
extraction behavior.
### Why are the changes needed?
Generator extraction currently happens over multiple analyzer iterations. If
a later generator's
children resolve before an earlier generator's children, the later generator
is extracted first.
That changes the nesting and evaluation order of the resulting `Generate`
operators. The difference
is observable when generator inputs contain nondeterministic expressions or
UDFs and can change query
results.
### Does this PR introduce _any_ user-facing change?
Yes. Independent generators are now evaluated in SELECT-list order by
default. For example, with an
identity UDF registered as `udf`:
```sql
SELECT
explode(array(sin(0) + udf(monotonically_increasing_id()))) AS left_value,
explode(array(10, 20)) AS right_value;
```
The new behavior returns `(0.0, 10)` and `(0.0, 20)`. The previous behavior
could return
`(0.0, 10)` and `(1.0, 20)` because the right generator was extracted first.
Setting
`spark.sql.generator.preserveSelectListOrder=false` restores the previous
behavior.
### How was this patch tested?
Added SQL golden tests for both configuration values. They cover Scala and
Python UDFs,
nondeterministic expressions, multiple generators, grouping analytics,
direct and transitive
lateral-column-alias dependencies, and a case combining all of them.
```bash
build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z
generators-resolution-edge-cases.sql"
build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite -- -z
generator-ordering"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
