sunchao opened a new pull request, #56226: URL: https://github.com/apache/spark/pull/56226
### Why are the changes needed? [SPARK-57175](https://issues.apache.org/jira/browse/SPARK-57175) follows [SPARK-57022](https://issues.apache.org/jira/browse/SPARK-57022), which added nested column pruning for `transform` over `array<struct>` inputs. The same optimization does not currently apply to the `exists` and `forall` higher-order array functions. For example: ```sql SELECT exists(rule_results, rule -> rule.rule_version > 10) FROM events ``` If `rule_results` contains additional fields, Spark currently retains the full element struct in the scan schema even though the predicate only reads `rule_version`. This causes unnecessary Parquet and ORC input reads for wide array element schemas. ### What changes were proposed in this PR? - Share the nested-field collection path introduced for `ArrayTransform` with `ArrayExists` and `ArrayForAll`. - Rewrite the bound lambda variable type and `GetStructField` ordinals against the projected element schema after pruning. - Keep the conservative fallback when a lambda consumes the whole element. - Add Catalyst and datasource tests covering schema discovery, ordinal rewrites, predicate-path schema merging, and whole-element fallback. `ArrayFilter` and `ArraySort` remain out of scope because they return original input elements and require a different downstream-schema design. ### Does this PR introduce _any_ user-facing change? Yes. Eligible queries using `exists` or `forall` over arrays of structs can read a narrower input schema. Query results and SQL APIs are unchanged. ### How was this patch tested? - `JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.SchemaPruningSuite" "sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.parquet.ParquetV2SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV2SchemaPruningSuite -- -z Array"` - `git diff --check` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Codex (GPT-5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
