hemanthboyina opened a new pull request, #58456: URL: https://github.com/apache/spark/pull/58456
**What changes were proposed in this pull request?**
This adds a new optimizer rule, RewriteSizeOfArrayStruct.
When a query calls size() (or array_size()) on a column that is an array
of structs, the rule rewrites it to take the size of a single field of that
struct instead of the whole array. For example,
size(events) becomes size of events with only one inner field selected. It
always picks the smallest field of the struct, following the same approach the
existing GenerateOptimization rule already uses for
the explode + count case.
The rewrite runs in the operator optimization batch, just before column
pruning, and only when nested schema pruning is enabled.
**Why are the changes needed?**
size() only needs to know how many elements are in the array. It never
looks at the values inside the struct. But today, when you write size() over a
whole array-of-struct column, Spark treats it as "the
entire column is needed," so nested column pruning and the Parquet/ORC
schema pruning cannot kick in. As a result, every field of the struct is read
from disk, even though none of them are used.
The nested pruning machinery only prunes when it sees a field being
accessed in the plan. A plain column reference has no field access, so it is
read in full. By rewriting the expression to access one small
field, we give the pruner the signal it needs, and it reads just that one
field.
**Does this PR introduce any user-facing change?**
No. The query results are unchanged; only the amount of data read from
disk is reduced.
**How was this patch tested?**
Added unit tests in NestedColumnAliasingSuite that check the rewritten
plan reads only a single nested field, and that arrays of primitives and
already-pruned expressions are left untouched.
**Was this patch authored or co-authored using generative AI tooling?**
Generated-by: Yes, used Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
