Eyck Troschke created SPARK-58735:
-------------------------------------
Summary: "size(array)" forces full read of nested array columns
and prevents column pruning for Parquet/Delta arrays
Key: SPARK-58735
URL: https://issues.apache.org/jira/browse/SPARK-58735
Project: Spark
Issue Type: Improvement
Components: Optimizer
Affects Versions: 4.1.2
Environment: Apache Spark 4.1.2
Databricks Runtime (Photon)
Reporter: Eyck Troschke
When applying `size()` to a nested array column stored in Parquet/Delta, Spark
reads the *entire* array column including all child struct fields, even though
only array offsets / repetition levels are required to compute the array
length. This results in significantly higher I/O compared to semantically
equivalent queries using `explode()` or accessing smaller sub‑arrays.
This behavior occurs in both Apache Spark and Databricks runtimes. It appears
to be caused by conservative Catalyst planning for nested arrays, which
prevents column pruning and page skipping.
*Example:*
Table size: ~100GB, ~18B rows
Column: `events` (ARRAY<STRUCT<...>>)
Sub‑column: `events.flag_a` (small ARRAY)
```
select sum(size(events)); -- 49s, ~100GB read
select sum(size(events.flag_a)); -- 8s, ~6GB read
select count(*) from t lateral view explode(events); -- 9s, ~10GB read
select count(*) from t lateral view explode(events.flag_a); -- 8s, ~6GB read
```
Although these queries are semantically equivalent, `size(array)` triggers a
full scan of all nested array fields, while `explode(array)` allows column
pruning.
*Expected Behavior:*
`size(array)` should only require reading array offsets / repetition levels and
should not force full read of all nested struct fields.
*Actual Behavior:*
Spark reads all nested array child columns, causing large and unnecessary I/O.
*Impact:*
Severe performance degradation for large nested arrays in Parquet/Delta tables.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
