Chao Sun created SPARK-60766:
--------------------------------
Summary: Prune nested fields through dynamic array indices and Map
keys
Key: SPARK-60766
URL: https://issues.apache.org/jira/browse/SPARK-60766
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
Nested schema pruning currently stops at GetMapValue/GetArrayItem/ElementAt
when the key or index cannot be evaluated as a constant. This preserves
correctness, but makes queries such as m[key_column].id and
items[index_column].id request the complete value/element Struct, including
unused payload fields.
A reproducible Parquet example in spark-shell:
{code:scala}
spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", "true")
val path = "/tmp/spark-dynamic-selector-pruning"
spark.sql("""SELECT 'chosen' AS lookup_key,
map('chosen', named_struct('id', 42, 'payload', 'unused')) AS m""")
.write.mode("overwrite").parquet(path)
val input = spark.read.parquet(path)
input.selectExpr("m['chosen'].id").explain()
input.selectExpr("m[lookup_key].id").explain()
{code}
Both queries return 42. The constant lookup can request only id from the Map
values, while the dynamic lookup requests both id and payload, plus lookup_key.
The desired dynamic read schema is
struct<lookup_key:string,m:map<string,struct<id:int>>>.
SPARK-42163 introduced the conservative restriction because the selector itself
may reference a nested field, and pruning must not discard that input. The
proposed change extends the existing additional-dependency traversal to collect
every key/index input while SelectedField follows the collection input.
ProjectionOverSchema also needs to rebind dynamic element_at expressions after
narrowing.
Scope: dynamic bracket Map/array access and element_at over scan-rooted
collections. Preserve selector dependencies even when they refer to the same
collection, occur beneath null checks, or interact with supported array
functions. Preserve field metadata, field ordinals, null results and ANSI/index
error behavior. No new settings, dependencies, or public API.
Validation should cover Catalyst schema/ordinal behavior and actual Parquet
read schemas/results across V1/V2 and vectorized/row readers. The existing
SPARK-42163 regression should require a narrower schema. Include renamed
field-ID paths and selector dependencies under IsNull/IsNotNull. This is a
missed optimization, not a reported end-to-end speedup.
Related: SPARK-42163 (original safety restriction), SPARK-40033 (element_at
pruning), SPARK-58772 (metadata preservation, already present on master).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
