[ https://issues.apache.org/jira/browse/SPARK-37450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
L. C. Hsieh reassigned SPARK-37450: ----------------------------------- Assignee: L. C. Hsieh > Spark SQL reads unnecessary nested fields (another type of pruning case) > ------------------------------------------------------------------------ > > Key: SPARK-37450 > URL: https://issues.apache.org/jira/browse/SPARK-37450 > Project: Spark > Issue Type: Improvement > Components: SQL > Affects Versions: 3.2.0 > Reporter: Jiri Humpolicek > Assignee: L. C. Hsieh > Priority: Major > > Based on this [SPARK-34638|https://issues.apache.org/jira/browse/SPARK-34638] > Maybe I found another nested fields pruning case. In this case I found full > read with `count` function > Example: > 1) Loading data > {code:scala} > val jsonStr = """{ > "items": [ > {"itemId": 1, "itemData": "a"}, > {"itemId": 2, "itemData": "b"} > ] > }""" > val df = spark.read.json(Seq(jsonStr).toDS) > df.write.format("parquet").mode("overwrite").saveAsTable("persisted") > {code} > 2) read query with explain > {code:scala} > val read = spark.table("persisted") > spark.conf.set("spark.sql.optimizer.nestedSchemaPruning.enabled", true) > read.select(explode($"items").as('item)).select(count(lit(true))).explain(true) > // ReadSchema: struct<items:array<struct<itemData:string,itemId:bigint>>> > {code} -- This message was sent by Atlassian Jira (v8.20.1#820001) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org