[
https://issues.apache.org/jira/browse/SPARK-60790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated SPARK-60790:
-----------------------------
Affects Version/s: 4.1.4
4.0.5
3.5.9
4.3.0
4.2.1
(was: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0,
5.0.0)
> Nested schema pruning rejects nonempty column defaults when reading older
> Parquet files
> ---------------------------------------------------------------------------------------
>
> Key: SPARK-60790
> URL: https://issues.apache.org/jira/browse/SPARK-60790
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.9, 4.3.0, 4.2.1, 4.0.5, 4.1.4
> Reporter: Chao Sun
> Priority: Major
>
> h3. Description
> When a column is added with a default, Spark uses that value to read rows
> from files written before the column existed. If the query selects only part
> of a nested column, Spark can attempt to coerce the full default value to the
> pruned type. A default that is valid for the declared column type then fails
> with {{INVALID_DEFAULT_VALUE.DATA_TYPE}}.
> h3. Reproduction
> Run the three SELECT statements separately with nested schema pruning
> enabled. Insert the row before adding the columns so that Spark must read
> their defaults.
> {code:sql}
> CREATE TABLE nested_default_repro (idx INT) USING PARQUET;
> INSERT INTO nested_default_repro VALUES (0);
> ALTER TABLE nested_default_repro ADD COLUMNS (
> arr ARRAY<STRUCT<a:INT,b:INT>>
> DEFAULT array(named_struct('a', 11, 'b', 22)),
> s STRUCT<a:INT,b:INT>
> DEFAULT named_struct('a', 11, 'b', 22)
> );
> SELECT arr[0].b FROM nested_default_repro;
> SELECT arr.b FROM nested_default_repro;
> SELECT s.b FROM nested_default_repro;
> {code}
> ||Projection||Expected||Actual on Spark master||
> |{{arr[0].b}}|{{22}}|{{INVALID_DEFAULT_VALUE.DATA_TYPE}}|
> |{{arr.b}}|{{[22]}}|{{INVALID_DEFAULT_VALUE.DATA_TYPE}}|
> |{{s.b}}|{{22}}|{{INVALID_DEFAULT_VALUE.DATA_TYPE}}|
> The failure reproduces with Parquet V1 and V2, using both row and vectorized
> readers. Empty array or Map defaults can mask the problem. Spark should
> resolve the default against the declared column type and then project the
> requested fields from that value.
> h3. References
> * [Review discussion and
> reproduction|https://github.com/apache/spark/pull/59972#discussion_r4433555037]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
