Chao Sun created SPARK-60790:
--------------------------------
Summary: Nested schema pruning rejects nonempty column defaults
when reading older Parquet files
Key: SPARK-60790
URL: https://issues.apache.org/jira/browse/SPARK-60790
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.2.0
Reporter: Chao Sun
h3. Description
When a column is added with a default, Spark uses that value to read rows from
files written before the column existed. If the query selects only part of a
nested column, Spark can attempt to coerce the full default value to the pruned
type. A default that is valid for the declared column type then fails with
{{INVALID_DEFAULT_VALUE.DATA_TYPE}}.
h3. Reproduction
Run the three SELECT statements separately with nested schema pruning enabled.
Insert the row before adding the columns so that Spark must read their defaults.
{code:sql}
CREATE TABLE nested_default_repro (idx INT) USING PARQUET;
INSERT INTO nested_default_repro VALUES (0);
ALTER TABLE nested_default_repro ADD COLUMNS (
arr ARRAY<STRUCT<a:INT,b:INT>>
DEFAULT array(named_struct('a', 11, 'b', 22)),
s STRUCT<a:INT,b:INT>
DEFAULT named_struct('a', 11, 'b', 22)
);
SELECT arr[0].b FROM nested_default_repro;
SELECT arr.b FROM nested_default_repro;
SELECT s.b FROM nested_default_repro;
{code}
||Projection||Expected||Actual on Spark master||
|{{arr[0].b}}|{{22}}|{{INVALID_DEFAULT_VALUE.DATA_TYPE}}|
|{{arr.b}}|{{[22]}}|{{INVALID_DEFAULT_VALUE.DATA_TYPE}}|
|{{s.b}}|{{22}}|{{INVALID_DEFAULT_VALUE.DATA_TYPE}}|
The failure reproduces with Parquet V1 and V2, using both row and vectorized
readers. Empty array or Map defaults can mask the problem. Spark should resolve
the default against the declared column type and then project the requested
fields from that value.
h3. References
* [Review discussion and
reproduction|https://github.com/apache/spark/pull/59972#discussion_r4433555037]
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
