[
https://issues.apache.org/jira/browse/SPARK-59251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Wang updated SPARK-59251:
-------------------------------
Issue Type: Improvement (was: Task)
> Reject incompatible Parquet row reader conversions
> --------------------------------------------------
>
> Key: SPARK-59251
> URL: https://issues.apache.org/jira/browse/SPARK-59251
> Project: Spark
> Issue Type: Improvement
> Components: Bug
> Affects Versions: 4.2.0
> Reporter: Jiayi Wang
> Priority: Major
>
> h1. Problem
> Spark's row-based Parquet reader and vectorized Parquet reader handle some
> incompatible requested
> schemas differently. The vectorized reader raises
> `SchemaColumnConvertNotSupportedException`, while
> the row-based reader silently interprets the physical value as the requested
> Catalyst type.
> Two examples are:
> |Parquet file type|Requested Spark type|Row-based reader|Vectorized reader|
> | | | | |
> |*FIXED_LEN_BYTE_ARRAY(4)*|*STRING*|Returns the raw bytes as UTF-8|Rejects
> the conversion|
> |*INT32 (DATE)*|*DECIMAL(10, 0)*|Returns the day count as a decimal|Rejects
> the conversion|
> The row reader's *ParquetRowConverter* currently accepts every binary-like
> primitive as a string
> and treats any INT32 or INT64 without decimal metadata as an unannotated
> integer-backed
> decimal. The latter ignores semantic logical annotations such as DATE.
> This is a correctness problem because changing
> `spark.sql.parquet.enableVectorizedReader` can
> change a query from failing cleanly to returning incorrectly interpreted data.
> h1. Expected behavior
> Both readers should reject these unsupported conversions with
> `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
> Supported conversions should remain unchanged, including Parquet `BINARY` to
> Spark `STRING` and
> unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible
> Spark decimal type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
