Jiayi Wang created SPARK-59251:
----------------------------------
Summary: Reject incompatible Parquet row reader conversions
Key: SPARK-59251
URL: https://issues.apache.org/jira/browse/SPARK-59251
Project: Spark
Issue Type: Bug
Components: Bug
Affects Versions: 4.2.0
Reporter: Jiayi Wang
### Problem
Spark's row-based Parquet reader and vectorized Parquet reader handle some
incompatible requested
schemas differently. The vectorized reader raises
`SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested
Catalyst type.
Two examples are:
| Parquet file type | Requested Spark type | Row-based reader | Vectorized
reader |
| --- | --- | --- | --- |
| `FIXED_LEN_BYTE_ARRAY(4)` | `STRING` | Returns the raw bytes as UTF-8 |
Rejects the conversion |
| `INT32 (DATE)` | `DECIMAL(10, 0)` | Returns the day count as a decimal |
Rejects the conversion |
The row reader's `ParquetRowConverter` currently accepts every binary-like
primitive as a string
and treats any `INT32` or `INT64` without decimal metadata as an unannotated
integer-backed
decimal. The latter ignores semantic logical annotations such as `DATE`.
This is a correctness problem because changing
`spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted data.
### Expected behavior
Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
Supported conversions should remain unchanged, including Parquet `BINARY` to
Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible
Spark decimal type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
