[
https://issues.apache.org/jira/browse/SPARK-59251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiayi Wang updated SPARK-59251:
-------------------------------
Description:
h1. Problem
Spark's row-based Parquet reader and vectorized Parquet reader handle some
incompatible requested
schemas differently. The vectorized reader raises
`SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested
Catalyst type.
Two examples are:
|Parquet file type|Requested Spark type|Row-based reader|Vectorized reader|
| | | | |
|*FIXED_LEN_BYTE_ARRAY(4)*|*STRING*|Returns the raw bytes as UTF-8|Rejects the
conversion|
|*INT32 (DATE)*|*DECIMAL(10, 0)*|Returns the day count as a decimal|Rejects the
conversion|
The row reader's *ParquetRowConverter* currently accepts every binary-like
primitive as a string
and treats any INT32 or INT64 without decimal metadata as an unannotated
integer-backed
decimal. The latter ignores semantic logical annotations such as DATE.
This is a correctness problem because changing
`spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted data.
h1. Expected behavior
Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
Supported conversions should remain unchanged, including Parquet `BINARY` to
Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible
Spark decimal type.
was:
### Problem
Spark's row-based Parquet reader and vectorized Parquet reader handle some
incompatible requested
schemas differently. The vectorized reader raises
`SchemaColumnConvertNotSupportedException`, while
the row-based reader silently interprets the physical value as the requested
Catalyst type.
Two examples are:
| Parquet file type | Requested Spark type | Row-based reader | Vectorized
reader |
| --- | --- | --- | --- |
| `FIXED_LEN_BYTE_ARRAY(4)` | `STRING` | Returns the raw bytes as UTF-8 |
Rejects the conversion |
| `INT32 (DATE)` | `DECIMAL(10, 0)` | Returns the day count as a decimal |
Rejects the conversion |
The row reader's `ParquetRowConverter` currently accepts every binary-like
primitive as a string
and treats any `INT32` or `INT64` without decimal metadata as an unannotated
integer-backed
decimal. The latter ignores semantic logical annotations such as `DATE`.
This is a correctness problem because changing
`spark.sql.parquet.enableVectorizedReader` can
change a query from failing cleanly to returning incorrectly interpreted data.
### Expected behavior
Both readers should reject these unsupported conversions with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
Supported conversions should remain unchanged, including Parquet `BINARY` to
Spark `STRING` and
unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible
Spark decimal type.
> Reject incompatible Parquet row reader conversions
> --------------------------------------------------
>
> Key: SPARK-59251
> URL: https://issues.apache.org/jira/browse/SPARK-59251
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 4.2.0
> Reporter: Jiayi Wang
> Priority: Major
>
> h1. Problem
> Spark's row-based Parquet reader and vectorized Parquet reader handle some
> incompatible requested
> schemas differently. The vectorized reader raises
> `SchemaColumnConvertNotSupportedException`, while
> the row-based reader silently interprets the physical value as the requested
> Catalyst type.
> Two examples are:
> |Parquet file type|Requested Spark type|Row-based reader|Vectorized reader|
> | | | | |
> |*FIXED_LEN_BYTE_ARRAY(4)*|*STRING*|Returns the raw bytes as UTF-8|Rejects
> the conversion|
> |*INT32 (DATE)*|*DECIMAL(10, 0)*|Returns the day count as a decimal|Rejects
> the conversion|
> The row reader's *ParquetRowConverter* currently accepts every binary-like
> primitive as a string
> and treats any INT32 or INT64 without decimal metadata as an unannotated
> integer-backed
> decimal. The latter ignores semantic logical annotations such as DATE.
> This is a correctness problem because changing
> `spark.sql.parquet.enableVectorizedReader` can
> change a query from failing cleanly to returning incorrectly interpreted data.
> h1. Expected behavior
> Both readers should reject these unsupported conversions with
> `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH`.
> Supported conversions should remain unchanged, including Parquet `BINARY` to
> Spark `STRING` and
> unannotated or signed-integer `INT32`/`INT64` to a sufficiently compatible
> Spark decimal type.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
