zahed1994 opened a new pull request, #59080:
URL: https://github.com/apache/spark/pull/59080
### What changes were proposed in this pull request?
Make the Parquet row-based reader (`ParquetRowConverter`) consistently
reject incompatible primitive type conversions that the vectorized reader
already rejects with `SchemaColumnConvertNotSupportedException`:
1. **Decimal Conversion Validation (`canReadAsDecimal`)**:
Check the logical type annotation for `INT32` and `INT64` columns when
converting to `DecimalType`. Conversions are accepted only when the logical
annotation is `DecimalLogicalTypeAnnotation`, unannotated (`null`), or a signed
integer (`IntLogicalTypeAnnotation` with `isSigned=true`). Semantic types such
as `DATE`, `TIMESTAMP`, `TIME`, and unsigned integers (`UINT32`) are rejected.
2. **Guarded String Conversions**:
Guard `case _: StringType` in `ParquetRowConverter` with
`parquetType.isPrimitive && parquetType.asPrimitiveType().getPrimitiveTypeName
== BINARY`. Other physical primitives (`FIXED_LEN_BYTE_ARRAY`, `INT96`,
`INT32`, `INT64`, `BOOLEAN`, `FLOAT`, `DOUBLE`) now throw
`SchemaColumnConvertNotSupportedException`. Complex non-primitive groups
requested as string cleanly throw
`QueryExecutionErrors.cannotCreateParquetConverterForDataTypeError`.
3. **Column Path Error Parameter Alignment**:
Thread `fieldPath: Seq[String]` through `ParquetRowConverter` and all
nested converters (`ParquetArrayConverter`, `ElementConverter`,
`ParquetMapConverter`, `KeyValueConverter`, `RepeatedPrimitiveConverter`,
`RepeatedGroupConverter`) so `columnPath` formats as `[c]` or `[parent,
child]`, matching `Arrays.toString(descriptor.getPath())` produced by the
vectorized reader.
### Why are the changes needed?
Previously, the row-based reader silently accepted incompatible conversions
or failed with internal decoding exceptions:
- `DATE` (`INT32`) was silently read as unscaled decimal values (e.g. date
`2020-01-01` read as decimal `18262`).
- `FIXED_LEN_BYTE_ARRAY` and `INT96` raw bytes were silently decoded as
corrupt UTF-8 strings.
- `INT32`/`INT64` to `STRING` failed with internal
`ParquetDecodingException` rather than Spark's standard column type mismatch
error.
Both Parquet readers should reject unsupported primitive conversions
consistently with `FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH` instead
of returning reader-dependent results or corrupt data.
### Does this PR introduce _any_ user-facing change?
Yes. When reading Parquet files with the row reader (e.g. with the
vectorized reader disabled or falling back to the row reader), the following
incompatible conversions now fail fast with
`FAILED_READ_FILE.PARQUET_COLUMN_DATA_TYPE_MISMATCH` instead of returning
corrupt data:
- `INT32` annotated as `DATE` -> `DecimalType`
- `INT64` annotated as `TIMESTAMP` / `TIME` -> `DecimalType`
- `INT32` annotated as `UINT32` -> `DecimalType`
- `FIXED_LEN_BYTE_ARRAY` -> `StringType`
- `INT96` -> `StringType`
- `INT32` / `INT64` / `BOOLEAN` / `FLOAT` / `DOUBLE` -> `StringType`
Valid conversions (e.g. `BINARY` -> `STRING`, unannotated `INT32`/`INT64` ->
`DECIMAL`) remain unaffected.
### How was this patch tested?
- Added `ParquetIOSuite`: `SPARK-59251: Parquet readers reject incompatible
primitive type conversions consistently` testing `INT32 DATE -> Decimal`,
`FIXED_LEN_BYTE_ARRAY -> String`, `INT96 -> String`, and `INT64 Timestamp ->
Decimal` across all Parquet readers.
- Updated `ParquetSchemaSuite`: unified `schema mismatch failure error
message` for both vectorized and row readers to check
`SchemaColumnConvertNotSupportedException` and parameter matches.
- Updated `ParquetTypeWideningSuite`: added `DateType -> DecimalType(10, 0)`
to the unsupported conversions table.
- Verified all targeted test suites pass locally via SBT.
- Verified `sql/scalastyle` passes cleanly with 0 errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
