akshaytayal opened a new issue, #13434:
URL: https://github.com/apache/gluten/issues/13434
Part of #13181 (sub-theme #13180). Found while re-enabling the Spark 4.2 UTs
disabled in #13163.
### Problem
Spark 4.2 (SPARK-54220) supports writing/reading `NullType` in Parquet. With
Gluten's native writer + native reader, `GlutenParquetIOSuite` `SPARK-54220:
NullType` fails:
```
VeloxRuntimeError: Converted type UNKNOWN is not allowed for requested type
INTEGER for file column '_1'
velox/dwio/parquet/reader/ParquetReader.cpp convertType
```
Spark-written NullType files read fine natively; vanilla Spark reads
Gluten-written files fine (but infers `IntegerType`).
### Root cause
1. **Velox writer** (`velox/dwio/parquet/writer/arrow/ArrowSchema.cpp`)
writes Arrow `NA` as `INT32 + UNKNOWN`; Spark writes `BOOLEAN + UNKNOWN` and
also stores its row schema (`org.apache.spark.sql.parquet.row.metadata`), which
Gluten's native writer doesn't. So Spark 4.2 infers `IntegerType` for
Gluten-written files (default
`spark.sql.parquet.reader.respectUnknownTypeAnnotation.enabled=false`).
2. **Velox reader** (`ReaderBase::convertType`) accepts an
`UNKNOWN`-annotated column only when the requested type is `UNKNOWN`.
parquet-mr/Spark ignore the annotation and read the all-null column as its
physical type. This also affects Arrow/pyarrow-written null columns.
### Proposed fix
- Velox reader: when the requested type isn't `UNKNOWN`, ignore the
`UNKNOWN` annotation and convert using the physical type (as parquet-mr does).
- Optional writer parity: write `NA` as `BOOLEAN + UNKNOWN` for Spark and
add Spark's row-metadata footer entry from Gluten's native writer, so the
inferred schema is `NullType` like vanilla Spark.
Interim option (Gluten-only): treat `NullType` as unsupported in
`VeloxBackendSettings` write validation so such writes fall back to Spark's
writer (identical files to vanilla Spark 4.2).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
