rdtr opened a new pull request, #13538: URL: https://github.com/apache/gluten/pull/13538
## What changes are proposed in this pull request? Spark 4.2 (SPARK-54220) writes and reads `NullType` columns in Parquet. With Gluten's native writer and reader, `GlutenParquetIOSuite` `SPARK-54220: NullType` failed with `VeloxRuntimeError: Converted type UNKNOWN is not allowed for requested type INTEGER for file column '_1'`, so it is excluded in the spark42 `VeloxTestSettings`. facebookincubator/velox#19461 (merged 2026-11-03) lets the Velox Parquet reader read a column with the UNKNOWN annotation as the requested type, like parquet-mr. This removes the spark42 exclude. The fix is not in the pinned Velox yet. `dft-2026_11_04` is based on upstream as of 2026-11-02, so this PR depends on the next daily Velox update. I'll rebase once it lands. `SPARK-54220: vectorized reader: missing all struct fields, struct with NullType only` stays excluded in spark41 and spark42. It runs the scan node alone with `VerifyNoAdditionalScanOutputExec`, which does not work for Gluten's scan node, and the Gluten rewrite of it in `GlutenParquetIOSuite` already passes. #12056 already re-enabled `SPARK-54220: NullType` on Spark 4.1. That leaves this one as the only exclude under #11921, so I think #11921 can be closed with this PR. Fixes #13434 cc @akshaytayal ## How was this patch tested? Built Gluten against the pinned Velox (`dft-2026_11_01`) with facebookincubator/velox#16966 and facebookincubator/velox#19461 cherry-picked, and ran `GlutenParquetIOSuite` on Spark 4.2 with `-Dspark.test.home` set. `SPARK-54220: NullType`, `Read external file with UNKNOWN type annotation (true)` and `(false)` pass. Two tests fail with `[UNSUPPORTED_TIME_TYPE]` from Spark's `DataSourceUtils.verifySchema` (`SPARK-11694 Parquet logical types are not being tested properly`, `Read TimeType for the logical TIME type`). That happens during analysis, before Gluten is involved. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
