dwsmith1983 commented on PR #6116: URL: https://github.com/apache/datafusion-comet/pull/6116#issuecomment-5819349038
The Spark 4.1 scans job failed the four tests that read a schema with field ids over a file without any. On 4.1.3 the `RuntimeException` from `ParquetReadSupport.getRequestedSchema` is still wrapped once by `FileDataSourceV2.attachFilePath` in `FileScanRDD.hasNext`, as `FAILED_READ_FILE.NO_HINT`, but that wrapper is the `SparkException` the test intercepts, with the `RuntimeException` as its direct cause. The same chain shows up on 4.0.4, so the helper's Spark 4 branch was wrong on both. It assumed one more `SparkException` above the wrapper and took the `RuntimeException` for it, which the Spark 3.5 job never sees because there the `RuntimeException` arrives bare. It now walks the cause chain from the intercepted exception and requires a link that is a plain `RuntimeException` whose message says the Parquet file schema has no field ids, so the number of layers above it no longer matters. The same check runs for Spark and for Comet. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
