uros-b commented on PR #56422: URL: https://github.com/apache/spark/pull/56422#issuecomment-5603141201
DSV2 never puts the new flag on the Hadoop conf ParquetFileFormat sets PARQUET_TIME_TYPE_ALLOW_IS_ADJUSTED_TO_UTC_READ next to inferTimestampNTZ and respectUnknownTypeAnnotation. ParquetScan does not, even though that block is documented as flags for ParquetToSparkSchemaConverter. Inference still works (it goes through mergeSchemasInParallel with SQLConf directly). Executor-side new ParquetToSparkSchemaConverter(conf) in ParquetReadSupport will keep the default false. That can throw on row-based convertField (for example array-element guessing) after inference already succeeded. Mirror the V1 hadoopConf.setBoolean in ParquetScan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
