awdavidson commented on code in PR #38312:
URL: https://github.com/apache/spark/pull/38312#discussion_r1004137291
##########
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaSuite.scala:
##########
@@ -198,6 +205,31 @@ abstract class ParquetSchemaTest extends ParquetTest with
SharedSparkSession {
}
class ParquetSchemaInferenceSuite extends ParquetSchemaTest {
+ testSchemaInference[Tuple1[Long]](
Review Comment:
So I've been looking further into it, it's because the message is different
between `1.10.1` and `1.12.3` - meaning the test would need to be different.
In `1.10.1` the message is
```
message schema {
required int64 attribute;
}
```
where as `1.12.3` the message is the same as the unit test
```
message schema {
required int64 attribute (TIMESTAMP(NANOS,true));
}
```
So in Spark 3.1.0 you end up hitting this block with returns a `LongType`
https://github.com/apache/spark/blob/branch-3.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L146
where as since 3.2 you hit
https://github.com/apache/spark/blob/branch-3.2/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L174
because a case for `TimeUnit.NANOS` is not covered
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
