vinishjail97 commented on PR #647:
URL: https://github.com/apache/incubator-xtable/pull/647#issuecomment-2768718713
The integration tests for source format as HUDI and target format as DELTA
are failing, because of the following exception.
Hudi writes parquet files with physicalType as INT64 and logicalType as
`TimestampType`, but when reading the delta table fails because it's expecting
a `TimestampNTZType` type.
```
Caused by:
org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException:
column: [timestamp_local_millis_nullable_field], physicalType: INT64,
logicalType: timestamp_ntz
at
org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.constructConvertNotSupportedException(ParquetVectorUpdaterFactory.java:1129)
at
org.apache.spark.sql.execution.datasources.parquet.ParquetVectorUpdaterFactory.getUpdater(ParquetVectorUpdaterFactory.java:191)
at
org.apache.spark.sql.execution.datasources.parquet.VectorizedColumnReader.readBatch(VectorizedColumnReader.java:175)
at
org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:328)
at
org.apache.spark.sql.execution.datasources.parquet.VectorizedParquetRecordReader.nextKeyValue(VectorizedParquetRecordReader.java:219)
at
org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:125)
at
org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:297)
... 19 more
```
The test will fail for hudi 0.x and will pass when hudi is upgraded to 1.x.
https://github.com/apache/hudi/blob/master/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala#L152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
