umehrot2 commented on issue #770: remove com.databricks:spark-avro to build spark avro schema by itself URL: https://github.com/apache/incubator-hudi/pull/770#issuecomment-532026904 > @umehrot2 Great analysis.. Would upgrading parquet-avro help? Good point @vinothchandar . Upon a quick look at `AvroSchemaConverter` code in `parquet-avro` it seems like handling of `LogicalType` conversion was added since parquet 1.8.2: https://github.com/apache/parquet-mr/blob/apache-parquet-1.8.2/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java#L379 It is not there in `parquet 1.8.1` which is what Hudi uses right now: https://github.com/apache/parquet-mr/blob/apache-parquet-1.8.1/parquet-avro/src/main/java/org/apache/parquet/avro/AvroSchemaConverter.java I will upgrade the parquet version and test. Will update here with what I find.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
