[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17313841#comment-17313841 ]
sivabalan narayanan commented on HUDI-774: ------------------------------------------ related issue : HUDI-1716 > Spark to Avro converter incorrectly generates optional fields > ------------------------------------------------------------- > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi > Issue Type: Bug > Affects Versions: 0.9.0 > Reporter: Alexander Filipchik > Priority: Major > Labels: pull-request-available, sev:critical, user-support-issues > Fix For: 0.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)