[jira] [Commented] (HUDI-774) Spark to Avro converter incorrectly generates optional fields
[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17411499#comment-17411499 ] ASF GitHub Bot commented on HUDI-774: - hudi-bot commented on pull request #1514: URL: https://github.com/apache/hudi/pull/1514#issuecomment-914631306 ## CI report: * 3baebd032231974a7e7d9410b5bfeb879c9790b1 UNKNOWN Bot commands @hudi-bot supports the following commands: - `@hudi-bot run travis` re-run the last Travis build - `@hudi-bot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Spark to Avro converter incorrectly generates optional fields > - > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Alexander Filipchik >Priority: Major > Labels: pull-request-available, sev:critical, user-support-issues > Fix For: 0.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-774) Spark to Avro converter incorrectly generates optional fields
[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325449#comment-17325449 ] sivabalan narayanan commented on HUDI-774: -- [~afilipchik]: Can you check if HUDI-1716 fixes this ticket. If yes, can we close this one out. > Spark to Avro converter incorrectly generates optional fields > - > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Alexander Filipchik >Priority: Major > Labels: pull-request-available, sev:critical, user-support-issues > Fix For: 0.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-774) Spark to Avro converter incorrectly generates optional fields
[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17313841#comment-17313841 ] sivabalan narayanan commented on HUDI-774: -- related issue : HUDI-1716 > Spark to Avro converter incorrectly generates optional fields > - > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi > Issue Type: Bug >Affects Versions: 0.9.0 >Reporter: Alexander Filipchik >Priority: Major > Labels: pull-request-available, sev:critical, user-support-issues > Fix For: 0.9.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-774) Spark to Avro converter incorrectly generates optional fields
[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17107473#comment-17107473 ] Vinoth Chandar commented on HUDI-774: - yes [~uditme] is going to look at it as well > Spark to Avro converter incorrectly generates optional fields > - > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi (incubating) > Issue Type: Bug >Reporter: Alexander Filipchik >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HUDI-774) Spark to Avro converter incorrectly generates optional fields
[ https://issues.apache.org/jira/browse/HUDI-774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106849#comment-17106849 ] Raymond Xu commented on HUDI-774: - I think this is a very important fix..+1 For any custom schema provider, union null and other type with default null is very common. This bug will break this kind of use cases. cc [~vinoth] > Spark to Avro converter incorrectly generates optional fields > - > > Key: HUDI-774 > URL: https://issues.apache.org/jira/browse/HUDI-774 > Project: Apache Hudi (incubating) > Issue Type: Bug >Reporter: Alexander Filipchik >Priority: Major > Labels: pull-request-available > Fix For: 0.6.0 > > Time Spent: 10m > Remaining Estimate: 0h > > I think https://issues.apache.org/jira/browse/SPARK-28008 is a good > descriptions of what is happening. > > It can cause a situation when schema in the MOR log files is incompatible > with the schema produced by RowBasedSchemaProvider, so compactions will stall. > > I have a fix which is a bit hacky -> postprocess schema produced by the > converter and > 1) Make sure unions with null types have those null types at position 0 > 2) They have default values set to null > I couldn't find a way to do a clean fix as some classes that are problematic > are from Hive and called from Spark. -- This message was sent by Atlassian Jira (v8.3.4#803005)
