DB Tsai created SPARK-27838:
-------------------------------
Summary: Support user provided non-nullable avro schema for
nullable catalyst schema without any null record
Key: SPARK-27838
URL: https://issues.apache.org/jira/browse/SPARK-27838
Project: Spark
Issue Type: New Feature
Components: SQL
Affects Versions: 2.4.3
Reporter: DB Tsai
Fix For: 3.0.0
When the data is read from the sources, the catalyst schema is always nullable.
Since Avro uses Union type to represent nullable, when any non-nullable avro
file is read and then written out, the schema will always be changed. This PR
provides a solution for users to keep the Avro schema without being forced to
use Union type.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
