[
https://issues.apache.org/jira/browse/SPARK-39775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nandini updated SPARK-39775:
----------------------------
Description:
With the upgrade in Avro version to 1.9.0, for schema evolution Avro added
https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default
values in schemas by default) which is causing regressions when user upgrades
their Spark verion.
Repro code:
{code:java}
import org.apache.spark.sql.avro.functions._
val avroTypeStruct = s"""
|{
| "type": "record",
| "name": "struct",
| "fields": [
| {"name": "id", "type": "long", "default": null}
| ]
|}""".stripMargin
val df = spark.range(10).select(struct('id).as("struct"))
val avroStructDF = df.select(to_avro('struct, avroTypeStruct).as("avro"))
display(avroStructDF.select(from_avro('avro, avroTypeStruct)))
{code}
Hive mitigated it by disabling this feature altogether in
https://issues.apache.org/jira/browse/HIVE-24797
Spark-Hive integration also imported the above changes in
https://issues.apache.org/jira/browse/SPARK-34512
Can we have a fix for all the senarios?
was:
With the upgrade in Avro version to 1.9.0, for schema evolution Avro added
https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default
values in schemas by default) which is causing regressions when user upgrades
their Spark verion.
Repro code:
{code:java}
import org.apache.spark.sql.avro.functions._
val avroTypeStruct = s"""
|{
| "type": "record",
| "name": "struct",
| "fields": [
| {"name": "id", "type": "long", "default": null}
| ]
|}""".stripMargin
val df = spark.range(10).select(struct('id).as("struct"))
{code}
Hive mitigated it by disabling this feature altogether in
https://issues.apache.org/jira/browse/HIVE-24797
Spark-Hive integration also imported the above changes in
https://issues.apache.org/jira/browse/SPARK-34512
Can we have a fix for all the senarios?
> Regression due to AVRO-2035
> ---------------------------
>
> Key: SPARK-39775
> URL: https://issues.apache.org/jira/browse/SPARK-39775
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.2.1
> Reporter: Nandini
> Priority: Major
>
> With the upgrade in Avro version to 1.9.0, for schema evolution Avro added
> https://issues.apache.org/jira/browse/AVRO-2035(enable validation of default
> values in schemas by default) which is causing regressions when user upgrades
> their Spark verion.
> Repro code:
> {code:java}
> import org.apache.spark.sql.avro.functions._
> val avroTypeStruct = s"""
> |{
> | "type": "record",
> | "name": "struct",
> | "fields": [
> | {"name": "id", "type": "long", "default": null}
> | ]
> |}""".stripMargin
> val df = spark.range(10).select(struct('id).as("struct"))
> val avroStructDF = df.select(to_avro('struct, avroTypeStruct).as("avro"))
> display(avroStructDF.select(from_avro('avro, avroTypeStruct)))
> {code}
> Hive mitigated it by disabling this feature altogether in
> https://issues.apache.org/jira/browse/HIVE-24797
> Spark-Hive integration also imported the above changes in
> https://issues.apache.org/jira/browse/SPARK-34512
> Can we have a fix for all the senarios?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
