[
https://issues.apache.org/jira/browse/SPARK-27733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17212451#comment-17212451
]
Ismaël Mejía commented on SPARK-27733:
--------------------------------------
[~sha...@uber.com] sorry I missed somehow the previous notification. If you
have a future parquet sync i would love to join to explain the full details,
otherwise, the tldr version is fix on Spark side is 'easy' the real deal is
that Spark gets Avro's 1.8 dependency via Hive and getting the fix on Hive has
proven difficult (already more than 1y in the making) and even with the fix
merged we still need it to be backported to the 2.3.x branch and have a release
that includes it, we need LOTS of good will and help from the Hive people so if
you guys know anyone there who can help that would be appreciated.
The issue is related to a more strict validation on unions with ill defined
defaults starting on Avro 1.9.x. There are multiple options to deal with this
(discussed in the ticket) but we shall probably do a fix on Avro side. I will
bring the update here once we agree on the fix, worse scenario it will require
a release of Avro too but this is a good time since we were already discussing
about a release soon.
> Upgrade to Avro 1.10.0
> ----------------------
>
> Key: SPARK-27733
> URL: https://issues.apache.org/jira/browse/SPARK-27733
> Project: Spark
> Issue Type: Improvement
> Components: Build, SQL
> Affects Versions: 3.1.0
> Reporter: Ismaël Mejía
> Priority: Minor
>
> Avro 1.9.2 was released with many nice features including reduced size (1MB
> less), and removed dependencies, no paranamer, no shaded guava, security
> updates, so probably a worth upgrade.
> Avro 1.10.0 was released and this is still not done.
> There is at the moment (2020/08) still a blocker because of Hive related
> transitive dependencies bringing older versions of Avro, so we could say that
> this is somehow still blocked until HIVE-21737 is solved.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
