[
https://issues.apache.org/jira/browse/SPARK-26045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782508#comment-16782508
]
Sean Owen commented on SPARK-26045:
-----------------------------------
Avro is already on 1.8.2 in 2.4.0. I think the problem may be that you're
adding a separate copy of Avro, which might end up in a different classloader.
You wouldn't want to do this, in general, but certainly won't likely work with
Spark 2.3, which was on Avro 1.7.7.
Try without adding an avro jar, and make sure you don't have a different Avro
dependency in your app or environment. If that's not it, my final guess is that
something in hive-exec has a copy of avro. But then I am not sure how the tests
would pass.
> Error in the spark 2.4 release package with the spark-avro_2.11 depdency
> ------------------------------------------------------------------------
>
> Key: SPARK-26045
> URL: https://issues.apache.org/jira/browse/SPARK-26045
> Project: Spark
> Issue Type: Bug
> Components: Build
> Affects Versions: 2.4.0
> Environment: 4.15.0-38-generic #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC
> 2018 x86_64 x86_64 x86_64 GNU/Linux
> Reporter: Oscar garcía
> Priority: Major
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> Hello I have been problems with the last spark 2.4 release, the read avro
> file feature does not seem to be working, I have fixed it in local building
> the source code and updating the *avro-1.8.2.jar* on the *$SPARK_HOME*/jars/
> dependencies.
> With the default spark 2.4 release when I try to read an avro file spark
> raise the following exception.
> {code:java}
> spark-shell --packages org.apache.spark:spark-avro_2.11:2.4.0
> scala> spark.read.format("avro").load("file.avro")
> java.lang.NoSuchMethodError:
> org.apache.avro.Schema.getLogicalType()Lorg/apache/avro/LogicalType;
> at
> org.apache.spark.sql.avro.SchemaConverters$.toSqlTypeHelper(SchemaConverters.scala:51)
> at
> org.apache.spark.sql.avro.SchemaConverters$.toSqlTypeHelper(SchemaConverters.scala:105
> {code}
> Checksum: spark-2.4.0-bin-without-hadoop.tgz: 7670E29B 59EAE7A8 5DBC9350
> 085DD1E0 F056CA13 11365306 7A6A32E9 B607C68E A8DAA666 EF053350 008D0254
> 318B70FB DE8A8B97 6586CA19 D65BA2B3 FD7F919E
>
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
