I’m trying to save a data frame in Avro format but am getting the following error:
java.lang.NoSuchMethodError: org.apache.avro.generic.GenericData.createDatumWriter(Lorg/apache/avro/Schema;)Lorg/apache/avro/io/DatumWriter; I found the following workaround https://github.com/databricks/spark-avro/issues/91 - which seems to say that this is from a mismatch in Avro versions. I have tried following both solutions detailed to no avail: - Manually downloading avro-1.7.7.jar and including it in /usr/lib/hadoop-mapreduce/ - Adding avro-1.7.7.jar to spark.driver.extraClassPath and spark.executor.extraClassPath - The same with avro-1.6.6 I am still getting the same error, and now I am just stabbing in the dark. Anyone else still running into this issue? I am using Pyspark 1.5.2 on EMR.
