Hi Tom There has to be a difference in classpaths in yarn-client and yarn-cluster mode. Perhaps a good starting point would be to print classpath as a first thing in SimpleApp.main. It should give clues around why it works in yarn-cluster mode.
Thanks, Aniket On Wed, Sep 9, 2015, 2:11 PM Tom Seddon <mr.tom.sed...@gmail.com> wrote: > Hi, > > I have a problem trying to get a fairly simple app working which makes use > of native avro libraries. The app runs fine on my local machine and in > yarn-cluster mode, but when I try to run it on EMR yarn-client mode I get > the error below. I'm aware this is a version problem, as EMR runs an > earlier version of avro, and I am trying to use avro-1.7.7. > > What's confusing me a great deal is the fact that this runs fine in > yarn-cluster mode. > > What is it about yarn-cluster mode that means the application has access > to the correct version of the avro library? I need to run in yarn-client > mode as I will be caching data to the driver machine in between batches. I > think in yarn-cluster mode the driver can run on any machine in the cluster > so this would not work. > > Grateful for any advice as I'm really stuck on this. AWS support are > trying but they don't seem to know why this is happening either! > > Just to note, I'm aware of Databricks spark-avro project and have used > it. This is an investigation to see if I can use RDDs instead of > dataframes. > > java.lang.NoSuchMethodError: > org.apache.avro.Schema$Parser.parse(Ljava/lang/String;[Ljava/lang/String;)Lorg/apache/avro/Schema; > at ophan.thrift.event.Event.<clinit>(Event.java:10) > at SimpleApp$.main(SimpleApp.scala:25) > at SimpleApp.main(SimpleApp.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:665) > at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:170) > at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:193) > at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:112) > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) > > Thanks, > > Tom > > >
