Hi I am quite beginner in spark and I have similar issue last week. I don't know if my issue is the same as yours. I found that my program's jar contain protobuf and when I remove this dependency on my program's pom.xml, rebuild my program and it works.
Here is how I solved my own issue. Environment: Spark 0.9, HDFS (Hadoop 2.3), Scala 2.10. My spark is hadoop 2 HDP2 prebuild version from http://spark.apache.org/downloads.html. I don't build spark by my own. Problem : I use spark 0.9 example folder's word count program to connect my hdfs file which is build on hadoop 2.3. The running command is "./bin/run-example org.apache.spark.examples.WordCount" It show "Caused by: java.lang.VerifyError". I survey a lot on web but cannot get any workable solution. How I Solve my issue I found that if I use spark 0.9's spark-shell and it can connect hdfs file without this problem. But if I use run-example command, it show java.lang.VerifyError. I think the main reason is these two command(spark-shell and run-example)'s classpath is different. Run-Example's classpath is $SPARK_HOME /examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar::$SPARK_HOME/conf:$SPARK_HOME/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar Spark-Home's classpath is :$SPARK_HOME/conf:$SPARK_HOME/assembly/target/scala-2.10/spark-assembly_2.10-0.9.0-incubating-hadoop2.2.0.jar The class path difference is $SPARK_HOME/examples/target/scala-2.10/spark-examples_2.10-assembly-0.9.0-incubating.jar and it is build by exmaple program. When I look into this jar file, I found that it contain two protobuf which I don't know where it is from. I remove all dependency from my example pom.xml and left only one dependncy "spark-core". I rebuild it and it success. I don't know if my issue is the same as yours. I hope it can help. Wisely Chen On Wed, Mar 26, 2014 at 7:10 AM, Patrick Wendell <pwend...@gmail.com> wrote: > Starting with Spark 0.9 the protobuf dependency we use is shaded and > cannot interfere with other protobuf libaries including those in > Hadoop. Not sure what's going on in this case. Would someone who is > having this problem post exactly how they are building spark? > > - Patrick > > On Fri, Mar 21, 2014 at 3:49 PM, Aureliano Buendia <buendia...@gmail.com> > wrote: > > > > > > > > On Tue, Mar 18, 2014 at 12:56 PM, Ognen Duzlevski > > <og...@plainvanillagames.com> wrote: > >> > >> > >> On 3/18/14, 4:49 AM, dmpou...@gmail.com wrote: > >>> > >>> On Sunday, 2 March 2014 19:19:49 UTC+2, Aureliano Buendia wrote: > >>>> > >>>> Is there a reason for spark using the older akka? > >>>> > >>>> > >>>> > >>>> > >>>> On Sun, Mar 2, 2014 at 1:53 PM, 1esha <alexey.r...@gmail.com> wrote: > >>>> > >>>> The problem is in akka remote. It contains files compiled with 2.4.*. > >>>> When > >>>> > >>>> you run it with 2.5.* in classpath it fails like above. > >>>> > >>>> > >>>> > >>>> Looks like moving to akka 2.3 will solve this issue. Check this issue > - > >>>> > >>>> > >>>> > https://www.assembla.com/spaces/akka/tickets/3154-use-protobuf-version-2-5-0#/activity/ticket > : > >>>> > >>>> > >>>> Is the solution to exclude the 2.4.*. dependency on protobuf or will > >>>> thi produce more complications? > >> > >> I am not sure I remember what the context was around this but I run > 0.9.0 > >> with hadoop 2.2.0 just fine. > > > > > > The problem is that spark depends on an older version of akka, which > depends > > on an older version of protobuf (2.4). > > > > This means people cannot use protobuf 2.5 with spark. > > > >> > >> Ognen > > > > >
