Just to provide more information on this issue. It seems that SPARK_HOME environment variable is causing the issue. If I unset the variable in spark-class script and run in the local mode my code runs fine without the exception. But if I run with SPARK_HOME, I get the exception mentioned below. I could run without setting SPARK_HOME but it is not possible to run in the cluster settings, as this tells where is spark on worker nodes. E.g. we are using Mesos as cluster manager, thus when set master to mesos we get the exception as SPARK_HOME is not set.
Just to mention again the pyspark works fine as well as spark-shell, only when we are running compiled jar it seems SPARK_HOME causes some java run time issues that we get class cast exception. Thanks, Gurvinder On 07/01/2014 09:28 AM, Gurvinder Singh wrote: > Hi, > > I am having issue in running scala example code. I have tested and able > to run successfully python example code, but when I run the scala code I > get this error > > java.lang.ClassCastException: cannot assign instance of > org.apache.spark.examples.SparkPi$$anonfun$1 to field > org.apache.spark.rdd.MappedRDD.f of type scala.Function1 in instance of > org.apache.spark.rdd.MappedRDD > > I have compiled spark from the github directly and running with the > command as > > spark-submit /usr/share/spark/lib/spark-examples_2.10-1.1.0-SNAPSHOT.jar > --class org.apache.spark.examples.SparkPi 5 --jars > /usr/share/spark/lib/spark-assembly-1.1.0-SNAPSHOT-hadoop2.h5.0.1.jar > > Any suggestions will be helpful. > > Thanks, > Gurvinder >
