Hi there, I'm trying out Spark Job Server (REST) to submit jobs to spark cluster. I believe that my problem is unrelated to this specific software, but otherwise generic issue with missing jars on paths. So every application implements the trait with SparkJob class:
/object LongPiJob extends SparkJob { ../ SparkJob class is available through the jar file, built by Spark Job Server Scala application. When I run all this with local Spark cluster, everything works fine after I add the export line into spark-env.sh: /export SPARK_CLASSPATH=$SPARK_HOME/job-server/spark-job-server.jar/ However, when I do the same on Spark cluster on EC2, I get the errors: /java.lang.NoClassDefFoundError: spark/jobserver/SparkJob/ I've added the path in spark-env.sh (on remote Spark master Amazon machine): /export MASTER=`cat /root/spark-ec2/cluster-url` *export SPARK_CLASSPATH=/root/spark/job-server/spark-job-server.jar* export SPARK_SUBMIT_LIBRARY_PATH="$SPARK_SUBMIT_LIBRARY_PATH:/root/ephemeral-hdfs/lib/native/" export SPARK_SUBMIT_CLASSPATH="$SPARK_CLASSPATH:$SPARK_SUBMIT_CLASSPATH:/root/ephemeral-hdfs/conf"/ Also, when I run ./bin/compute-classpath.sh, I can see the required jar, defining "missing" class at the first place: /bin]$ ./compute-classpath.sh Spark assembly has been built with Hive, including Datanucleus jars on classpath /root/spark/job-server/spark-job-server.jar:/root/spark/job-server/spark-job-server.jar::/root/ephemeral-hdfs/conf:/root/spark/conf:/root/spark/lib/spark-assembly-1.2.0-hadoop1.0.4.jar:/root/spark/lib/datanucleus-core-3.2.10.jar:/root/spark/lib/datanucleus-api-jdo-3.2.6.jar:/root/spark/lib/datanucleus-rdbms-3.2.9.jar/ What am I missing? I'd greatly appreciate your help -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Submitting-jobs-on-Spark-EC2-cluster-class-not-found-even-if-it-s-on-CLASSPATH-tp21864.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org