Please use --conf spark.executor.extraClassPath=XXX to specify dependent jars.
On Tue, Dec 15, 2015 at 3:57 AM, vimal dinakaran <vimal3...@gmail.com> wrote: > I am running spark using cluster mode for deployment . Below is the command > > > JARS=$JARS_HOME/amqp-client-3.5.3.jar,$JARS_HOME/nscala-time_2.10-2.0.0.jar,\ > $JARS_HOME/kafka_2.10-0.8.2.1.jar,$JARS_HOME/kafka-clients-0.8.2.1.jar,\ > $JARS_HOME/spark-streaming-kafka_2.10-1.4.1.jar,\ > $JARS_HOME/zkclient-0.3.jar,$JARS_HOME/protobuf-java-2.4.0a.jar > > dse spark-submit -v --conf > "spark.serializer=org.apache.spark.serializer.KryoSerializer" \ > --executor-memory 512M \ > --total-executor-cores 3 \ > --deploy-mode "cluster" \ > --master spark://$MASTER:7077 \ > --jars=$JARS \ > --supervise \ > --class "com.testclass" $APP_JAR input.json \ > --files "/home/test/input.json" > > The above command is working fine in client mode. But when I use it in > cluster mode I get class not found exception > > Exception in thread "main" java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.apache.spark.deploy.worker.DriverWrapper$.main(DriverWrapper.scala:58) > at > org.apache.spark.deploy.worker.DriverWrapper.main(DriverWrapper.scala) > Caused by: java.lang.NoClassDefFoundError: > org/apache/spark/streaming/kafka/KafkaUtils$ > > In client mode the dependent jars are getting copied to the > /var/lib/spark/work directory whereas in cluster mode it is not. > I am using nfs and I have mounted the same directory on all the spark > nodes under same name. Still I get the error. > > From the verbose logs of dse spark-submit, I see the classpath elements > are missing here . > > ---------------------------------------------------------------------------------------------- > spark.hadoop.cassandra.input.native.ssl.trust.store.password -> cassandra > spark.cassandra.connection.ssl.trustStore.password -> cassandra > spark.ssl.keyStorePassword -> cassandra > spark.cassandra.auth.username -> cassandra > spark.hadoop.fs.har.impl -> org.apache.hadoop.fs.HarFileSystem > Classpath elements: > > > WARN 2015-12-15 17:08:48 org.apache.spark.util.Utils: Your hostname, > demeter-dev-node2 resolves to a loopback address: 127.0.1.1; using > 10.29.23.170 instead (on interface eth0) > WARN 2015-12-15 17:08:48 org.apache.spark.util.Utils: Set SPARK_LOCAL_IP > if you need to bind to another addres > > How it is able to pick the application jar which is also under same > directory but not the dependent jars ? > Please help me in getting this solved. > >
