Hi, spark-users:
I'm using spark-submit to submit multiple jars and files(all in HDFS) to run a
job, with the following command:
Spark-submit
--class myClass
--master spark://localhost:7077/
--deploy-mode cluster
--jars hdfs://localhost/1.jar, hdfs://localhost/2.jar
--files hdfs://localhost/1.txt, hdfs://localhost/2.txt
hdfs://localhost/main.jar
the stderr in the driver showed java.lang.ClassNotDefException for a class in
1.jar.
I checked the log that spark has added these jars:
INFO SparkContext: Added JAR hdfs:// ...1.jar
INFO SparkContext: Added JAR hdfs:// ...2.jar
In the folder of the driver, I only saw the main.jar is copied to that place,
but the other jars and files were not there
Could someone explain how should I pass the jars and files needed by the main
jar to spark?
If my class in main.jar refer to these files with a relative path, will spark
copy these files into one folder?
BTW, my class works in a client mode with all jars and files in local.
Thanks
Dong Lei
