hi all: I found a problem using spark . WHEN I use spark-submit to launch a task. it works
spark-submit --num-executors 8 --executor-memory 8G --class com.icbc.nss.spark.PfsjnlSplit --master yarn-cluster /home/nssbatch/nss_schedual/jar/SparkBigtableJoinSqlJava.jar /user/nss/nss-20151018-pfsjnl-004_024.txt /user/nss/output_join /user/nss/output_join2 but when i use the command created by spark-class as below /home/nssbatch/huaweiclient/hadoopclient/JDK/jdk/bin/java -Djava.security.krb5.conf=/home/nssbatch/huaweiclient/hadoopclient/KrbClient/kerberos/var/krb5kdc/krb5.conf -Dzookeeper.server.principal=zookeeper/hadoop.hadoop.com -Djava.security.auth.login.config=/home/nssbatch/huaweiclient/hadoopclient/Spark/adapter/client/controller/jaas.conf -Dzookeeper.kinit=/home/nssbatch/huaweiclient/hadoopclient/KrbClient/kerberos/bin/kinit -cp /home/nssbatch/huaweiclient/hadoopclient/Spark/spark/conf/:/home/nssbatch/huaweiclient/hadoopclient/Spark/spark/lib/spark-assembly-1.3.0-hadoop2.7.1.jar:/home/nssbatch/huaweiclient/hadoopclient/Spark/spark/lib/datanucleus-core-3.2.10.jar:/home/nssbatch/huaweiclient/hadoopclient/Spark/spark/lib/datanucleus-rdbms-3.2.9.jar:/home/nssbatch/huaweiclient/hadoopclient/Spark/spark/lib/datanucleus-api-jdo-3.2.6.jar:/home/nssbatch/huaweiclient/hadoopclient/Spark/spark/conf/:/home/nssbatch/huaweiclient/hadoopclient/Yarn/config/ org.apache.spark.deploy.SparkSubmit --master yarn-cluster --class com.icbc.nss.spark.PfsjnlSplit --num-executors 8 --executor-memory 8G /home/nssbatch/nss_schedual/jar/SparkBigtableJoinSqlJava.jar /user/nss/nss-20151018-pfsjnl-004_024.txt /user/nss/output_join /user/nss/output_join2 it didn't work. i compare the log.and found that: 16/05/10 22:23:24 INFO Client: Uploading resource file:/tmp/spark-a4457754-7183-44ce-bd0d-32a071757c92/__hadoop_conf__4372868703234608846.zip -> hdfs://hacluster/user/nss/.sparkStaging/application_1462442311990_0057/__hadoop_conf__4372868703234608846.zip the conf_file uploaded into hdfs was different. why is this happened? where can i find the resource file to be uploading?