If anybody knows the reason, please help me. thanks a lot.
Thanks
Best Regard
LEO HU
CDSP
SAP LABS CHINA
From: Hu, Leo [mailto:leo.h...@sap.com]
Sent: Friday, December 05, 2014 10:23 AM
To: u...@spark.incubator.apache.org
Subject: spark assembly jar caused changed on src filesystem error
Hi all
when I execute:
/spark-1.1.1-bin-hadoop2.4/bin/spark-submit --verbose --master yarn-cluster
--class spark.SimpleApp --jars
/spark-1.1.1-bin-hadoop2.4/lib/spark-assembly-1.1.1-hadoop2.4.0.jar
--executor-memory 1G --num-executors 2
/spark-1.1.1-bin-hadoop2.4/testfile/simple-project-1.0.jar
It comes to this error:
appDiagnostics: Application application_1417686359838_0011 failed 2 times due
to AM Container for appattempt_1417686359838_0011_02 exited with exitCode:
-1000 due to: Resource
hdfs://192.168.70.23:9000/user/root/.sparkStaging/application_1417686359838_0011/spark-assembly-1.1.1-hadoop2.4.0.jar
changed on src filesystem (expected 1417745205278, was 1417745206168
when I remove the --jars option, namely execute:
/spark-1.1.1-bin-hadoop2.4/bin/spark-submit --verbose --master yarn-cluster
--class spark.SimpleApp --executor-memory 1G --num-executors 2
/spark-1.1.1-bin-hadoop2.4/testfile/simple-project-1.0.jar
Then it comes to the error below:
appDiagnostics: Application application_1417686359838_0012 failed 2 times due
to AM Container for appattempt_1417686359838_0012_02 exited with exitCode:
-1000 due to: File does not exist:
hdfs://192.168.70.23:9000/user/root/.sparkStaging/application_1417686359838_0012/spark-assembly-1.1.1-hadoop2.4.0.jar
.Failing this attempt.. Failing the application.
So my question is :
1. Will spark automatically upload the assembly jar to HDFS? Is it needed
to upload it by using --jars option manually?
2. If needed to upload with --jars option, what is the reason of the
changed on src filesystem error? How to solve it? And I have try to delete
all the jar files in .sparkStaging folder, but the error still exists.
I'll appreciate it very much if anyone help me with this. Thanks in advance.
Thanks
Best Regard
LEO HU
CDSP
SAP LABS CHINA