RE: spark assembly jar caused changed on src filesystem error
If anybody knows the reason, please help me. thanks a lot. Thanks Best Regard LEO HU CDSP SAP LABS CHINA From: Hu, Leo [mailto:leo.h...@sap.com] Sent: Friday, December 05, 2014 10:23 AM To: u...@spark.incubator.apache.org Subject: spark assembly jar caused changed on src filesystem error Hi all when I execute: /spark-1.1.1-bin-hadoop2.4/bin/spark-submit --verbose --master yarn-cluster --class spark.SimpleApp --jars /spark-1.1.1-bin-hadoop2.4/lib/spark-assembly-1.1.1-hadoop2.4.0.jar --executor-memory 1G --num-executors 2 /spark-1.1.1-bin-hadoop2.4/testfile/simple-project-1.0.jar It comes to this error: appDiagnostics: Application application_1417686359838_0011 failed 2 times due to AM Container for appattempt_1417686359838_0011_02 exited with exitCode: -1000 due to: Resource hdfs://192.168.70.23:9000/user/root/.sparkStaging/application_1417686359838_0011/spark-assembly-1.1.1-hadoop2.4.0.jar changed on src filesystem (expected 1417745205278, was 1417745206168 when I remove the --jars option, namely execute: /spark-1.1.1-bin-hadoop2.4/bin/spark-submit --verbose --master yarn-cluster --class spark.SimpleApp --executor-memory 1G --num-executors 2 /spark-1.1.1-bin-hadoop2.4/testfile/simple-project-1.0.jar Then it comes to the error below: appDiagnostics: Application application_1417686359838_0012 failed 2 times due to AM Container for appattempt_1417686359838_0012_02 exited with exitCode: -1000 due to: File does not exist: hdfs://192.168.70.23:9000/user/root/.sparkStaging/application_1417686359838_0012/spark-assembly-1.1.1-hadoop2.4.0.jar .Failing this attempt.. Failing the application. So my question is : 1. Will spark automatically upload the assembly jar to HDFS? Is it needed to upload it by using --jars option manually? 2. If needed to upload with --jars option, what is the reason of the changed on src filesystem error? How to solve it? And I have try to delete all the jar files in .sparkStaging folder, but the error still exists. I'll appreciate it very much if anyone help me with this. Thanks in advance. Thanks Best Regard LEO HU CDSP SAP LABS CHINA
spark assembly jar caused changed on src filesystem error
Hi all when I execute: /spark-1.1.1-bin-hadoop2.4/bin/spark-submit --verbose --master yarn-cluster --class spark.SimpleApp --jars /spark-1.1.1-bin-hadoop2.4/lib/spark-assembly-1.1.1-hadoop2.4.0.jar --executor-memory 1G --num-executors 2 /spark-1.1.1-bin-hadoop2.4/testfile/simple-project-1.0.jar It comes to this error: appDiagnostics: Application application_1417686359838_0011 failed 2 times due to AM Container for appattempt_1417686359838_0011_02 exited with exitCode: -1000 due to: Resource hdfs://192.168.70.23:9000/user/root/.sparkStaging/application_1417686359838_0011/spark-assembly-1.1.1-hadoop2.4.0.jar changed on src filesystem (expected 1417745205278, was 1417745206168 when I remove the --jars option, namely execute: /spark-1.1.1-bin-hadoop2.4/bin/spark-submit --verbose --master yarn-cluster --class spark.SimpleApp --executor-memory 1G --num-executors 2 /spark-1.1.1-bin-hadoop2.4/testfile/simple-project-1.0.jar Then it comes to the error below: appDiagnostics: Application application_1417686359838_0012 failed 2 times due to AM Container for appattempt_1417686359838_0012_02 exited with exitCode: -1000 due to: File does not exist: hdfs://192.168.70.23:9000/user/root/.sparkStaging/application_1417686359838_0012/spark-assembly-1.1.1-hadoop2.4.0.jar .Failing this attempt.. Failing the application. So my question is : 1. Will spark automatically upload the assembly jar to HDFS? Is it needed to upload it by using --jars option manually? 2. If needed to upload with --jars option, what is the reason of the changed on src filesystem error? How to solve it? And I have try to delete all the jar files in .sparkStaging folder, but the error still exists. I'll appreciate it very much if anyone help me with this. Thanks in advance. Thanks Best Regard LEO HU CDSP SAP LABS CHINA
spark1.0.1 hadoop2.2.0 issue
Hi all Have anyone encounter such problem below, and how to solve it ? any help would be appreciated. Caused by: java.lang.UnsatisfiedLinkError: org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V at org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native Method) at org.apache.hadoop.security.JniBasedUnixGroupsMapping.clinit(JniBasedUnixGroupsMapping.java:49) at org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.init(JniBasedUnixGroupsMappingWithFallback.java:38) Best Regards Leo Hu