: Wed, 9 Sep 2015 22:29
To: shahab;
Cc: user@spark.apache.org;
Subject:Re: [Spark on Amazon EMR] : File does not exist:
hdfs://ip-x-x-x-x:/.../spark-assembly-1.4.1-hadoop2.6.0-amzn-0.jar
If you log into the cluster, do you see the file if you type:
hdfs dfs -ls
hdfs://ipx-x-x-x:8020/user/hadoop
Thank you all for the comments, but my problem still exists.
@Dean,@Ewan yes, I do have hadoop file system installed and working
@Sujit: the last version of EMR (version 4) does not need manual copying
of jar file to the server. The blog that you pointed out refers to older
version (3.x) of EMR.
15 22:29
> To: shahab;
> Cc: user@spark.apache.org;
> Subject:Re: [Spark on Amazon EMR] : File does not exist:
> hdfs://ip-x-x-x-x:/.../spark-assembly-1.4.1-hadoop2.6.0-amzn-0.jar
> If you log into the cluster, do you see the file if you type:
> hdfs dfs -ls
> hdfs://ipx-x-x-x:802
Hi,
I am using Spark on Amazon EMR. So far I have not succeeded to submit the
application successfully, not sure what's problem. In the log file I see
the followings.
java.io.FileNotFoundException: File does not exist:
If you log into the cluster, do you see the file if you type:
hdfs dfs
-ls
hdfs://ipx-x-x-x:8020/user/hadoop/.sparkStaging/application_123344567_0018/spark-assembly-1.4.1-hadoop2.6.0-amzn-0.jar
(with the correct server address for "ipx-x-x-x"). If not, is the server
address correct and routable
