Just to note that in cluster mode the spark driver might run on any node of the cluster, hence you need to make sure that the file exists on *all* nodes. Push the file on all nodes or use client deploy-mode.
Best, Anastasios Am 24.06.2017 23:24 schrieb "Holden Karau" <hol...@pigscanfly.ca>: > addFile is supposed to not depend on a shared FS unless the semantics have > changed recently. > > On Sat, Jun 24, 2017 at 11:55 AM varma dantuluri <dvsnva...@gmail.com> > wrote: > >> Hi Sudhir, >> >> I believe you have to use a shared file system that is accused by all >> nodes. >> >> >> On Jun 24, 2017, at 1:30 PM, sudhir k <k.sudhi...@gmail.com> wrote: >> >> >> I am new to Spark and i need some guidance on how to fetch files from >> --files option on Spark-Submit. >> >> I read on some forums that we can fetch the files from >> Spark.getFiles(fileName) and can use it in our code and all nodes should >> read it. >> >> But i am facing some issue >> >> Below is the command i am using >> >> spark-submit --deploy-mode cluster --class com.check.Driver --files >> /home/sql/first.sql test.jar 20170619 >> >> so when i use SparkFiles.get(first.sql) , i should be able to read the >> file Path but it is throwing File not Found exception. >> >> I tried SpackContext.addFile(/home/sql/first.sql) and then >> SparkFiles.get(first.sql) but still the same error. >> >> Its working on the stand alone mode but not on cluster mode. Any help is >> appreciated.. Using Spark 2.1.0 and Scala 2.11 >> >> Thanks. >> >> >> Regards, >> Sudhir K >> >> >> >> -- >> Regards, >> Sudhir K >> >> >> -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau >