If you put a * in the path, Spark will look for a file or directory named *. To read all the files in a directory, just remove the star.
-- Michael Mior michael.m...@gmail.com On Jun 22, 2017 17:21, "saatvikshah1994" <saatvikshah1...@gmail.com> wrote: > Hi, > > I've downloaded and kept the same set of data files on all my cluster > nodes, > in the same absolute path - say /home/xyzuser/data/*. I am now trying to > perform an operation(say open(filename).read()) on all these files in > spark, > but by passing local file paths. I was under the assumption that as long as > the worker can find the file path it will be able to execute it. However, > my > Spark tasks fail with the error(/home/xyzuser/data/* is not present) - and > Im sure its present on all my worker nodes. > > If this experiment was successful I was planning to setup a NFS (actually > more like a read-only cloud persistent disk connected to my cluster nodes > in > dataproc) and use that instead. > > What exactly is going wrong here? > > Thanks > > > > -- > View this message in context: http://apache-spark-user-list. > 1001560.n3.nabble.com/Using-Spark-with-Local-File-System-NFS-tp28781.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
