I had face a similar issue. The actual problem was not in the file name. We run Spark on Yarn. The actual problem was seen in the logs by running the command: $ yarn logs -applicationId <app-id>
Scroll from the beginning to know the actual error. ~Pratik On Fri, Oct 23, 2015 at 11:40 AM kali.tumm...@gmail.com < kali.tumm...@gmail.com> wrote: > Hi All, > > got this weird error when I tried to run spark on YARN-CLUSTER mode , I > have > 33 files and I am looping spark in bash one by one most of them worked ok > except few files. > > Is this below error HDFS or spark error ? > > Exception in thread "Driver" java.lang.IllegalArgumentException: Pathname > /user/myid/-u/12:51/_temporary/0 from > hdfs://dev/user/myid/-u/12:51/_temporary/0 is not a valid DFS filename. > > File Name which I passed to spark , does file name causes issue ? > > > hdfs://dev/data/20151019/sipmktdata.ColorDataArchive.UTD.P4_M-P.v5.2015-09-18.txt.20150918 > > Thanks > Sri > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Saprk-error-Not-a-valid-DFS-File-name-tp25186.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
