Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

General note: The /root is a protected local directory, meaning that if your program spawns as a non-root user, it will never be able to access the file. On Sat, Dec 12, 2015 at 12:21 AM Zhan Zhang wrote: > As Sean mentioned, you cannot referring to the local file in

how to access local file from Spark sc.textFile("file:///path to/myfile")

Hi, I have problem accessing local file, with such example: sc.textFile("file:///root/2008.csv").count() with error: File file:/root/2008.csv does not exist. The file clearly exists since, since if I missed type the file name to an non-existing one, it will show: Error: Input path does not

Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

*From:* Vijay Gharge [mailto:vijay.gha...@gmail.com] >> *Sent:* Friday, December 11, 2015 12:31 PM >> *To:* Lin, Hao >> *Cc:* user@spark.apache.org >> *Subject:* Re: how to access local file from Spark >> sc.textFile("file:///path to/myfile") >> >>

Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

Hm, are you referencing a local file from your remote workers? That won't work as the file only exists in one machine (I presume). On Fri, Dec 11, 2015 at 5:19 PM, Lin, Hao wrote: > Hi, > > > > I have problem accessing local file, with such example: > > > >

RE: how to access local file from Spark sc.textFile("file:///path to/myfile")

Here you go, thanks. -rw-r--r-- 1 root root 658M Dec 9 2014 /root/2008.csv From: Vijay Gharge [mailto:vijay.gha...@gmail.com] Sent: Friday, December 11, 2015 12:31 PM To: Lin, Hao Cc: user@spark.apache.org Subject: Re: how to access local file from Spark sc.textFile("file:///path to/m

Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

jay Gharge [mailto:vijay.gha...@gmail.com] >>> *Sent:* Friday, December 11, 2015 12:31 PM >>> *To:* Lin, Hao >>> *Cc:* user@spark.apache.org >>> *Subject:* Re: how to access local file from Spark >>> sc.textFile("file:///path

Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

Can you provide output of "ls -lh /root/2008.csv" ? On Friday 11 December 2015, Lin, Hao wrote: > Hi, > > > > I have problem accessing local file, with such example: > > > > sc.textFile("file:///root/2008.csv").count() > > > > with error: File file:/root/2008.csv does not

Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

> > > > *From:* Vijay Gharge [mailto:vijay.gha...@gmail.com > <javascript:_e(%7B%7D,'cvml','vijay.gha...@gmail.com');>] > *Sent:* Friday, December 11, 2015 12:31 PM > *To:* Lin, Hao > *Cc:* user@spark.apache.org > <javascript:_e(%7B%7D,'cvml','user@spark.apache.org

RE: how to access local file from Spark sc.textFile("file:///path to/myfile")

: how to access local file from Spark sc.textFile("file:///path to/myfile") One more question. Are you also running spark commands using root user ? Meanwhile am trying to simulate this locally. On Friday 11 December 2015, Lin, Hao <hao@finra.org<mailto:hao@finra.org>&

RE: how to access local file from Spark sc.textFile("file:///path to/myfile")

To: Lin, Hao Cc: user@spark.apache.org Subject: Re: how to access local file from Spark sc.textFile("file:///path to/myfile") Hm, are you referencing a local file from your remote workers? That won't work as the file only exists in one machine (I presume). On Fri, Dec 11, 2015 at 5:19 PM

Re: how to access local file from Spark sc.textFile("file:///path to/myfile")

As Sean mentioned, you cannot referring to the local file in your remote machine (executors). One walk around is to copy the file to all machines within same directory. Thanks. Zhan Zhang On Dec 11, 2015, at 10:26 AM, Lin, Hao > wrote: of the