General note: The /root is a protected local directory, meaning that if
your program spawns as a non-root user, it will never be able to access the
file.
On Sat, Dec 12, 2015 at 12:21 AM Zhan Zhang wrote:
> As Sean mentioned, you cannot referring to the local file in
*From:* Vijay Gharge [mailto:vijay.gha...@gmail.com]
>> *Sent:* Friday, December 11, 2015 12:31 PM
>> *To:* Lin, Hao
>> *Cc:* user@spark.apache.org
>> *Subject:* Re: how to access local file from Spark
>> sc.textFile("file:///path to/myfile")
>>
>>
Hm, are you referencing a local file from your remote workers? That
won't work as the file only exists in one machine (I presume).
On Fri, Dec 11, 2015 at 5:19 PM, Lin, Hao wrote:
> Hi,
>
>
>
> I have problem accessing local file, with such example:
>
>
>
>
Here you go, thanks.
-rw-r--r-- 1 root root 658M Dec 9 2014 /root/2008.csv
From: Vijay Gharge [mailto:vijay.gha...@gmail.com]
Sent: Friday, December 11, 2015 12:31 PM
To: Lin, Hao
Cc: user@spark.apache.org
Subject: Re: how to access local file from Spark sc.textFile("file:///path
to/m
jay Gharge [mailto:vijay.gha...@gmail.com]
>>> *Sent:* Friday, December 11, 2015 12:31 PM
>>> *To:* Lin, Hao
>>> *Cc:* user@spark.apache.org
>>> *Subject:* Re: how to access local file from Spark
>>> sc.textFile("file:///path
Can you provide output of "ls -lh /root/2008.csv" ?
On Friday 11 December 2015, Lin, Hao wrote:
> Hi,
>
>
>
> I have problem accessing local file, with such example:
>
>
>
> sc.textFile("file:///root/2008.csv").count()
>
>
>
> with error: File file:/root/2008.csv does not
>
>
>
> *From:* Vijay Gharge [mailto:vijay.gha...@gmail.com
> <javascript:_e(%7B%7D,'cvml','vijay.gha...@gmail.com');>]
> *Sent:* Friday, December 11, 2015 12:31 PM
> *To:* Lin, Hao
> *Cc:* user@spark.apache.org
> <javascript:_e(%7B%7D,'cvml','user@spark.apache.org
vascript:_e(%7B%7D,'cvml','user@spark.apache.org');>
Subject: Re: how to access local file from Spark sc.textFile("file:///path
to/myfile")
Can you provide output of "ls -lh /root/2008.csv" ?
On Friday 11 December 2015, Lin, Hao
<hao@finra.org<javascript:_e(%7B%7D,'cvml',
To: Lin, Hao
Cc: user@spark.apache.org
Subject: Re: how to access local file from Spark sc.textFile("file:///path
to/myfile")
Hm, are you referencing a local file from your remote workers? That won't work
as the file only exists in one machine (I presume).
On Fri, Dec 11, 2015 at 5:19 PM
As Sean mentioned, you cannot referring to the local file in your remote
machine (executors). One walk around is to copy the file to all machines within
same directory.
Thanks.
Zhan Zhang
On Dec 11, 2015, at 10:26 AM, Lin, Hao
> wrote:
of the
10 matches
Mail list logo