Read a HDFS file from Spark source code

2014-11-11 Thread rapelly kartheek 
Hi

I am trying to access a file in HDFS from spark source code. Basically, I
am tweaking the spark source code. I need to access a file in HDFS from the
source code of the spark. I am really not understanding how to go about
doing this.

Can someone please help me out in this regard.
Thank you!!
Karthik

Re: Read a HDFS file from Spark source code

2014-11-11 Thread Samarth Mailinglist 
Instead of a file path, use a HDFS URI.
For example: (In Python)



data = sc.textFile(hdfs://localhost/user/someuser/data)

​

On Wed, Nov 12, 2014 at 10:12 AM, rapelly kartheek kartheek.m...@gmail.com
wrote:

 Hi

 I am trying to access a file in HDFS from spark source code. Basically,
 I am tweaking the spark source code. I need to access a file in HDFS from
 the source code of the spark. I am really not understanding how to go about
 doing this.

 Can someone please help me out in this regard.
 Thank you!!
 Karthik

Re: Read a HDFS file from Spark source code

2014-11-11 Thread rapelly kartheek 
Hi Sean,
I was following this link;

http://mund-consulting.com/Blog/Posts/file-operations-in-HDFS-using-java.aspx

But, I was facing FileSystem ambiguity error. I really don't have any idea
as to how to go about doing this.
Can you please help me how to start off with this?


On Wed, Nov 12, 2014 at 11:26 AM, Samarth Mailinglist 
mailinglistsama...@gmail.com wrote:

 Instead of a file path, use a HDFS URI.
 For example: (In Python)



 data = sc.textFile(hdfs://localhost/user/someuser/data)

 ​

 On Wed, Nov 12, 2014 at 10:12 AM, rapelly kartheek 
 kartheek.m...@gmail.com wrote:

 Hi

 I am trying to access a file in HDFS from spark source code. Basically,
 I am tweaking the spark source code. I need to access a file in HDFS from
 the source code of the spark. I am really not understanding how to go about
 doing this.

 Can someone please help me out in this regard.
 Thank you!!
 Karthik