Read a HDFS file from Spark using HDFS API
Hi, I am trying to read a HDFS file from Spark scheduler code. I could find how to write hdfs read/writes in java. But I need to access hdfs from spark using scala. Can someone please help me in this regard.
Re: Read a HDFS file from Spark using HDFS API
like this? val file = sc.textFile(hdfs://localhost:9000/sigmoid/input.txt) Thanks Best Regards On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek kartheek.m...@gmail.com wrote: Hi, I am trying to read a HDFS file from Spark scheduler code. I could find how to write hdfs read/writes in java. But I need to access hdfs from spark using scala. Can someone please help me in this regard.
Re: Read a HDFS file from Spark using HDFS API
Can you not create SparkContext inside the scheduler code? If you are looking just to access hdfs then you can use the following object with it, you can create/read/write files. val hdfs = org.apache.hadoop.fs.FileSystem.get(new URI(hdfs://localhost:9000), hadoopConf) Thanks Best Regards On Fri, Nov 14, 2014 at 9:12 PM, rapelly kartheek kartheek.m...@gmail.com wrote: No. I am not accessing hdfs from either shell or a spark application. I want to access from spark Scheduler code. I face an error when I use sc.textFile() as SparkContext wouldn't have been created yet. So, error says: sc not found. On Fri, Nov 14, 2014 at 9:07 PM, Akhil Das ak...@sigmoidanalytics.com wrote: like this? val file = sc.textFile(hdfs://localhost:9000/sigmoid/input.txt) Thanks Best Regards On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek kartheek.m...@gmail.com wrote: Hi, I am trying to read a HDFS file from Spark scheduler code. I could find how to write hdfs read/writes in java. But I need to access hdfs from spark using scala. Can someone please help me in this regard.
Re: Read a HDFS file from Spark using HDFS API
[image: Inline image 1] Thanks Best Regards On Fri, Nov 14, 2014 at 9:18 PM, Bui, Tri tri@verizonwireless.com.invalid wrote: It should be val file = sc.textFile(hdfs:///localhost:9000/sigmoid/input.txt) 3 “///” Thanks Tri *From:* rapelly kartheek [mailto:kartheek.m...@gmail.com] *Sent:* Friday, November 14, 2014 9:42 AM *To:* Akhil Das; user@spark.apache.org *Subject:* Re: Read a HDFS file from Spark using HDFS API No. I am not accessing hdfs from either shell or a spark application. I want to access from spark Scheduler code. I face an error when I use sc.textFile() as SparkContext wouldn't have been created yet. So, error says: sc not found. On Fri, Nov 14, 2014 at 9:07 PM, Akhil Das ak...@sigmoidanalytics.com wrote: like this? val file = sc.textFile(hdfs://localhost:9000/sigmoid/input.txt) Thanks Best Regards On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek kartheek.m...@gmail.com wrote: Hi, I am trying to read a HDFS file from Spark scheduler code. I could find how to write hdfs read/writes in java. But I need to access hdfs from spark using scala. Can someone please help me in this regard.
Re: Read a HDFS file from Spark using HDFS API
I'll just try out with object Akhil provided. There was no problem working in shell with sc.textFile. Thank you Akhil and Tri. On Fri, Nov 14, 2014 at 9:21 PM, Akhil Das ak...@sigmoidanalytics.com wrote: [image: Inline image 1] Thanks Best Regards On Fri, Nov 14, 2014 at 9:18 PM, Bui, Tri tri@verizonwireless.com.invalid wrote: It should be val file = sc.textFile(hdfs:///localhost:9000/sigmoid/input.txt) 3 “///” Thanks Tri *From:* rapelly kartheek [mailto:kartheek.m...@gmail.com] *Sent:* Friday, November 14, 2014 9:42 AM *To:* Akhil Das; user@spark.apache.org *Subject:* Re: Read a HDFS file from Spark using HDFS API No. I am not accessing hdfs from either shell or a spark application. I want to access from spark Scheduler code. I face an error when I use sc.textFile() as SparkContext wouldn't have been created yet. So, error says: sc not found. On Fri, Nov 14, 2014 at 9:07 PM, Akhil Das ak...@sigmoidanalytics.com wrote: like this? val file = sc.textFile(hdfs://localhost:9000/sigmoid/input.txt) Thanks Best Regards On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek kartheek.m...@gmail.com wrote: Hi, I am trying to read a HDFS file from Spark scheduler code. I could find how to write hdfs read/writes in java. But I need to access hdfs from spark using scala. Can someone please help me in this regard.
Re: Read a HDFS file from Spark using HDFS API
Hi Akhil, I face error: not found : value URI On Fri, Nov 14, 2014 at 9:29 PM, rapelly kartheek kartheek.m...@gmail.com wrote: I'll just try out with object Akhil provided. There was no problem working in shell with sc.textFile. Thank you Akhil and Tri. On Fri, Nov 14, 2014 at 9:21 PM, Akhil Das ak...@sigmoidanalytics.com wrote: [image: Inline image 1] Thanks Best Regards On Fri, Nov 14, 2014 at 9:18 PM, Bui, Tri tri@verizonwireless.com.invalid wrote: It should be val file = sc.textFile(hdfs:///localhost:9000/sigmoid/input.txt) 3 “///” Thanks Tri *From:* rapelly kartheek [mailto:kartheek.m...@gmail.com] *Sent:* Friday, November 14, 2014 9:42 AM *To:* Akhil Das; user@spark.apache.org *Subject:* Re: Read a HDFS file from Spark using HDFS API No. I am not accessing hdfs from either shell or a spark application. I want to access from spark Scheduler code. I face an error when I use sc.textFile() as SparkContext wouldn't have been created yet. So, error says: sc not found. On Fri, Nov 14, 2014 at 9:07 PM, Akhil Das ak...@sigmoidanalytics.com wrote: like this? val file = sc.textFile(hdfs://localhost:9000/sigmoid/input.txt) Thanks Best Regards On Fri, Nov 14, 2014 at 9:02 PM, rapelly kartheek kartheek.m...@gmail.com wrote: Hi, I am trying to read a HDFS file from Spark scheduler code. I could find how to write hdfs read/writes in java. But I need to access hdfs from spark using scala. Can someone please help me in this regard.
