I have a file in HDFS inside my HortonWorks HDP 2.3_1 VirtualBox VM.
If I go into the guest spark-shell and refer to the file thus, it works fine
val words=sc.textFile("hdfs:///tmp/people.txt")
words.count
However if I try to access it from a local Spark app on my Windows host, it
doesn't work
val conf = new SparkConf().setMaster("local").setAppName("My App")
val sc = new SparkContext(conf)
val words=sc.textFile("hdfs://localhost:8020/tmp/people.txt")
words.count
Emits
The port 8020 is open, and if I choose the wrong file name, it will tell me
My pom has
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>1.4.1</version>
<scope>provided</scope>
</dependency>
Am I doing something wrong?
Thanks.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Local-Spark-talking-to-remote-HDFS-tp24425.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
