I am trying to play a little bit with apache-spark cluster mode.
So my cluster consists of a driver in my machine and a worker and manager in
host machine(separate machine).
I send a textfile using `sparkContext.addFile(filepath)` where the filepath
is the path of my text file in local machine for which I get the following
output:
INFO Utils: Copying /home/files/data.txt to
/tmp/spark-b2e2bb22-487b-412b-831d-19d7aa96f275/userFiles-147c9552-1a77-427e-9b17-cb0845807860/data.txt
INFO SparkContext: Added file /home/files/data.txt at
http://192.XX.XX.164:58143/files/data.txt with timestamp 1457432207649
But when I try to access the same file using `SparkFiles.get("data.txt")`, I
get the path to file in my driver instead of worker.
I am setting my file like this
SparkConf conf = new
SparkConf().setAppName("spark-play").setMaster("spark://192.XX.XX.172:7077");
conf.setJars(new String[]{"jars/SparkWorker.jar"});
JavaSparkContext sparkContext = new JavaSparkContext(conf);
sparkContext.addFile("/home/files/data.txt");
List<String> file
=sparkContext.textFile(SparkFiles.get("data.txt")).collect();
I am getting FileNotFoundException here.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/SparkFiles-get-returns-with-driver-path-Instead-of-Worker-Path-tp26428.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
