I am trying to play a little bit with apache-spark cluster mode. So my cluster consists of a driver in my machine and a worker and manager in host machine(separate machine).
I send a textfile using `sparkContext.addFile(filepath)` where the filepath is the path of my text file in local machine for which I get the following output: INFO Utils: Copying /home/files/data.txt to /tmp/spark-b2e2bb22-487b-412b-831d-19d7aa96f275/userFiles-147c9552-1a77-427e-9b17-cb0845807860/data.txt INFO SparkContext: Added file /home/files/data.txt at http://192.XX.XX.164:58143/files/data.txt with timestamp 1457432207649 But when I try to access the same file using `SparkFiles.get("data.txt")`, I get the path to file in my driver instead of worker. I am setting my file like this SparkConf conf = new SparkConf().setAppName("spark-play").setMaster("spark://192.XX.XX.172:7077"); conf.setJars(new String[]{"jars/SparkWorker.jar"}); JavaSparkContext sparkContext = new JavaSparkContext(conf); sparkContext.addFile("/home/files/data.txt"); List<String> file =sparkContext.textFile(SparkFiles.get("data.txt")).collect(); I am getting FileNotFoundException here. -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkFiles-get-returns-with-driver-path-Instead-of-Worker-Path-tp26428.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org