make sure
"/mnt/local/1024gbxvdf1/all_adleads_cleaned_commas_in_quotes_good_file.csv" is
accessible on your slave node.
--
Ali
On Nov 9, 2015, at 6:06 PM, Sanjay Subramanian
<sanjaysubraman...@yahoo.com.INVALID> wrote:
> hey guys
>
> I have a 2 node SparkR (1 master 1 slave)cluster on AWS using
> spark-1.5.1-bin-without-hadoop.tgz
>
> Running the SparkR job on the master node
>
> /opt/spark-1.5.1-bin-hadoop2.6/bin/sparkR --master
> spark://ip-xx-ppp-vv-ddd:7077 --packages com.databricks:spark-csv_2.10:1.2.0
> --executor-cores 16 --num-executors 8 --executor-memory 8G --driver-memory 8g
> myRprogram.R
>
>
> org.apache.spark.SparkException: Job aborted due to stage failure: Task 17
> in stage 1.0 failed 4 times, most recent failure: Lost task 17.3 in stage 1.0
> (TID 103, xx.ff.rr.tt): java.io.FileNotFoundException: File
> file:/mnt/local/1024gbxvdf1/all_adleads_cleaned_commas_in_quotes_good_file.csv
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:534)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:747)
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:524)
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:409)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140)
> at
> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
> at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecord
>
>
>
>
>
> myRprogram.R
>
> library(SparkR)
>
> sc <- sparkR.init(appName="SparkR-CancerData-example")
> sqlContext <- sparkRSQL.init(sc)
>
> lds <- read.df(sqlContext,
> "file:///mnt/local/1024gbxvdf1/all_adleads_cleaned_commas_in_quotes_good_file.csv",
> "com.databricks.spark.csv", header="true")
> sink("file:///mnt/local/1024gbxvdf1/leads_new_data_analyis.txt")
> summary(lds)
>
>
> This used to run when we had a single node SparkR installation
>
> regards
>
> sanjay
>
>
