Please ignore typo. I meant root "permissions" Regards, Vijay Gharge
On Fri, Dec 11, 2015 at 11:30 PM, Vijay Gharge <vijay.gha...@gmail.com> wrote: > This issue is due to file permission issue. You need to execute spark > operations using root command only. > > > > Regards, > Vijay Gharge > > > > On Fri, Dec 11, 2015 at 11:20 PM, Vijay Gharge <vijay.gha...@gmail.com> > wrote: > >> One more question. Are you also running spark commands using root user ? >> Meanwhile am trying to simulate this locally. >> >> >> On Friday 11 December 2015, Lin, Hao <hao....@finra.org> wrote: >> >>> Here you go, thanks. >>> >>> >>> >>> -rw-r--r-- 1 root root 658M Dec 9 2014 /root/2008.csv >>> >>> >>> >>> *From:* Vijay Gharge [mailto:vijay.gha...@gmail.com] >>> *Sent:* Friday, December 11, 2015 12:31 PM >>> *To:* Lin, Hao >>> *Cc:* user@spark.apache.org >>> *Subject:* Re: how to access local file from Spark >>> sc.textFile("file:///path to/myfile") >>> >>> >>> >>> Can you provide output of "ls -lh /root/2008.csv" ? >>> >>> On Friday 11 December 2015, Lin, Hao <hao....@finra.org> wrote: >>> >>> Hi, >>> >>> >>> >>> I have problem accessing local file, with such example: >>> >>> >>> >>> sc.textFile("file:///root/2008.csv").count() >>> >>> >>> >>> with error: File file:/root/2008.csv does not exist. >>> >>> The file clearly exists since, since if I missed type the file name to >>> an non-existing one, it will show: >>> >>> >>> >>> Error: Input path does not exist >>> >>> >>> >>> Please help! >>> >>> >>> >>> The following is the error message: >>> >>> >>> >>> scala> sc.textFile("file:///root/2008.csv").count() >>> >>> 15/12/11 17:12:08 WARN TaskSetManager: Lost task 15.0 in stage 8.0 (TID >>> 498, 10.162.167.24): java.io.FileNotFoundException: File >>> file:/root/2008.csv does not exist >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511) >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724) >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501) >>> >>> at >>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397) >>> >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:137) >>> >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339) >>> >>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) >>> >>> at >>> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108) >>> >>> at >>> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) >>> >>> at >>> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:239) >>> >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:216) >>> >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) >>> >>> at >>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>> >>> at org.apache.spark.scheduler.Task.run(Task.scala:88) >>> >>> at >>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> >>> 15/12/11 17:12:08 ERROR TaskSetManager: Task 9 in stage 8.0 failed 4 >>> times; aborting job >>> >>> org.apache.spark.SparkException: Job aborted due to stage failure: Task >>> 9 in stage 8.0 failed 4 times, most recent failure: Lost task 9.3 in stage >>> 8.0 (TID 547, 10.162.167.23): java.io.FileNotFoundException: File >>> file:/root/2008.csv does not exist >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511) >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724) >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501) >>> >>> at >>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397) >>> >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:137) >>> >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339) >>> >>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) >>> >>> at >>> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108) >>> >>> at >>> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) >>> >>> at >>> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:239) >>> >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:216) >>> >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) >>> >>> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) >>> >>> at >>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>> >>> at org.apache.spark.scheduler.Task.run(Task.scala:88) >>> >>> at >>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> >>> Driver stacktrace: >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler.org[org.apache.spark.scheduler.DAGScheduler.org] >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.scheduler.DAGScheduler.org&d=CwMFaQ&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=DrYQ-tpUMZyhNn2eTUcbxSrkZEun1MPFJ0GOpQDM_QY&s=UOpsCySW-Pmg6pcqzATD8m0RW5tv4SpkJcBc4tn27ug&e=> >>> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1283) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1271) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1270) >>> >>> at >>> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) >>> >>> at >>> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) >>> >>> at scala.Option.foreach(Option.scala:236) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) >>> >>> at >>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496) >>> >>> at >>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458) >>> >>> at >>> org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447) >>> >>> at >>> org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) >>> >>> at >>> org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567) >>> >>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) >>> >>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837) >>> >>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850) >>> >>> at org.apache.spark.SparkContext.runJob(SparkContext.scala:1921) >>> >>> at org.apache.spark.rdd.RDD.count(RDD.scala:1125) >>> >>> at >>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:25) >>> >>> at >>> $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:30) >>> >>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:32) >>> >>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:34) >>> >>> at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:36) >>> >>> at $iwC$$iwC$$iwC$$iwC$$iwC.<init>(<console>:38) >>> >>> at $iwC$$iwC$$iwC$$iwC.<init>(<console>:40) >>> >>> at $iwC$$iwC$$iwC.<init>(<console>:42) >>> >>> at $iwC$$iwC.<init>(<console>:44) >>> >>> at $iwC.<init>(<console>:46) >>> >>> at <init>(<console>:48) >>> >>> at .<init>(<console>:52) >>> >>> at .<clinit>(<console>) >>> >>> at .<init>(<console>:7) >>> >>> at .<clinit>(<console>) >>> >>> at $print(<console>) >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> >>> at >>> org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) >>> >>> at >>> org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) >>> >>> at >>> org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) >>> >>> at >>> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) >>> >>> at >>> org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) >>> >>> at >>> org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) >>> >>> at >>> org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) >>> >>> at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) >>> >>> at >>> org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) >>> >>> at >>> org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) >>> >>> at >>> org.apache.spark.repl.SparkILoop.org[org.apache.spark.repl.SparkILoop.org] >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.repl.SparkILoop.org&d=CwMFaQ&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=DrYQ-tpUMZyhNn2eTUcbxSrkZEun1MPFJ0GOpQDM_QY&s=f5Ft3xWMAILo1b-3HzZThdEDDl3jO5JAh27ljFXmv_c&e=> >>> $apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670) >>> >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997) >>> >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >>> >>> at >>> org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) >>> >>> at >>> scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) >>> >>> at >>> org.apache.spark.repl.SparkILoop.org[org.apache.spark.repl.SparkILoop.org] >>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__org.apache.spark.repl.SparkILoop.org&d=CwMFaQ&c=XK1GVu0Y2HvWRiFNJ9Hesw&r=uIybaSSiVvR1Uni2EecKYCQDa6UHqV0QDbyaKNjHuMM&m=DrYQ-tpUMZyhNn2eTUcbxSrkZEun1MPFJ0GOpQDM_QY&s=f5Ft3xWMAILo1b-3HzZThdEDDl3jO5JAh27ljFXmv_c&e=> >>> $apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) >>> >>> at >>> org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) >>> >>> at org.apache.spark.repl.Main$.main(Main.scala:31) >>> >>> at org.apache.spark.repl.Main.main(Main.scala) >>> >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> >>> at >>> org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674) >>> >>> at >>> org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180) >>> >>> at >>> org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205) >>> >>> at >>> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120) >>> >>> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) >>> >>> Caused by: java.io.FileNotFoundException: File file:/root/2008.csv does >>> not exist >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:511) >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724) >>> >>> at >>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501) >>> >>> at >>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397) >>> >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:137) >>> >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339) >>> >>> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:764) >>> >>> at >>> org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:108) >>> >>> at >>> org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67) >>> >>> at >>> org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:239) >>> >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:216) >>> >>> at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:101) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) >>> >>> at >>> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) >>> >>> at >>> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:300) >>> >>> at org.apache.spark.rdd.RDD.iterator(RDD.scala:264) >>> >>> at >>> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) >>> >>> at org.apache.spark.scheduler.Task.run(Task.scala:88) >>> >>> at >>> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> >>> at java.lang.Thread.run(Thread.java:745) >>> >>> >>> >>> Confidentiality Notice:: This email, including attachments, may include >>> non-public, proprietary, confidential or legally privileged information. If >>> you are not an intended recipient or an authorized agent of an intended >>> recipient, you are hereby notified that any dissemination, distribution or >>> copying of the information contained in or transmitted with this e-mail is >>> unauthorized and strictly prohibited. If you have received this email in >>> error, please notify the sender by replying to this message and permanently >>> delete this e-mail, its attachments, and any copies of it immediately. You >>> should not retain, copy or use this e-mail or any attachment for any >>> purpose, nor disclose all or any part of the contents to any other person. >>> Thank you. >>> >>> >>> >>> -- >>> >>> Regards, >>> >>> Vijay Gharge >>> >>> >>> Confidentiality Notice:: This email, including attachments, may include >>> non-public, proprietary, confidential or legally privileged information. If >>> you are not an intended recipient or an authorized agent of an intended >>> recipient, you are hereby notified that any dissemination, distribution or >>> copying of the information contained in or transmitted with this e-mail is >>> unauthorized and strictly prohibited. If you have received this email in >>> error, please notify the sender by replying to this message and permanently >>> delete this e-mail, its attachments, and any copies of it immediately. You >>> should not retain, copy or use this e-mail or any attachment for any >>> purpose, nor disclose all or any part of the contents to any other person. >>> Thank you. >>> >> >> >> -- >> Regards, >> Vijay Gharge >> >> >> >> >
