Ignore the question. There was a Hadoop setting that needed to be set to get it working.
-- Kannan On Wed, Apr 1, 2015 at 1:37 PM, Kannan Rajah <kra...@maprtech.com> wrote: > Running a simple word count job in standalone mode as a non root user from > spark-shell. The spark master, worker services are running as root user. > > The problem is the _temporary under /user/krajah/output2/_temporary/0 dir > is being created with root permission even when running the job as non root > user - krajah in this case. The higher level directories are getting > created with right permission though. There was a similar question posted > long time back, but there is no answer: > http://mail-archives.apache.org/mod_mbox/mesos-user/201408.mbox/%3CCAAeYHL2M9J9xEotf_0zXmZXy2_x-oBHa=xxl2naft203o6u...@mail.gmail.com%3E > > > *Wrong permission for child directory* > drwxr-xr-x - root root 0 2015-04-01 11:20 > /user/krajah/output2/_temporary/0/_temporary > > > *Right permission for parent directories* > hadoop fs -ls -R /user/krajah/my_output > drwxr-xr-x - krajah krajah 1 2015-04-01 11:46 > /user/krajah/my_output/_temporary > drwxr-xr-x - krajah krajah 3 2015-04-01 11:46 > /user/krajah/my_output/_temporary/0 > > *Job and Stacktrace* > > scala> val file = sc.textFile("/user/krajah/junk.txt") > scala> val counts = file.flatMap(line => line.split(" ")) > scala> .map(word => (word, 1)) > scala> .reduceByKey(_ + _) > > scala> counts.saveAsTextFile("/user/krajah/count2") > java.io.IOException: Error: Permission denied > at com.mapr.fs.MapRFileSystem.rename(MapRFileSystem.java:926) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:362) > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) > at > org.apache.hadoop.mapred.FileOutputCommitter.commitJob(FileOutputCommitter.java:136) > at > org.apache.spark.SparkHadoopWriter.commitJob(SparkHadoopWriter.scala:127) > at > org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopDataset(PairRDDFunctions.scala:1079) > at > org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:944) > at > org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:853) > at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1199) > at $iwC$$iwC$$iwC$$iwC.<init>(<console>:17) > at $iwC$$iwC$$iwC.<init>(<console>:22) > at $iwC$$iwC.<init>(<console>:24) > at $iwC.<init>(<console>:26) > at <init>(<console>:28) > at .<init>(<console>:32) > at .<clinit>(<console>) > at .<init>(<console>:7) > at .<clinit>(<console>) > at $print(<console>) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > -- > Kannan >
