ah
this is what mapred.sytem.dir defaults to
<property>
<name>mapred.system.dir</name>
<value>${hadoop.tmp.dir}/mapred/system</value>
<description>The directory where MapReduce stores control files.
</description>
</property>
So thats why its trying to write to /data/hadoop/tmp/hadoop-mapred/mapred/system
So if you want hadoop.tmp.dir to be /data/hadoop/tmp/hadoop-${user.name} then
i would suggest that create /data/hadoop/tmp on hdfs and chmod -R 777 /data or
you can remove the hadoop.tmp.dir from your configs and let it be set to the
default value of
<property>
<name>hadoop.tmp.dir</name>
<value>/tmp/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
So to fix your problem you can do the above or set mapred.system.dir to
/tmp/mapred/system in your mapred-site.xml.
--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/
On Apr 30, 2013, at 7:55 AM, "Kevin Burton" <rkevinbur...@charter.net> wrote:
> In core-site.xml I have:
>
> <property>
> <name>fs.default.name</name>
> <value>hdfs://devubuntu05:9000</value>
> <description>The name of the default file system. A URI whose scheme and
> authority determine the FileSystem implementation. </description>
> </property>
>
> In hdfs-site.xml I have
>
> <property>
> <name>hadoop.tmp.dir</name>
> <value>/data/hadoop/tmp/hadoop-${user.name}</value>
> <description>Hadoop temporary folder</description>
> </property>
>
>
> From: Arpit Gupta [mailto:ar...@hortonworks.com]
> Sent: Tuesday, April 30, 2013 9:48 AM
> To: Kevin Burton
> Cc: user@hadoop.apache.org
> Subject: Re: Permission problem
>
> Based on the logs your system dir is set to
>
> hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system
>
>
> what is your fs.default.name and hadoop.tmp.dir in core-site.xml set to?
>
>
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Apr 30, 2013, at 7:39 AM, "Kevin Burton" <rkevinbur...@charter.net> wrote:
>
>
>
> Thank you.
>
> mapred.system.dir is not set. I am guessing that it is whatever the default
> is. What should I set it to?
>
> /tmp is already 777
>
> kevin@devUbuntu05:~$ hadoop fs -ls /tmp
> Found 1 items
> drwxr-xr-x - hdfs supergroup 0 2013-04-29 15:45 /tmp/mapred
> kevin@devUbuntu05:~$ hadoop fs -ls -d /tmp
> Found 1 items
> drwxrwxrwt - hdfs supergroup 0 2013-04-29 15:45 /tmp
>
> But notice that the mapred folder in the /tmp folder is 755.
> So I changed it:
>
> kevin@devUbuntu05 $ hadoop fs -ls -d /tmp
> drwxrwxrwt - hdfs supergroup 0 2013-04-29 15:45 /tmp
>
> kevin@devUbuntu05 $ hadoop fs -ls -R /tmp
> drwxr-xr-x - mapred supergroup 0 2013-04-29 15:45 /tmp/mapred
> drwxr-xr-x - mapred supergroup 0 2013-04-29 15:45
> /tmp/mapred/system
>
> I still get the errors in the log file:
>
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Failed to
> operate on mapred.system.dir
> (hdfs://devubuntu05:9000/data/hadoop/tmp/hadoop-mapred/mapred/system) because
> of permissions.
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: This
> directory should be owned by the user 'mapred (auth:SIMPLE)'
> 2013-04-30 09:35:11,609 WARN org.apache.hadoop.mapred.JobTracker: Bailing out
> ...
> . . . . .
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . .
> Caused by:
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException):
> Permission denied: user=mapred, access=WRITE,
> inode="/":hdfs:supergroup:drwxrwxr-x
> 2013-04-30 09:35:11,610 FATAL org.apache.hadoop.mapred.JobTracker:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
> . . . . . .
>
>
> From: Arpit Gupta [mailto:ar...@hortonworks.com]
> Sent: Tuesday, April 30, 2013 9:25 AM
> To: user@hadoop.apache.org
> Subject: Re: Permission problem
>
> what is your mapred.system.dir set to in mapred-site.xml?
>
> By default it will write to /tmp on hdfs.
>
> So you can do the following
>
> create /tmp on hdfs and chmod it to 777 as user hdfs and then restart
> jobtracker and tasktrackers.
>
> In case its set to /mapred/something then create /mapred and chown it to user
> mapred.
>
>
> --
> Arpit Gupta
> Hortonworks Inc.
> http://hortonworks.com/
>
> On Apr 30, 2013, at 6:36 AM, "Kevin Burton" <rkevinbur...@charter.net> wrote:
>
>
>
> To further complicate the issue the log file in
> (/var/log/hadoop-0.20-mapreduce/hadoop-hadoop-jobtracker-devUbuntu05.log) is
> owned by mapred:mapred and the name of the file seems to indicate some other
> lineage (hadoop,hadoop). I am out of my league in understanding the
> permission structure for hadoop hdfs and mr. Ideas?
>
> From: Kevin Burton [mailto:rkevinbur...@charter.net]
> Sent: Tuesday, April 30, 2013 8:31 AM
> To: user@hadoop.apache.org
> Cc: 'Mohammad Tariq'
> Subject: RE: Permission problem
>
> That is what I perceive as the problem. The hdfs file system was created with
> the user ‘hdfs’ owning the root (‘/’) but for some reason with a M/R job the
> user ‘mapred’ needs to have write permission to the root. I don’t know how to
> satisfy both conditions. That is one reason that I relaxed the permission to
> 775 so that the group would also have write permission but that didn’t seem
> to help.
>
> From: Mohammad Tariq [mailto:donta...@gmail.com]
> Sent: Tuesday, April 30, 2013 8:20 AM
> To: Kevin Burton
> Subject: Re: Permission problem
>
> user?"ls" shows "hdfs" and the log says "mapred"..
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Tue, Apr 30, 2013 at 6:22 PM, Kevin Burton <rkevinbur...@charter.net>
> wrote:
> I have relaxed it even further so now it is 775
>
> kevin@devUbuntu05:/var/log/hadoop-0.20-mapreduce$ hadoop fs -ls -d /
> Found 1 items
> drwxrwxr-x - hdfs supergroup 0 2013-04-29 15:43 /
>
> But I still get this error:
>
> 2013-04-30 07:43:02,520 FATAL org.apache.hadoop.mapred.JobTracker:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=mapred, access=WRITE, inode="/":hdfs:supergroup:drwxrwxr-x
>
>
> From: Mohammad Tariq [mailto:donta...@gmail.com]
> Sent: Monday, April 29, 2013 5:10 PM
> To: user@hadoop.apache.org
> Subject: Re: Incompartible cluserIDS
>
> make it 755.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
>
