What is your config set to for mapred local dirs? And what are the permissions to those directories?
All users need executable permissions in all the paths up to the local-dir so that they can create their own directories in there. For e.g. if one of the mapred local dir is /a/b/c/mapred, then all of /a, /a/b, /a/b/c etc need to be executable by everyone - an executable permission is needed in a linux dir for someone to be able to create files/dir in some of the sub-directories. Thanks, +Vinod Kumar Vavilapalli Hortonworks Inc. http://hortonworks.com/ On Sep 18, 2013, at 7:26 AM, Christopher Penney wrote: > I have a test environment with hadoop 1.1.1 setup with Kerberos and yesterday > I zapped my mapred.local.dir on the job and task trackers as part of some > cleanup. When I started the task trackers back up I was unable to run MR > jobs. This seems like a permission issue, but I can't figure out what it > would be since it auto creates everything. I didn't make any changes to > taskcontroller.cfg or mapred-site.xml. Below is a log from the task tracker. > > Chris > > 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: > LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_0 task's > state:UNASSIGNED > 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: Trying to > launch : attempt_201309180916_0024_m_000002_0 which needs 1 slots > 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: In > TaskLauncher, current free slots : 16 and trying to launch > attempt_201309180916_0024_m_000002_0 which needs 1 slots > 2013-09-18 10:21:28,524 WARN org.apache.hadoop.mapred.TaskTracker: Error > initializing attempt_201309180916_0024_m_000002_0: > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find > taskTracker/cpenney/jobcache/job_201309180916_0024/job.xml in any of the > configured local directories > at > org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) > at > org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1341) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568) > at java.lang.Thread.run(Thread.java:662) > > 2013-09-18 10:21:28,525 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to > set finish time for task attempt_201309180916_0024_m_000002_0 when no start > time is set, stackTrace is : java.lang.Exception > at org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145) > at > org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3285) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2578) > at java.lang.Thread.run(Thread.java:662) > > 2013-09-18 10:21:28,525 INFO org.apache.hadoop.mapred.TaskTracker: > addFreeSlot : current free slots : 16 > 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: > LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_1 task's > state:UNASSIGNED > 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: Trying to > launch : attempt_201309180916_0024_m_000002_1 which needs 1 slots > 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: In > TaskLauncher, current free slots : 16 and trying to launch > attempt_201309180916_0024_m_000002_1 which needs 1 slots > 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Reading > task controller config from /etc/hadoop/taskcontroller.cfg > 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main : > command provided 0 > 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main : > user is cpenney > 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Good > mapred-local-dirs are /tmp/hadoop/mapred > 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Can't > open > /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken > for output - File exists > 2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker: Exception > while localization java.io.IOException: Job initialization failed (255) with > output: Reading task controller config from /etc/hadoop/taskcontroller.cfg > main : command provided 0 > main : user is cpenney > Good mapred-local-dirs are /tmp/hadoop/mapred > Can't open > /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken > for output - File exists > > at > org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1298) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.hadoop.util.Shell$ExitCodeException: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) > at org.apache.hadoop.util.Shell.run(Shell.java:182) > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) > at > org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186) > ... 8 more > > 2013-09-18 10:21:28,596 ERROR > org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException > as:cpenney cause:java.io.IOException: Job initialization failed (255) with > output: Reading task controller config from /etc/hadoop/taskcontroller.cfg > main : command provided 0 > main : user is cpenney > Good mapred-local-dirs are /tmp/hadoop/mapred > Can't open > /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken > for output - File exists > > 2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker: Error > initializing attempt_201309180916_0024_m_000002_1: > java.io.IOException: Job initialization failed (255) with output: Reading > task controller config from /etc/hadoop/taskcontroller.cfg > main : command provided 0 > main : user is cpenney > Good mapred-local-dirs are /tmp/hadoop/mapred > Can't open > /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken > for output - File exists > > at > org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193) > at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) > at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1298) > at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213) > at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568) > at java.lang.Thread.run(Thread.java:662) > Caused by: org.apache.hadoop.util.Shell$ExitCodeException: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) > at org.apache.hadoop.util.Shell.run(Shell.java:182) > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) > at > org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186) > ... 8 more > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
signature.asc
Description: Message signed with OpenPGP using GPGMail
