I have a test environment with hadoop 1.1.1 setup with Kerberos and yesterday I zapped my mapred.local.dir on the job and task trackers as part of some cleanup. When I started the task trackers back up I was unable to run MR jobs. This seems like a permission issue, but I can't figure out what it would be since it auto creates everything. I didn't make any changes to taskcontroller.cfg or mapred-site.xml. Below is a log from the task tracker.
Chris 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_0 task's state:UNASSIGNED 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201309180916_0024_m_000002_0 which needs 1 slots 2013-09-18 10:21:27,040 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 16 and trying to launch attempt_201309180916_0024_m_000002_0 which needs 1 slots 2013-09-18 10:21:28,524 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201309180916_0024_m_000002_0: org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find taskTracker/cpenney/jobcache/job_201309180916_0024/job.xml in any of the configured local directories at org.apache.hadoop.fs.LocalDirAllocator$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:429) at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead(LocalDirAllocator.java:160) at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1341) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213) at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568) at java.lang.Thread.run(Thread.java:662) 2013-09-18 10:21:28,525 ERROR org.apache.hadoop.mapred.TaskStatus: Trying to set finish time for task attempt_201309180916_0024_m_000002_0 when no start time is set, stackTrace is : java.lang.Exception at org.apache.hadoop.mapred.TaskStatus.setFinishTime(TaskStatus.java:145) at org.apache.hadoop.mapred.TaskTracker$TaskInProgress.kill(TaskTracker.java:3285) at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2578) at java.lang.Thread.run(Thread.java:662) 2013-09-18 10:21:28,525 INFO org.apache.hadoop.mapred.TaskTracker: addFreeSlot : current free slots : 16 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: LaunchTaskAction (registerTask): attempt_201309180916_0024_m_000002_1 task's state:UNASSIGNED 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: Trying to launch : attempt_201309180916_0024_m_000002_1 which needs 1 slots 2013-09-18 10:21:28,554 INFO org.apache.hadoop.mapred.TaskTracker: In TaskLauncher, current free slots : 16 and trying to launch attempt_201309180916_0024_m_000002_1 which needs 1 slots 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Reading task controller config from /etc/hadoop/taskcontroller.cfg 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main : command provided 0 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: main : user is cpenney 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Good mapred-local-dirs are /tmp/hadoop/mapred 2013-09-18 10:21:28,595 INFO org.apache.hadoop.mapred.TaskController: Can't open /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken for output - File exists 2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker: Exception while localization java.io.IOException: Job initialization failed (255) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg main : command provided 0 main : user is cpenney Good mapred-local-dirs are /tmp/hadoop/mapred Can't open /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken for output - File exists at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1298) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213) at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) at org.apache.hadoop.util.Shell.run(Shell.java:182) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186) ... 8 more 2013-09-18 10:21:28,596 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:cpenney cause:java.io.IOException: Job initialization failed (255) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg main : command provided 0 main : user is cpenney Good mapred-local-dirs are /tmp/hadoop/mapred Can't open /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken for output - File exists 2013-09-18 10:21:28,596 WARN org.apache.hadoop.mapred.TaskTracker: Error initializing attempt_201309180916_0024_m_000002_1: java.io.IOException: Job initialization failed (255) with output: Reading task controller config from /etc/hadoop/taskcontroller.cfg main : command provided 0 main : user is cpenney Good mapred-local-dirs are /tmp/hadoop/mapred Can't open /tmp/hadoop/mapred/taskTracker/cpenney/jobcache/job_201309180916_0024/jobToken for output - File exists at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:193) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1323) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1298) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1213) at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2568) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:255) at org.apache.hadoop.util.Shell.run(Shell.java:182) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:375) at org.apache.hadoop.mapred.LinuxTaskController.initializeJob(LinuxTaskController.java:186) ... 8 more
