Arpit Gupta created MAPREDUCE-5354: -------------------------------------- Summary: some job submission fail when one of the configured local disks on jt is read only Key: MAPREDUCE-5354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5354 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Arpit Gupta
Following exception is seen in the jt {code} 2013-06-25 06:58:26,988 INFO org.apache.hadoop.mapred.JobTracker: jobToken generated and stored with users keys in /mapred/system/job_201306250645_0002/jobToken 2013-06-25 06:58:26,997 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:user cause:java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system) 2013-06-25 06:58:26,997 INFO org.apache.hadoop.ipc.Server: IPC Server handler 33 on 50300, call submitJob(job_201306250645_0002, hdfs://hostname:8020/user/hrt_qa/.staging/job_201306250645_0002, org.apache.hadoop.security.Credentials@34e77781) from 68.142.244.30:56631: error: java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system) java.io.IOException: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438) Caused by: java.io.FileNotFoundException: /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file system) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:194) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:193) at org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:260) at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381) at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536) at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229) at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1248) at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1229) at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:411) at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613) ... 12 more 2013-06-25 06:58:27,079 INFO org.apache.hadoop.mapred.JobInitializationPoller: Passing to Initializer Job Id :job_2013062506 {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira