[ 
https://issues.apache.org/jira/browse/MAPREDUCE-5354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13693508#comment-13693508
 ] 

Arpit Gupta commented on MAPREDUCE-5354:
----------------------------------------

So some job submissions will go through and some will fail based on which ones 
end up using the read only disk. It would be better if jobtracker could detect 
this bad disk and ignore it for future use. Otherwise users will have remove 
this dir from the config and restart jobtracker and thus loose all the jobs in 
progress or if they have recovery enabled they will start from scratch.
                
> some job submission fail when one of the configured local disks on jt is read 
> only
> ----------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-5354
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5354
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>    Affects Versions: 1.3.0
>            Reporter: Arpit Gupta
>
> Following exception is seen in the jt
> {code}
> 2013-06-25 06:58:26,988 INFO org.apache.hadoop.mapred.JobTracker: jobToken 
> generated and stored with users keys in 
> /mapred/system/job_201306250645_0002/jobToken
> 2013-06-25 06:58:26,997 ERROR 
> org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException 
> as:user cause:java.io.IOException: java.io.FileNotFoundException: 
> /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file 
> system)
> 2013-06-25 06:58:26,997 INFO org.apache.hadoop.ipc.Server: IPC Server handler 
> 33 on 50300, call submitJob(job_201306250645_0002, 
> hdfs://hostname:8020/user/hrt_qa/.staging/job_201306250645_0002, 
> org.apache.hadoop.security.Credentials@34e77781) from 68.142.244.30:56631: 
> error: java.io.IOException: java.io.FileNotFoundException: 
> /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file 
> system)
> java.io.IOException: java.io.FileNotFoundException: 
> /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file 
> system)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3615)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3561)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1444)
>         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1440)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1438)
> Caused by: java.io.FileNotFoundException: 
> /grid/2/hdp/mapred/local/jobTracker/job_201306250645_0002.xml (Read-only file 
> system)
>         at java.io.FileOutputStream.open(Native Method)
>         at java.io.FileOutputStream.<init>(FileOutputStream.java:194)
>         at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:193)
>         at 
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileOutputStream.<init>(RawLocalFileSystem.java:189)
>         at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:260)
>         at 
> org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:241)
>         at 
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSOutputSummer.<init>(ChecksumFileSystem.java:335)
>         at 
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:381)
>         at 
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:364)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:555)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:536)
>         at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:443)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:229)
>         at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:163)
>         at 
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1248)
>         at 
> org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:1229)
>         at 
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:411)
>         at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3613)
>         ... 12 more
> 2013-06-25 06:58:27,079 INFO 
> org.apache.hadoop.mapred.JobInitializationPoller: Passing to Initializer Job 
> Id :job_2013062506
> {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Reply via email to