[jira] [Commented] (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13016880#comment-13016880 ] Hudson commented on MAPREDUCE-1382: --- Integrated in Hadoop-Mapreduce-trunk #643 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/]) > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch, MAPREDUCE-1382.5.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch, > MAPREDUCE-1382.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12986091#action_12986091 ] Hudson commented on MAPREDUCE-1382: --- Integrated in Hadoop-Mapreduce-trunk-Commit #581 (See [https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/581/]) MAPREDUCE-1382. MRAsyncDiscService should tolerate missing local.dir. Contributed by Zheng Shao and tomwhite. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Fix For: 0.22.0 > > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch, MAPREDUCE-1382.5.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch, > MAPREDUCE-1382.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985935#action_12985935 ] Todd Lipcon commented on MAPREDUCE-1382: +1, seems like a good fix. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch, MAPREDUCE-1382.5.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch, > MAPREDUCE-1382.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12983576#action_12983576 ] Hadoop QA commented on MAPREDUCE-1382: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12467776/MAPREDUCE-1382.5.patch against trunk revision 1058374. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestControlledMapReduceJob org.apache.hadoop.mapred.TestSetupTaskScheduling -1 contrib tests. The patch failed contrib unit tests. -1 system test framework. The patch failed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/14//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/14//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/14//console This message is automatically generated. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch, MAPREDUCE-1382.5.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12980583#action_12980583 ] Todd Lipcon commented on MAPREDUCE-1382: +1, looks good to me. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch, MAPREDUCE-1382.5.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907814#action_12907814 ] Scott Chen commented on MAPREDUCE-1382: --- Zheng, Is it possible that you can add a simple test for this bug? Use some non-exist volume name to construct a MRAsyncDiskService. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[
https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907812#action_12907812
]
Scott Chen commented on MAPREDUCE-1382:
---
The try { } block in cleanupAllVolumes() also needs the fix Dan pointed out.
> MRAsyncDiscService should tolerate missing local.dir
>
>
> Key: MAPREDUCE-1382
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Reporter: Scott Chen
>Assignee: Zheng Shao
> Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch,
> MAPREDUCE-1382.3.patch, MAPREDUCE-1382.4.patch,
> MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch
>
>
> Currently when some of the local.dir do not exist, MRAsyncDiscService will
> fail. It should only fail when all directories don't work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907494#action_12907494 ] Dan Adkins commented on MAPREDUCE-1382: --- I have applied the latest patch to trunk and *still* I cannot start a task tracker if the the directory does not exist: 10/09/09 00:36:29 WARN mapred.TaskTracker: Task Tracker local can not create directory: /mnt/data8/mr20 10/09/09 00:36:29 WARN util.MRAsyncDiskService: Cannot create toBeDeleted in /mnt/data8/mr20. Ignored. 10/09/09 00:36:29 ERROR mapred.TaskTracker: Can not start task tracker because java.io.FileNotFoundException: File /mnt/data8/mr20/toBeDeleted does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:301) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1071) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1096) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:494) at org.apache.hadoop.mapreduce.util.MRAsyncDiskService.(MRAsyncDiskService.java:103) at org.apache.hadoop.mapreduce.util.MRAsyncDiskService.(MRAsyncDiskService.java:127) at org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:616) at org.apache.hadoop.mapred.TaskTracker.(TaskTracker.java:1323) at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:3508) > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907489#action_12907489 ] Scott Chen commented on MAPREDUCE-1382: --- +1, the patch looks good to me > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907441#action_12907441 ] Zheng Shao commented on MAPREDUCE-1382: --- I believe other logic in TaskTracker/JobTracker will fail and report in that case. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12907439#action_12907439 ] Scott Chen commented on MAPREDUCE-1382: --- Does it make sense to throw IOE if all volumes in the constructor are missing? > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, MAPREDUCE-1382.2.patch, > MAPREDUCE-1382.3.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1382) MRAsyncDiscService should tolerate missing local.dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-1382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803467#action_12803467 ] Todd Lipcon commented on MAPREDUCE-1382: Hi Zheng, I wonder whether localFileSystem.listStatus will ever throw IOE in the case of a bad volume? Yours handles the case where the volume doesn't exist at all, but I can imagine if it's on a broken mount, the listStatus will get an I/O Error from the underlying filesystem and throw at that point. Do you need to catch this and handle in the same way as if it were null? Thanks. > MRAsyncDiscService should tolerate missing local.dir > > > Key: MAPREDUCE-1382 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1382 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Scott Chen >Assignee: Zheng Shao > Attachments: MAPREDUCE-1382.1.patch, > MAPREDUCE-1382.branch-0.20.on.top.of.MAPREDUCE-1302.1.patch > > > Currently when some of the local.dir do not exist, MRAsyncDiscService will > fail. It should only fail when all directories don't work. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
