[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13564086#comment-13564086 ] Matt Foley commented on MAPREDUCE-4396: --- The patch for HADOOP-8734 is the same in LocalJobRunner.java. However, that patch includes a proper unit test change in TestMRWithDistributedCache.java. @Yu, your comment above that This patch can be verified by running unit test TestMRWithDistributedCache in a user directory with restrictive permission (e.g. 700) is correct, but the conclusion that therefore no new unit test is necessary is incorrect. What's necessary is a unit test that does precisely that. The patch from HADOOP-8734 provides such a patch. I've merged the unit test patch to hadoop-1 and hadoop-1.1. Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13532790#comment-13532790 ] Eric Yang commented on MAPREDUCE-4396: -- +1 looks good. Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13532791#comment-13532791 ] Eric Yang commented on MAPREDUCE-4396: -- I just committed this. Thanks Ivan and Yu. Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13529474#comment-13529474 ] Yu Gao commented on MAPREDUCE-4396: --- @[~eyang] Trunk does not have this issue. Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13529477#comment-13529477 ] Yu Gao commented on MAPREDUCE-4396: --- Hmm, the patch here is the same as that in HADOOP-8734 for LocalJobRunner, so either way is ok. Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13528577#comment-13528577 ] Ivan Mitic commented on MAPREDUCE-4396: --- This was fixed with HADOOP-8734 in branch-1-win. Maybe just integrate the same patch to branch-1? Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13527640#comment-13527640 ] Eric Yang commented on MAPREDUCE-4396: -- Does trunk need a patch? Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4396) Make LocalJobRunner work with private distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-4396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492762#comment-13492762 ] Yu Gao commented on MAPREDUCE-4396: --- Attached ant test and test-patch result. This patch can be verified by running unit test TestMRWithDistributedCache in a user directory with restrictive permission (e.g. 700), so no new test is added. Before this patch, TestMRWithDistributedCache#testLocalJobRunner could fail when the test directory - test.build.data - is in a path where any ancestor doesn't have +x permissions for others, with following exception: java.io.FileNotFoundException: File build/test/mapred/local/archive/5659425182264546915_1878220893_1556512037/file/home/biadmin/Hadoop_upstream/branch-1/build/test/data/distributed.first does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:402) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) at org.apache.hadoop.filecache.TestMRWithDistributedCache$DistributedCacheChecker.setup(TestMRWithDistributedCache.java:93) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212) This results from private distributed cache not localized by LocalJobRunner. The patch adds the localization for private distributed cache, so that no matter whether the test directory permission is too restrictive or not, distributed cache (public/private) will be localized. Make LocalJobRunner work with private distributed cache --- Key: MAPREDUCE-4396 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4396 Project: Hadoop Map/Reduce Issue Type: Bug Components: client Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor Attachments: mapreduce-4396-branch-1.patch, test-afterpatch.result, test-beforepatch.result, test-patch.result Some LocalJobRunner related unit tests fails if user directory permission and/or umask is too restrictive. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
