[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847788#action_12847788 ] Hudson commented on MAPREDUCE-890: -- Integrated in Hadoop-Mapreduce-trunk #264 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/264/]) Move from 0.21 to 0.22 in changelog to match commit > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.22.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, > MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847729#action_12847729 ] Hudson commented on MAPREDUCE-890: -- Integrated in Hadoop-Mapreduce-trunk-Commit #285 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/285/]) > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.22.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.20S.patch, MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, > MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842597#action_12842597 ] Vinod K V commented on MAPREDUCE-890: - I've just committed this to trunk. Thanks Ravi! The trunk patch isn't applying to 0.21 branch. I think we'll need a new one, can you work on it? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12842579#action_12842579 ] Vinod K V commented on MAPREDUCE-890: - For some reason, Hudson's couldn't report which tests failed on its web portal, even though the console output is still there. I've downloaded the console output and saw that all the tests passed except TestTaskTrackerMemoryManager. And this failure is because of the same wierd java.lang.NoClassDefFoundError on some core classes that's been plaguing other test runs, this time on org.apache.hadoop.http.HttpServer$QuotingInputFilter. I could run TestTaskTrackerMemoryManager successfully. I am going to check this in. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841839#action_12841839 ] Hadoop QA commented on MAPREDUCE-890: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console This message is automatically generated. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.2.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841747#action_12841747 ] Vinod K V commented on MAPREDUCE-890: - The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why? Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841226#action_12841226 ] Hadoop QA commented on MAPREDUCE-890: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437589/MR890.v1.1.patch against trunk revision 918864. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/console This message is automatically generated. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12841102#action_12841102 ] Vinod K V commented on MAPREDUCE-890: - Ravi, can you run tests and test-patch? Hudson is not picking patches up, so.. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12840174#action_12840174 ] Vinod K V commented on MAPREDUCE-890: - Patch looks good to me. +1, pending MAPREDUCE-1421. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.1.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839600#action_12839600 ] Vinod K V commented on MAPREDUCE-890: - The patch is not applying over the latest patch at MAPREDUCE-1421 which is a blocker for this JIRA issue. Can you please upload an updated patch? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch, MR890.v1.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828564#action_12828564 ] Ravi Gummadi commented on MAPREDUCE-890: I don't see much value in changing all LinuxTaskController based tests to take default value for taskcontroller-ugi when taskcontroller-path is specified(anyway user has to specify taskcontroller-path for the tests to be run) and taskcontroller-ugi is not specified. Do you really think it is useful ? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828056#action_12828056 ] Amareshwari Sriramadasu commented on MAPREDUCE-890: --- why don't we run LinuxTaskController tests as the test-launcher(thereby as TT), if nothing is passed in taskcontroller-ugi ? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12828053#action_12828053 ] Ravi Gummadi commented on MAPREDUCE-890: As MAPREDUCE-1429 will take care of running the linux task controller based tests with both different user and tt_user(and also changing the source code of all tests is not maintainable --- as future linux task controller based testcases should also consciously follow this duplication of testcases for both users), I will drop this(Vinod's 3rd comment) from this patch. Incorporating first 2 comments. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt, > MR890.patch > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12805856#action_12805856
]
Vinod K V commented on MAPREDUCE-890:
-
Looked at the patch. Got some comments:
- After patch, at +643, in {{initialize_user()}} function, the patch
incorrectly passes 0 to {{secure_path()}}'s shouldCheckOwnership argument.
- TestTrackerDistributedCacheManager* tests will need modifications too, I
guess.
- It's already a pain running the task-controller tests. With this patch, the
expectation is to run these 5-odd tests with $tt_user and $$non_tt_user, that
is two times. I think we should modify the tests themselves so that this double
test-run is automatically done.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Reporter: Karam Singh
>Assignee: Ravi Gummadi
>Priority: Blocker
> Fix For: 0.21.0
>
> Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt,
> MR890.patch
>
>
> Even setup and cleanup task of job fails due exception -: It fails to create
> job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804963#action_12804963 ] Ravi Gummadi commented on MAPREDUCE-890: Yes Hemanth. Check is in task-controller. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804636#action_12804636 ] Hemanth Yamijala commented on MAPREDUCE-890: I would like to give it some more thought. But for now, I think Approach 1 seems to be the right way to go (in spite of the special check). I am assuming the special check is in the task-controller ? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12804375#action_12804375 ] Ravi Gummadi commented on MAPREDUCE-890: Another solution is to call DefaultTaskController's methods from LinuxTaskContrller's methods if user(i.e. jobOwner) is tt_user. This will keep 700 permissions for userDir, jobcache, distcache dirs and so are secure and are accessible to jobOwner. As I don't see any issues with either of these 2 solutions, currently I am planning to implement 1st approach(i.e. LinuxTaskController setting 770 permissions for jobcache and distcache dirs if jobOwner is tt_user). Thoughts ? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Ravi Gummadi >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12802718#action_12802718 ] Vinod K V commented on MAPREDUCE-890: - The patch above that implements the 2nd proposal needs more changes. bq. Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir With this, unwarranted files/dirs with improper permissions in job-dir will prevent TT from recursively deleting all the files - a manifestation of MAPREDUCE-896 in the job-dir. This should also be fixed in this patch itself. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Vinod K V >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12776390#action_12776390 ] Hadoop QA commented on MAPREDUCE-890: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419014/MAPREDUCE-890-20090909.txt against trunk revision 834284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/console This message is automatically generated. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Vinod K V >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12758191#action_12758191 ] Vinod K V commented on MAPREDUCE-890: - The same (latest) patch works post feature-freeze even with all the patches that went in. > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh >Assignee: Vinod K V >Priority: Blocker > Fix For: 0.21.0 > > Attachments: MAPREDUCE-890-20090904.txt, MAPREDUCE-890-20090909.txt > > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745784#action_12745784
]
Vinod K V commented on MAPREDUCE-890:
-
In other words, the problem is the following:
{code}
$ mkdir testing
$ ls -ld testing/
drwxr-xr-x 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
$ touch testing/t1.txt
$ echo $?
0
$ chmod 0570 testing/
$ ls -ld testing/
dr-xrwx--- 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
$ touch testing/t2.txt
touch: cannot touch `testing/t2.txt': Permission denied
{code}
I searched and hunted this down in the Linux Kernel filesystem code:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/namei.c;h=f3c5b278895a0d3e0f23fe6fd474e2728a1c6cb6;hb=6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f
. Particularly the generic_permission() function where we check if
current_fsuid() == inode->i_uid.
> After HADOOP-4491, the user who started mapred system is not able to run job.
> -
>
> Key: MAPREDUCE-890
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-890
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Reporter: Karam Singh
>Assignee: Vinod K V
>
> Even setup and cleanup task of job fails due exception -: It fails to create
> job and related directories under mapred.local.dir/taskTracker/jobcache
> Directories are created as -:
> [dr-xrws--- mapred hadoop ] job_200908190916_0002
> mapred is not wrtie under this. Even manually I failed to touch file.
> mapred is use of started mr cluster
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-890) After HADOOP-4491, the user who started mapred system is not able to run job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12745783#action_12745783 ] Vinod K V commented on MAPREDUCE-890: - As part of MAPREDUCE-842, job directories are set to have the following permissions: ||permissions||user ownership||group ownership||file/dir name|| |dr-xrws--- |$job-owner| $tt_group |job_200908190916_0002| CASE I: the $job-owner is other than the $tt_user $tt_user is part of $tt_group, and so can create attempt directories inside the job directory as part of the task-localization because job_directory is group writable. CASE II: the $job-owner is same as the $tt_user TT cannot create attempt directories inside the job directory!! Because Linux seems to check uid of the process with the fsuid of the directory and return error if the directory is not user writable! CASE II is what is causing the current bug. Alternative solutions at hand: - Leave the code as is, and live with the fact that user's cannot submit jobs as mapred user - Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir - Set drwxrws-- if the $job-owner is same as the $tt_user, set dr-xrws--- otherwise. Handles both the cases, but complicates code very slightly. Thoughts? > After HADOOP-4491, the user who started mapred system is not able to run job. > - > > Key: MAPREDUCE-890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-890 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Reporter: Karam Singh > > Even setup and cleanup task of job fails due exception -: It fails to create > job and related directories under mapred.local.dir/taskTracker/jobcache > Directories are created as -: > [dr-xrws--- mapred hadoop ] job_200908190916_0002 > mapred is not wrtie under this. Even manually I failed to touch file. > mapred is use of started mr cluster -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
