[jira] Commented: (MAPREDUCE-2239) BlockPlacementPolicyRaid should call getBlockLocations only when necessary
[ https://issues.apache.org/jira/browse/MAPREDUCE-2239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002493#comment-13002493 ] Ramkumar Vadali commented on MAPREDUCE-2239: +1 Patch looks good > BlockPlacementPolicyRaid should call getBlockLocations only when necessary > -- > > Key: MAPREDUCE-2239 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2239 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/raid >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2239-1.txt, MAPREDUCE-2239-2.txt, > MAPREDUCE-2239-3.txt, MAPREDUCE-2239.txt > > > Currently BlockPlacementPolicyRaid calls getBlockLocations for every > chooseTarget(). > This puts pressure on NameNode. We should avoid calling if this file is not > raided or a parity file. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2345) Optimize jobtracker's memory usage
[ https://issues.apache.org/jira/browse/MAPREDUCE-2345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] MengWang updated MAPREDUCE-2345: Description: Too many tasks will eat up a considerable amount of JobTracker's heap space. According to our observation, 50GB heap size can support to 5,000,000 tasks, so we should optimize jobtracker's memory usage for more jobs and tasks. Yourkit java profile show that counters, duplicate strings, task waste too much memory. Our optimization around these three points reduced jobtracker's memory to 1/3. (was: To many tasks will eat up a considerable amount of JobTracker's heap space. According to our observation, 50GB heap size can support to 5,000,000 tasks, so we should optimize jobtracker's memory usage for more jobs and tasks. Yourkit java profile show that counters, duplicate strings, Task waste too much memory. Our optimization around these three points reduced jobtracker's memory to 1/3. ) > Optimize jobtracker's memory usage > - > > Key: MAPREDUCE-2345 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2345 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobtracker >Affects Versions: 0.21.0 >Reporter: MengWang > Labels: hadoop > Fix For: 0.23.0 > > Attachments: jt-memory-useage.bmp > > > Too many tasks will eat up a considerable amount of JobTracker's heap space. > According to our observation, 50GB heap size can support to 5,000,000 tasks, > so we should optimize jobtracker's memory usage for more jobs and tasks. > Yourkit java profile show that counters, duplicate strings, task waste too > much memory. Our optimization around these three points reduced jobtracker's > memory to 1/3. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2331) Add coverage of task graph servlet to fair scheduler system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002467#comment-13002467 ] Hadoop QA commented on MAPREDUCE-2331: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472626/mapreduce-2331.txt against trunk revision 1076804. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/115//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/115//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/115//console This message is automatically generated. > Add coverage of task graph servlet to fair scheduler system test > > > Key: MAPREDUCE-2331 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2331 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: mapreduce-2331.txt > > > Would be useful to hit the TaskGraph servlet in the fair scheduler system > test. This way, when run under JCarder, it will check for any lock inversions > in this code. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2331) Add coverage of task graph servlet to fair scheduler system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002449#comment-13002449 ] Tom White commented on MAPREDUCE-2331: -- +1 looks good. > Add coverage of task graph servlet to fair scheduler system test > > > Key: MAPREDUCE-2331 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2331 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: mapreduce-2331.txt > > > Would be useful to hit the TaskGraph servlet in the fair scheduler system > test. This way, when run under JCarder, it will check for any lock inversions > in this code. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-437) JobTracker must ask for a new FS instance and close it when terminated.
[ https://issues.apache.org/jira/browse/MAPREDUCE-437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-437: Status: Open (was: Patch Available) > JobTracker must ask for a new FS instance and close it when terminated. > --- > > Key: MAPREDUCE-437 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-437 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 0.21.0, 0.20.1, 0.22.0, 0.23.0 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Fix For: 0.23.0 > > Attachments: MAPREDUCE-437.patch > > Original Estimate: 1.5h > Time Spent: 2h > Remaining Estimate: 0h > > This is something I've been experimenting with HADOOP-3268; I'm not sure what > the right action is here. > -currently, the JobTracker does not close() its filesystem when it is shut > down. This will cause it to leak filesystem references if JobTrackers are > started and stopped in the same process. > -The TestMRServerPorts test explicitly closes the filesystem > jt.fs.close(); > jt.stopTracker(); > -If you move the close() operation into the stopTracker()/terminate logic, > the filesystem gets cleaned up, but > TestRackAwareTaskPlacement and TestMultipleLevelCaching fail with a > FilesystemClosed error (stack traces to follow) > Should the JobTracker close its filesystem whenever it is terminated? If so, > there are some tests that need to be reworked slightly to not expect the > fileystem to be live after the jobtracker is taken down. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-1242) Chain APIs error misleading
[ https://issues.apache.org/jira/browse/MAPREDUCE-1242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1242: - Resolution: Fixed Fix Version/s: (was: 0.22.0) 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. Thanks Harsh! > Chain APIs error misleading > --- > > Key: MAPREDUCE-1242 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-1242 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Amogh Vasekar >Assignee: Harsh J Chouraria >Priority: Trivial > Fix For: 0.23.0 > > Attachments: MAPREDUCE-1242.patch, MAPREDUCE-1242.r2.patch > > > Hi, > I was using the Chain[Mapper/Reducer] APIs , and in Class Chain line 207 the > error thrown : > "The Mapper output key class does not match the previous Mapper input key > class" > Shouldn't this be "The Mapper *input* key class does not match the previous > Mapper *Output* key class" ? Sort of misleads :) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2241) ClusterWithLinuxTaskController should accept relative path on the command line
[ https://issues.apache.org/jira/browse/MAPREDUCE-2241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002446#comment-13002446 ] Tom White commented on MAPREDUCE-2241: -- +1 > ClusterWithLinuxTaskController should accept relative path on the command line > -- > > Key: MAPREDUCE-2241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2241 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: task-controller, test >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Attachments: mr-2241.txt > > > Currently if you pass a relative path for the -Dtaskcontroller-path option > when running these tests, it fails in a fairly unintuitive way. We should > absolutize it inside the tests to make it easier for people to run them. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2323) Add metrics to the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002435#comment-13002435 ] Hadoop QA commented on MAPREDUCE-2323: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472627/mr-2323.txt against trunk revision 1076804. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/116//console This message is automatically generated. > Add metrics to the fair scheduler > - > > Key: MAPREDUCE-2323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt > > > It would be useful to be able to monitor various metrics in the fair > scheduler, like demand, fair share, min share, and running task count. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2323) Add metrics to the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-2323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2323: --- Attachment: mr-2323.txt Slightly improved patch - the main code is the same, but this fixes some flaky behavior in the tests. I previously wasn't unregistering the metrics updater in fairsched's termination method, so different test cases were interacting poorly with eachother. This should be ready for review. > Add metrics to the fair scheduler > - > > Key: MAPREDUCE-2323 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2323 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.23.0 > > Attachments: mr-2323-20.txt, mr-2323.txt, mr-2323.txt > > > It would be useful to be able to monitor various metrics in the fair > scheduler, like demand, fair share, min share, and running task count. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2331) Add coverage of task graph servlet to fair scheduler system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2331: --- Attachment: mapreduce-2331.txt > Add coverage of task graph servlet to fair scheduler system test > > > Key: MAPREDUCE-2331 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2331 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: mapreduce-2331.txt > > > Would be useful to hit the TaskGraph servlet in the fair scheduler system > test. This way, when run under JCarder, it will check for any lock inversions > in this code. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2331) Add coverage of task graph servlet to fair scheduler system test
[ https://issues.apache.org/jira/browse/MAPREDUCE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated MAPREDUCE-2331: --- Status: Patch Available (was: Open) > Add coverage of task graph servlet to fair scheduler system test > > > Key: MAPREDUCE-2331 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2331 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 0.22.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: 0.22.0 > > Attachments: mapreduce-2331.txt > > > Would be useful to hit the TaskGraph servlet in the fair scheduler system > test. This way, when run under JCarder, it will check for any lock inversions > in this code. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2351) mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-2351: - Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) I've just committed this. > mapred.job.tracker.history.completed.location should support an arbitrary > filesystem URI > > > Key: MAPREDUCE-2351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2351 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2351.patch > > > Currently, mapred.job.tracker.history.completed.location is resolved relative > to the default filesystem. If not set it defaults to history/done in the > local log directory. There is no way to set it to another local filesystem > location (with a file:// URI) or an arbitrary Hadoop filesystem. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2353) Make the MR changes to reflect the API changes in SecureIO library
[ https://issues.apache.org/jira/browse/MAPREDUCE-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13001976#comment-13001976 ] Devaraj Das commented on MAPREDUCE-2353: bq. Patch for trunk (from the MR changes introduced by the commit in the 20.100 security branch - ref HADOOP-7119) I meant HADOOP-7115 > Make the MR changes to reflect the API changes in SecureIO library > -- > > Key: MAPREDUCE-2353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2353 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: security, task, tasktracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das > Attachments: MR-2353.patch > > > Make the MR changes to reflect the API changes in SecureIO library. > Specifically, the 'group' argument is never used in the SecureIO library, and > hence the API changes. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (MAPREDUCE-2353) Make the MR changes to reflect the API changes in SecureIO library
[ https://issues.apache.org/jira/browse/MAPREDUCE-2353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated MAPREDUCE-2353: --- Attachment: MR-2353.patch Patch for trunk (from the MR changes introduced by the commit in the 20.100 security branch - ref HADOOP-7119) > Make the MR changes to reflect the API changes in SecureIO library > -- > > Key: MAPREDUCE-2353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2353 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: security, task, tasktracker >Affects Versions: 0.22.0 >Reporter: Devaraj Das > Attachments: MR-2353.patch > > > Make the MR changes to reflect the API changes in SecureIO library. > Specifically, the 'group' argument is never used in the SecureIO library, and > hence the API changes. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (MAPREDUCE-2353) Make the MR changes to reflect the API changes in SecureIO library
Make the MR changes to reflect the API changes in SecureIO library -- Key: MAPREDUCE-2353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2353 Project: Hadoop Map/Reduce Issue Type: Improvement Components: security, task, tasktracker Affects Versions: 0.22.0 Reporter: Devaraj Das Make the MR changes to reflect the API changes in SecureIO library. Specifically, the 'group' argument is never used in the SecureIO library, and hence the API changes. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
