[jira] [Updated] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-3375: - Attachment: MAPREDUCE-3375.v3.patch Added one more scenario for total heap ratio. Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149514#comment-13149514 ] Vinay Kumar Thota commented on MAPREDUCE-3375: -- Ran tests with latest patch and all the tests are passed. TEST-org.apache.hadoop.mapred.gridmix.TestMemEmulForMapsAndReducesWithCustomIntrvl.txt:Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 400.59 sec TEST-org.apache.hadoop.mapred.gridmix.TestMemEmulForMapsAndReducesWithDefaultIntrvl.txt:Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 440.233 sec TEST-org.apache.hadoop.mapred.gridmix.TestMemEmulForMapsWithCustomHeapMemoryRatio.txt:Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 650.089 sec TEST-org.apache.hadoop.mapred.gridmix.TestMemEmulForMapsWithCustomIntrvl.txt:Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 674.838 sec TEST-org.apache.hadoop.mapred.gridmix.TestMemEmulForMapsWithDefaultIntrvl.txt:Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 605.841 sec Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3397) Support no sort dataflow in map output and reduce merge phrase
Support no sort dataflow in map output and reduce merge phrase -- Key: MAPREDUCE-3397 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3397 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Affects Versions: 0.20.205.0 Reporter: Binglin Chang Assignee: Binglin Chang In our experience, many data aggregation style queries/jobs don't need to sort the intermediate data. In fact reducer side can use hashmap or even array to do application level aggregations. For example, consider computing CTR using display log click log in sponsored search. Map side just emit (adv_id, clk_cnt, dis_cnt), reduce side aggregate clk_cnt and dis_cnt for every adv_id, cause adv_id is integer, we can partition adv_id by range: ** reduce0: 0-10 ** reduce1: 10-20 ** ... ** reduceM: xxx-max adv-id Then the reducer can use an array(for example: int [100][2]) to store the aggregated clk_cnt dis_cnt, and we don't need the framework to sort intermediate data anymore. By supporting no sort, we can gain a lot of performance improvements: # Eliminate map side sort merge. KV paris need to sort by partition first, but this can be done using a liner time counting sort, which is much faster than quick sort. Just merge spill segments one by one, doesn't need to use heap merge. # Eliminate shuffle phrase barrier, reducer can start to processing data before all map output data are copied merged. For most cases, memory won't be a problem, cause keys are divided to many partitions, each reducers only process a small subset of the global key set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3397) Support no sort dataflow in map output and reduce merge phrase
[ https://issues.apache.org/jira/browse/MAPREDUCE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-3397: - Attachment: MAPREDUCE-3397-nosort.v1.patch A preview patch supporting no sort. Support no sort dataflow in map output and reduce merge phrase -- Key: MAPREDUCE-3397 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3397 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Affects Versions: 0.20.205.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3397-nosort.v1.patch In our experience, many data aggregation style queries/jobs don't need to sort the intermediate data. In fact reducer side can use hashmap or even array to do application level aggregations. For example, consider computing CTR using display log click log in sponsored search. Map side just emit (adv_id, clk_cnt, dis_cnt), reduce side aggregate clk_cnt and dis_cnt for every adv_id, cause adv_id is integer, we can partition adv_id by range: ** reduce0: 0-10 ** reduce1: 10-20 ** ... ** reduceM: xxx-max adv-id Then the reducer can use an array(for example: int [100][2]) to store the aggregated clk_cnt dis_cnt, and we don't need the framework to sort intermediate data anymore. By supporting no sort, we can gain a lot of performance improvements: # Eliminate map side sort merge. KV paris need to sort by partition first, but this can be done using a liner time counting sort, which is much faster than quick sort. Just merge spill segments one by one, doesn't need to use heap merge. # Eliminate shuffle phrase barrier, reducer can start to processing data before all map output data are copied merged. For most cases, memory won't be a problem, cause keys are divided to many partitions, each reducers only process a small subset of the global key set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3121) NodeManager should handle disk-failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3121: Attachment: 3121.v1.patch Attaching new patch incorporating most of the review comments. Will discuss the remaining minor comments with Vinod soon and upload another patch. Added a new configuration property for minimum fraction of number of disks that are to be healthy for considering a node to be healthy interms of disks. It is emyarn.nodemanager.disk-health-checker.min-healthy-disks/em. It's default value is 0.05. i.e. By default, a node is considered unhealthy if there are only less than 5% of disks are healthy. NodeManager should handle disk-failures --- Key: MAPREDUCE-3121 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Gummadi Fix For: 0.23.1 Attachments: 3121.patch, 3121.v1.patch This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to minimize the impact of transient/permanent disk failures on containers. With larger number of disks per node, the ability to continue to run containers on other disks is crucial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3121) NodeManager should handle disk-failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3121: Attachment: 3121.v1.1.patch Attaching new patch with a minor fix to earlier patch. NodeManager should handle disk-failures --- Key: MAPREDUCE-3121 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Gummadi Fix For: 0.23.1 Attachments: 3121.patch, 3121.v1.1.patch, 3121.v1.patch This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to minimize the impact of transient/permanent disk failures on containers. With larger number of disks per node, the ability to continue to run containers on other disks is crucial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-3375: - Attachment: MAPREDUCE-3375.v4.patch Uploading another new patch by addressing some more offline comments from Amar. Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149590#comment-13149590 ] Vinay Kumar Thota commented on MAPREDUCE-3375: -- Ran test-patch and no issues found. +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 18 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3121) NodeManager should handle disk-failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149596#comment-13149596 ] Hadoop QA commented on MAPREDUCE-3121: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503598/3121.v1.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 44 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1743 javac compiler warnings (more than the trunk's current 1742 warnings). -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1301//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1301//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1301//console This message is automatically generated. NodeManager should handle disk-failures --- Key: MAPREDUCE-3121 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Gummadi Fix For: 0.23.1 Attachments: 3121.patch, 3121.v1.1.patch, 3121.v1.patch This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to minimize the impact of transient/permanent disk failures on containers. With larger number of disks per node, the ability to continue to run containers on other disks is crucial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3309: --- Status: Open (was: Patch Available) Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149669#comment-13149669 ] Jonathan Eagles commented on MAPREDUCE-3309: Canceling patch while looking into the test failure Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinay Kumar Thota updated MAPREDUCE-3375: - Attachment: MAPREDUCE-3375.v5.patch Amar, incorporated your comments. Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3147) Handle leaf queues with the same name properly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149715#comment-13149715 ] Ravi Prakash commented on MAPREDUCE-3147: - There are two approaches to solve this problem: 1. Disallow queues with the same name. 2. Use the complete hierarchical name when specifying queues. Do any of you have comments which way it ought to be fixed? I'm leaning towards option 2. Handle leaf queues with the same name properly -- Key: MAPREDUCE-3147 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3147 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Fix For: 0.23.1 If there are two leaf queues with the same name, there is ambiguity while submitting jobs, displaying queue info. When such ambiguity exists, the system should ask for clarification / show disambiguated information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3328: Attachment: MAPREDUCE-3328.branch-0.23.patch Attaching patch to fix the problem. mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149719#comment-13149719 ] jirapos...@reviews.apache.org commented on MAPREDUCE-3328: -- bq. On 2011-11-11 20:44:17, Jonathan Eagles wrote: bq. looks good to me. i prefer parentqueue.childqueue notation instead of indentation format which i think will look better with deep queue hierarchies. Thanks Jon! :) I'm leaning towards fixing the notation (parentqueue.childqueue) in MAPREDUCE-3147. - Ravi --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2804/#review3180 --- On 2011-11-11 15:53:53, Ravi Prakash wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2804/ bq. --- bq. bq. (Updated 2011-11-11 15:53:53) bq. bq. bq. Review request for Tom Graves, Robert Evans, Eric Payne, Jonathan Eagles, Mark Holderbaugh, and johnvijoe. bq. bq. bq. Summary bq. --- bq. bq. Output is like this now: bq. bq. == bq. Queue Name : boston bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : default bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : kansas bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : styx bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : boston bq. Queue State : running bq. Scheduling Info : Capacity: 50.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : kansas bq. Queue State : running bq. Scheduling Info : Capacity: 50.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. bq. (MaximumCapacity was undefined in my setup) bq. bq. bq. This addresses bug MAPREDUCE-3328. bq. https://issues.apache.org/jira/browse/MAPREDUCE-3328 bq. bq. bq. Diffs bq. - bq. bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java PRE-CREATION bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2804/diff bq. bq. bq. Testing bq. --- bq. bq. - Unit tests ran. bq. - test-patch ran with +1. bq. - command executed. bq. bq. bq. Thanks, bq. bq. Ravi bq. bq. mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149733#comment-13149733 ] Vinay Kumar Thota commented on MAPREDUCE-3375: -- Ran test-patch manually. +1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 20 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version ) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3328: Status: Patch Available (was: Open) mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3309: --- Attachment: MAPREDUCE-3309.patch Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3309: --- Status: Patch Available (was: Open) Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149740#comment-13149740 ] Jonathan Eagles commented on MAPREDUCE-3309: Fixed the test error for TestRMWebApp. Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149751#comment-13149751 ] jirapos...@reviews.apache.org commented on MAPREDUCE-3309: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2822/ --- Review request for Tom Graves, Robert Evans and Mark Holderbaugh. Summary --- Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- The big idea is to add the application master to the jobhistory ui so that can be sort and increase search capabilities. Currently, adding anything to the jobhistory apps page means to add to the job history file name since that is parsed to get job meta data. This is done for performance reasons to prevent reading all jobhistory files. This addresses bug MAPREDUCE-3309. http://issues.apache.org/jira/browse/MAPREDUCE-3309 Diffs - hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/MockAsm.java c4ef938 hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java a27ba15 hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/DefaultSchedulerPage.java 8db4caf hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmController.java 700fdb3 hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java 7c175d1 hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java 1e9215f hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsList.java 57e695c hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java ca4ab18 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java d412a63 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsJobsBlock.java 0a6b969 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsView.java ddb904d hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/JobIndexInfo.java 3c0f3bb hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/jobhistory/TestFileNameIndexUtils.java 3a35045 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/jobhistory/FileNameIndexUtils.java 0d0271c hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java ae886cf hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java 4abcd34 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java dd19ed0 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java 5a67576 hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java f82c1d5 Diff: https://reviews.apache.org/r/2822/diff Testing --- Manual test done: 1. Verify correct application master on job history apps page with jobhistory files using new format 2. Verify old job history files are still usable to retrieve meta data minus the application master 3. Verify
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149754#comment-13149754 ] Hadoop QA commented on MAPREDUCE-3328: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503635/MAPREDUCE-3328.branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1302//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1302//console This message is automatically generated. mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test Queue State : running Scheduling Info : queueName: test, capacity: 0.2, maximumCapacity: -1.0, currentCapacity: 0.0, state: Q_RUNNING, == Queue Name : test2 Queue State : running Scheduling Info : queueName: test2, capacity: 0.1, maximumCapacity: 5.0, currentCapacity: 0.0, state: Q_RUNNING, == here default is configured to have capacity=70% and maximum capacity = 90% -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3309) Report the AM of an application in the UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149752#comment-13149752 ] Hadoop QA commented on MAPREDUCE-3309: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503637/MAPREDUCE-3309.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1303//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1303//console This message is automatically generated. Report the AM of an application in the UI - Key: MAPREDUCE-3309 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3309 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Attachments: MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch, MAPREDUCE-3309.patch Make provision to report the AM hostname of an application in the RM/JHS UI. It is difficult to trace back the AM on which an app ran when there are 100+ jobs in history. Digging through the logs is an option but since there is no consistency maintained in naming of apps in UI/logs/local dirs (MAPREDUCE-2793), debugging is all the more harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3194) mapred mradmin command is broken in mrv2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149759#comment-13149759 ] Ravi Prakash commented on MAPREDUCE-3194: - This is what I get noformat $ mapred mradmin -refreshQueues 2011-11-14 11:18:10,962 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 0 time(s). 2011-11-14 11:18:11,969 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 1 time(s). 2011-11-14 11:18:12,990 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 2 time(s). 2011-11-14 11:18:13,993 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 3 time(s). 2011-11-14 11:18:15,081 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 4 time(s). 2011-11-14 11:18:16,209 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 5 time(s). 2011-11-14 11:18:17,212 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 6 time(s). 2011-11-14 11:18:18,215 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 7 time(s). 2011-11-14 11:18:19,218 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 8 time(s). 2011-11-14 11:18:20,221 INFO [main] ipc.Client (Client.java:handleConnectionFailure(671)) - Retrying connect to server: localhost.localdomain:192.168.1.2:50030. Already tried 9 time(s). refreshQueues: Call From localhost.localdomain:192.168.1.2 to localhost.localdomain:50030 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused noformat mapred mradmin command is broken in mrv2 -- Key: MAPREDUCE-3194 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3194 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth $mapred mradmin Exception in thread main java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/tools/MRAdmin Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.tools.MRAdmin at java.net.URLClassLoader$1.run(URLClassLoader.java:202) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:190) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) Could not find the main class: org.apache.hadoop.mapred.tools.MRAdmin. Program will exit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3343) TaskTracker Out of Memory because of distributed cache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149761#comment-13149761 ] Eli Collins commented on MAPREDUCE-3343: How does the test cover that the job is removed from the archives? Looks like it should pass even if we remove the call to removeTaskDistributedCacheManager in TT and the test. TaskTracker Out of Memory because of distributed cache -- Key: MAPREDUCE-3343 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3343 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.205.0 Reporter: Ahmed Radwan Assignee: zhaoyunjiong Labels: mapreduce, patch Attachments: MAPREDUCE-3343_rev2.patch, mapreduce-3343-release-0.20.205.0.patch This Out of Memory happens when you run large number of jobs (using the distributed cache) on a TaskTracker. Seems the basic issue is with the distributedCacheManager (instance of TrackerDistributedCacheManager in TaskTracker.java), this gets created during TaskTracker.initialize(), and it keeps references to TaskDistributedCacheManager for every submitted job via the jobArchives Map, also references to CacheStatus via cachedArchives map. I am not seeing these cleaned up between jobs, so this can out of memory problems after really large number of jobs are submitted. We have seen this issue in a number of cases. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3395) Add mapred.disk.healthChecker.interval to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149778#comment-13149778 ] Eli Collins commented on MAPREDUCE-3395: Thanks Harsh! I left this option out of the node health script variables section because it's unrelated to the health script. ammended your commit on branch-20-security. Add mapred.disk.healthChecker.interval to mapred-default.xml Key: MAPREDUCE-3395 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3395 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Affects Versions: 0.20.205.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Trivial Fix For: 0.20.206.0 Attachments: mapreduce-3395-1.patch, mapreduce-3395-2.patch Let's add mapred.disk.healthChecker.interval to mapred-default.xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3015) Add local dir failure info to metrics and the web UI
[ https://issues.apache.org/jira/browse/MAPREDUCE-3015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved MAPREDUCE-3015. Resolution: Fixed Fix Version/s: 0.20.206.0 Hadoop Flags: Reviewed Since the changes to the previous patch were trivial I went ahead and committed this. Thanks Todd. Add local dir failure info to metrics and the web UI Key: MAPREDUCE-3015 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3015 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: tasktracker Affects Versions: 0.20.204.0 Reporter: Eli Collins Assignee: Eli Collins Fix For: 0.20.206.0 Attachments: mapreduce-3015-1.patch, mapreduce-3015-2.patch Like HDFS-811/HDFS-1850 but for the TT. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[
https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149819#comment-13149819
]
Thomas Graves commented on MAPREDUCE-2863:
--
A few changes to the mapreduce am and history server apis:
removing these that were similar to the web ui links:
{jobid}/counters/{group}/{counterName}
{jobid}/tasks/{taskid}/counters/{group}/{counterName}
and just adding one that is:
{jobid}/tasks/{taskid}/attempts/{attemptid}/counters
Support web-services for RM NM
Key: MAPREDUCE-2863
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, nodemanager, resourcemanager
Reporter: Arun C Murthy
Assignee: Thomas Graves
Attachments: MAPREDUCE-2863.patch, nmoutput.txt, rmoutput.txt
It will be very useful for RM and NM to support web-services to export
json/xml.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149827#comment-13149827 ] Hitesh Shah commented on MAPREDUCE-2863: Regarding the json output for http://virt09-pv1.tgraves.pool.corp.sp2.yahoo.com:8088/ws/v1/cluster/apps, it looks like the output is for a single app and not all the apps. Just wanted to ensure that the output should always be an array with root elem apps nesting an array of app objects so that the handler for that request's output would not need to change based on no. of elements in the response. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves Attachments: MAPREDUCE-2863.patch, nmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149835#comment-13149835 ] Arpit Gupta commented on MAPREDUCE-2863: @Thomas I would suggest that we keep the schema closer :) for example in xml /apps/app and json is /app What about adding a root element in json called /apps. Here you can see a yql example where the xml and json schemas match more closely. json - http://goo.gl/G5cvd xml - http://goo.gl/vyvXl Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves Attachments: MAPREDUCE-2863.patch, nmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3393) TestMRJobs, TestMROldApiJobs, and TestUberAM failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149844#comment-13149844 ] Hitesh Shah commented on MAPREDUCE-3393: @Thomas, sorry for the delay in getting back to you. Re-ran multiple times both on local mac laptop as well as a linux host ( as order of tests sometimes differ). All tests are simply mvn test install from the top of hadoop-mapreduce-project and go through across multiple runs. TestMRJobs, TestMROldApiJobs, and TestUberAM failures - Key: MAPREDUCE-3393 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3393 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Hitesh Shah Attachments: MR-3393.1.patch, MR-3393.2.patch, org.apache.hadoop.mapreduce.v2.TestMRJobs-output.txt Check out branch 0.23 and run mvn test from hadoop-mapreduce-project directory --- T E S T S --- Running org.apache.hadoop.mapred.TestClientServiceDelegate Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.717 sec Running org.apache.hadoop.mapred.TestClientRedirect Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.436 sec Running org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.975 sec Running org.apache.hadoop.mapreduce.v2.TestMRJobs Tests run: 4, Failures: 3, Errors: 1, Skipped: 0, Time elapsed: 67.999 sec FAILURE! Running org.apache.hadoop.mapreduce.v2.TestYARNRunner Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.976 sec Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 31.879 sec FAILURE! Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService ^NRunning org.apache.hadoop.mapreduce.v2.TestUberAM Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.096 sec FAILURE! Results : Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs) testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs) testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs) testJobSucceed(org.apache.hadoop.mapreduce.v2.TestMROldApiJobs): Job expected to succeed failed testJobFail(org.apache.hadoop.mapreduce.v2.TestMROldApiJobs) Tests in error: testFailingMapper(org.apache.hadoop.mapreduce.v2.TestMRJobs): 0 org.apache.hadoop.mapreduce.v2.TestUberAM: Failed to Start org.apache.hadoop.mapreduce.v2.TestMRJobs Tests run: 19, Failures: 5, Errors: 2, Skipped: 0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149847#comment-13149847 ] Thomas Graves commented on MAPREDUCE-2863: -- The intention was to always have the array even in single app case, not sure what happened there. ok, will make sure they match with the root elements having the /apps there also and similarly across other outputs. Will upload the new output once I have it working. let me know if here are further comments. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves Attachments: MAPREDUCE-2863.patch, nmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149893#comment-13149893 ] Hitesh Shah commented on MAPREDUCE-2863: Taking a look at the output formats only (apart from the comment for having all outputs be a closer match for json/xml): A minor comment: - totalMemoryNeeded : 2048, - totalMemoryNeededMB - there is a mix of bytes, MB and GB in use in various diff outputs which could probably do with some form of standardization. Not sure if it needs to be done as part of this but will affect the element names later if the value types change. - Is anyone expected to use the version info? If yes, would they likely use it as is or would they need to parse it to strip out the actual version no? If the latter, would 2 values - versionNum and versionInfo help with the former just have the basic version string and the latter the full build version info? Also, I am assuming the framework correctly handles escaping all data that could be in elements which depict urls or other info such as the diagnostics? Apart from that, the output generally looks good. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves Attachments: MAPREDUCE-2863.patch, nmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3121) NodeManager should handle disk-failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149914#comment-13149914 ] Hitesh Shah commented on MAPREDUCE-3121: Some comments: - DISKS_FAILED 144 is probably not a good idea. It clashes with SIGUSR1. We could use EIO or any other relevant exit code related to file system errors. Another option is to use a non-clashing exit code along the lines of container aborted ( -100 ). Anyone have any preferences on which approach to leverage? The latter will obviously be a more clear indicator on what the failure was and allow easy blacklisting of this node/re-scheduling on other nodes. - Should the the failed disks error information be propagated into the app/container diagnostics? - Should there be a check for whether there are any good dirs left in ResourceLocalizationService before starting of localizing the resources? NodeManager should handle disk-failures --- Key: MAPREDUCE-3121 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3121 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Gummadi Fix For: 0.23.1 Attachments: 3121.patch, 3121.v1.1.patch, 3121.v1.patch This is akin to MAPREDUCE-2413 but for YARN's NodeManager. We want to minimize the impact of transient/permanent disk failures on containers. With larger number of disks per node, the ability to continue to run containers on other disks is crucial. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3371) Review and improve the yarn-api javadocs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3371: Status: Patch Available (was: Open) Review and improve the yarn-api javadocs. - Key: MAPREDUCE-3371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3371 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Minor Labels: documentation Fix For: 0.23.1 Attachments: MAPREDUCE-3371.branch-0.23.patch Review and improve the yarn-api javadocs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3324: - Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Affects Version/s: (was: 0.24.0) Fix Version/s: 0.23.1 +1 looks good to me. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3324: - Resolution: Fixed Target Version/s: 0.24.0, 0.23.1 (was: 0.23.1, 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this. Thanks Jonathan. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149950#comment-13149950 ] Hudson commented on MAPREDUCE-3324: --- Integrated in Hadoop-Hdfs-trunk-Commit #1342 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1342/]) MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201914 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149951#comment-13149951 ] jirapos...@reviews.apache.org commented on MAPREDUCE-3328: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2804/#review3237 --- Ship it! - Eric On 2011-11-11 15:53:53, Ravi Prakash wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2804/ bq. --- bq. bq. (Updated 2011-11-11 15:53:53) bq. bq. bq. Review request for Tom Graves, Robert Evans, Eric Payne, Jonathan Eagles, Mark Holderbaugh, and johnvijoe. bq. bq. bq. Summary bq. --- bq. bq. Output is like this now: bq. bq. == bq. Queue Name : boston bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : default bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : kansas bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : styx bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : boston bq. Queue State : running bq. Scheduling Info : Capacity: 50.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : kansas bq. Queue State : running bq. Scheduling Info : Capacity: 50.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. bq. (MaximumCapacity was undefined in my setup) bq. bq. bq. This addresses bug MAPREDUCE-3328. bq. https://issues.apache.org/jira/browse/MAPREDUCE-3328 bq. bq. bq. Diffs bq. - bq. bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java PRE-CREATION bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2804/diff bq. bq. bq. Testing bq. --- bq. bq. - Unit tests ran. bq. - test-patch ran with +1. bq. - command executed. bq. bq. bq. Thanks, bq. bq. Ravi bq. bq. mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of maximum capacity doesn't match the format of the current capacity and capacity. the latter two use float while the maximum is specified as int: Queue Name : default Queue State : running Scheduling Info : queueName: default, capacity: 0.7, maximumCapacity: 90.0, currentCapacity: 0.0, state:
[jira] [Commented] (MAPREDUCE-3328) mapred queue -list output inconsistent and missing child queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149952#comment-13149952 ] jirapos...@reviews.apache.org commented on MAPREDUCE-3328: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2804/#review3230 --- branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java https://reviews.apache.org/r/2804/#comment7191 Line 461: Is queueInfo.getChildQueues() recursive? Will this get children of children? It does, right? - Eric On 2011-11-11 15:53:53, Ravi Prakash wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/2804/ bq. --- bq. bq. (Updated 2011-11-11 15:53:53) bq. bq. bq. Review request for Tom Graves, Robert Evans, Eric Payne, Jonathan Eagles, Mark Holderbaugh, and johnvijoe. bq. bq. bq. Summary bq. --- bq. bq. Output is like this now: bq. bq. == bq. Queue Name : boston bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : default bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : kansas bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : styx bq. Queue State : running bq. Scheduling Info : Capacity: 25.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : boston bq. Queue State : running bq. Scheduling Info : Capacity: 50.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. == bq. Queue Name : kansas bq. Queue State : running bq. Scheduling Info : Capacity: 50.0, MaximumCapacity: UNDEFINED, CurrentCapacity: 0.0 bq. bq. (MaximumCapacity was undefined in my setup) bq. bq. bq. This addresses bug MAPREDUCE-3328. bq. https://issues.apache.org/jira/browse/MAPREDUCE-3328 bq. bq. bq. Diffs bq. - bq. bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueClient.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobQueueInfo.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobQueueClient.java PRE-CREATION bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ResourceMgrDelegate.java 1200893 bq. branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/2804/diff bq. bq. bq. Testing bq. --- bq. bq. - Unit tests ran. bq. - test-patch ran with +1. bq. - command executed. bq. bq. bq. Thanks, bq. bq. Ravi bq. bq. mapred queue -list output inconsistent and missing child queues --- Key: MAPREDUCE-3328 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3328 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Priority: Critical Attachments: MAPREDUCE-3328.branch-0.23.patch When running mapred queue -list on a 0.23.0 cluster with capacity scheduler configured with child queues. In my case I have queues default, test1, and test2. test1 has subqueues of a1, a2. test2 has subqueues of a3 and a4. - the child queues do not show up - The output of
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149953#comment-13149953 ] Hudson commented on MAPREDUCE-3324: --- Integrated in Hadoop-Common-trunk-Commit #1268 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1268/]) MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201914 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3371) Review and improve the yarn-api javadocs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149960#comment-13149960 ] Hadoop QA commented on MAPREDUCE-3371: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12502835/MAPREDUCE-3371.branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1304//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1304//console This message is automatically generated. Review and improve the yarn-api javadocs. - Key: MAPREDUCE-3371 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3371 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation, mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Ravi Prakash Priority: Minor Labels: documentation Fix For: 0.23.1 Attachments: MAPREDUCE-3371.branch-0.23.patch Review and improve the yarn-api javadocs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149961#comment-13149961 ] Hudson commented on MAPREDUCE-3324: --- Integrated in Hadoop-Hdfs-0.23-Commit #167 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/167/]) MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (Jonathan Eagles via mahadev) - Merging r1201914 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201916 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149962#comment-13149962 ] Hudson commented on MAPREDUCE-3324: --- Integrated in Hadoop-Common-0.23-Commit #168 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/168/]) MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (Jonathan Eagles via mahadev) - Merging r1201914 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201916 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3327) RM web ui scheduler link doesn't show correct max value for queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3327: - Status: Open (was: Patch Available) Anupam, Can you include a small test case just to make sure all the properties are read properly in the leaf queues? RM web ui scheduler link doesn't show correct max value for queues -- Key: MAPREDUCE-3327 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3327 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Anupam Seth Priority: Critical Attachments: MAPREDUCE-3327-branch_0_23.patch Configure a cluster to use the capacity scheduler and then specifying a maximum-capacity 100% for a queue. If you go to the RM Web UI and hover over the queue, it always shows the max at 100%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149969#comment-13149969 ] Hudson commented on MAPREDUCE-3324: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1290 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1290/]) MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (Jonathan Eagles via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201914 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149971#comment-13149971 ] Mahadev konar commented on MAPREDUCE-3325: -- +1 looks good. Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149978#comment-13149978 ] Hudson commented on MAPREDUCE-3324: --- Integrated in Hadoop-Mapreduce-0.23-Commit #179 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/179/]) MAPREDUCE-3324. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers (Jonathan Eagles via mahadev) - Merging r1201914 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201916 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsNavBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NavBlock.java Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149984#comment-13149984 ] Hudson commented on MAPREDUCE-3325: --- Integrated in Hadoop-Common-trunk-Commit #1269 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1269/]) MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201925 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.1 Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3325: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this. Thanks Thomas. Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.1 Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3393) TestMRJobs, TestMROldApiJobs, and TestUberAM failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah reassigned MAPREDUCE-3393: -- Assignee: Thomas Graves (was: Hitesh Shah) Re-assigning back to Thomas to investigate more as not reproducible. TestMRJobs, TestMROldApiJobs, and TestUberAM failures - Key: MAPREDUCE-3393 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3393 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: MR-3393.1.patch, MR-3393.2.patch, org.apache.hadoop.mapreduce.v2.TestMRJobs-output.txt Check out branch 0.23 and run mvn test from hadoop-mapreduce-project directory --- T E S T S --- Running org.apache.hadoop.mapred.TestClientServiceDelegate Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.717 sec Running org.apache.hadoop.mapred.TestClientRedirect Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.436 sec Running org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.975 sec Running org.apache.hadoop.mapreduce.v2.TestMRJobs Tests run: 4, Failures: 3, Errors: 1, Skipped: 0, Time elapsed: 67.999 sec FAILURE! Running org.apache.hadoop.mapreduce.v2.TestYARNRunner Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.976 sec Running org.apache.hadoop.mapreduce.v2.TestMROldApiJobs Tests run: 2, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 31.879 sec FAILURE! Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithHistoryService ^NRunning org.apache.hadoop.mapreduce.v2.TestUberAM Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 101.096 sec FAILURE! Results : Failed tests: testSleepJob(org.apache.hadoop.mapreduce.v2.TestMRJobs) testRandomWriter(org.apache.hadoop.mapreduce.v2.TestMRJobs) testDistributedCache(org.apache.hadoop.mapreduce.v2.TestMRJobs) testJobSucceed(org.apache.hadoop.mapreduce.v2.TestMROldApiJobs): Job expected to succeed failed testJobFail(org.apache.hadoop.mapreduce.v2.TestMROldApiJobs) Tests in error: testFailingMapper(org.apache.hadoop.mapreduce.v2.TestMRJobs): 0 org.apache.hadoop.mapreduce.v2.TestUberAM: Failed to Start org.apache.hadoop.mapreduce.v2.TestMRJobs Tests run: 19, Failures: 5, Errors: 2, Skipped: 0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3102) NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/MAPREDUCE-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3102: - Status: Open (was: Patch Available) Hitesh, The patch looks good. One minor nit, can you remove the TODO's in InvalidContainerExecutor.java? thanks NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor --- Key: MAPREDUCE-3102 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3102 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3102.1.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149995#comment-13149995 ] Hudson commented on MAPREDUCE-3325: --- Integrated in Hadoop-Hdfs-0.23-Commit #168 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/168/]) MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves via mahadev) - Merging r1201925 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201927 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.1 Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13149996#comment-13149996 ] Hudson commented on MAPREDUCE-3325: --- Integrated in Hadoop-Common-0.23-Commit #169 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/169/]) MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves via mahadev) - Merging r1201925 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201927 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.1 Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1315#comment-1315 ] Hudson commented on MAPREDUCE-3325: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1291 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1291/]) MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201925 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.1 Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3325) Improvements to CapacityScheduler doc
[ https://issues.apache.org/jira/browse/MAPREDUCE-3325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150014#comment-13150014 ] Hudson commented on MAPREDUCE-3325: --- Integrated in Hadoop-Mapreduce-0.23-Commit #180 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/180/]) MAPREDUCE-3325. Improvements to CapacityScheduler doc. (Thomas Graves via mahadev) - Merging r1201925 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201927 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm Improvements to CapacityScheduler doc - Key: MAPREDUCE-3325 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3325 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Fix For: 0.23.1 Attachments: MAPREDUCE-3325.patch I noticed the following issues with the capacity scheduler doc: ./hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm - In overview section, 3rd paragraph, sentence There is an added benefit that an organization can access any excess capacity no being used by others. No should be not. - in overview section, 4th paragraph. dispropotionate misspelled - in features section, under multitenancy - monopolizing is misspelled. - in features section, under operability - it doesn't say if you can delete queues at runtime? I see there is a note at the end but perhaps that can be added into the other sections to since its easy to miss that Note at the very end. - in features section - hierarchy and Hierarchical mispelled. - under configuration section the class to turn on to use capacity scheduler should be: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler - section on setting up queues, 4th sentence - hierarchy misspelled as heirarcy and heirarchy. - I think specifying how a user has to specify the queue when running a job/app would be useful information. Especially with the new hierarchical queues. Does the user have to specify the entire path like a.b.c or can they just specify c. - under Running and Pending Application Limits section, property yarn.scheduler.capacity.maximum-applications, they are referred to them as jobs, I believe that should be applications. - misspelled concurrently as concurently in same section of maximum-applications. - I think it should specify the defaults (if any) for the config vars. Also what format are they specified in - int, float,etc? - might be nice to say it doesn't support preemption. - under admin options yarn.scheduler.capacity.queue-path.state - queues misspelled as queueus - under changing queue configuration it should have yarn in front of the rmadmin -refreshQueues. Similarly a few lines down at $YARN_HOME/bin/rmadmin -refreshQueues -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150017#comment-13150017 ] Mahadev konar commented on MAPREDUCE-3341: -- +1 looks good. Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3341: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this. Thanks Anupam. Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150025#comment-13150025 ] Hudson commented on MAPREDUCE-3341: --- Integrated in Hadoop-Hdfs-trunk-Commit #1344 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1344/]) MAPREDUCE-3341. Enhance logging of initalized queue limit values. (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201947 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150027#comment-13150027 ] Hudson commented on MAPREDUCE-3341: --- Integrated in Hadoop-Common-trunk-Commit #1270 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1270/]) MAPREDUCE-3341. Enhance logging of initalized queue limit values. (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201947 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150028#comment-13150028 ] Jonathan Eagles commented on MAPREDUCE-3045: I think what's going on here is that the filter on time is searching on both the text data of the cell as well as title. I think the intention is to sort by hidden element title and filter based on text data of the cell. Elapsed time filter on jobhistory server displays incorrect table entries - Key: MAPREDUCE-3045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Priority: Minor Fix For: 0.23.1 The elapsed time filter on the jobhistory server filters incorrect information. For e.g. on a cluster where the elapsed time of all the tasks is either 7 or 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mahadev konar updated MAPREDUCE-3243:
-
Resolution: Fixed
Fix Version/s: 0.23.1
Status: Resolved (was: Patch Available)
Just committed this. Thanks Jonathan.
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150036#comment-13150036
]
Hudson commented on MAPREDUCE-3243:
---
Integrated in Hadoop-Hdfs-0.23-Commit #169 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/169/])
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev) - Merging r1201951 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201952
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150037#comment-13150037 ] Hudson commented on MAPREDUCE-3341: --- Integrated in Hadoop-Hdfs-0.23-Commit #169 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/169/]) MAPREDUCE-3341. Enhance logging of initalized queue limit values. (Anupam Seth via mahadev) - Merging r1201947 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201949 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150039#comment-13150039 ] Hudson commented on MAPREDUCE-3341: --- Integrated in Hadoop-Common-0.23-Commit #170 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/170/]) MAPREDUCE-3341. Enhance logging of initalized queue limit values. (Anupam Seth via mahadev) - Merging r1201947 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201949 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150038#comment-13150038
]
Hudson commented on MAPREDUCE-3243:
---
Integrated in Hadoop-Common-0.23-Commit #170 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/170/])
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev) - Merging r1201951 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201952
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150045#comment-13150045
]
Hudson commented on MAPREDUCE-3243:
---
Integrated in Hadoop-Common-trunk-Commit #1271 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1271/])
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201951
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150049#comment-13150049 ] Hudson commented on MAPREDUCE-3341: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1292 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1292/]) MAPREDUCE-3341. Enhance logging of initalized queue limit values. (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201947 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2905) CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job)
[ https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150050#comment-13150050 ] Eli Collins commented on MAPREDUCE-2905: Hi Matei, Are you +1 on Todd's patch or just the approach, ie does the patch look good to go? CapBasedLoadManager incorrectly allows assignment when assignMultiple is true (was: assignmultiple per job) --- Key: MAPREDUCE-2905 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 0.20.2 Reporter: Jeff Bean Assignee: Jeff Bean Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2, mr-2905.txt, screenshot-1.jpg We encountered a situation where in the same cluster, large jobs benefit from mapred.fairscheduler.assignmultiple, but small jobs with small numbers of mappers do not: the mappers all clump to fully occupy just a few nodes, which causes those nodes to saturate and bottleneck. The desired behavior is to spread the job across more nodes so that a relatively small job doesn't saturate any node in the cluster. Testing has shown that setting mapred.fairscheduler.assignmultiple to false gives the desired behavior for small jobs, but is unnecessary for large jobs. However, since this is a cluster-wide setting, we can't properly tune. It'd be nice if jobs can set a param similar to mapred.fairscheduler.assignmultiple on submission to better control the task distribution of a particular job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150043#comment-13150043
]
Hudson commented on MAPREDUCE-3243:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1345 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1345/])
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201951
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150058#comment-13150058
]
Hudson commented on MAPREDUCE-3243:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #181 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/181/])
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev) - Merging r1201951 from trunk.
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201952
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3341) Enhance logging of initalized queue limit values
[ https://issues.apache.org/jira/browse/MAPREDUCE-3341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150059#comment-13150059 ] Hudson commented on MAPREDUCE-3341: --- Integrated in Hadoop-Mapreduce-0.23-Commit #181 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/181/]) MAPREDUCE-3341. Enhance logging of initalized queue limit values. (Anupam Seth via mahadev) - Merging r1201947 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201949 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java Enhance logging of initalized queue limit values Key: MAPREDUCE-3341 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3341 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Anupam Seth Assignee: Anupam Seth Fix For: 0.23.1 Attachments: MAPREDUCE-3341-branch_0_23.patch, MAPREDUCE-3341-branch_0_23.patch Currently the RM log shows only a partial set of the limits that are configured when a queue is initialized / reinitialized. For example, this is what is currently shown in the RM log for an initialized queue: # datestamp INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue: Initializing default, capacity=0.25, asboluteCapacity=0.25, maxCapacity=25.0, asboluteMaxCapacity=0.25, userLimit=100, userLimitFactor=20.0, maxApplications=2500, maxApplicationsPerUser=5, state=RUNNING, acls=ADMINISTER_QUEUE:*SUBMIT_JOB:*ADMINISTER_JOBS:* Breaking down the line above, shows: capacity=0.25 asboluteCapacity=0.25 maxCapacity=25.0 asboluteMaxCapacity=0.25 userLimit=100 userLimitFactor=20.0 maxApplications=2500 maxApplicationsPerUser=5 It might be nice if we could include more information such as maxActiveApplications, maxActiveApplicationsPerUser, utilization, and usedCapacity along with information on how each of these is computed (i.e. formulae used) (Thanks to Phil Su for requesting this). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3102) NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/MAPREDUCE-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3102: --- Attachment: MR-3102.2.patch Addressed comments. NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor --- Key: MAPREDUCE-3102 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3102 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3102.1.patch, MR-3102.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3102) NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/MAPREDUCE-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3102: --- Status: Patch Available (was: Open) NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor --- Key: MAPREDUCE-3102 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3102 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3102.1.patch, MR-3102.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150070#comment-13150070
]
Hudson commented on MAPREDUCE-3243:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1293 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1293/])
MAPREDUCE-3243. Invalid tracking URL for streaming jobs (Jonathan Eagles
via mahadev)
mahadev :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201951
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/streaming/src/java/org/apache/hadoop/streaming/StreamJob.java
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3045) Elapsed time filter on jobhistory server displays incorrect table entries
[ https://issues.apache.org/jira/browse/MAPREDUCE-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles reassigned MAPREDUCE-3045: -- Assignee: Jonathan Eagles Elapsed time filter on jobhistory server displays incorrect table entries - Key: MAPREDUCE-3045 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3045 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Ramya Sunil Assignee: Jonathan Eagles Priority: Minor Fix For: 0.23.1 The elapsed time filter on the jobhistory server filters incorrect information. For e.g. on a cluster where the elapsed time of all the tasks is either 7 or 8sec, the filter displays non null table entries for 1sec or 3sec -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3243) Invalid tracking URL for streaming jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150080#comment-13150080
]
Jonathan Eagles commented on MAPREDUCE-3243:
Thanks, Mahadev and Vinod for taking a look.
Invalid tracking URL for streaming jobs
---
Key: MAPREDUCE-3243
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3243
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/streaming, mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Assignee: Jonathan Eagles
Fix For: 0.23.1
Attachments: MAPREDUCE-3243.patch
The tracking URL for streaming jobs currently display http://N/A;
{noformat}
INFO streaming.StreamJob: To kill this job, run:
INFO streaming.StreamJob: hadoop job -kill jobID
INFO streaming.StreamJob: Tracking URL: http://N/A
INFO mapreduce.Job: Running job: jobID
INFO mapreduce.Job: map 0% reduce 0%
INFO mapred.ClientServiceDelegate: Tracking Url of JOB is host:port
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3324) Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150079#comment-13150079 ] Jonathan Eagles commented on MAPREDUCE-3324: Thanks, Mahadev. Not All HttpServer tools links (stacks,logs,config,metrics) are accessible through all UI servers - Key: MAPREDUCE-3324 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3324 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, nodemanager Affects Versions: 0.23.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3324.patch Nodemanager has no tools listed under tools UI. Jobhistory server has no logs tool listed under tools UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3102) NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor
[ https://issues.apache.org/jira/browse/MAPREDUCE-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150083#comment-13150083 ] Hadoop QA commented on MAPREDUCE-3102: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12503688/MR-3102.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1305//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1305//console This message is automatically generated. NodeManager should fail fast with wrong configuration or permissions for LinuxContainerExecutor --- Key: MAPREDUCE-3102 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3102 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, security Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Hitesh Shah Fix For: 0.23.1 Attachments: MR-3102.1.patch, MR-3102.2.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3397) Support no sort dataflow in map output and reduce merge phrase
[ https://issues.apache.org/jira/browse/MAPREDUCE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150093#comment-13150093 ] Aaron T. Myers commented on MAPREDUCE-3397: --- Is this not a duplicate of MAPREDUCE-1639? Support no sort dataflow in map output and reduce merge phrase -- Key: MAPREDUCE-3397 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3397 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Affects Versions: 0.20.205.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3397-nosort.v1.patch In our experience, many data aggregation style queries/jobs don't need to sort the intermediate data. In fact reducer side can use hashmap or even array to do application level aggregations. For example, consider computing CTR using display log click log in sponsored search. Map side just emit (adv_id, clk_cnt, dis_cnt), reduce side aggregate clk_cnt and dis_cnt for every adv_id, cause adv_id is integer, we can partition adv_id by range: ** reduce0: 0-10 ** reduce1: 10-20 ** ... ** reduceM: xxx-max adv-id Then the reducer can use an array(for example: int [100][2]) to store the aggregated clk_cnt dis_cnt, and we don't need the framework to sort intermediate data anymore. By supporting no sort, we can gain a lot of performance improvements: # Eliminate map side sort merge. KV paris need to sort by partition first, but this can be done using a liner time counting sort, which is much faster than quick sort. Just merge spill segments one by one, doesn't need to use heap merge. # Eliminate shuffle phrase barrier, reducer can start to processing data before all map output data are copied merged. For most cases, memory won't be a problem, cause keys are divided to many partitions, each reducers only process a small subset of the global key set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3391) Connecting to CM is logged as Connecting to RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-3391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3391:
-
Fix Version/s: (was: 0.23.0)
0.23.1
Connecting to CM is logged as Connecting to RM
--
Key: MAPREDUCE-3391
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3391
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster
Affects Versions: 0.23.0
Reporter: Subroto Sanyal
Assignee: Subroto Sanyal
Priority: Minor
Labels: newbie
Fix For: 0.23.1
In class
*org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster*
{code}
private void connectToCM() {
String cmIpPortStr = container.getNodeId().getHost() + :
+ container.getNodeId().getPort();
InetSocketAddress cmAddress = NetUtils.createSocketAddr(cmIpPortStr);
LOG.info(Connecting to ResourceManager at + cmIpPortStr);
this.cm = ((ContainerManager) rpc.getProxy(ContainerManager.class,
cmAddress, conf));
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3394) Add log guard for a debug message in ReduceTask
[
https://issues.apache.org/jira/browse/MAPREDUCE-3394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon resolved MAPREDUCE-3394.
Resolution: Fixed
Fix Version/s: 0.20.206.0
Hadoop Flags: Reviewed
Committed to branch-20-security for 0.20.206. Thanks Eli.
Add log guard for a debug message in ReduceTask
---
Key: MAPREDUCE-3394
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3394
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: task
Affects Versions: 0.20.205.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Trivial
Fix For: 0.20.206.0
Attachments: mr-3394.txt
There's a LOG.debug message in ReduceTask that stringifies a task ID and uses
a non-negligible amount of CPU in some cases. We should guard it with
{{isDebugEnabled}}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150155#comment-13150155 ] Mahadev konar commented on MAPREDUCE-3331: -- Thanks Anupam. It would be good to add a single node secure deployment instructions as well. Want to open another jira for that? Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amar Kamat resolved MAPREDUCE-3375. --- Tags: gridmix system-tests memory-emulation Resolution: Fixed Fix Version/s: 0.23.1 0.24.0 Target Version/s: 0.24.0 Release Note: Added system tests to test the memory emulation feature in Gridmix. Hadoop Flags: Reviewed I just committed this to trunk and branch-0.23. Thanks Vinay! Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2863) Support web-services for RM NM
[ https://issues.apache.org/jira/browse/MAPREDUCE-2863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150165#comment-13150165 ] Thomas Graves commented on MAPREDUCE-2863: -- thanks for the review. Yeah I agree I don't like the mix of types but unfortunately its not consistent in yarn. Some store things as MB, some as GB, some as bytes. Perhaps for everything memory related I can make web services all common with MB? I'll break the version info out more similar to actual VersionInfo class. My understanding was that jersey/jaxb handles escaping but I will test to verify. Support web-services for RM NM Key: MAPREDUCE-2863 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2863 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2, nodemanager, resourcemanager Reporter: Arun C Murthy Assignee: Thomas Graves Attachments: MAPREDUCE-2863.patch, nmoutput.txt, rmoutput.txt It will be very useful for RM and NM to support web-services to export json/xml. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3331: - Resolution: Fixed Fix Version/s: 0.23.1 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Just committed this. Thanks Anupam. Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150175#comment-13150175 ] Hudson commented on MAPREDUCE-3375: --- Integrated in Hadoop-Hdfs-trunk-Commit #1347 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1347/]) MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. (Vinay Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201997 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithCustomIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithDefaultIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomHeapMemoryRatio.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithDefaultIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case1.json.gz * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case2.json.gz Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150177#comment-13150177 ] Hudson commented on MAPREDUCE-3331: --- Integrated in Hadoop-Hdfs-trunk-Commit #1347 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1347/]) MAPREDUCE-3331. Improvement to single node cluster setup documentation for 0.23 (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1202002 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150178#comment-13150178 ] Hudson commented on MAPREDUCE-3375: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1296 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1296/]) MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. (Vinay Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201997 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithCustomIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithDefaultIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomHeapMemoryRatio.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithDefaultIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case1.json.gz * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case2.json.gz Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150181#comment-13150181 ] Hudson commented on MAPREDUCE-3331: --- Integrated in Hadoop-Common-trunk-Commit #1273 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1273/]) MAPREDUCE-3331. Improvement to single node cluster setup documentation for 0.23 (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1202002 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150179#comment-13150179 ] Hudson commented on MAPREDUCE-3375: --- Integrated in Hadoop-Common-trunk-Commit #1273 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1273/]) MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. (Vinay Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201997 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithCustomIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithDefaultIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomHeapMemoryRatio.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithDefaultIntrvl.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case1.json.gz * /hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case2.json.gz Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150185#comment-13150185 ] Hudson commented on MAPREDUCE-3375: --- Integrated in Hadoop-Hdfs-0.23-Commit #170 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/170/]) MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. (Vinay Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201999 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithCustomIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithDefaultIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomHeapMemoryRatio.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithDefaultIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case1.json.gz * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case2.json.gz Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150186#comment-13150186 ] Hudson commented on MAPREDUCE-3331: --- Integrated in Hadoop-Hdfs-0.23-Commit #170 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/170/]) MAPREDUCE-3331. Improvement to single node cluster setup documentation for 0.23 (Anupam Seth via mahadev) - Merging r1202002 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1202004 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150189#comment-13150189 ] Hudson commented on MAPREDUCE-3375: --- Integrated in Hadoop-Common-0.23-Commit #171 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/171/]) MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. (Vinay Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201999 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithCustomIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithDefaultIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomHeapMemoryRatio.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithDefaultIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case1.json.gz * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case2.json.gz Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150190#comment-13150190 ] Hudson commented on MAPREDUCE-3331: --- Integrated in Hadoop-Common-0.23-Commit #171 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/171/]) MAPREDUCE-3331. Improvement to single node cluster setup documentation for 0.23 (Anupam Seth via mahadev) - Merging r1202002 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1202004 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3375) Memory Emulation system tests.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150191#comment-13150191 ] Hudson commented on MAPREDUCE-3375: --- Integrated in Hadoop-Mapreduce-0.23-Commit #182 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/182/]) MAPREDUCE-3375. [Gridmix] Memory Emulation system tests. (Vinay Thota via amarrk) amarrk : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1201999 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithCustomIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsAndReducesWithDefaultIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomHeapMemoryRatio.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithCustomIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/TestMemEmulForMapsWithDefaultIntrvl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridMixConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/org/apache/hadoop/mapred/gridmix/test/system/GridmixJobVerification.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case1.json.gz * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/contrib/gridmix/src/test/system/resources/mem_emul_case2.json.gz Memory Emulation system tests. -- Key: MAPREDUCE-3375 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3375 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Vinay Kumar Thota Assignee: Vinay Kumar Thota Fix For: 0.24.0, 0.23.1 Attachments: MAPREDUCE-3375.v1.patch, MAPREDUCE-3375.v2.patch, MAPREDUCE-3375.v3.patch, MAPREDUCE-3375.v4.patch, MAPREDUCE-3375.v5.patch 1. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 2. Test the Gridmix memory emulation feature for gridmix jobs with custom progress interval, different input data, submission policies and user resolver modes . Verify the maps phase of total heap usage of gridmix jobs with corresponding the original job in the trace. 3. Test the Gridmix memory emulation feature for gridmix jobs with default progress interval, different input data, submission policies and user resolver modes. Verify the maps and reduces phase of total heap usage metric of gridmix jobs with corresponding the original job in the trace. 4. Disable Gridmix memory emulation option and verify the jobs whether it emulates the heap memory or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150202#comment-13150202 ] Hudson commented on MAPREDUCE-3331: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1297 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1297/]) MAPREDUCE-3331. Improvement to single node cluster setup documentation for 0.23 (Anupam Seth via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1202002 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3397) Support no sort dataflow in map output and reduce merge phrase
[ https://issues.apache.org/jira/browse/MAPREDUCE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150201#comment-13150201 ] Binglin Chang commented on MAPREDUCE-3397: -- No, grouping is not the same as no sort: # Grouping still needs shuffle phrase barrier; # In grouping kv pairs of the same key are grouped together, but in no sort kv pairs of the same key may not grouped together, framework only promise they are in the same partition(reduce). Support no sort dataflow in map output and reduce merge phrase -- Key: MAPREDUCE-3397 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3397 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Affects Versions: 0.20.205.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3397-nosort.v1.patch In our experience, many data aggregation style queries/jobs don't need to sort the intermediate data. In fact reducer side can use hashmap or even array to do application level aggregations. For example, consider computing CTR using display log click log in sponsored search. Map side just emit (adv_id, clk_cnt, dis_cnt), reduce side aggregate clk_cnt and dis_cnt for every adv_id, cause adv_id is integer, we can partition adv_id by range: ** reduce0: 0-10 ** reduce1: 10-20 ** ... ** reduceM: xxx-max adv-id Then the reducer can use an array(for example: int [100][2]) to store the aggregated clk_cnt dis_cnt, and we don't need the framework to sort intermediate data anymore. By supporting no sort, we can gain a lot of performance improvements: # Eliminate map side sort merge. KV paris need to sort by partition first, but this can be done using a liner time counting sort, which is much faster than quick sort. Just merge spill segments one by one, doesn't need to use heap merge. # Eliminate shuffle phrase barrier, reducer can start to processing data before all map output data are copied merged. For most cases, memory won't be a problem, cause keys are divided to many partitions, each reducers only process a small subset of the global key set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3331) Improvement to single node cluster setup documentation for 0.23
[ https://issues.apache.org/jira/browse/MAPREDUCE-3331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150205#comment-13150205 ] Hudson commented on MAPREDUCE-3331: --- Integrated in Hadoop-Mapreduce-0.23-Commit #183 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/183/]) MAPREDUCE-3331. Improvement to single node cluster setup documentation for 0.23 (Anupam Seth via mahadev) - Merging r1202002 from trunk mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1202004 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm Improvement to single node cluster setup documentation for 0.23 --- Key: MAPREDUCE-3331 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3331 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Reporter: Anupam Seth Assignee: Anupam Seth Priority: Minor Fix For: 0.23.1 Attachments: MRV2doc_review.patch This JIRA is to track some minor corrections and suggestions for improvement for the documentation for the setup of a single node cluster using 0.23 currently available at http://people.apache.org/~acmurthy/hadoop-0.23/hadoop-yarn/hadoop-yarn-site/SingleCluster.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3397) Support no sort dataflow in map output and reduce merge phrase
[ https://issues.apache.org/jira/browse/MAPREDUCE-3397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13150208#comment-13150208 ] Binglin Chang commented on MAPREDUCE-3397: -- I think why no sort make sense is that, in many cases application has a more efficient way to process data(such as do aggregation on the fly), they don't want the framework to do some sort of heavy weighted data preprocessing, cause they have better prior knowledge/understanding about the data and the goal. Support no sort dataflow in map output and reduce merge phrase -- Key: MAPREDUCE-3397 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3397 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: task Affects Versions: 0.20.205.0 Reporter: Binglin Chang Assignee: Binglin Chang Attachments: MAPREDUCE-3397-nosort.v1.patch In our experience, many data aggregation style queries/jobs don't need to sort the intermediate data. In fact reducer side can use hashmap or even array to do application level aggregations. For example, consider computing CTR using display log click log in sponsored search. Map side just emit (adv_id, clk_cnt, dis_cnt), reduce side aggregate clk_cnt and dis_cnt for every adv_id, cause adv_id is integer, we can partition adv_id by range: ** reduce0: 0-10 ** reduce1: 10-20 ** ... ** reduceM: xxx-max adv-id Then the reducer can use an array(for example: int [100][2]) to store the aggregated clk_cnt dis_cnt, and we don't need the framework to sort intermediate data anymore. By supporting no sort, we can gain a lot of performance improvements: # Eliminate map side sort merge. KV paris need to sort by partition first, but this can be done using a liner time counting sort, which is much faster than quick sort. Just merge spill segments one by one, doesn't need to use heap merge. # Eliminate shuffle phrase barrier, reducer can start to processing data before all map output data are copied merged. For most cases, memory won't be a problem, cause keys are divided to many partitions, each reducers only process a small subset of the global key set. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
