[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given
[ https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3829: Resolution: Fixed Status: Resolved (was: Patch Available) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given - Key: MAPREDUCE-3829 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/gridmix Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 3829.v2.patch Instead of throwing exception messages on to the console, Gridmix should give better error message when input-data directory already exists and -generate option is given. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Attachment: 3953.v1.patch Attaching patch with the code changes mentioned in the previous comment. Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Release Note: Fixes NPE and makes Gridmix simulate succeeded-jobs-with-failed-tasks. All tasks of such simulated jobs(including the failed ones of original job) will succeed. Status: Patch Available (was: Open) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13230973#comment-13230973 ] Hadoop QA commented on MAPREDUCE-3953: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518631/3953.v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2065//console This message is automatically generated. Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231161#comment-13231161 ] Steve Loughran commented on MAPREDUCE-3431: --- voting for this myself, +1, it's a trivial robustness tweak to shutdowns with tests NPE in Resource Manager shutdown Key: MAPREDUCE-3431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Environment: OS/X w/ java6 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Labels: yarn Attachments: MAPREDUCE-3431.patch bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3431: -- Resolution: Fixed Fix Version/s: 0.23.3 0.24.0 Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Resolved (was: Patch Available) NPE in Resource Manager shutdown Key: MAPREDUCE-3431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Environment: OS/X w/ java6 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Labels: yarn Fix For: 0.24.0, 0.23.3 Attachments: MAPREDUCE-3431.patch bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231183#comment-13231183 ] Hudson commented on MAPREDUCE-3431: --- Integrated in Hadoop-Common-trunk-Commit #1887 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1887/]) MAPREDUCE-3431 (Revision 1301500) MAPREDUCE-3431 (Revision 1301496) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301500 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301496 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewerLifecycle.java NPE in Resource Manager shutdown Key: MAPREDUCE-3431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Environment: OS/X w/ java6 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Labels: yarn Fix For: 0.24.0, 0.23.3 Attachments: MAPREDUCE-3431.patch bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[ https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231228#comment-13231228 ] Hudson commented on MAPREDUCE-3970: --- Integrated in Hadoop-Mapreduce-0.23-Commit #699 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/699/]) MAPREDUCE-3970 not a big fix, an improvement (Revision 1301495) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301495 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt Add ServiceOperations class to aid working with Services Key: MAPREDUCE-3970 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Steve Loughran Assignee: Steve Loughran Fix For: 0.24.0, 0.23.3 Attachments: MAPREDUCE-3970.patch, MAPREDUCE-3970.patch Add Helper methods to move things through lifecycles. init-start is common, stop-if-service!=null another. Some static methods can execute these, and even call stop() if init() raises an exception. These could go into a class ServiceOps in the same package. These can be used by those services that wrap other services, and help manage more robust shutdowns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231230#comment-13231230 ] Hudson commented on MAPREDUCE-3431: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1895 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1895/]) MAPREDUCE-3431 (Revision 1301500) MAPREDUCE-3431 (Revision 1301496) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301500 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301496 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewerLifecycle.java NPE in Resource Manager shutdown Key: MAPREDUCE-3431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Environment: OS/X w/ java6 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Labels: yarn Fix For: 0.24.0, 0.23.3 Attachments: MAPREDUCE-3431.patch bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231229#comment-13231229 ] Hudson commented on MAPREDUCE-3431: --- Integrated in Hadoop-Mapreduce-0.23-Commit #699 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/699/]) MAPREDUCE-3431 (Revision 1301502) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301502 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewerLifecycle.java NPE in Resource Manager shutdown Key: MAPREDUCE-3431 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 0.23.0, 0.24.0 Environment: OS/X w/ java6 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Labels: yarn Fix For: 0.24.0, 0.23.3 Attachments: MAPREDUCE-3431.patch bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4010: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Alejandro, I just checked this into 0.23.3 and trunk. I also added a small comment in the test about why we are skipping the deprecated values. TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231248#comment-13231248 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Hdfs-0.23-Commit #683 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/683/]) svn merge -c 1301551 from trunk to branch-0.23 FIXES MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301553) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301553 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231250#comment-13231250 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Common-0.23-Commit #692 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/692/]) svn merge -c 1301551 from trunk to branch-0.23 FIXES MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301553) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301553 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231254#comment-13231254 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Hdfs-trunk-Commit #1962 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1962/]) MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301551) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301551 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231262#comment-13231262 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Common-trunk-Commit #1888 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1888/]) MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301551) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301551 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4021) Make yarn Resource Manager services robust against shutdown
Make yarn Resource Manager services robust against shutdown --- Key: MAPREDUCE-4021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4021 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Split MAPREDUCE-3502 patches to make the RM code more resilient to being stopped more than once, or before started. This depends on MAPREDUCE-4014. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231290#comment-13231290 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Mapreduce-0.23-Commit #700 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/700/]) svn merge -c 1301551 from trunk to branch-0.23 FIXES MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301553) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301553 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4021) Make yarn Resource Manager services robust against shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4021: -- Attachment: MAPREDUCE-4021.patch patch not ready to apply to trunk, pending a version of MAPREDUCE 4014 going in Make yarn Resource Manager services robust against shutdown --- Key: MAPREDUCE-4021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4021 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: MAPREDUCE-4021.patch Split MAPREDUCE-3502 patches to make the RM code more resilient to being stopped more than once, or before started. This depends on MAPREDUCE-4014. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231291#comment-13231291 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1896 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1896/]) MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301551) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301551 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java TestWritableJobConf fails on trunk -- Key: MAPREDUCE-4010 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.24.0 Reporter: Jason Lowe Assignee: Alejandro Abdelnur Priority: Critical Fix For: 0.23.3 Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, MAPREDUCE-4010.patch TestWritableJobConf is currently failing two tests on trunk: * testEmptyConfiguration * testNonEmptyConfiguration Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4022) Make Yarn Node Manager services robust against shutdown
Make Yarn Node Manager services robust against shutdown --- Key: MAPREDUCE-4022 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4022 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.2 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Add the nodemanager bits of MAPREDUCE-3502 to shut down the Nodemanager services. This is done by checking for fields being non-null before shutting down/closing etc, and setting the fields to null afterwards -to be resilient against re-entrancy. No tests other than manual review. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4022) Make Yarn Node Manager services robust against shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4022: -- Attachment: MAPREDUCE-4022.patch depends on MAPREDUCE-4014 Make Yarn Node Manager services robust against shutdown --- Key: MAPREDUCE-4022 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4022 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.2 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: MAPREDUCE-4022.patch Add the nodemanager bits of MAPREDUCE-3502 to shut down the Nodemanager services. This is done by checking for fields being non-null before shutting down/closing etc, and setting the fields to null afterwards -to be resilient against re-entrancy. No tests other than manual review. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4023) Make Yarn Client service shutdown operations robust
[ https://issues.apache.org/jira/browse/MAPREDUCE-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4023: -- Attachment: MAPREDUCE-4023.patch depends on MAPREDUCE-4014 Make Yarn Client service shutdown operations robust --- Key: MAPREDUCE-4023 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4023 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.2 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: MAPREDUCE-4023.patch Make the yarn client services more robust against being shut down while not started, or shutdown more than once, by null-checking fields before closing them, setting to null afterwards to prevent double-invocation. This is a subset of MAPREDUCE-3502 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4023) Make Yarn Client service shutdown operations robust
Make Yarn Client service shutdown operations robust --- Key: MAPREDUCE-4023 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4023 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.2 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: MAPREDUCE-4023.patch Make the yarn client services more robust against being shut down while not started, or shutdown more than once, by null-checking fields before closing them, setting to null afterwards to prevent double-invocation. This is a subset of MAPREDUCE-3502 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4008) ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
[
https://issues.apache.org/jira/browse/MAPREDUCE-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231316#comment-13231316
]
Jason Lowe commented on MAPREDUCE-4008:
---
The TestWritableJobConf failure is a known issue, see MAPREDUCE-4010.
ResourceManager throws MetricsException on start up saying QueueMetrics MBean
already exists
Key: MAPREDUCE-4008
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4008
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager, scheduler
Affects Versions: 0.24.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4008.patch
{code:xml}
2012-03-14 15:22:23,089 WARN org.apache.hadoop.metrics2.util.MBeans: Error
creating MBean object name:
Hadoop:service=ResourceManager,name=QueueMetrics,q0=default
org.apache.hadoop.metrics2.MetricsException:
org.apache.hadoop.metrics2.MetricsException:
Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:117)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
at $Proxy6.postStart(Unknown Source)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:183)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:155)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:454)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:588)
Caused by: org.apache.hadoop.metrics2.MetricsException:
Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
at
org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:113)
... 19 more
2012-03-14 15:22:23,090 WARN org.apache.hadoop.metrics2.util.MBeans: Failed
to register MBean null
javax.management.RuntimeOperationsException: Exception occurred trying to
register the MBean
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:969)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
at
org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
at $Proxy6.postStart(Unknown Source)
at
[jira] [Updated] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3621: Attachment: MAPREDUCE-3621.patch Splitting out the changes from hadoop-common and hadoop-hdfs Can someone please review and commit these? TestDBJob and TestDataDrivenDBInputFormat ant tests fail Key: MAPREDUCE-3621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Thomas Graves Assignee: Ravi Prakash Fix For: 0.23.0 Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch The following mapred ant tests fail and have been failing for a very long time: [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4024) RM webservices can't query on finalStatus
RM webservices can't query on finalStatus - Key: MAPREDUCE-4024 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4024 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Thomas Graves Assignee: Thomas Graves The resource manager web service api to get the list of apps doesn't have a query parameter for finalStatus. It has one for the state but since that isn't what is reported by app master so we really need to be able to query on both state and finalStatus. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4024) RM webservices can't query on finalStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4024: - Status: Patch Available (was: Open) RM webservices can't query on finalStatus - Key: MAPREDUCE-4024 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4024 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: MAPREDUCE-4024.patch The resource manager web service api to get the list of apps doesn't have a query parameter for finalStatus. It has one for the state but since that isn't what is reported by app master so we really need to be able to query on both state and finalStatus. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4024) RM webservices can't query on finalStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4024: - Attachment: MAPREDUCE-4024.patch add in the finalStatus query parameter to RM apps api and re-orders a bit so we only create appinfo object if we really need to, I also added in a state query parameter for history server jobs api. RM webservices can't query on finalStatus - Key: MAPREDUCE-4024 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4024 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Thomas Graves Assignee: Thomas Graves Attachments: MAPREDUCE-4024.patch The resource manager web service api to get the list of apps doesn't have a query parameter for finalStatus. It has one for the state but since that isn't what is reported by app master so we really need to be able to query on both state and finalStatus. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231469#comment-13231469 ] Patrick Hunt commented on MAPREDUCE-3451: - Patrick unfortunately while the patch applies to the current trunk, it does not compile. Could you update? Thanks. Port Fair Scheduler to MR2 -- Key: MAPREDUCE-3451 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 Project: Hadoop Map/Reduce Issue Type: New Feature Components: mrv2, scheduler Reporter: Patrick Wendell Assignee: Patrick Wendell Attachments: MAPREDUCE-3451.v1.patch.txt The Fair Scheduler is in widespread use today in MR1 clusters, but not yet ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231508#comment-13231508 ] Siddharth Seth commented on MAPREDUCE-4005: --- +1. Looks good. AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4005: -- Resolution: Fixed Fix Version/s: 0.23.2 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-0.23 and branch-0.23.2. Thanks Jason. AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231515#comment-13231515 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Common-0.23-Commit #696 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/696/]) merge MAPREDUCE-4005 from trunk (Revision 1301693) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301693 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231519#comment-13231519 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Hdfs-0.23-Commit #687 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/687/]) merge MAPREDUCE-4005 from trunk (Revision 1301693) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301693 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231531#comment-13231531 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Common-trunk-Commit #1892 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1892/]) MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager Application Info page. (Contributed by Jason Lowe) (Revision 1301691) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301691 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231532#comment-13231532 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Hdfs-trunk-Commit #1966 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1966/]) MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager Application Info page. (Contributed by Jason Lowe) (Revision 1301691) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301691 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3983) TestTTResourceReporting can fail, and should just be deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3983: Attachment: MAPREDUCE-3983.patch Hi Bobby! Here's a patch to do that. Can you please commit it? Thanks Ravi TestTTResourceReporting can fail, and should just be deleted Key: MAPREDUCE-3983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3983 Project: Hadoop Map/Reduce Issue Type: Test Components: mrv1 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans Attachments: MAPREDUCE-3983.patch TestTTResourceReporting can fail. It is an ant test for task trackers which shoudl just be removed because task trackers are no longer supported outside of the ant tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3983) TestTTResourceReporting can fail, and should just be deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3983: Status: Patch Available (was: Open) TestTTResourceReporting can fail, and should just be deleted Key: MAPREDUCE-3983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3983 Project: Hadoop Map/Reduce Issue Type: Test Components: mrv1 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans Attachments: MAPREDUCE-3983.patch TestTTResourceReporting can fail. It is an ant test for task trackers which shoudl just be removed because task trackers are no longer supported outside of the ant tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Status: Patch Available (was: Open) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Attachment: MAPREDUCE-3353-branch-0.23.patch New patch. I have not change ConcurrentMap implementations. They use ConcurrentHashmap that have safe iterators, although java docs claim that the iterator itself can be accessed on 1 thread. Need a RM-AM channel to inform AMs about faulty/unhealthy/lost nodes - Key: MAPREDUCE-3353 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch When a node gets lost or turns faulty, AM needs to know about that event so that it can take some action like for e.g. re-executing map tasks whose intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3851) Allow more aggressive action on detection of the jetty issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231549#comment-13231549 ] Robert Joseph Evans commented on MAPREDUCE-3851: The patch looks good to me. The only thing that I can think of might be documentation. Do we want to have this documented? It will be removed in 0.23 when we no longer use Jetty so that makes me lean towards no, but the default config value is to have this disabled mapreduce.reduce.shuffle.catch.exception.percent.limit.fatal is 0.0 by default. which if it is something that is off by default users who have this problem will need to know how they can turn this on. Allow more aggressive action on detection of the jetty issue Key: MAPREDUCE-3851 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3851 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 1.0.0 Reporter: Kihwal Lee Assignee: Thomas Graves Fix For: 1.1.0, 1.0.1 Attachments: MAPREDUCE-3851.patch, MAPREDUCE-3851.patch, MAPREDUCE-3851.patch, MAPREDUCE-3851.patch MAPREDUCE-2529 added the useful failure detection mechanism. In this jira, I propose we add a periodic check inside TT and configurable action to self-destruct. Blacklisting helps but is not enough. Hung jetty still accepts connection and it takes very long time for clients to fail out. Short jobs are delayed for hours because of this. This feature will be a nice companion to MAPREDUCE-3184. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4009) AM container log links need to be clicked twice to get to the actual log file
[ https://issues.apache.org/jira/browse/MAPREDUCE-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash resolved MAPREDUCE-4009. - Resolution: Duplicate Duping to MAPREDUCE-4005 AM container log links need to be clicked twice to get to the actual log file - Key: MAPREDUCE-4009 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4009 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, webapps Affects Versions: 0.23.2 Reporter: Ravi Prakash Priority: Minor On the RM page-click on an application-Click on the link for AM Container logs This page contains links to stdout, stderr and syslog (i.e. hostname/node/containerlogs/container_1331751290995_0001_01_01/*stdout*/?start=-4096 ) Clicking on any of them still shows the same page. NOW clicking on any of them will take you to the log. e.g. hostname/node/containerlogs/container_1331751290995_0001_01_01/*stdout/stdout*/?start=-4096 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231565#comment-13231565 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1900 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1900/]) MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager Application Info page. (Contributed by Jason Lowe) (Revision 1301691) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301691 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231564#comment-13231564 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Mapreduce-0.23-Commit #704 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/704/]) merge MAPREDUCE-4005 from trunk (Revision 1301693) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301693 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java AM container logs URL is broken for completed apps when log aggregation is enabled -- Key: MAPREDUCE-4005 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 0.23.2 Attachments: MAPREDUCE-4005.patch With log aggregation enabled and yarn.log.server.url pointing to the job history server, the AM container logs URL for a completed application fails with the error Cannot get container logs without an app owner. Looking at the code in the nodemanager to handle redirects to the log server, it appears the AM container log URL is missing a user name for the job. I verified that tacking on the app's user name after the AM container log URL reported by the RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231572#comment-13231572 ] Thomas Graves commented on MAPREDUCE-4006: -- +1. Thanks Sid! I'll commit this shortly. history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231584#comment-13231584 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Common-trunk-Commit #1893 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1893/]) MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301731 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231582#comment-13231582 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Hdfs-trunk-Commit #1967 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1967/]) MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301731 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3983) TestTTResourceReporting can fail, and should just be deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231586#comment-13231586 ] Hadoop QA commented on MAPREDUCE-3983: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518723/MAPREDUCE-3983.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2068//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2068//console This message is automatically generated. TestTTResourceReporting can fail, and should just be deleted Key: MAPREDUCE-3983 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3983 Project: Hadoop Map/Reduce Issue Type: Test Components: mrv1 Affects Versions: 0.23.2 Reporter: Robert Joseph Evans Attachments: MAPREDUCE-3983.patch TestTTResourceReporting can fail. It is an ant test for task trackers which shoudl just be removed because task trackers are no longer supported outside of the ant tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4006: - Resolution: Fixed Fix Version/s: 0.23.2 Status: Resolved (was: Patch Available) committed to trunk, branch-0.23, and branch-0.23.2. history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Fix For: 0.23.2 Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231590#comment-13231590 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Hdfs-0.23-Commit #688 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/688/]) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301734 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Fix For: 0.23.2 Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231594#comment-13231594 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Common-0.23-Commit #697 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/697/]) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301734 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Fix For: 0.23.2 Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231620#comment-13231620 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1901 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1901/]) MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301731 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Fix For: 0.23.2 Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
AM can crash if task attempt reports bogus progress value
-
Key: MAPREDUCE-4025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
If a task attempt reports a bogus progress value (e.g.: something above 1.0)
then the AM can crash like this:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 12
at
org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
at
org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231629#comment-13231629 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Mapreduce-0.23-Commit #705 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/705/]) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1301734 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java history server container log web UI sometimes combines stderr/stdout/syslog contents together - Key: MAPREDUCE-4006 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Assignee: Siddharth Seth Fix For: 0.23.2 Attachments: MR4006.txt When log aggregation is enabled, going to the job history server UI for the AM container log can show the log contents combined together. Examples I've seen are portions of the syslog contents appended to either the stderr or stdout contents. The log corruption does not occur when using the mapred job -logs command, so this appears to be something specific to the history server web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3812: --- Target Version/s: 0.23.3 Fix Version/s: (was: 0.23.2) Setting a target version instead of a fix version. Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
Lower minimum-allocation-mb to sensible defaults Key: MAPREDUCE-4026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Harsh J Assignee: Harsh J The CapacityScheduler's minimum-allocation-mb is set to 1024. The FIFO's minimum-allocation-mb meanwhile, is 128. I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes slots on NMs - and I also do not see why CS has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4027) Document the minimum-allocation-mb configurations
Document the minimum-allocation-mb configurations - Key: MAPREDUCE-4027 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4027 Project: Hadoop Map/Reduce Issue Type: Improvement Components: resourcemanager Affects Versions: 0.23.3 Reporter: Harsh J Assignee: Harsh J Priority: Minor None of the current yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.capacity.minimum-allocation-mb are documented anywhere. Without knowledge of these params, one can't change the default allocations. And the default allocations are pretty high btw (MAPREDUCE-4026). We should document these in the Cluster Setup page at least. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4025:
--
Attachment: MAPREDUCE-4025.patch
Patch to simply clamp the progress value to 0.0-1.0 before we try to update the
counters.
Needs a test. I've manually verified the fix.
AM can crash if task attempt reports bogus progress value
-
Key: MAPREDUCE-4025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
Attachments: MAPREDUCE-4025.patch
If a task attempt reports a bogus progress value (e.g.: something above 1.0)
then the AM can crash like this:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 12
at
org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
at
org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231714#comment-13231714 ] Arun C Murthy commented on MAPREDUCE-4026: -- You can't really do much in jvm with 128M heap. Maybe 512M ... Lower minimum-allocation-mb to sensible defaults Key: MAPREDUCE-4026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Harsh J Assignee: Harsh J The CapacityScheduler's minimum-allocation-mb is set to 1024. The FIFO's minimum-allocation-mb meanwhile, is 128. I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes slots on NMs - and I also do not see why CS has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231832#comment-13231832 ] Harsh J commented on MAPREDUCE-4026: Arun, It is only the min-alloc am changing, not the job defaults (mapreduce.map.memory.mb and such) which are presently 1024 or higher, and will need a new JIRA if to be lowered as a default (agree that 128 is too low - either 200 or 512 should be it for those). I don't think lowering min alloc default would change default job requirements, but will at least allow folks to pass -D options for memory needs that are lower than 1024 - for more tasks in parallel. Or am I missing something? Lower minimum-allocation-mb to sensible defaults Key: MAPREDUCE-4026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Harsh J Assignee: Harsh J The CapacityScheduler's minimum-allocation-mb is set to 1024. The FIFO's minimum-allocation-mb meanwhile, is 128. I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes slots on NMs - and I also do not see why CS has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13231844#comment-13231844 ] Amar Kamat commented on MAPREDUCE-3953: --- +1. Patch looks good to me. Not sure why the patch failed. Ravi, can you kindly update the test-patch and junit test results. Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task --- Key: MAPREDUCE-3953 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Ravi Gummadi Assignee: Ravi Gummadi Attachments: 3953.v1.patch In a trace file, if a succeeded job contains a failed task, then that task's taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is ignoring/not-considering such jobs for simulation. The job could succeed even with failed tasks if the job submitter in original cluster configured that job to tolerate failures using mapreduce.map.failures.maxpercent and mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
