[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231844#comment-13231844 ] Amar Kamat commented on MAPREDUCE-3953: --- +1. Patch looks good to me. Not sure why the patch failed. Ravi, can you kindly update the test-patch and junit test results. > Gridmix throws NPE and does not simulate a job if the trace contains null > taskStatus for a task > --- > > Key: MAPREDUCE-3953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3953.v1.patch > > > In a trace file, if a succeeded job contains a failed task, then that task's > taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is > ignoring/not-considering such jobs for simulation. The job could succeed even > with failed tasks if the job submitter in original cluster configured that > job to tolerate failures using mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231832#comment-13231832 ] Harsh J commented on MAPREDUCE-4026: Arun, It is only the min-alloc am changing, not the job defaults (mapreduce.map.memory.mb and such) which are presently 1024 or higher, and will need a new JIRA if to be lowered as a default (agree that 128 is too low - either 200 or 512 should be it for those). I don't think lowering min alloc default would change default job requirements, but will at least allow folks to pass -D options for memory needs that are lower than 1024 - for more tasks in parallel. Or am I missing something? > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231714#comment-13231714 ] Arun C Murthy commented on MAPREDUCE-4026: -- You can't really do much in jvm with 128M heap. Maybe 512M ... > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
[
https://issues.apache.org/jira/browse/MAPREDUCE-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4025:
--
Attachment: MAPREDUCE-4025.patch
Patch to simply clamp the progress value to 0.0-1.0 before we try to update the
counters.
Needs a test. I've manually verified the fix.
> AM can crash if task attempt reports bogus progress value
> -
>
> Key: MAPREDUCE-4025
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am, mrv2
>Affects Versions: 0.23.2
>Reporter: Jason Lowe
> Attachments: MAPREDUCE-4025.patch
>
>
> If a task attempt reports a bogus progress value (e.g.: something above 1.0)
> then the AM can crash like this:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 12
> at
> org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
> at
> org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:619)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4027) Document the minimum-allocation-mb configurations
Document the minimum-allocation-mb configurations - Key: MAPREDUCE-4027 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4027 Project: Hadoop Map/Reduce Issue Type: Improvement Components: resourcemanager Affects Versions: 0.23.3 Reporter: Harsh J Assignee: Harsh J Priority: Minor None of the current yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.capacity.minimum-allocation-mb are documented anywhere. Without knowledge of these params, one can't change the default allocations. And the default allocations are pretty high btw (MAPREDUCE-4026). We should document these in the Cluster Setup page at least. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
Lower minimum-allocation-mb to sensible defaults Key: MAPREDUCE-4026 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.3 Reporter: Harsh J Assignee: Harsh J The CapacityScheduler's minimum-allocation-mb is set to 1024. The FIFO's minimum-allocation-mb meanwhile, is 128. I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3812: --- Target Version/s: 0.23.3 Fix Version/s: (was: 0.23.2) Setting a target version instead of a fix version. > Change default memory slot sizes to be 1.5GB > > > Key: MAPREDUCE-3812 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, performance >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-3812-20120205.txt, > MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, > MAPREDUCE-3812.patch, MAPREDUCE-3812.patch > > > After a few performance improvements tracked at MAPREDUCE-3561, like > MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within > 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work > around the issues with AM heap. Now that those are fixed, we should go back > to 1GB. > This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231629#comment-13231629 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Mapreduce-0.23-Commit #705 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/705/]) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Fix For: 0.23.2 > > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4025) AM can crash if task attempt reports bogus progress value
AM can crash if task attempt reports bogus progress value
-
Key: MAPREDUCE-4025
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4025
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.2
Reporter: Jason Lowe
If a task attempt reports a bogus progress value (e.g.: something above 1.0)
then the AM can crash like this:
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 12
at
org.apache.hadoop.mapred.PeriodicStatsAccumulator.extend(PeriodicStatsAccumulator.java:185)
at
org.apache.hadoop.mapred.WrappedPeriodicStatsAccumulator.extend(WrappedPeriodicStatsAccumulator.java:31)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.updateProgressSplits(TaskAttemptImpl.java:1043)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.access$4100(TaskAttemptImpl.java:136)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1509)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$StatusUpdater.transition(TaskAttemptImpl.java:1490)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:931)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:135)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:886)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:878)
at
org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
at
org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
at java.lang.Thread.run(Thread.java:619)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231620#comment-13231620 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1901 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1901/]) MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = ABORTED tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Fix For: 0.23.2 > > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231594#comment-13231594 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Common-0.23-Commit #697 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/697/]) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Fix For: 0.23.2 > > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231590#comment-13231590 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Hdfs-0.23-Commit #688 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/688/]) merge -r 1301730:1301731 from trunk to branch-0.23. FIXES: MAPREDUCE-4006 (Revision 1301734) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301734 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Fix For: 0.23.2 > > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4006: - Resolution: Fixed Fix Version/s: 0.23.2 Status: Resolved (was: Patch Available) committed to trunk, branch-0.23, and branch-0.23.2. > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Fix For: 0.23.2 > > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3983) TestTTResourceReporting can fail, and should just be deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231586#comment-13231586 ] Hadoop QA commented on MAPREDUCE-3983: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518723/MAPREDUCE-3983.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2068//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2068//console This message is automatically generated. > TestTTResourceReporting can fail, and should just be deleted > > > Key: MAPREDUCE-3983 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3983 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: mrv1 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > Attachments: MAPREDUCE-3983.patch > > > TestTTResourceReporting can fail. It is an ant test for task trackers which > shoudl just be removed because task trackers are no longer supported outside > of the ant tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231582#comment-13231582 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Hdfs-trunk-Commit #1967 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1967/]) MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231584#comment-13231584 ] Hudson commented on MAPREDUCE-4006: --- Integrated in Hadoop-Common-trunk-Commit #1893 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1893/]) MAPREDUCE-4006. History server container log web UI sometimes combines stderr/stdout/syslog contents together (Siddharth Seth via tgraves) (Revision 1301731) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301731 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/AggregatedLogFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/logaggregation/TestAggregatedLogFormat.java > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4006) history server container log web UI sometimes combines stderr/stdout/syslog contents together
[ https://issues.apache.org/jira/browse/MAPREDUCE-4006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231572#comment-13231572 ] Thomas Graves commented on MAPREDUCE-4006: -- +1. Thanks Sid! I'll commit this shortly. > history server container log web UI sometimes combines stderr/stdout/syslog > contents together > - > > Key: MAPREDUCE-4006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Siddharth Seth > Attachments: MR4006.txt > > > When log aggregation is enabled, going to the job history server UI for the > AM container log can show the log contents combined together. Examples I've > seen are portions of the syslog contents appended to either the stderr or > stdout contents. The log corruption does not occur when using the mapred job > -logs command, so this appears to be something specific to the history server > web UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231564#comment-13231564 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Mapreduce-0.23-Commit #704 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/704/]) merge MAPREDUCE-4005 from trunk (Revision 1301693) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301693 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231565#comment-13231565 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1900 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1900/]) MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager Application Info page. (Contributed by Jason Lowe) (Revision 1301691) Result = ABORTED sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301691 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4009) AM container log links need to be clicked twice to get to the actual log file
[ https://issues.apache.org/jira/browse/MAPREDUCE-4009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash resolved MAPREDUCE-4009. - Resolution: Duplicate Duping to MAPREDUCE-4005 > AM container log links need to be clicked twice to get to the actual log file > - > > Key: MAPREDUCE-4009 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4009 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, webapps >Affects Versions: 0.23.2 >Reporter: Ravi Prakash >Priority: Minor > > On the RM page->click on an application->Click on the link for "AM Container > logs" > This page contains links to stdout, stderr and syslog (i.e. > /node/containerlogs/container_1331751290995_0001_01_01/*stdout*/?start=-4096 > ) > Clicking on any of them still shows the same page. NOW clicking on any of > them will take you to the log. e.g. > hostname/node/containerlogs/container_1331751290995_0001_01_01/*stdout/stdout*/?start=-4096 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3851) Allow more aggressive action on detection of the jetty issue
[ https://issues.apache.org/jira/browse/MAPREDUCE-3851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231549#comment-13231549 ] Robert Joseph Evans commented on MAPREDUCE-3851: The patch looks good to me. The only thing that I can think of might be documentation. Do we want to have this documented? It will be removed in 0.23 when we no longer use Jetty so that makes me lean towards no, but the default config value is to have this disabled "mapreduce.reduce.shuffle.catch.exception.percent.limit.fatal" is 0.0 by default. which if it is something that is off by default users who have this problem will need to know how they can turn this on. > Allow more aggressive action on detection of the jetty issue > > > Key: MAPREDUCE-3851 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3851 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.0.0 >Reporter: Kihwal Lee >Assignee: Thomas Graves > Fix For: 1.1.0, 1.0.1 > > Attachments: MAPREDUCE-3851.patch, MAPREDUCE-3851.patch, > MAPREDUCE-3851.patch, MAPREDUCE-3851.patch > > > MAPREDUCE-2529 added the useful failure detection mechanism. In this jira, I > propose we add a periodic check inside TT and configurable action to > self-destruct. Blacklisting helps but is not enough. Hung jetty still accepts > connection and it takes very long time for clients to fail out. Short jobs > are delayed for hours because of this. This feature will be a nice companion > to MAPREDUCE-3184. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Attachment: MAPREDUCE-3353-branch-0.23.patch New patch. I have not change ConcurrentMap implementations. They use ConcurrentHashmap that have safe iterators, although java docs claim that the iterator itself can be accessed on 1 thread. > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3353) Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3353: -- Status: Patch Available (was: Open) > Need a RM->AM channel to inform AMs about faulty/unhealthy/lost nodes > - > > Key: MAPREDUCE-3353 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3353 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch, > MAPREDUCE-3353-branch-0.23.patch, MAPREDUCE-3353-branch-0.23.patch > > > When a node gets lost or turns faulty, AM needs to know about that event so > that it can take some action like for e.g. re-executing map tasks whose > intermediate output live on that faulty node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3983) TestTTResourceReporting can fail, and should just be deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3983: Status: Patch Available (was: Open) > TestTTResourceReporting can fail, and should just be deleted > > > Key: MAPREDUCE-3983 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3983 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: mrv1 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > Attachments: MAPREDUCE-3983.patch > > > TestTTResourceReporting can fail. It is an ant test for task trackers which > shoudl just be removed because task trackers are no longer supported outside > of the ant tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3983) TestTTResourceReporting can fail, and should just be deleted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3983: Attachment: MAPREDUCE-3983.patch Hi Bobby! Here's a patch to do that. Can you please commit it? Thanks Ravi > TestTTResourceReporting can fail, and should just be deleted > > > Key: MAPREDUCE-3983 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3983 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: mrv1 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > Attachments: MAPREDUCE-3983.patch > > > TestTTResourceReporting can fail. It is an ant test for task trackers which > shoudl just be removed because task trackers are no longer supported outside > of the ant tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231532#comment-13231532 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Hdfs-trunk-Commit #1966 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1966/]) MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager Application Info page. (Contributed by Jason Lowe) (Revision 1301691) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301691 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231531#comment-13231531 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Common-trunk-Commit #1892 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1892/]) MAPREDUCE-4005. Fixes broken AM container logs URL on ResourceManager Application Info page. (Contributed by Jason Lowe) (Revision 1301691) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301691 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231519#comment-13231519 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Hdfs-0.23-Commit #687 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/687/]) merge MAPREDUCE-4005 from trunk (Revision 1301693) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301693 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231515#comment-13231515 ] Hudson commented on MAPREDUCE-4005: --- Integrated in Hadoop-Common-0.23-Commit #696 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/696/]) merge MAPREDUCE-4005 from trunk (Revision 1301693) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301693 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/AppInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesApps.java > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-4005: -- Resolution: Fixed Fix Version/s: 0.23.2 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-0.23 and branch-0.23.2. Thanks Jason. > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.2 > > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4005) AM container logs URL is broken for completed apps when log aggregation is enabled
[ https://issues.apache.org/jira/browse/MAPREDUCE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231508#comment-13231508 ] Siddharth Seth commented on MAPREDUCE-4005: --- +1. Looks good. > AM container logs URL is broken for completed apps when log aggregation is > enabled > -- > > Key: MAPREDUCE-4005 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4005 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4005.patch > > > With log aggregation enabled and yarn.log.server.url pointing to the job > history server, the AM container logs URL for a completed application fails > with the error "Cannot get container logs without an app owner". Looking at > the code in the nodemanager to handle redirects to the log server, it appears > the AM container log URL is missing a user name for the job. I verified that > tacking on the app's user name after the AM container log URL reported by the > RM works. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231469#comment-13231469 ] Patrick Hunt commented on MAPREDUCE-3451: - Patrick unfortunately while the patch applies to the current trunk, it does not compile. Could you update? Thanks. > Port Fair Scheduler to MR2 > -- > > Key: MAPREDUCE-3451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, scheduler >Reporter: Patrick Wendell >Assignee: Patrick Wendell > Attachments: MAPREDUCE-3451.v1.patch.txt > > > The Fair Scheduler is in widespread use today in MR1 clusters, but not yet > ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and > will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4024) RM webservices can't query on finalStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4024: - Attachment: MAPREDUCE-4024.patch add in the finalStatus query parameter to RM apps api and re-orders a bit so we only create appinfo object if we really need to, I also added in a state query parameter for history server jobs api. > RM webservices can't query on finalStatus > - > > Key: MAPREDUCE-4024 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4024 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4024.patch > > > The resource manager web service api to get the list of apps doesn't have a > query parameter for finalStatus. It has one for the state but since that > isn't what is reported by app master so we really need to be able to query on > both state and finalStatus. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4024) RM webservices can't query on finalStatus
[ https://issues.apache.org/jira/browse/MAPREDUCE-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4024: - Status: Patch Available (was: Open) > RM webservices can't query on finalStatus > - > > Key: MAPREDUCE-4024 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4024 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Assignee: Thomas Graves > Attachments: MAPREDUCE-4024.patch > > > The resource manager web service api to get the list of apps doesn't have a > query parameter for finalStatus. It has one for the state but since that > isn't what is reported by app master so we really need to be able to query on > both state and finalStatus. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4024) RM webservices can't query on finalStatus
RM webservices can't query on finalStatus - Key: MAPREDUCE-4024 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4024 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.2 Reporter: Thomas Graves Assignee: Thomas Graves The resource manager web service api to get the list of apps doesn't have a query parameter for finalStatus. It has one for the state but since that isn't what is reported by app master so we really need to be able to query on both state and finalStatus. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3621: Attachment: MAPREDUCE-3621.patch Splitting out the changes from hadoop-common and hadoop-hdfs Can someone please review and commit these? > TestDBJob and TestDataDrivenDBInputFormat ant tests fail > > > Key: MAPREDUCE-3621 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Thomas Graves >Assignee: Ravi Prakash > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch > > > The following mapred ant tests fail and have been failing for a very long > time: > [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob > [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4008) ResourceManager throws MetricsException on start up saying QueueMetrics MBean already exists
[
https://issues.apache.org/jira/browse/MAPREDUCE-4008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231316#comment-13231316
]
Jason Lowe commented on MAPREDUCE-4008:
---
The TestWritableJobConf failure is a known issue, see MAPREDUCE-4010.
> ResourceManager throws MetricsException on start up saying QueueMetrics MBean
> already exists
>
>
> Key: MAPREDUCE-4008
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4008
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager, scheduler
>Affects Versions: 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-4008.patch
>
>
> {code:xml}
> 2012-03-14 15:22:23,089 WARN org.apache.hadoop.metrics2.util.MBeans: Error
> creating MBean object name:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default
> org.apache.hadoop.metrics2.MetricsException:
> org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:117)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newMBeanName(DefaultMetricsSystem.java:102)
> at org.apache.hadoop.metrics2.util.MBeans.getMBeanName(MBeans.java:91)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:55)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invoke(MetricsSystemImpl.java:288)
> at $Proxy6.postStart(Unknown Source)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.start(MetricsSystemImpl.java:183)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.init(MetricsSystemImpl.java:155)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.init(DefaultMetricsSystem.java:54)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.initialize(DefaultMetricsSystem.java:50)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.start(ResourceManager.java:454)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:588)
> Caused by: org.apache.hadoop.metrics2.MetricsException:
> Hadoop:service=ResourceManager,name=QueueMetrics,q0=default already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newObjectName(DefaultMetricsSystem.java:113)
> ... 19 more
> 2012-03-14 15:22:23,090 WARN org.apache.hadoop.metrics2.util.MBeans: Failed
> to register MBean "null"
> javax.management.RuntimeOperationsException: Exception occurred trying to
> register the MBean
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:969)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)
> at
> com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)
> at
> com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)
> at org.apache.hadoop.metrics2.util.MBeans.register(MBeans.java:57)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.startMBeans(MetricsSourceAdapter.java:218)
> at
> org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.start(MetricsSourceAdapter.java:93)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.registerSource(MetricsSystemImpl.java:243)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$1.postStart(MetricsSystemImpl.java:227)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl$3.invok
[jira] [Created] (MAPREDUCE-4023) Make Yarn Client service shutdown operations robust
Make Yarn Client service shutdown operations robust --- Key: MAPREDUCE-4023 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4023 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.2 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Attachments: MAPREDUCE-4023.patch Make the yarn client services more robust against being shut down while not started, or shutdown more than once, by null-checking fields before closing them, setting to null afterwards to prevent double-invocation. This is a subset of MAPREDUCE-3502 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4023) Make Yarn Client service shutdown operations robust
[ https://issues.apache.org/jira/browse/MAPREDUCE-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4023: -- Attachment: MAPREDUCE-4023.patch depends on MAPREDUCE-4014 > Make Yarn Client service shutdown operations robust > --- > > Key: MAPREDUCE-4023 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4023 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: MAPREDUCE-4023.patch > > > Make the yarn client services more robust against being shut down while not > started, or shutdown more than once, by null-checking fields before closing > them, setting to null afterwards to prevent double-invocation. This is a > subset of MAPREDUCE-3502 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4022) Make Yarn Node Manager services robust against shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4022: -- Attachment: MAPREDUCE-4022.patch depends on MAPREDUCE-4014 > Make Yarn Node Manager services robust against shutdown > --- > > Key: MAPREDUCE-4022 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4022 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.24.0, 0.23.2 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: MAPREDUCE-4022.patch > > > Add the nodemanager bits of MAPREDUCE-3502 to shut down the Nodemanager > services. This is done by checking for fields being non-null before shutting > down/closing etc, and setting the fields to null afterwards -to be resilient > against re-entrancy. > No tests other than manual review. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4022) Make Yarn Node Manager services robust against shutdown
Make Yarn Node Manager services robust against shutdown --- Key: MAPREDUCE-4022 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4022 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.2 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Add the nodemanager bits of MAPREDUCE-3502 to shut down the Nodemanager services. This is done by checking for fields being non-null before shutting down/closing etc, and setting the fields to null afterwards -to be resilient against re-entrancy. No tests other than manual review. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231291#comment-13231291 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1896 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1896/]) MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301551) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301551 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231290#comment-13231290 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Mapreduce-0.23-Commit #700 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/700/]) svn merge -c 1301551 from trunk to branch-0.23 FIXES MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301553) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301553 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4021) Make yarn Resource Manager services robust against shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-4021: -- Attachment: MAPREDUCE-4021.patch patch not ready to apply to trunk, pending a version of MAPREDUCE 4014 going in > Make yarn Resource Manager services robust against shutdown > --- > > Key: MAPREDUCE-4021 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4021 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.24.0, 0.23.3 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Attachments: MAPREDUCE-4021.patch > > > Split MAPREDUCE-3502 patches to make the RM code more resilient to being > stopped more than once, or before started. > This depends on MAPREDUCE-4014. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4021) Make yarn Resource Manager services robust against shutdown
Make yarn Resource Manager services robust against shutdown --- Key: MAPREDUCE-4021 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4021 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.24.0, 0.23.3 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Split MAPREDUCE-3502 patches to make the RM code more resilient to being stopped more than once, or before started. This depends on MAPREDUCE-4014. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231262#comment-13231262 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Common-trunk-Commit #1888 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1888/]) MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301551) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301551 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231254#comment-13231254 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Hdfs-trunk-Commit #1962 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1962/]) MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301551) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301551 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231250#comment-13231250 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Common-0.23-Commit #692 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/692/]) svn merge -c 1301551 from trunk to branch-0.23 FIXES MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301553) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301553 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231248#comment-13231248 ] Hudson commented on MAPREDUCE-4010: --- Integrated in Hadoop-Hdfs-0.23-Commit #683 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/683/]) svn merge -c 1301551 from trunk to branch-0.23 FIXES MAPREDUCE-4010. TestWritableJobConf fails on trunk (tucu via bobby) (Revision 1301553) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301553 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestWritableJobConf.java > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4010) TestWritableJobConf fails on trunk
[ https://issues.apache.org/jira/browse/MAPREDUCE-4010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4010: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks Alejandro, I just checked this into 0.23.3 and trunk. I also added a small comment in the test about why we are skipping the deprecated values. > TestWritableJobConf fails on trunk > -- > > Key: MAPREDUCE-4010 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4010 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Jason Lowe >Assignee: Alejandro Abdelnur >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-4010.patch, MAPREDUCE-4010.patch, > MAPREDUCE-4010.patch > > > TestWritableJobConf is currently failing two tests on trunk: > * testEmptyConfiguration > * testNonEmptyConfiguration > Appears to have been caused by HADOOP-8167. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231229#comment-13231229 ] Hudson commented on MAPREDUCE-3431: --- Integrated in Hadoop-Mapreduce-0.23-Commit #699 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/699/]) MAPREDUCE-3431 (Revision 1301502) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301502 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewerLifecycle.java > NPE in Resource Manager shutdown > > > Key: MAPREDUCE-3431 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.0, 0.24.0 > Environment: OS/X w/ java6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: yarn > Fix For: 0.24.0, 0.23.3 > > Attachments: MAPREDUCE-3431.patch > > > bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3970) Add ServiceOperations class to aid working with Services
[ https://issues.apache.org/jira/browse/MAPREDUCE-3970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231228#comment-13231228 ] Hudson commented on MAPREDUCE-3970: --- Integrated in Hadoop-Mapreduce-0.23-Commit #699 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/699/]) MAPREDUCE-3970 not a big fix, an improvement (Revision 1301495) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301495 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt > Add ServiceOperations class to aid working with Services > > > Key: MAPREDUCE-3970 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3970 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.1, 0.24.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Fix For: 0.24.0, 0.23.3 > > Attachments: MAPREDUCE-3970.patch, MAPREDUCE-3970.patch > > > Add Helper methods to move things through lifecycles. init->start is common, > stop-if-service!=null another. Some static methods can execute these, and > even call stop() if init() raises an exception. These could go into a class > ServiceOps in the same package. These can be used by those services that wrap > other services, and help manage more robust shutdowns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231230#comment-13231230 ] Hudson commented on MAPREDUCE-3431: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1895 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1895/]) MAPREDUCE-3431 (Revision 1301500) MAPREDUCE-3431 (Revision 1301496) Result = ABORTED stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301500 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301496 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewerLifecycle.java > NPE in Resource Manager shutdown > > > Key: MAPREDUCE-3431 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.0, 0.24.0 > Environment: OS/X w/ java6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: yarn > Fix For: 0.24.0, 0.23.3 > > Attachments: MAPREDUCE-3431.patch > > > bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231183#comment-13231183 ] Hudson commented on MAPREDUCE-3431: --- Integrated in Hadoop-Common-trunk-Commit #1887 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1887/]) MAPREDUCE-3431 (Revision 1301500) MAPREDUCE-3431 (Revision 1301496) Result = SUCCESS stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301500 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt stevel : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1301496 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/security/DelegationTokenRenewer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestDelegationTokenRenewerLifecycle.java > NPE in Resource Manager shutdown > > > Key: MAPREDUCE-3431 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.0, 0.24.0 > Environment: OS/X w/ java6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: yarn > Fix For: 0.24.0, 0.23.3 > > Attachments: MAPREDUCE-3431.patch > > > bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated MAPREDUCE-3431: -- Resolution: Fixed Fix Version/s: 0.23.3 0.24.0 Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Resolved (was: Patch Available) > NPE in Resource Manager shutdown > > > Key: MAPREDUCE-3431 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.0, 0.24.0 > Environment: OS/X w/ java6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: yarn > Fix For: 0.24.0, 0.23.3 > > Attachments: MAPREDUCE-3431.patch > > > bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3431) NPE in Resource Manager shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231161#comment-13231161 ] Steve Loughran commented on MAPREDUCE-3431: --- voting for this myself, +1, it's a trivial robustness tweak to shutdowns with tests > NPE in Resource Manager shutdown > > > Key: MAPREDUCE-3431 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3431 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: resourcemanager >Affects Versions: 0.23.0, 0.24.0 > Environment: OS/X w/ java6 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Minor > Labels: yarn > Attachments: MAPREDUCE-3431.patch > > > bringing up a resource manager failed; shutdown triggered an NPE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13230973#comment-13230973 ] Hadoop QA commented on MAPREDUCE-3953: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518631/3953.v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2065//console This message is automatically generated. > Gridmix throws NPE and does not simulate a job if the trace contains null > taskStatus for a task > --- > > Key: MAPREDUCE-3953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3953.v1.patch > > > In a trace file, if a succeeded job contains a failed task, then that task's > taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is > ignoring/not-considering such jobs for simulation. The job could succeed even > with failed tasks if the job submitter in original cluster configured that > job to tolerate failures using mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Release Note: Fixes NPE and makes Gridmix simulate succeeded-jobs-with-failed-tasks. All tasks of such simulated jobs(including the failed ones of original job) will succeed. Status: Patch Available (was: Open) > Gridmix throws NPE and does not simulate a job if the trace contains null > taskStatus for a task > --- > > Key: MAPREDUCE-3953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3953.v1.patch > > > In a trace file, if a succeeded job contains a failed task, then that task's > taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is > ignoring/not-considering such jobs for simulation. The job could succeed even > with failed tasks if the job submitter in original cluster configured that > job to tolerate failures using mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3953) Gridmix throws NPE and does not simulate a job if the trace contains null taskStatus for a task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Gummadi updated MAPREDUCE-3953: Attachment: 3953.v1.patch Attaching patch with the code changes mentioned in the previous comment. > Gridmix throws NPE and does not simulate a job if the trace contains null > taskStatus for a task > --- > > Key: MAPREDUCE-3953 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3953 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Ravi Gummadi >Assignee: Ravi Gummadi > Attachments: 3953.v1.patch > > > In a trace file, if a succeeded job contains a failed task, then that task's > taskStatus will be null. This is causing NPE in Gridmix and then Gridmix is > ignoring/not-considering such jobs for simulation. The job could succeed even > with failed tasks if the job submitter in original cluster configured that > job to tolerate failures using mapreduce.map.failures.maxpercent and > mapreduce.reduce.failures.maxpercent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
