[jira] [Commented] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be
[ https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181261#comment-13181261 ] Hudson commented on MAPREDUCE-3326: --- Integrated in Hadoop-Hdfs-trunk #917 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/917/]) MAPREDUCE-3326. Added detailed information about queue's to the CapacityScheduler web-ui. Contributed by Jason Lowe. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227788 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java RM web UI scheduler link not as useful as should be --- Key: MAPREDUCE-3326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Thomas Graves Assignee: Jason Lowe Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3326.patch, Screen shot 2012-01-05 at 11.23.26 AM.png The resource manager web ui page for scheduler doesn't have all the information about the configuration like the jobtracker page used to have. The things it seems to show you are the current queues - each queues used, set, and max percent and then what apps are running in that queue. It doesn't list any of yarn.scheduler.capacity.maximum-applications, yarn.scheduler.capacity.maximum-am-resource-percent, yarn.scheduler.capacity.queue-path.user-limit-factor, yarn.scheduler.capacity.queue-path.minimum-user-limit-percent, queue state, active users and percent used by user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3548) write unit tests for web services for mapreduce app master and job history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181265#comment-13181265 ] Hudson commented on MAPREDUCE-3548: --- Integrated in Hadoop-Hdfs-trunk #917 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/917/]) MAPREDUCE-3548. Added more unit tests for MR AM JHS web-services. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227801 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/CounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobCounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptCounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskCounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskAttemptsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesTasks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java *
[jira] [Commented] (MAPREDUCE-3617) Remove yarn default values for resource manager and nodemanager principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181274#comment-13181274 ] Hudson commented on MAPREDUCE-3617: --- Integrated in Hadoop-Hdfs-trunk #917 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/917/]) MAPREDUCE-3617. Removed wrong default value for yarn.resourcemanager.principal and yarn.nodemanager.principal. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227808 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml Remove yarn default values for resource manager and nodemanager principal - Key: MAPREDUCE-3617 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3617 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3617.patch, MAPREDUCE-3617.patch Default values should be empty since no use can be made of them without correct values defined. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3615) mapred ant test failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181273#comment-13181273 ] Hudson commented on MAPREDUCE-3615: --- Integrated in Hadoop-Hdfs-trunk #917 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/917/]) MAPREDUCE-3615. Fix some ant test failures. (Contributed by Thomas Graves) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227741 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java mapred ant test failures Key: MAPREDUCE-3615 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3615 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3615.patch The following mapred ant tests are failing. This started on December 22nd. [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Running org.apache.hadoop.mapred.TestMiniMRDFSSort [junit] Running org.apache.hadoop.mapred.TestBadRecords [junit] Running org.apache.hadoop.mapred.TestClusterMRNotification [junit] Running org.apache.hadoop.mapred.TestDebugScript [junit] Running org.apache.hadoop.mapred.TestJobCleanup [junit] Running org.apache.hadoop.mapred.TestJobClient [junit] Running org.apache.hadoop.mapred.TestJobHistory [junit] Running org.apache.hadoop.mapred.TestJobInProgressListener [junit] Running org.apache.hadoop.mapred.TestJobKillAndFail [junit] Running org.apache.hadoop.mapred.TestJvmReuse [junit] Running org.apache.hadoop.mapred.TestKillSubProcesses [junit] Running org.apache.hadoop.mapred.TestNodeRefresh [junit] Running org.apache.hadoop.mapred.TestSetupAndCleanupFailure [junit] Running org.apache.hadoop.mapred.TestTaskFail [junit] Running org.apache.hadoop.mapred.TestTaskOutputSize [junit] Running org.apache.hadoop.mapred.TestTaskTrackerSlotManagement [junit] Running org.apache.hadoop.mapreduce.TestMRJobClient [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3620) Some times all the gridmix jobs are not processing.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181276#comment-13181276
]
Karam Singh commented on MAPREDUCE-3620:
The issue reported Vinay Kumar Thota does not seems to be MRV2. It is more
related to GridMix statistics which gridmix prints.
The trace is having 1205 jobs, gridmix process 1200 whereas rejecting next 4-5
as those are failed/killed jobs in trace.
In boths runs reported by Vinay number of jobs process were 1200.
But In run 1 :
{
6 jobs failed, 1193 jobs succeeded and one job was lost (as NodeManager on
which AppMaster of that job running lost).
At the GridMix statistics at the end print 1196 jobs processed, 6 jobs failed
and 1190 jobs passed (which is wrong w.r.t actual figure).
}
Whereas In run 2 :
{
4 jobs failed, 1196 jobs succeeded.
At the GridMix statistics at the end print 1200 jobs processed, 4 jobs failed
and 1196 jobs passed (which is correct).
}
So issue clearly belongs GridMix statistics reporting at end not job processing.
And lost jobs was due to AM lost
Some times all the gridmix jobs are not processing.
---
Key: MAPREDUCE-3620
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3620
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Priority: Critical
Courtesy [~vinaythota]
{quote}
Job trace contains 1205 jobs and Gridmix start processing 1200 jobs after
processing. However, after completion of
gridmix run, execution summary details, it showed 1196 jobs are processed and
remaining 4 jobs are missing. One log shows 1196 jobs processed and another
log shows 1200 jobs are processed.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3326) RM web UI scheduler link not as useful as should be
[ https://issues.apache.org/jira/browse/MAPREDUCE-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181299#comment-13181299 ] Hudson commented on MAPREDUCE-3326: --- Integrated in Hadoop-Mapreduce-trunk #950 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/950/]) MAPREDUCE-3326. Added detailed information about queue's to the CapacityScheduler web-ui. Contributed by Jason Lowe. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227788 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/ResponseInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerLeafQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java RM web UI scheduler link not as useful as should be --- Key: MAPREDUCE-3326 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3326 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Thomas Graves Assignee: Jason Lowe Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3326.patch, Screen shot 2012-01-05 at 11.23.26 AM.png The resource manager web ui page for scheduler doesn't have all the information about the configuration like the jobtracker page used to have. The things it seems to show you are the current queues - each queues used, set, and max percent and then what apps are running in that queue. It doesn't list any of yarn.scheduler.capacity.maximum-applications, yarn.scheduler.capacity.maximum-am-resource-percent, yarn.scheduler.capacity.queue-path.user-limit-factor, yarn.scheduler.capacity.queue-path.minimum-user-limit-percent, queue state, active users and percent used by user -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3548) write unit tests for web services for mapreduce app master and job history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-3548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181303#comment-13181303 ] Hudson commented on MAPREDUCE-3548: --- Integrated in Hadoop-Mapreduce-trunk #950 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/950/]) MAPREDUCE-3548. Added more unit tests for MR AM JHS web-services. Contributed by Thomas Graves. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227801 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/AppInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/ConfInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/CounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobCounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskAttemptCounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/JobTaskCounterInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskAttemptsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/dao/TaskInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAMWebServicesTasks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/JAXBContextResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/AMAttemptInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesAttempts.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobConf.java *
[jira] [Commented] (MAPREDUCE-3615) mapred ant test failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181311#comment-13181311 ] Hudson commented on MAPREDUCE-3615: --- Integrated in Hadoop-Mapreduce-trunk #950 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/950/]) MAPREDUCE-3615. Fix some ant test failures. (Contributed by Thomas Graves) sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227741 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/TaskAttemptUnsuccessfulCompletionEvent.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/java/org/apache/hadoop/mapred/JobInProgress.java mapred ant test failures Key: MAPREDUCE-3615 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3615 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3615.patch The following mapred ant tests are failing. This started on December 22nd. [junit] Running org.apache.hadoop.mapred.TestTrackerBlacklistAcrossJobs [junit] Running org.apache.hadoop.mapred.TestMiniMRDFSSort [junit] Running org.apache.hadoop.mapred.TestBadRecords [junit] Running org.apache.hadoop.mapred.TestClusterMRNotification [junit] Running org.apache.hadoop.mapred.TestDebugScript [junit] Running org.apache.hadoop.mapred.TestJobCleanup [junit] Running org.apache.hadoop.mapred.TestJobClient [junit] Running org.apache.hadoop.mapred.TestJobHistory [junit] Running org.apache.hadoop.mapred.TestJobInProgressListener [junit] Running org.apache.hadoop.mapred.TestJobKillAndFail [junit] Running org.apache.hadoop.mapred.TestJvmReuse [junit] Running org.apache.hadoop.mapred.TestKillSubProcesses [junit] Running org.apache.hadoop.mapred.TestNodeRefresh [junit] Running org.apache.hadoop.mapred.TestSetupAndCleanupFailure [junit] Running org.apache.hadoop.mapred.TestTaskFail [junit] Running org.apache.hadoop.mapred.TestTaskOutputSize [junit] Running org.apache.hadoop.mapred.TestTaskTrackerSlotManagement [junit] Running org.apache.hadoop.mapreduce.TestMRJobClient [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3617) Remove yarn default values for resource manager and nodemanager principal
[ https://issues.apache.org/jira/browse/MAPREDUCE-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181312#comment-13181312 ] Hudson commented on MAPREDUCE-3617: --- Integrated in Hadoop-Mapreduce-trunk #950 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/950/]) MAPREDUCE-3617. Removed wrong default value for yarn.resourcemanager.principal and yarn.nodemanager.principal. Contributed by Jonathan Eagles. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1227808 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/resources/yarn-default.xml Remove yarn default values for resource manager and nodemanager principal - Key: MAPREDUCE-3617 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3617 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Jonathan Eagles Assignee: Jonathan Eagles Fix For: 0.23.0 Attachments: MAPREDUCE-3617.patch, MAPREDUCE-3617.patch Default values should be empty since no use can be made of them without correct values defined. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1938) Ability for having user's classes take precedence over the system classes for tasks' classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1938: --- Fix Version/s: (was: 0.23.0) This isn't present in 0.23. Removing from Fix Version. I'll take Arun's word that it is not needed there either (though it'd look like API breakage, hrm?) It should be included in 0.22 though. Ability for having user's classes take precedence over the system classes for tasks' classpath -- Key: MAPREDUCE-1938 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938 Project: Hadoop Map/Reduce Issue Type: New Feature Components: job submission, task, tasktracker Reporter: Devaraj Das Assignee: Krishna Ramachandran Priority: Blocker Fix For: 0.20.204.0 Attachments: mapred-1938-1, mapred-1938-2.patch, mapred-1938-3.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3312) Make MR AM not send a stopContainer w/o corresponding start container
[ https://issues.apache.org/jira/browse/MAPREDUCE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181356#comment-13181356 ] Robert Joseph Evans commented on MAPREDUCE-3312: Will fix the findbugs warning and resubmit. Make MR AM not send a stopContainer w/o corresponding start container - Key: MAPREDUCE-3312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3312 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.23.1 Attachments: MR-3312.txt, MR-3312.txt This is a follow on to MAPREDUCE-3274. It is possible, although rare, for the MR AM to send a stop container before it sends a start container. This needs to stop that from happening. If a stop is found first it should prevent the start from being sent. It tries to do this, but only if the stop is currently pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3312) Make MR AM not send a stopContainer w/o corresponding start container
[ https://issues.apache.org/jira/browse/MAPREDUCE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3312: --- Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) Make MR AM not send a stopContainer w/o corresponding start container - Key: MAPREDUCE-3312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3312 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.23.1 Attachments: MR-3312.txt, MR-3312.txt This is a follow on to MAPREDUCE-3274. It is possible, although rare, for the MR AM to send a stop container before it sends a start container. This needs to stop that from happening. If a stop is found first it should prevent the start from being sent. It tries to do this, but only if the stop is currently pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3633) MAPREDUCE-1938 isn't present in 0.22 nor 0.23 nor trunk. Breaks API.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-3633: --- Target Version/s: 0.23.1, 0.22.1 MAPREDUCE-1938 isn't present in 0.22 nor 0.23 nor trunk. Breaks API. Key: MAPREDUCE-3633 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3633 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Harsh J Priority: Blocker MAPREDUCE-1938 introduced an API call in JobConf and the likes. This is not present in 0.22 nor 0.23. Even if it isn't needed, the method must be still present as @Deprecated and perhaps made defunct inside if it should not do anything. Not having this method is API breakage. Let me know if am grossly wrong here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3633) MAPREDUCE-1938 isn't present in 0.22 nor 0.23 nor trunk. Breaks API.
MAPREDUCE-1938 isn't present in 0.22 nor 0.23 nor trunk. Breaks API. Key: MAPREDUCE-3633 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3633 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0, 0.22.0 Reporter: Harsh J Priority: Blocker MAPREDUCE-1938 introduced an API call in JobConf and the likes. This is not present in 0.22 nor 0.23. Even if it isn't needed, the method must be still present as @Deprecated and perhaps made defunct inside if it should not do anything. Not having this method is API breakage. Let me know if am grossly wrong here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1938) Ability for having user's classes take precedence over the system classes for tasks' classpath
[ https://issues.apache.org/jira/browse/MAPREDUCE-1938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-1938: --- Fix Version/s: 0.23.0 Sorry, my bad. I meant to target another JIRA. Adding back the Fix Version. Ability for having user's classes take precedence over the system classes for tasks' classpath -- Key: MAPREDUCE-1938 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1938 Project: Hadoop Map/Reduce Issue Type: New Feature Components: job submission, task, tasktracker Reporter: Devaraj Das Assignee: Krishna Ramachandran Priority: Blocker Fix For: 0.20.204.0, 0.23.0 Attachments: mapred-1938-1, mapred-1938-2.patch, mapred-1938-3.patch, mr-1938-bp20.1.patch, mr-1938-bp20.patch It would be nice to have the ability in MapReduce to allow users to specify for their jobs alternate implementations of classes that are already defined in the MapReduce libraries. For example, an alternate implementation for CombineFileInputFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3633) MAPREDUCE-1938 isn't present in 0.22 nor 0.23 nor trunk. Breaks API.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-3633. Resolution: Invalid Target Version/s: 0.23.1, 0.22.1 (was: 0.22.1, 0.23.1) Nevermind. This was an oversight, looking at some wrong sources. MAPREDUCE-1938 isn't present in 0.22 nor 0.23 nor trunk. Breaks API. Key: MAPREDUCE-3633 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3633 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Harsh J Priority: Blocker MAPREDUCE-1938 introduced an API call in JobConf and the likes. This is not present in 0.22 nor 0.23. Even if it isn't needed, the method must be still present as @Deprecated and perhaps made defunct inside if it should not do anything. Not having this method is API breakage. Let me know if am grossly wrong here. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3312) Make MR AM not send a stopContainer w/o corresponding start container
[ https://issues.apache.org/jira/browse/MAPREDUCE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3312: --- Fix Version/s: 0.24.0 Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) Make MR AM not send a stopContainer w/o corresponding start container - Key: MAPREDUCE-3312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3312 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.23.1, 0.24.0 Attachments: MR-3312.txt, MR-3312.txt, MR-3312.txt This is a follow on to MAPREDUCE-3274. It is possible, although rare, for the MR AM to send a stop container before it sends a start container. This needs to stop that from happening. If a stop is found first it should prevent the start from being sent. It tries to do this, but only if the stop is currently pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3312) Make MR AM not send a stopContainer w/o corresponding start container
[ https://issues.apache.org/jira/browse/MAPREDUCE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3312: --- Attachment: MR-3312.txt This patch fixes the findbugs issue, but it looks like fixing it unmasked another findbugs issue that is not related to this patch. I will file a separate JIRA for that issue once test-patch completes. Make MR AM not send a stopContainer w/o corresponding start container - Key: MAPREDUCE-3312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3312 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.23.1, 0.24.0 Attachments: MR-3312.txt, MR-3312.txt, MR-3312.txt This is a follow on to MAPREDUCE-3274. It is possible, although rare, for the MR AM to send a stop container before it sends a start container. This needs to stop that from happening. If a stop is found first it should prevent the start from being sent. It tries to do this, but only if the stop is currently pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3628) DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181391#comment-13181391
]
Kihwal Lee commented on MAPREDUCE-3628:
---
I believe it was TestDFSIO.
{noformat}
12/01/05 07:50:01 INFO fs.TestDFSIO: TestDFSIO.0.0.6
12/01/05 07:50:01 INFO fs.TestDFSIO: nrFiles = 2037
12/01/05 07:50:01 INFO fs.TestDFSIO: fileSize (MB) = 1024.0
12/01/05 07:50:01 INFO fs.TestDFSIO: bufferSize = 10485760
12/01/05 07:50:01 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
12/01/05 07:50:02 INFO fs.TestDFSIO: creating control file: 1073741824 bytes,
2037 files
12/01/05 07:50:36 INFO fs.TestDFSIO: created control files for: 2037 files
{noformat}
The input data set wasn't that big. So many factors external to DFS and disk
I/O could have caused this. I asked Vinay who ran the test for more details.
DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350
nodes size cluster.
-
Key: MAPREDUCE-3628
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3628
Project: Hadoop Map/Reduce
Issue Type: Task
Components: mrv2
Affects Versions: 0.23.0
Reporter: Amol Kekre
Priority: Blocker
Fix For: 0.23.1
DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350
nodes size cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3553: - Attachment: MAPREDUCE-3553.patch Patch rebased and added more tests. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON - Key: MAPREDUCE-3553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch When the web services apis for rm, nm, app master, and job history server throw an exception - like bad request, not found, they always return the data in JSON format. It would be nice to return based on what they requested - xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3553: - Status: Patch Available (was: Open) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON - Key: MAPREDUCE-3553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch When the web services apis for rm, nm, app master, and job history server throw an exception - like bad request, not found, they always return the data in JSON format. It would be nice to return based on what they requested - xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Attachment: MAPREDUCE-3299.patch Web Services unit tests are now passing. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Status: Patch Available (was: Open) Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3312) Make MR AM not send a stopContainer w/o corresponding start container
[ https://issues.apache.org/jira/browse/MAPREDUCE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181406#comment-13181406 ] Hadoop QA commented on MAPREDUCE-3312: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509687/MR-3312.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1553//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1553//console This message is automatically generated. Make MR AM not send a stopContainer w/o corresponding start container - Key: MAPREDUCE-3312 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3312 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0, 0.24.0 Reporter: Robert Joseph Evans Assignee: Robert Joseph Evans Fix For: 0.23.1, 0.24.0 Attachments: MR-3312.txt, MR-3312.txt, MR-3312.txt This is a follow on to MAPREDUCE-3274. It is possible, although rare, for the MR AM to send a stop container before it sends a start container. This needs to stop that from happening. If a stop is found first it should prevent the start from being sent. It tries to do this, but only if the stop is currently pending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3543) Mavenize Gridmix.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181409#comment-13181409 ] Alejandro Abdelnur commented on MAPREDUCE-3543: --- Amar, Thxs for the clarification. If that is the case, then Rumen has to be recoded to use MR2 stuff for trunk/0.23 and Gridmix would be on hold till then. Right? Mavenize Gridmix. - Key: MAPREDUCE-3543 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3543 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Mahadev konar Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3543v1.patch, MAPREDUCE-3543v1.sh Gridmix codebase still resides in src/contrib and needs to be compiled via ant. We should move it to maven. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3620) Some times all the gridmix jobs are not processing.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-3620:
Component/s: (was: mrv2)
contrib/gridmix
Some times all the gridmix jobs are not processing.
---
Key: MAPREDUCE-3620
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3620
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Priority: Critical
Courtesy [~vinaythota]
{quote}
Job trace contains 1205 jobs and Gridmix start processing 1200 jobs after
processing. However, after completion of
gridmix run, execution summary details, it showed 1196 jobs are processed and
remaining 4 jobs are missing. One log shows 1196 jobs processed and another
log shows 1200 jobs are processed.
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181417#comment-13181417 ] Hadoop QA commented on MAPREDUCE-3553: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509689/MAPREDUCE-3553.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1554//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1554//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1554//console This message is automatically generated. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON - Key: MAPREDUCE-3553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch When the web services apis for rm, nm, app master, and job history server throw an exception - like bad request, not found, they always return the data in JSON format. It would be nice to return based on what they requested - xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3553) Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON
[ https://issues.apache.org/jira/browse/MAPREDUCE-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181419#comment-13181419 ] Thomas Graves commented on MAPREDUCE-3553: -- findbugs was pre-existing see jira 3622. Add support for data returned when exceptions thrown from web service apis to be in either xml or in JSON - Key: MAPREDUCE-3553 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3553 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Minor Attachments: MAPREDUCE-3553.patch, MAPREDUCE-3553.patch When the web services apis for rm, nm, app master, and job history server throw an exception - like bad request, not found, they always return the data in JSON format. It would be nice to return based on what they requested - xml or JSON. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181426#comment-13181426 ] Hadoop QA commented on MAPREDUCE-3299: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509691/MAPREDUCE-3299.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1555//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1555//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1555//console This message is automatically generated. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181430#comment-13181430 ] Jonathan Eagles commented on MAPREDUCE-3299: Findbugs ContainerRemoteLaunchEvent defines equals but not hashCode is pre-existing Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Patch Available (was: Open) JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Attachment: MAPREDUCE-3354.patch Updated patch to address ClusterSetup.apt.vm and hadoop-mapreduce-project/INSTALL aligning the documentation with the new method of starting the history server. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181452#comment-13181452 ] Hadoop QA commented on MAPREDUCE-3354: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509693/MAPREDUCE-3354.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1556//console This message is automatically generated. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181459#comment-13181459 ] Jonathan Eagles commented on MAPREDUCE-3354: Patch modifies hadoop-assemblies so patch apply failure is expected. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3634) All daemons should crash instead of hanging around when their EventHandlers get exceptions
All daemons should crash instead of hanging around when their EventHandlers get exceptions -- Key: MAPREDUCE-3634 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3634 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 We should make sure that the daemons crash in case the dispatchers get exceptions and stop processing. That way we will be debugging RM/NM/AM crashes instead of hard-to-track hanging jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3596) Sort benchmark got hang after completion of 99% map phase
[
https://issues.apache.org/jira/browse/MAPREDUCE-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181505#comment-13181505
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3596:
Looks like MAPREDUCE-3530 didn't work, seeing the same exception again.
Orthogonal to this, we should make sure RM crashes when the dispatcher gets an
exception. Created MAPREDUCE-3634 for this.
Sort benchmark got hang after completion of 99% map phase
-
Key: MAPREDUCE-3596
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3596
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.0
Reporter: Ravi Prakash
Assignee: Vinod Kumar Vavilapalli
Priority: Blocker
Attachments: logs.tar.bz2, logs.tar.bz2
Courtesy [~vinaythota]
{quote}
Ran sort benchmark couple of times and every time the job got hang after
completion 99% map phase. There are some map tasks failed. Also it's not
scheduled some of the pending map tasks.
Cluster size is 350 nodes.
Build Details:
==
Compiled: Fri Dec 9 16:25:27 PST 2011 by someone from
branches/branch-0.23/hadoop-common-project/hadoop-common
ResourceManager version:revision 1212681 by someone source checksum
on Fri Dec 9 16:52:07 PST 2011
Hadoop version: revision 1212592 by someone Fri Dec 9 16:25:27 PST
2011
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3628) DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181510#comment-13181510 ] Todd Lipcon commented on MAPREDUCE-3628: k, my hunch is that it's probably MR-related, then. I've seen a lot of cases where differences in scheduler performance give either better or worse looking DFSIO results on the same HDFS version. And my benchmarks of HDFS 0.23 show it significantly _faster_ than 0.20. DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. - Key: MAPREDUCE-3628 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3628 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0 Reporter: Amol Kekre Priority: Blocker Fix For: 0.23.1 DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3635) Improve Hadoop subcomponent integration in Hadoop 0.23
Improve Hadoop subcomponent integration in Hadoop 0.23 -- Key: MAPREDUCE-3635 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3635 Project: Hadoop Map/Reduce Issue Type: Improvement Components: build, client Affects Versions: 0.23.0 Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Fix For: 0.23.1 Please see HADOOP-7939 for a complete description and discussion. This JIRA is for patch tracking purposes only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-3360:
--
Attachment: MAPREDUCE-3360-4.patch
Hi Arun,
Thanks for looking into this issue.
findbugs' warning is not related this patch.
{noformat}org.apache.hadoop.mapreduce.v2.app.launcher.ContainerRemoteLaunchEvent
defines equals but not hashCode{noformat}
Attaching a patch which provies the information through web-services api as
well.
Provide information about lost nodes in the UI.
---
Key: MAPREDUCE-3360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Attachments: LostNodes.png, MAPREDUCE-3360-1.patch,
MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch,
MAPREDUCE-3360.patch, lostNodes.png
Currently there is no information provided about *lost nodes*. Provide
information in the UI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-3360: -- Status: Patch Available (was: Open) Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3628) DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181528#comment-13181528 ] Amol Kekre commented on MAPREDUCE-3628: --- Todd, thanks for the update. We are taking a look at our current data. DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. - Key: MAPREDUCE-3628 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3628 Project: Hadoop Map/Reduce Issue Type: Task Components: mrv2 Affects Versions: 0.23.0 Reporter: Amol Kekre Priority: Blocker Fix For: 0.23.1 DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3624:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Mahadev!
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181539#comment-13181539
]
Hudson commented on MAPREDUCE-3624:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1583 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1583/])
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar.
Contributed by Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228354
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181538#comment-13181538
]
Hudson commented on MAPREDUCE-3624:
---
Integrated in Hadoop-Common-0.23-Commit #351 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/351/])
Merge -c 1228354 from trunk to branch-0.23 to fix MAPREDUCE-3624. Remove
unnecessary dependency on JDK's tools.jar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228355
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181542#comment-13181542
]
Hudson commented on MAPREDUCE-3624:
---
Integrated in Hadoop-Common-trunk-Commit #1510 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1510/])
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar.
Contributed by Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228354
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181545#comment-13181545
]
Hudson commented on MAPREDUCE-3624:
---
Integrated in Hadoop-Hdfs-0.23-Commit #341 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/341/])
Merge -c 1228354 from trunk to branch-0.23 to fix MAPREDUCE-3624. Remove
unnecessary dependency on JDK's tools.jar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228355
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181555#comment-13181555
]
Hudson commented on MAPREDUCE-3624:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #362 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/362/])
Merge -c 1228354 from trunk to branch-0.23 to fix MAPREDUCE-3624. Remove
unnecessary dependency on JDK's tools.jar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228355
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181556#comment-13181556 ] Hadoop QA commented on MAPREDUCE-3360: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509705/MAPREDUCE-3360-4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1557//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1557//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1557//console This message is automatically generated. Provide information about lost nodes in the UI. --- Key: MAPREDUCE-3360 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Environment: NA Reporter: Bhallamudi Venkata Siva Kamesh Assignee: Bhallamudi Venkata Siva Kamesh Priority: Critical Attachments: LostNodes.png, MAPREDUCE-3360-1.patch, MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch, MAPREDUCE-3360.patch, lostNodes.png Currently there is no information provided about *lost nodes*. Provide information in the UI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3624) bin/yarn script adds jdk tools.jar to the classpath.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181559#comment-13181559
]
Hudson commented on MAPREDUCE-3624:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1530 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1530/])
MAPREDUCE-3624. Remove unnecessary dependency on JDK's tools.jar.
Contributed by Mahadev Konar.
acmurthy :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1228354
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/bin/yarn
bin/yarn script adds jdk tools.jar to the classpath.
Key: MAPREDUCE-3624
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3624
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.1
Attachments: MAPREDUCE-3624.patch
Thanks to Roman for pointing it out. Looks like we have the following lines
in bin/yarn:
{code}
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
{code}
We dont really have a dependency on the tools jar. We should remove this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash reassigned MAPREDUCE-3382: --- Assignee: Ravi Prakash Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Fix For: 0.23.1 MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3549) write api documentation for web service apis for RM, NM, mapreduce app master, and job history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-3549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181580#comment-13181580 ] Thomas Graves commented on MAPREDUCE-3549: -- also make sure docs include changes from MAPREDUCE-3299. write api documentation for web service apis for RM, NM, mapreduce app master, and job history server - Key: MAPREDUCE-3549 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3549 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical write api documentation for web service apis for RM, NM, mapreduce app master, and job history server. web services were added in MAPREDUCE-2863 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3636) Apply audience and stability annotations to classes in MapReduce for 1.x
Apply audience and stability annotations to classes in MapReduce for 1.x Key: MAPREDUCE-3636 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3636 Project: Hadoop Map/Reduce Issue Type: Improvement Components: documentation Reporter: Tom White Port MAPREDUCE-1623 to branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181603#comment-13181603 ] Thomas Graves commented on MAPREDUCE-3299: -- Downloaded and tested,a couple comments: - the webservice output for logs link is coming out funny (missing http:/): logsLink : /host.com:/node/containerlogs/container_1325883370639_0003_01_01, I think you want to add http:// in the AMAttemptInfo and remove it from the JobBlock. - there is new helper routine in AMWebServices to get the job and handle error checking - Job job = getJobFromJobIdString(jid, appCtx); - JAXBContextResolver.java - wrap line to 80 characters long - checkout the newest revision of AMAttemptInfo, it has a couple extra null checks. I'll add the api docs for this under MAPREDUCE-3549. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work stopped] (MAPREDUCE-2765) DistCp Rewrite
[ https://issues.apache.org/jira/browse/MAPREDUCE-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-2765 stopped by Mithun Radhakrishnan. DistCp Rewrite -- Key: MAPREDUCE-2765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2765 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distcp, mrv2 Affects Versions: 0.20.203.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: distcpv2.20.203.patch, distcpv2_hadoop-0.23.1.patch, distcpv2_hadoop-0.23.1.patch, distcpv2_hadoop-trunk.patch, distcpv2_patch_0.23.1-SNAPSHOT_tucu_reviewed.patch, distcpv2_patch_hadoop-trunk_tucu_reviewed.patch, distcpv2_trunk.patch, distcpv2_trunk_post_review_1.patch This is a slightly modified version of the DistCp rewrite that Yahoo uses in production today. The rewrite was ground-up, with specific focus on: 1. improved startup time (postponing as much work as possible to the MR job) 2. support for multiple copy-strategies 3. new features (e.g. -atomic, -async, -bandwidth.) 4. improved programmatic use Some effort has gone into refactoring what used to be achieved by a single large (1.7 KLOC) source file, into a design that (hopefully) reads better too. The proposed DistCpV2 preserves command-line-compatibility with the old version, and should be a drop-in replacement. New to v2: 1. Copy-strategies and the DynamicInputFormat: A copy-strategy determines the policy by which source-file-paths are distributed between map-tasks. (These boil down to the choice of the input-format.) If no strategy is explicitly specified on the command-line, the policy chosen is uniform size, where v2 behaves identically to old-DistCp. (The number of bytes transferred by each map-task is roughly equal, at a per-file granularity.) Alternatively, v2 ships with a dynamic copy-strategy (in the DynamicInputFormat). This policy acknowledges that (a) dividing files based only on file-size might not be an even distribution (E.g. if some datanodes are slower than others, or if some files are skipped.) (b) a static association of a source-path to a map increases the likelihood of long-tails during copy. The dynamic strategy divides the list-of-source-paths into a number ( nMaps) of smaller parts. When each map completes its current list of paths, it picks up a new list to process, if available. So if a map-task is stuck on a slow (and not necessarily large) file, other maps can pick up the slack. The thinner the file-list is sliced, the greater the parallelism (and the lower the chances of long-tails). Within reason, of course: the number of these short-lived list-files is capped at an overridable maximum. Internal benchmarks against source/target clusters with some slow(ish) datanodes have indicated significant performance gains when using the dynamic-strategy. Gains are most pronounced when nFiles greatly exceeds nMaps. Please note that the DynamicInputFormat might prove useful outside of DistCp. It is hence available as a mapred/lib, unfettered to DistCpV2. Also note that the copy-strategies have no bearing on the CopyMapper.map() implementation. 2. Improved startup-time and programmatic use: When the old-DistCp runs with -update, and creates the list-of-source-paths, it attempts to filter out files that might be skipped (by comparing file-sizes, checksums, etc.) This significantly increases the startup time (or the time spent in serial processing till the MR job is launched), blocking the calling-thread. This becomes pronounced as nFiles increases. (Internal benchmarks have seen situations where more time is spent setting up the job than on the actual transfer.) DistCpV2 postpones as much work as possible to the MR job. The file-listing isn't filtered until the map-task runs (at which time, identical files are skipped). DistCpV2 can now be run asynchronously. The program quits at job-launch, logging the job-id for tracking. Programmatically, the DistCp.execute() returns a Job instance for progress-tracking. 3. New features: (a) -async: As described in #2. (b) -atomic: Data is copied to a (user-specifiable) tmp-location, and then moved atomically to destination. (c) -bandwidth: Enforces a limit on the bandwidth consumed per map. (d) -strategy: As above. A more comprehensive description the newer features, how the dynamic-strategy works, etc. is available in src/site/xdoc/, and in the pdf that's generated therefrom, during the build. High on the list of things to do is support to parallelize copies on a per-block level. (i.e. Incorporation of HDFS-222.) I look forward to comments, suggestions and discussion that
[jira] [Work started] (MAPREDUCE-2765) DistCp Rewrite
[ https://issues.apache.org/jira/browse/MAPREDUCE-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-2765 started by Mithun Radhakrishnan. DistCp Rewrite -- Key: MAPREDUCE-2765 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2765 Project: Hadoop Map/Reduce Issue Type: New Feature Components: distcp, mrv2 Affects Versions: 0.20.203.0 Reporter: Mithun Radhakrishnan Assignee: Mithun Radhakrishnan Attachments: distcpv2.20.203.patch, distcpv2_hadoop-0.23.1.patch, distcpv2_hadoop-0.23.1.patch, distcpv2_hadoop-trunk.patch, distcpv2_patch_0.23.1-SNAPSHOT_tucu_reviewed.patch, distcpv2_patch_hadoop-trunk_tucu_reviewed.patch, distcpv2_trunk.patch, distcpv2_trunk_post_review_1.patch This is a slightly modified version of the DistCp rewrite that Yahoo uses in production today. The rewrite was ground-up, with specific focus on: 1. improved startup time (postponing as much work as possible to the MR job) 2. support for multiple copy-strategies 3. new features (e.g. -atomic, -async, -bandwidth.) 4. improved programmatic use Some effort has gone into refactoring what used to be achieved by a single large (1.7 KLOC) source file, into a design that (hopefully) reads better too. The proposed DistCpV2 preserves command-line-compatibility with the old version, and should be a drop-in replacement. New to v2: 1. Copy-strategies and the DynamicInputFormat: A copy-strategy determines the policy by which source-file-paths are distributed between map-tasks. (These boil down to the choice of the input-format.) If no strategy is explicitly specified on the command-line, the policy chosen is uniform size, where v2 behaves identically to old-DistCp. (The number of bytes transferred by each map-task is roughly equal, at a per-file granularity.) Alternatively, v2 ships with a dynamic copy-strategy (in the DynamicInputFormat). This policy acknowledges that (a) dividing files based only on file-size might not be an even distribution (E.g. if some datanodes are slower than others, or if some files are skipped.) (b) a static association of a source-path to a map increases the likelihood of long-tails during copy. The dynamic strategy divides the list-of-source-paths into a number ( nMaps) of smaller parts. When each map completes its current list of paths, it picks up a new list to process, if available. So if a map-task is stuck on a slow (and not necessarily large) file, other maps can pick up the slack. The thinner the file-list is sliced, the greater the parallelism (and the lower the chances of long-tails). Within reason, of course: the number of these short-lived list-files is capped at an overridable maximum. Internal benchmarks against source/target clusters with some slow(ish) datanodes have indicated significant performance gains when using the dynamic-strategy. Gains are most pronounced when nFiles greatly exceeds nMaps. Please note that the DynamicInputFormat might prove useful outside of DistCp. It is hence available as a mapred/lib, unfettered to DistCpV2. Also note that the copy-strategies have no bearing on the CopyMapper.map() implementation. 2. Improved startup-time and programmatic use: When the old-DistCp runs with -update, and creates the list-of-source-paths, it attempts to filter out files that might be skipped (by comparing file-sizes, checksums, etc.) This significantly increases the startup time (or the time spent in serial processing till the MR job is launched), blocking the calling-thread. This becomes pronounced as nFiles increases. (Internal benchmarks have seen situations where more time is spent setting up the job than on the actual transfer.) DistCpV2 postpones as much work as possible to the MR job. The file-listing isn't filtered until the map-task runs (at which time, identical files are skipped). DistCpV2 can now be run asynchronously. The program quits at job-launch, logging the job-id for tracking. Programmatically, the DistCp.execute() returns a Job instance for progress-tracking. 3. New features: (a) -async: As described in #2. (b) -atomic: Data is copied to a (user-specifiable) tmp-location, and then moved atomically to destination. (c) -bandwidth: Enforces a limit on the bandwidth consumed per map. (d) -strategy: As above. A more comprehensive description the newer features, how the dynamic-strategy works, etc. is available in src/site/xdoc/, and in the pdf that's generated therefrom, during the build. High on the list of things to do is support to parallelize copies on a per-block level. (i.e. Incorporation of HDFS-222.) I look forward to comments, suggestions and discussion that
[jira] [Created] (MAPREDUCE-3637) Health checker interval does not appear to work - according to web ui
Health checker interval does not appear to work - according to web ui - Key: MAPREDUCE-3637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3637 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical reported by QE, I think this is probably a web ui not updating issue, but not positive. 1. The property yarn.nodemanager.health-checker.interval-ms on the RM and the DN are set to 135000 ms, or 2 minutes and 15 sec. $ grep -A 2 health-checker.interval-ms yarn-site.xml nameyarn.nodemanager.health-checker.interval-ms/name value135000/value /property 2. According to the web ui (e.g http://rm:8088/cluster/nodes), the 'last health-update' column does not get updated, except when the RM is restated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3638) Yarn trying to download cacheFile to container but Path is a local file
Yarn trying to download cacheFile to container but Path is a local file --- Key: MAPREDUCE-3638 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3638 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical It looks like the AM, which is running on host1.com, is trying to access a local file but the file is on host2.com (where the command was run). ran: hadoop --config conf/hadoop/ jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* -input Streaming/streaming-610/input.txt -mapper 'xargs cat' -reducer cat -output Streaming/streaming-610/Output -cacheFile file://Streaming/data/streaming-610//InputFile#testlink -jobconf mapred.map.tasks=1 -jobconf mapred.reduce.tasks=1 -jobconf mapred.job.name=streamingTest-610 -jobconf mapreduce.job.acl-view-job=* failure: 11/11/10 07:48:06 INFO mapreduce.Job: Job job_1320887371559_0215 failed with state FAILED due to: Application application_1320887371559_0215 failed 1 times due to AM Container for appattempt_1320887371559_0215_01 exited with exitCode: -1000 due to: java.io.FileNotFoundException: File file:/Streaming/data/streaming-610/InputFile does not exist at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:431) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:315) at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:85) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:152) at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:50) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Status: Open (was: Patch Available) Canceling while addressing comments from @Thomas Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3382: Attachment: MAPREDUCE-3382.branch-0.23.patch Attaching a patch to enable the configuration of an HTTP Proxy. Can someone please review and commit? Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3382: Status: Patch Available (was: Open) Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3637) Health checker interval does not appear to work - according to web ui
[ https://issues.apache.org/jira/browse/MAPREDUCE-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved MAPREDUCE-3637. -- Resolution: Cannot Reproduce seems to be working. Health checker interval does not appear to work - according to web ui - Key: MAPREDUCE-3637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3637 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Priority: Critical reported by QE, I think this is probably a web ui not updating issue, but not positive. 1. The property yarn.nodemanager.health-checker.interval-ms on the RM and the DN are set to 135000 ms, or 2 minutes and 15 sec. $ grep -A 2 health-checker.interval-ms yarn-site.xml nameyarn.nodemanager.health-checker.interval-ms/name value135000/value /property 2. According to the web ui (e.g http://rm:8088/cluster/nodes), the 'last health-update' column does not get updated, except when the RM is restated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181662#comment-13181662 ] Hadoop QA commented on MAPREDUCE-3382: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509721/MAPREDUCE-3382.branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1558//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1558//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1558//console This message is automatically generated. Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3380) Token infrastructure for running clients which are not kerberos authenticated
[ https://issues.apache.org/jira/browse/MAPREDUCE-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3380: - Attachment: MAPREDUCE-3380.patch Finally was able to get a secure oozie running and test this patch out. The patch works. The only issue is that oozie still reports the job as running although the launcher and action job complete. Will have to debug that with some oozie folks but as for the token this should be it. Token infrastructure for running clients which are not kerberos authenticated - Key: MAPREDUCE-3380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3380 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380_v1.patch The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job. What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3380) Token infrastructure for running clients which are not kerberos authenticated
[ https://issues.apache.org/jira/browse/MAPREDUCE-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3380: - Status: Patch Available (was: Open) Token infrastructure for running clients which are not kerberos authenticated - Key: MAPREDUCE-3380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3380 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380_v1.patch The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job. What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Attachment: MAPREDUCE-3299.patch Latest patch addresses above comments Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3299: --- Status: Patch Available (was: Open) Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3380) Token infrastructure for running clients which are not kerberos authenticated
[ https://issues.apache.org/jira/browse/MAPREDUCE-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181674#comment-13181674 ] Hadoop QA commented on MAPREDUCE-3380: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509733/MAPREDUCE-3380.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed the unit tests build +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1559//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1559//console This message is automatically generated. Token infrastructure for running clients which are not kerberos authenticated - Key: MAPREDUCE-3380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3380 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380_v1.patch The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job. What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3380) Token infrastructure for running clients which are not kerberos authenticated
[ https://issues.apache.org/jira/browse/MAPREDUCE-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3380: - Status: Open (was: Patch Available) Ill address the test/javac/findbugs failures. Token infrastructure for running clients which are not kerberos authenticated - Key: MAPREDUCE-3380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3380 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380_v1.patch The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job. What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181707#comment-13181707 ] Hadoop QA commented on MAPREDUCE-3299: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509735/MAPREDUCE-3299.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1560//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1560//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1560//console This message is automatically generated. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181715#comment-13181715 ] Ravi Prakash commented on MAPREDUCE-3382: - The findbugs warning (org.apache.hadoop.mapreduce.v2.app.launcher.ContainerRemoteLaunchEvent defines equals but not hashCode) is unrelated to this patch. The same patch also applies to trunk Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch, MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3382: Attachment: MAPREDUCE-3382.branch-0.23.patch Adding support for configuring SOCKS proxy. By default it assumes HTTP proxy. Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch, MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3634) All daemons should crash instead of hanging around when their EventHandlers get exceptions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amol Kekre updated MAPREDUCE-3634: -- Component/s: mrv2 All daemons should crash instead of hanging around when their EventHandlers get exceptions -- Key: MAPREDUCE-3634 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3634 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 We should make sure that the daemons crash in case the dispatchers get exceptions and stop processing. That way we will be debugging RM/NM/AM crashes instead of hard-to-track hanging jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated MAPREDUCE-3382: Priority: Critical (was: Major) Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch, MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3360) Provide information about lost nodes in the UI.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181729#comment-13181729
]
Jason Lowe commented on MAPREDUCE-3360:
---
Thanks for adding the web services support. A few issues I noticed:
1) Copy-n-paste error on line 234 of TestRMWebServicesNodes.java in
testNodesQueryStateLost():
{code}
WebServicesTestUtils.checkStringMatch(nodeHTTPAddress, rmNode
.getState().toString(), info.getString(state));
{code}
{{nodeHTTPAddress}} should be {{state}}. This also occurs on line 264.
2) In testNodesQueryStateLost() the test is assuming that the order of nodes
returned is deterministic, but I'm not sure that's guaranteed by the API. In
addition, it simply checks if the node ID is what it's looking for, and if it
isn't, it silently allows the test to pass. Since both nodes should be lost at
this point in the test, we could just loop over the returned nodes and verify
we see the fields we expect for each node entry.
3) In testSingleNodeQueryStateLost() the code similarly checks for the expected
node ID. If it doesn't match it should fail.
Provide information about lost nodes in the UI.
---
Key: MAPREDUCE-3360
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3360
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Environment: NA
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Priority: Critical
Attachments: LostNodes.png, MAPREDUCE-3360-1.patch,
MAPREDUCE-3360-2.patch, MAPREDUCE-3360-3.patch, MAPREDUCE-3360-4.patch,
MAPREDUCE-3360.patch, lostNodes.png
Currently there is no information provided about *lost nodes*. Provide
information in the UI.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3582) Move successfully passing MR1 tests to MR2 maven tree.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181732#comment-13181732 ] Alejandro Abdelnur commented on MAPREDUCE-3582: --- From layout build perspective: GetGroups, DistCp, DistCh, Logalyzer are being moved to hadoop-mapreduce-client-jobclient, shouldn't they go into hadoop-tools? Also there is a new version of Distcp about to be committed, should we ignore the old one and let it die. Also, rumen stuff (which is needed for gridmix) it OK to go into main/src/java or that is a tool as well? The MemoryCalculatorPlugin classes is being moved to src/main/java, is really required to be there? As it seems it is only used by testcases, no? The examples POM has version in a few dependencies, they should be removed and the corresponding dependency with version should be added to the hadoop-project/pom.xml in the dependency management section. Other than that looks good. Thxs Move successfully passing MR1 tests to MR2 maven tree. -- Key: MAPREDUCE-3582 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3582 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Reporter: Ahmed Radwan Assignee: Ahmed Radwan Attachments: MAPREDUCE-3582-part1_rev3.patch, MAPREDUCE-3582-part1_rev4.patch, MAPREDUCE-3582-part2_rev3.patch, MAPREDUCE-3582-part2_rev4.patch, MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev3.patch, MAPREDUCE-3582-whole_rev4.patch, MAPREDUCE-3582.patch, MAPREDUCE-3582_rev2.patch, mv_script_MR-3582.sh, mv_script_MR-3582_rev2.sh, mv_script_MR-3582_rev3.sh, mv_script_MR-3582_rev4.sh This ticket will track moving mr1 tests that are passing successfully to mr2 maven tree. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3382) Network ACLs can prevent AMs to ping the Job-end notification URL
[ https://issues.apache.org/jira/browse/MAPREDUCE-3382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181740#comment-13181740 ] Hadoop QA commented on MAPREDUCE-3382: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509748/MAPREDUCE-3382.branch-0.23.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1561//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1561//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1561//console This message is automatically generated. Network ACLs can prevent AMs to ping the Job-end notification URL - Key: MAPREDUCE-3382 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3382 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Ravi Prakash Priority: Critical Fix For: 0.23.1 Attachments: MAPREDUCE-3382.branch-0.23.patch, MAPREDUCE-3382.branch-0.23.patch MAPREDUCE-3028 added support for job-end notification from MR AMs after the job finishes. Network ACLs can have an implication on this one - outgoing connections from the compute nodes may be restricted in some settings and so job-end notification( that can originate from the AMs which may run on random nodes in the cluster) may have issues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3299) Add AMInfo table to the AM job page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181788#comment-13181788 ] Jonathan Eagles commented on MAPREDUCE-3299: Findbugs warning is unrelated to this patch. Add AMInfo table to the AM job page --- Key: MAPREDUCE-3299 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3299 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Jonathan Eagles Priority: Minor Attachments: MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch, MAPREDUCE-3299.patch JobHistory has a table to list all AMs. A similar table can be added to the AM for info on past failed AMs and the current running one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3639) TokenCache likely broken for FileSystems which don't issue delegation tokens
TokenCache likely broken for FileSystems which don't issue delegation tokens Key: MAPREDUCE-3639 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3639 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Priority: Critical Ref HADOOP-7963. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3476) Optimize YARN API calls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3476: - Priority: Blocker (was: Critical) Optimize YARN API calls --- Key: MAPREDUCE-3476 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3476 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Ravi Prakash Assignee: Vinod Kumar Vavilapalli Priority: Blocker Several YARN API calls are taking inordinately long. This might be a performance blocker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3380) Token infrastructure for running clients which are not kerberos authenticated
[ https://issues.apache.org/jira/browse/MAPREDUCE-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3380: - Status: Patch Available (was: Open) Token infrastructure for running clients which are not kerberos authenticated - Key: MAPREDUCE-3380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3380 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380_v1.patch The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job. What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3380) Token infrastructure for running clients which are not kerberos authenticated
[ https://issues.apache.org/jira/browse/MAPREDUCE-3380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13181879#comment-13181879 ] Hadoop QA commented on MAPREDUCE-3380: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12509768/MAPREDUCE-3380.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1562//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1562//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1562//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1562//console This message is automatically generated. Token infrastructure for running clients which are not kerberos authenticated - Key: MAPREDUCE-3380 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3380 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Alejandro Abdelnur Assignee: Mahadev konar Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.branch-0.23.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380.patch, MAPREDUCE-3380_v1.patch The JobClient.getDelegationToken() method is returning NULL, this makes Oozie fail when trying to get the delegation token to use it for starting a job. What is seems to be happing is that Jobclient.getDelegationToken() calls Cluster.getDelegationToken() that calls YarnRunner.getDelegationToken() that calls ResourceMgrDelegate.getDelegationToken(). And the last one is not implemented. (Thanks Ahmed for tracing this in MR2 code) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
