[jira] [Commented] (MAPREDUCE-3747) Memory Total is not refreshed until an app is launched
[ https://issues.apache.org/jira/browse/MAPREDUCE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201244#comment-13201244 ] Hudson commented on MAPREDUCE-3747: --- Integrated in Hadoop-Hdfs-trunk #948 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/948/]) MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1240886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java Memory Total is not refreshed until an app is launched -- Key: MAPREDUCE-3747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3747 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch Memory Total on the RM UI is not refreshed until an application is launched. This is a problem when the cluster is started for the first time or when there are any lost/decommissioned NMs. When the cluster is started for the first time, Active Nodes is 0 but the Memory Total=0. Also when there are any lost/decommissioned nodes, Memory Total has wrong value. This is a useful tool for cluster admins and has to be updated correctly without having the need to submit an app each time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3746) Nodemanagers are not automatically shut down after decommissioning
[ https://issues.apache.org/jira/browse/MAPREDUCE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201247#comment-13201247 ] Hudson commented on MAPREDUCE-3746: --- Integrated in Hadoop-Hdfs-0.23-Build #161 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/161/]) Merge -c 1240886 from trunk to branch-0.23 to fix MAPREDUCE-3746. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1240887 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java Nodemanagers are not automatically shut down after decommissioning -- Key: MAPREDUCE-3746 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3746 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Critical Fix For: 0.23.1 Nodemanagers are not automatically shutdown after decommissioning. MAPREDUCE-2775 does not seem to fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3746) Nodemanagers are not automatically shut down after decommissioning
[ https://issues.apache.org/jira/browse/MAPREDUCE-3746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201266#comment-13201266 ] Hudson commented on MAPREDUCE-3746: --- Integrated in Hadoop-Mapreduce-0.23-Build #183 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/183/]) Merge -c 1240886 from trunk to branch-0.23 to fix MAPREDUCE-3746. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1240887 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java Nodemanagers are not automatically shut down after decommissioning -- Key: MAPREDUCE-3746 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3746 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Devaraj K Priority: Critical Fix For: 0.23.1 Nodemanagers are not automatically shutdown after decommissioning. MAPREDUCE-2775 does not seem to fix the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3747) Memory Total is not refreshed until an app is launched
[ https://issues.apache.org/jira/browse/MAPREDUCE-3747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201295#comment-13201295 ] Hudson commented on MAPREDUCE-3747: --- Integrated in Hadoop-Mapreduce-trunk #981 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/981/]) MAPREDUCE-3747. Initialize queue metrics upfront and added start/finish time to RM Web-UI. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1240886 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/AppsBlock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RmView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/ClusterMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/UserMetricsInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestNodesPage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java Memory Total is not refreshed until an app is launched -- Key: MAPREDUCE-3747 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3747 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch, MAPREDUCE-3747.patch Memory Total on the RM UI is not refreshed until an application is launched. This is a problem when the cluster is started for the first time or when there are any lost/decommissioned NMs. When the cluster is started for the first time, Active Nodes is 0 but the Memory Total=0. Also when there are any lost/decommissioned nodes, Memory Total has wrong value. This is a useful tool for cluster admins and has to be updated correctly without having the need to submit an app each time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201329#comment-13201329
]
Robert Joseph Evans commented on MAPREDUCE-3802:
That appears to be the case. We are getting an NPE which is caused by calling
RecoverService.getTaskAttemptInfo() and getting a null back.
RecoveryService.getTaskAttemptInfo() first gets a task info, and then gets a
task attempt info from inside that task. It looks like the task info is parsed
and populated just fine, but the task attempt info is not. That seems to be
caused by no TaskAttemptStarted events being put in the history log at all
during the recovery process. This also seems like no MapAttemptFinishedEvents,
ReduceAttemp0tFinishedEvents, TaskAttemptFailedEvents nor
TaskAttemptFinishedEvents are in the log either, or we would get null pointer
exceptions while parsing them too.
If an MR AM dies twice it looks like the process freezes
-
Key: MAPREDUCE-3802
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Robert Joseph Evans
Priority: Critical
Attachments: syslog
It looks like recovering from an RM AM dieing works very well on a single
failure. But if it fails multiple times we appear to get into a live lock
situation.
{noformat}
yarn jar
hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
input output
12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
17
12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
application_1328302034486_0003 to ResourceManager at HOST/IP:8040
12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
http://HOST:8088/proxy/application_1328302034486_0003/
12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
uber mode : false
12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
#KILLED AM with kill -9 here
12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
#killed AM with kill -9 here
12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 0 time(s).
12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 1 time(s).
12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 2 time(s).
12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
#It never makes any more progress...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3816) capacity scheduler web ui bar graphs for used capacity wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3816: - Attachment: capacityschedulerusedcap.png In this screen shots, this is single node cluster 8 GB total memory. The default queue capacity is 20% which would be 1638MB allocated to it. You will see that its currently using 2G so the usage is 125% - which is the utilization. capacity scheduler web ui bar graphs for used capacity wrong Key: MAPREDUCE-3816 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3816 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Thomas Graves Priority: Critical Attachments: capacityschedulerusedcap.png The capacity scheduler web ui has bar graphs showing the capacity/used capacity/max capacity for each queue. The used capacity it is showing is actually the % of its parents queue it is using, which doesn't make sense on the bar graphs when compared to the capacity and max capacity of that particular queue. The bar graphs should be using utilization so that the user can see that its using x% or the y% allocated to that queue. I will attach some screen shots showing the issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Attachment: MAPREDUCE-3034-3.patch NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Open (was: Patch Available) NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3034: - Target Version/s: 0.23.0, 0.24.0 (was: 0.24.0, 0.23.0) Status: Patch Available (was: Open) Thanks a lot Robert for taking look at the patch. I have updated the patch with the above comments fixed. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201360#comment-13201360 ] Eric Payne commented on MAPREDUCE-3034: --- +1 (non-binding) I approve the changes introduced in MAPREDUCE-3034-3.patch NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Attachment: mr-3773-trunk.patch This patch applies to 0.23 and trunk. It renames the property to yarn.resourcemanager.metrics.runtime.buckets I had to make the CapacityScheduler Configurable so that it could get the YarnConfiguration. I also needed to add the YarnConfiguration to QueueMetrics.forQueue so that the QueueMetrics object would have access to the YarnConfiguration. Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Target Version/s: 0.23.1, 1.1.0 Status: Patch Available (was: Open) Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201383#comment-13201383 ] Hadoop QA commented on MAPREDUCE-3773: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513435/mr-3773-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.TestResourceTrackerService org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebApp org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodes org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher org.apache.hadoop.yarn.server.resourcemanager.webapp.TestNodesPage org.apache.hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits org.apache.hadoop.yarn.server.resourcemanager.TestRM org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing org.apache.hadoop.yarn.server.resourcemanager.TestResourceManager org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1788//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1788//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1788//console This message is automatically generated. Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201386#comment-13201386 ] Hadoop QA commented on MAPREDUCE-3034: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513433/MAPREDUCE-3034-3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.mapred.TestJobCounters +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1787//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1787//console This message is automatically generated. NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3802) If an MR AM dies twice it looks like the process freezes
[
https://issues.apache.org/jira/browse/MAPREDUCE-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201414#comment-13201414
]
Robert Joseph Evans commented on MAPREDUCE-3802:
I have been looking for differences in the jhist files between the initial
kill, and the second one, and I cannot find much as far as the content is
concerned. So I think it might have something to do with the order of the
events.
In the second jhist file there are two AMStarted events. I assume that is to
allow for the history server to show how many have been started.
The mapFinishTime of all the new MAP_ATTEMPT_FINISH_EVENTS is 0. The finishTime
for these events are all showing the same finish time which I think is when the
attempt was recovered, not the original event finish time. The state of the
event changed from map (which seems like a bug) to SUCCEEDED which looks
more correct to me. And finally the clockSplits counters are all 0 in the new
one as well.
Similarly the TASK_FINISH_EVENTS have the finish time of the recovery, not the
actual finish time.
I could not find anything else that is significantly different.
If an MR AM dies twice it looks like the process freezes
-
Key: MAPREDUCE-3802
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3802
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: applicationmaster, mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Robert Joseph Evans
Priority: Critical
Attachments: syslog
It looks like recovering from an RM AM dieing works very well on a single
failure. But if it fails multiple times we appear to get into a live lock
situation.
{noformat}
yarn jar
hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*-SNAPSHOT.jar
wordcount -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.job.reduces=30
input output
12/02/03 21:06:57 WARN conf.Configuration: fs.default.name is deprecated.
Instead, use fs.defaultFS
12/02/03 21:06:57 WARN conf.Configuration: mapred.used.genericoptionsparser
is deprecated. Instead, use mapreduce.client.genericoptionsparser.used
12/02/03 21:06:57 INFO input.FileInputFormat: Total input paths to process :
17
12/02/03 21:06:57 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/02/03 21:06:57 WARN snappy.LoadSnappy: Snappy native library not loaded
12/02/03 21:06:57 INFO mapreduce.JobSubmitter: number of splits:17
12/02/03 21:06:57 INFO mapred.ResourceMgrDelegate: Submitted application
application_1328302034486_0003 to ResourceManager at HOST/IP:8040
12/02/03 21:06:57 INFO mapreduce.Job: The url to track the job:
http://HOST:8088/proxy/application_1328302034486_0003/
12/02/03 21:06:57 INFO mapreduce.Job: Running job: job_1328302034486_0003
12/02/03 21:07:03 INFO mapreduce.Job: Job job_1328302034486_0003 running in
uber mode : false
12/02/03 21:07:03 INFO mapreduce.Job: map 0% reduce 0%
12/02/03 21:07:09 INFO mapreduce.Job: map 5% reduce 0%
12/02/03 21:07:10 INFO mapreduce.Job: map 17% reduce 0%
#KILLED AM with kill -9 here
12/02/03 21:07:16 INFO mapreduce.Job: map 29% reduce 0%
12/02/03 21:07:17 INFO mapreduce.Job: map 35% reduce 0%
12/02/03 21:07:30 INFO mapreduce.Job: map 52% reduce 0%
12/02/03 21:07:35 INFO mapreduce.Job: map 58% reduce 0%
12/02/03 21:07:37 INFO mapreduce.Job: map 70% reduce 0%
12/02/03 21:07:41 INFO mapreduce.Job: map 76% reduce 0%
12/02/03 21:07:43 INFO mapreduce.Job: map 82% reduce 0%
12/02/03 21:07:44 INFO mapreduce.Job: map 88% reduce 0%
12/02/03 21:07:47 INFO mapreduce.Job: map 94% reduce 0%
12/02/03 21:07:49 INFO mapreduce.Job: map 100% reduce 0%
12/02/03 21:07:53 INFO mapreduce.Job: map 100% reduce 3%
12/02/03 21:08:00 INFO mapreduce.Job: map 100% reduce 6%
12/02/03 21:08:06 INFO mapreduce.Job: map 100% reduce 10%
12/02/03 21:08:12 INFO mapreduce.Job: map 100% reduce 13%
12/02/03 21:08:18 INFO mapreduce.Job: map 100% reduce 16%
#killed AM with kill -9 here
12/02/03 21:08:20 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 0 time(s).
12/02/03 21:08:21 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 1 time(s).
12/02/03 21:08:22 INFO ipc.Client: Retrying connect to server: HOST/IP:44223.
Already tried 2 time(s).
12/02/03 21:08:26 INFO mapreduce.Job: map 64% reduce 16%
#It never makes any more progress...
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Gupta updated MAPREDUCE-3817: --- Attachment: MAPREDUCE-3817.patch added same options available in bin/hadoop also to bin/mapred for distcp and archives bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli reassigned MAPREDUCE-3808:
--
Assignee: Robert Joseph Evans
Rober, can you please take a crack at this one?
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3817: - Status: Patch Available (was: Open) bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Attachment: MR-3709.1.patch @Eli, given that MAPREDUCE-3765 is now in, the latest patch should be able to use smaller sized containers which hopefully should solve the problem. Could you please try the patch and let me know if it fixes the issue in your env. I am also downgrading this to a non-blocker as it is not reproducible in most environments. Please let me know if you see any issues with doing this. TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Priority: Blocker Attachments: MR-3709.1.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Status: Patch Available (was: Open) TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Attachments: MR-3709.1.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Priority: Major (was: Blocker) TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Attachments: MR-3709.1.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201471#comment-13201471
]
Robert Joseph Evans commented on MAPREDUCE-3808:
Sure I can look into it. It appears that for some reason
mapreduce.output.fileoutputformat.outputdir is not set in the config.
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201474#comment-13201474
]
Robert Joseph Evans commented on MAPREDUCE-3808:
OK I am a bit confused here. Why is it allowed to run a map only job that does
not have an output directory set? The FileOutputFormat blows up if there are
any reduces and the output directory is not set, but for a map only job it is
fine. What is supposed to happen in that situation? I can replicate it, but I
don't know what the previous behavior was.
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201476#comment-13201476
]
Robert Joseph Evans commented on MAPREDUCE-3808:
Well looking at the rest of the code it looks like the FileOutputCommitter
becomes a NOOP when the output dir is not set, so I need to replicate that here
too.
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201480#comment-13201480 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3813: Allen, this is not for hostname resolution. This is the rack resolver plugin which finds out the racks for a given set of hosts. Not every rack resolver plugin is fast enough, so we need a cache to keep track of hosts whose racks are already resolved. RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Attachment: MAPREDUCE-3812-20120206.txt Correct patch applicable to trunk. Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Status: Open (was: Patch Available) Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Status: Patch Available (was: Open) Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS returns IPs in getFileBlockLocations
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201497#comment-13201497 ] Siddharth Seth commented on MAPREDUCE-3815: --- Both - will create a separate hdfs jira. This one is for a check in MR for such situations. Data Locality suffers if HDFS returns IPs in getFileBlockLocations -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3815) Data Locality suffers if HDFS returns IPs in getFileBlockLocations
[ https://issues.apache.org/jira/browse/MAPREDUCE-3815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201498#comment-13201498 ] Siddharth Seth commented on MAPREDUCE-3815: --- Both - will create a separate hdfs jira. This one is for a check in MR for such situations. Data Locality suffers if HDFS returns IPs in getFileBlockLocations -- Key: MAPREDUCE-3815 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3815 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical BlockLocation.getHosts() returns IP addresses occasionally. Data locality is affected - since the RM requires hostnames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hitesh Shah updated MAPREDUCE-3709: --- Status: Open (was: Patch Available) TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Attachments: MR-3709.1.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.24.0
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201506#comment-13201506
]
Aaron T. Myers commented on MAPREDUCE-3818:
---
Seems very likely this is due to the recent commit of HDFS-2895.
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.24.0
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Attachment: mr-3773-trunk.patch Fixed findbugs warnings and add missing Configurable interface from CapacityScheduler. Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Target Version/s: 0.23.1, 1.1.0 (was: 1.1.0, 0.23.1) Status: Patch Available (was: Open) resubmitting Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-3773: - Target Version/s: 0.23.1, 1.1.0 (was: 1.1.0, 0.23.1) Status: Open (was: Patch Available) Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3697: - Attachment: apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh I reviewed the patch for compatibility with Hadoop 1.x. I ran a SigTest compatibility test (like in HADOOP-7738) to see what the differences between the mapred counters are between 1.0.0 and trunk with this patch (see attached). Most are false positives due to the inheritance hierarchy changing. Here are the ones that I think need fixing: * Counters.getGroupNames returns an IterableString not a CollectionString as it does in 1.x. I think we should change it back to return CollectionString - we can always add a getGroupNamesIterable() later if needed. * method public java.lang.String org.apache.hadoop.mapred.Counters.makeCompactString() is missing * public static int org.apache.hadoop.mapred.Counters.MAX_COUNTER_LIMIT is missing (it's not used in 23, but we should still define it for compatibility) Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201517#comment-13201517 ] Hadoop QA commented on MAPREDUCE-3697: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513480/apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1793//console This message is automatically generated. Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3697) Hadoop Counters API limits Oozie's working across different hadoop versions
[ https://issues.apache.org/jira/browse/MAPREDUCE-3697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201516#comment-13201516 ] Mahadev konar commented on MAPREDUCE-3697: -- Thanks for doing this Tom. Ill take care of those! Hadoop Counters API limits Oozie's working across different hadoop versions --- Key: MAPREDUCE-3697 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3697 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: John George Assignee: Mahadev konar Priority: Blocker Attachments: MAPREDUCE-3697-facade.patch, MAPREDUCE-3697-facadeV2.patch, MAPREDUCE-3697-facadeV3.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, MAPREDUCE-3697.patch, apicheck-hadoop-1.0.0-0.24.0-SNAPSHOT.sh Oozie uses Hadoop Counters API, by invoking Counters.getGroup(). However, in hadoop 23, org.apache.hadoop.mapred.Counters does not implement getGroup(). Its parent class AbstractCounters implements it. This is different from hadoop20X. As a result, Oozie compiled with either hadoop version does not work with the other version. A specific scenario, Oozie compiled with .23 and run against 205, does not update job status owing to a Counters API exception. Will explicit re-compilation against the relevant hadoop jars be required each time? This will prevent launching a uniform Oozie version across different clusters. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3808:
---
Attachment: MR-3808.txt
This patch should fix the issue. It adds in a check for null on the output in
several places when building paths and short curcits those methods to return
null. It also adds in the a null output check in recoverTask and causes it to
be a noop when output is null. Finally it adds in tests to validate that the
FileOutputCommitter handles that case correctly.
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3808:
---
Target Version/s: 0.23.1, 0.24.0
Affects Version/s: 0.24.0
Status: Patch Available (was: Open)
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201522#comment-13201522 ] Hadoop QA commented on MAPREDUCE-3817: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513459/MAPREDUCE-3817.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1789//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1789//console This message is automatically generated. bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3817) bin/mapred command cannot run distcp and archive jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201528#comment-13201528 ] Arpit Gupta commented on MAPREDUCE-3817: no tests added as changes are to shell scripts. Ran distcp and archive jobs using bin/mapred with these changes successfully. bin/mapred command cannot run distcp and archive jobs - Key: MAPREDUCE-3817 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3817 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1, 0.24.0 Reporter: Arpit Gupta Assignee: Arpit Gupta Attachments: MAPREDUCE-3817.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201534#comment-13201534 ] Hadoop QA commented on MAPREDUCE-3812: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513476/MAPREDUCE-3812-20120206.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1791//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1791//console This message is automatically generated. Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3794: - Attachment: MAPREDUCE-3794.patch Thanks Mahadev. Here's a corrected patch. Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3709) TestDistributedShell is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201541#comment-13201541 ] Hadoop QA commented on MAPREDUCE-3709: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513463/MR-3709.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1790//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1790//console This message is automatically generated. TestDistributedShell is failing --- Key: MAPREDUCE-3709 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3709 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.1 Reporter: Eli Collins Assignee: Hitesh Shah Attachments: MR-3709.1.patch TestDistributedShell#testDSShell is failing the assert on line 90 on branch-23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3773) Add queue metrics with buckets for job run times
[ https://issues.apache.org/jira/browse/MAPREDUCE-3773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201543#comment-13201543 ] Hadoop QA commented on MAPREDUCE-3773: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513479/mr-3773-trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1792//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1792//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1792//console This message is automatically generated. Add queue metrics with buckets for job run times Key: MAPREDUCE-3773 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3773 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: mr-3773-trunk.patch, mr-3773-trunk.patch, mr-3773.patch It would be nice to have queue metrics that reflect the number of jobs in each queue that have been running for different ranges of time. Reasonable time ranges are probably 0-1 hr, 1-5 hr, 5-24 hr, 24+ hrs; but they should be configurable. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3628) DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201547#comment-13201547 ] Amol Kekre commented on MAPREDUCE-3628: --- Throughput decrease was 10% with 6% CPU slowdown a few wks ago. Waiting for new numbers later this wk. DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. - Key: MAPREDUCE-3628 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3628 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2 Affects Versions: 0.23.0 Reporter: Amol Kekre Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 DFSIO read throughput is decreased by 16% in 0.23 than Hadoop-0.20.204 on 350 nodes size cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3034) NM should act on a REBOOT command from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-3034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201551#comment-13201551 ] Robert Joseph Evans commented on MAPREDUCE-3034: The patch looks good to me too. But it is a complex enough change that I don't feel comfortable committing it myself, I think Arun or Mahadev should look at the patch. Also could you look at the TestJobCounters test failure. It looks like the failure is sporadic, and not related to this patch, but I could not find an existing JIRA for it. Could you at least file a new JIRA for it? NM should act on a REBOOT command from RM - Key: MAPREDUCE-3034 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3034 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.0, 0.24.0 Reporter: Vinod Kumar Vavilapalli Assignee: Devaraj K Priority: Critical Attachments: MAPREDUCE-3034-1.patch, MAPREDUCE-3034-2.patch, MAPREDUCE-3034-3.patch, MAPREDUCE-3034.patch, MR-3034.txt RM sends a reboot command to NM in some cases, like when it gets lost and rejoins back. In such a case, NM should act on the command and reboot/reinitalize itself. This is akin to TT reinitialize on order from JT. We will need to shutdown all the services properly and reinitialize - this should automatically take care of killing of containers, cleaning up local temporary files etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated MAPREDUCE-3818:
---
Attachment: MAPREDUCE-3818.txt
DFSClient, ClientProtocol etc. is is classified as private interface. The tests
should be using FileSystem interface. We should create a bug to track this.
Vinod, can you please run a quick validation for this patch and commit it. I do
not have my mapreduce env completely setup.
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Priority: Blocker
Fix For: 0.24.0
Attachments: MAPREDUCE-3818.txt
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas reassigned MAPREDUCE-3818:
--
Assignee: Suresh Srinivas
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Suresh Srinivas
Priority: Blocker
Fix For: 0.24.0
Attachments: MAPREDUCE-3818.txt
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3819) Exception while renewing delegation token
Exception while renewing delegation token - Key: MAPREDUCE-3819 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3819 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3819) Exception while renewing delegation token
[
https://issues.apache.org/jira/browse/MAPREDUCE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201557#comment-13201557
]
Ramya Sunil commented on MAPREDUCE-3819:
Exception seen in the RM logs:
{noformat}
ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:RM principal/RMhostname@REALM
(auth:KERBEROS) cause:java.io.IOException: Failed to specify server's Kerberos
principal name
WARN org.apache.hadoop.ipc.Client: Couldn't setup connection for RM
principal/RMhostname@REALM to null
ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:RM principal/RMhostname@REALM
(auth:KERBEROS) cause:java.io.IOException: Couldn't setup connection for RM
principal/RMhostname@REALM to null
ERROR org.apache.hadoop.security.UserGroupInformation:
PriviledgedActionException as:RM principal/RMhostname@REALM
(auth:KERBEROS) cause:java.io.IOException: Failed on local exception:
java.io.IOException: Couldn't setup connection for RM
principal/RMhostname@REALM to null; Host Details : local host is: RM
hostname/RM IP; destination host is: NN hostname
:8020;
ERROR
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer:
Exception renewing tokenIdent: x y z, Kind: HDFS_DELEGATION_TOKEN, Service: NN
IP:8020. Not rescheduled
java.io.IOException: Failed on local exception: java.io.IOException: Couldn't
setup connection for RM principal/RMhostname@REALM to null; Host Details
: local host is: RM hostname/RM IP;
destination host is: NN hostnamet:8020;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:724)
at org.apache.hadoop.ipc.Client.call(Client.java:1090)
at
org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:193)
at $Proxy24.renewDelegationToken(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient$Renewer.renew(DFSClient.java:619)
at org.apache.hadoop.security.token.Token.renew(Token.java:312)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask$1.run(DelegationTokenRenewer.java:317)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask$1.run(DelegationTokenRenewer.java:313)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.yarn.server.resourcemanager.security.DelegationTokenRenewer$RenewalTimerTask.run(DelegationTokenRenewer.java:312)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)
Caused by: java.io.IOException: Couldn't setup connection for RM
principal/RMhostname@REALM to null
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:528)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:500)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:584)
at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:206)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1196)
at org.apache.hadoop.ipc.Client.call(Client.java:1066)
... 12 more
Caused by: java.io.IOException: Failed to specify server's Kerberos principal
name
at
org.apache.hadoop.security.SaslRpcClient.init(SaslRpcClient.java:88)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:407)
at org.apache.hadoop.ipc.Client$Connection.access$1200(Client.java:206)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:577)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:574)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:573)
... 15 more
{noformat}
Seems harmless at the moment but need to debug.
Exception while renewing delegation token
-
Key: MAPREDUCE-3819
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3819
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Priority: Minor
--
This message is automatically
[jira] [Created] (MAPREDUCE-3820) Improve logging when containers run beyond memory limits
Improve logging when containers run beyond memory limits Key: MAPREDUCE-3820 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3820 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Minor When containers run beyond memory limits, they are killed without logging any useful message. Diagnostics message reads Task attemptID failed 0 times and the console output reads INFO mapreduce.Job: Job jobID failed with state KILLED due to: which are both not useful. This message has to be improved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3821) NPE while running Shuffle benchmark
NPE while running Shuffle benchmark
---
Key: MAPREDUCE-3821
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3821
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Priority: Critical
hadoop jar hadoop-mapreduce-test.jar loadgen -outKey org.apache.hadoop.io.Text
-outValue org.apache.hadoop.io.Text
The tasks fail with the following exception:
{noformat}
Error: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.init(Path.java:69)
at org.apache.hadoop.fs.Path.init(Path.java:58)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3821) NPE while running Shuffle benchmark
[
https://issues.apache.org/jira/browse/MAPREDUCE-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans resolved MAPREDUCE-3821.
Resolution: Duplicate
NPE while running Shuffle benchmark
---
Key: MAPREDUCE-3821
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3821
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Priority: Critical
hadoop jar hadoop-mapreduce-test.jar loadgen -outKey
org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
The tasks fail with the following exception:
{noformat}
Error: java.lang.NullPointerException
at org.apache.hadoop.fs.Path.init(Path.java:69)
at org.apache.hadoop.fs.Path.init(Path.java:58)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3719) Make gridmix performance on YARN+MR to match or exceed that on 1.0
[ https://issues.apache.org/jira/browse/MAPREDUCE-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201576#comment-13201576 ] Amol Kekre commented on MAPREDUCE-3719: --- initial runs for gmv3 are matching 1.0 Make gridmix performance on YARN+MR to match or exceed that on 1.0 -- Key: MAPREDUCE-3719 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3719 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Reporter: Vinod Kumar Vavilapalli -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201578#comment-13201578 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-3814: Looks good to me too. Build passes after the patch. +1. Pushing this in. MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Open (was: Patch Available) Upmerging to the patch JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3814: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I just committed this to trunk and branch-0.23. Thanks Arun! MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-3822) TestJobCounters is failing intermittently on trunk and 0.23.
TestJobCounters is failing intermittently on trunk and 0.23. Key: MAPREDUCE-3822 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3822 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Mahadev konar Assignee: Mahadev konar Priority: Critical Fix For: 0.23.1 TestJobCounters fails sometimes on trunk. I have tracked it down to stats issue in FileSystem. Still working on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3822) TestJobCounters is failing intermittently on trunk and 0.23.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3822: - Component/s: mrv2 TestJobCounters is failing intermittently on trunk and 0.23. Key: MAPREDUCE-3822 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3822 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Reporter: Mahadev konar Assignee: Mahadev konar Priority: Critical Fix For: 0.23.1 TestJobCounters fails sometimes on trunk. I have tracked it down to stats issue in FileSystem. Still working on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3819) Exception while renewing delegation token
[ https://issues.apache.org/jira/browse/MAPREDUCE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3819: - Priority: Critical (was: Major) Exception while renewing delegation token - Key: MAPREDUCE-3819 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3819 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil Priority: Critical -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3819) Exception while renewing delegation token
[ https://issues.apache.org/jira/browse/MAPREDUCE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3819: - Priority: Major (was: Minor) I dont think this is harmless. We should find out whats the issue here. Exception while renewing delegation token - Key: MAPREDUCE-3819 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3819 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.1 Reporter: Ramya Sunil -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201595#comment-13201595 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Hdfs-trunk-Commit #1742 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1742/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241177 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201597#comment-13201597
]
Hadoop QA commented on MAPREDUCE-3808:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513481/MR-3808.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1794//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1794//console
This message is automatically generated.
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201598#comment-13201598 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Common-trunk-Commit #1670 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1670/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241177 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3818) Trunk MRV1 compilation is broken.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3818:
---
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
+1. This looks good. I can confirm that the compilation passes with the patch
and fails without.
Jenkins won't run these tests, but just submitting in case it catches any
warnings. Will commit it post that.
Trunk MRV1 compilation is broken.
-
Key: MAPREDUCE-3818
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3818
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: build, test
Affects Versions: 0.24.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Suresh Srinivas
Priority: Blocker
Fix For: 0.24.0
Attachments: MAPREDUCE-3818.txt
Seeing this:
{code}
[javac]
/Users/vinodkv/Workspace/eclipse-workspace/apache-git/hadoop-common/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:155:
cannot find symbol
[javac] symbol : class ClientNamenodeWireProtocol
[javac] location: class org.apache.hadoop.mapred.TestSubmitJob
[javac] RPC.getProxy(ClientNamenodeWireProtocol.class,
[javac]^
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201602#comment-13201602 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Hdfs-0.23-Commit #487 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/487/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) svn merge --ignore-ancestry -c 1241177 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241181 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson updated MAPREDUCE-3804: - Attachment: MAPREDUCE-3804.patch Patch fixes above findbug issue. Also corrects vulnerability when containers are present. yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Patch Available (was: Open) JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Attachment: MAPREDUCE-3354.patch JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201608#comment-13201608 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Common-0.23-Commit #497 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/497/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) svn merge --ignore-ancestry -c 1241177 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241181 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dave Thompson updated MAPREDUCE-3804: - Status: Patch Available (was: Open) yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201607#comment-13201607 ] Jonathan Eagles commented on MAPREDUCE-3354: Verified that mapred-daemon.sh can both start and stop the job history server. Verified that start-mapred.sh and stop-mapred.sh also work. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Open (was: Patch Available) JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Attachment: MAPREDUCE-3354.patch JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-3354: --- Status: Patch Available (was: Open) JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3794) Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility
[ https://issues.apache.org/jira/browse/MAPREDUCE-3794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201620#comment-13201620 ] Hadoop QA commented on MAPREDUCE-3794: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513490/MAPREDUCE-3794.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1795//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1795//console This message is automatically generated. Support mapred.Task.Counter and mapred.JobInProgress.Counter enums for compatibility Key: MAPREDUCE-3794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3794 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-3794.patch, MAPREDUCE-3794.patch The new counters are mapreduce.TaskCounter and mapreduce.JobCounter, but we should support the old ones too since they are public in Hadoop 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201628#comment-13201628 ] Hadoop QA commented on MAPREDUCE-3354: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513502/MAPREDUCE-3354.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1798//console This message is automatically generated. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Status: Open (was: Patch Available) Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3819) Exception while renewing delegation token
[
https://issues.apache.org/jira/browse/MAPREDUCE-3819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201638#comment-13201638
]
Daryn Sharp commented on MAPREDUCE-3819:
No, Mahadev is right, this is not harmless. The interesting part of the log is:
{{Couldn't setup connection for RM principal/RMhostname@REALM to *null*}}
There was a similar issue with the JHS where the to null error occurred when
the service protocol wasn't correctly defined.
Exception while renewing delegation token
-
Key: MAPREDUCE-3819
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3819
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Ramya Sunil
Priority: Critical
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes to be 1.5GB
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3812: --- Status: Patch Available (was: Open) Change default memory slot sizes to be 1.5GB Key: MAPREDUCE-3812 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB. This is just a configuration change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201645#comment-13201645 ] Hadoop QA commented on MAPREDUCE-3804: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12513499/MAPREDUCE-3804.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1797//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1797//console This message is automatically generated. yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201646#comment-13201646 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Mapreduce-trunk-Commit #1685 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1685/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241177 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3814) MR1 compile fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-3814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201647#comment-13201647 ] Hudson commented on MAPREDUCE-3814: --- Integrated in Hadoop-Mapreduce-0.23-Commit #508 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/508/]) MAPREDUCE-3814. Fixed MRV1 compilation. (Arun C Murthy via vinodkv) svn merge --ignore-ancestry -c 1241177 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241181 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/build.xml MR1 compile fails - Key: MAPREDUCE-3814 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3814 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Affects Versions: 0.23.1 Reporter: Arun C Murthy Assignee: Arun C Murthy Fix For: 0.23.1 Attachments: MAPREDUCE-3814.patch $ ant veryclean all-jars -Dversion=0.23.1 -Dresolvers=internal BUILD FAILED /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/build.xml:537: srcdir /grid/0/dev/acm/hadoop-0.23/hadoop-mapreduce-project/src/test/mapred/testjar does not exist! -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201652#comment-13201652 ] Robert Joseph Evans commented on MAPREDUCE-3804: The patch looks good to me +1. Could you please file a separate JIRA for the failure in org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainersMonitor. It appears to not be related to this fix at all, but I could not find another JIRA for it, even though I found others where it also failed for them. yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3354) JobHistoryServer should be started by bin/mapred and not by bin/yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201654#comment-13201654 ] Jonathan Eagles commented on MAPREDUCE-3354: Patch failure is expected due to hadoop-assemblies change. JobHistoryServer should be started by bin/mapred and not by bin/yarn Key: MAPREDUCE-3354 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3354 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Jonathan Eagles Priority: Blocker Attachments: MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch, MAPREDUCE-3354.patch JobHistoryServer belongs to mapreduce land. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3804) yarn webapp interface vulnerable to cross scripting attacks
[ https://issues.apache.org/jira/browse/MAPREDUCE-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201656#comment-13201656 ] Mahadev konar commented on MAPREDUCE-3804: -- @bobby/Dave, The reason TestContainersMonitor fails is due to MAPREDUCE-3583. I'd suggest re running through hudson just to make sure everything else is fine. yarn webapp interface vulnerable to cross scripting attacks --- Key: MAPREDUCE-3804 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3804 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2, resourcemanager Affects Versions: 0.23.0 Reporter: Dave Thompson Assignee: Dave Thompson Fix For: 0.23.1 Attachments: MAPREDUCE-3804.patch, MAPREDUCE-3804.patch, MAPREDUCE_3804_br_0.23.0.patch Yarn webapp interface may be vulnerable to certain cross scripting attacks, injected through URL request. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3810: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Vinod! MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3813: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Vinod! RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201670#comment-13201670
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3808:
This looks good. Simple test too.
+1. Am pushing this in.
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201681#comment-13201681 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Hdfs-trunk-Commit #1744 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1744/]) MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201682#comment-13201682 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Hdfs-trunk-Commit #1744 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1744/]) MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241205 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3808) NPE in FileOutputCommitter when running a 0 reduce job
[
https://issues.apache.org/jira/browse/MAPREDUCE-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod Kumar Vavilapalli updated MAPREDUCE-3808:
---
Resolution: Fixed
Fix Version/s: 0.23.1
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Release Note: Fixed an NPE in FileOutputCommitter for jobs with maps
but no reduces.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I just committed this to trunk and branch-0.23. Thanks Robert!
NPE in FileOutputCommitter when running a 0 reduce job
--
Key: MAPREDUCE-3808
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3808
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0, 0.24.0
Reporter: Siddharth Seth
Assignee: Robert Joseph Evans
Priority: Blocker
Fix For: 0.23.1
Attachments: MR-3808.txt
This was while running LoadGen.
{noformat}
Error: java.lang.NullPointerException at
org.apache.hadoop.fs.Path.init(Path.java:67)
at org.apache.hadoop.fs.Path.init(Path.java:56)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingJobAttemptsPath(FileOutputCommitter.java:118)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:167)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getJobAttemptPath(FileOutputCommitter.java:149)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getPendingTaskAttemptsPath(FileOutputCommitter.java:185)
at
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:209)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:100)
at
org.apache.hadoop.mapred.FileOutputCommitter.getTaskAttemptPath(FileOutputCommitter.java:94)
at
org.apache.hadoop.mapred.FileOutputCommitter.needsTaskCommit(FileOutputCommitter.java:176)
at
org.apache.hadoop.mapred.OutputCommitter.needsTaskCommit(OutputCommitter.java:248)
at org.apache.hadoop.mapred.Task.isCommitRequired(Task.java:955)
at org.apache.hadoop.mapred.Task.done(Task.java:912)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:331)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201690#comment-13201690 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Common-0.23-Commit #498 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/498/]) Merge -c 1241205 from trunk to branch-0.23 to fix MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241208 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201689#comment-13201689 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Common-0.23-Commit #498 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/498/]) Merge -c 1241209 from trunk to branch-0.23 to fix MAPREDUCE-3813. Added a cache for resolved racks. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241210 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3810) MR AM's ContainerAllocator is assigning the allocated containers very slowly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201696#comment-13201696 ] Hudson commented on MAPREDUCE-3810: --- Integrated in Hadoop-Common-trunk-Commit #1672 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1672/]) MAPREDUCE-3810. Performance tweaks - reduced logging in AM and defined hascode/equals for ResourceRequest Priority. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241205 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/Priority.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ResourceRequest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/PriorityPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourceRequestPBImpl.java MR AM's ContainerAllocator is assigning the allocated containers very slowly Key: MAPREDUCE-3810 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3810 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Fix For: 0.23.1 Attachments: MAPREDUCE-3810-20120205.txt This is mostly due to logging and other not-so-cheap operations we are doing as part of the AM-RM heartbeat cycle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3813) RackResolver should maintain a cache to avoid repetitive lookups.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13201694#comment-13201694 ] Hudson commented on MAPREDUCE-3813: --- Integrated in Hadoop-Common-trunk-Commit #1672 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1672/]) MAPREDUCE-3813. Added a cache for resolved racks. Contributed by Vinod K V. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1241209 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/RackResolver.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/TestRackResolver.java RackResolver should maintain a cache to avoid repetitive lookups. - Key: MAPREDUCE-3813 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3813 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: mrv2, performance Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Fix For: 0.23.1 Attachments: MAPREDUCE-3813-20120205.txt With the current code, during task creation, we repeatedly resolve hosts and RackResolver doesn't cache any of the results. Caching will improve performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
