[jira] [Updated] (MAPREDUCE-4637) Killing an unassigned task attempt causes the job to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4637: - Status: Open (was: Patch Available) Killing an unassigned task attempt causes the job to fail - Key: MAPREDUCE-4637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4637 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Mayank Bansal Attachments: MAPREDUCE-4637-trunk.patch, MAPREDUCE-4637-trunk-v2.patch, MAPREDUCE-4637-trunk-v3.patch, MAPREDUCE-4637-trunk-v4.patch, MAPREDUCE-4637-trunk-v5.patch, MAPREDUCE-4637-trunk-v6.patch, MapReduce.png Attempting to kill a task attempt that has been scheduled but is not running causes an invalid state transition and the AM to stop with an error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4637) Killing an unassigned task attempt causes the job to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4637: - Attachment: MAPREDUCE-4637-trunk-v6.patch Fixing Test. Thanks, Mayank Killing an unassigned task attempt causes the job to fail - Key: MAPREDUCE-4637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4637 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Mayank Bansal Attachments: MAPREDUCE-4637-trunk.patch, MAPREDUCE-4637-trunk-v2.patch, MAPREDUCE-4637-trunk-v3.patch, MAPREDUCE-4637-trunk-v4.patch, MAPREDUCE-4637-trunk-v5.patch, MAPREDUCE-4637-trunk-v6.patch, MapReduce.png Attempting to kill a task attempt that has been scheduled but is not running causes an invalid state transition and the AM to stop with an error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4637) Killing an unassigned task attempt causes the job to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-4637: - Status: Patch Available (was: Open) Killing an unassigned task attempt causes the job to fail - Key: MAPREDUCE-4637 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4637 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.0-alpha Reporter: Tom White Assignee: Mayank Bansal Attachments: MAPREDUCE-4637-trunk.patch, MAPREDUCE-4637-trunk-v2.patch, MAPREDUCE-4637-trunk-v3.patch, MAPREDUCE-4637-trunk-v4.patch, MAPREDUCE-4637-trunk-v5.patch, MAPREDUCE-4637-trunk-v6.patch, MapReduce.png Attempting to kill a task attempt that has been scheduled but is not running causes an invalid state transition and the AM to stop with an error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4637) Killing an unassigned task attempt causes the job to fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-4637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480666#comment-13480666
]
Hadoop QA commented on MAPREDUCE-4637:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550133/MAPREDUCE-4637-trunk-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2950//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2950//console
This message is automatically generated.
Killing an unassigned task attempt causes the job to fail
-
Key: MAPREDUCE-4637
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4637
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Tom White
Assignee: Mayank Bansal
Attachments: MAPREDUCE-4637-trunk.patch,
MAPREDUCE-4637-trunk-v2.patch, MAPREDUCE-4637-trunk-v3.patch,
MAPREDUCE-4637-trunk-v4.patch, MAPREDUCE-4637-trunk-v5.patch,
MAPREDUCE-4637-trunk-v6.patch, MapReduce.png
Attempting to kill a task attempt that has been scheduled but is not running
causes an invalid state transition and the AM to stop with an error.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4398) Fix mapred.system.dir permission error with FairScheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu resolved MAPREDUCE-4398. Resolution: Duplicate Fix Version/s: 1.2.0 The changes in MAPREDUCE-4451 will resolve this issue as well, whether security is enabled or not. Fix mapred.system.dir permission error with FairScheduler - Key: MAPREDUCE-4398 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4398 Project: Hadoop Map/Reduce Issue Type: Bug Components: contrib/fair-share Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Fix For: 1.2.0 Incorrect job initialization logic in FairScheduler causes mysterious intermittent mapred.system.dir permission errors. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3470) Jobtracker sets permissions on mapred.system.dir to 700 preventing non-superusers from submitting jobs to multi-user cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480678#comment-13480678 ] Luke Lu commented on MAPREDUCE-3470: You must be using fairscheduler? If so the problem is resolved by MAPREDUCE-4451 (yes it affects non-kerberos config as well). Jobtracker sets permissions on mapred.system.dir to 700 preventing non-superusers from submitting jobs to multi-user cluster Key: MAPREDUCE-3470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3470 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.205.0, 1.0.0 Environment: Debian 6.0 x64_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: stephen mulcahy (See thread discussing here - https://mail-archives.apache.org/mod_mbox/hadoop-common-user/20.mbox/%3c4ebac2b3.6090...@deri.org%3E) I have installed a hadoop 0.20.205.0 cluster for use by multiple users, each of which will submit their jobs from remote client systems. I have disabled security property namehadoop.security.authorization/name valuefalse/value /property property namehadoop.security.authentication/name valuesimple/value /property When a user other than super-user attempts to submit a job, they get the following error 11/11/09 16:32:53 INFO mapred.FileInputFormat: Total input paths to process : 2 11/11/09 16:32:53 INFO mapred.JobClient: Running job: job_20091731_0003 11/11/09 16:32:54 INFO mapred.JobClient: map 0% reduce 0% 11/11/09 16:32:54 INFO mapred.JobClient: Job complete: job_20091731_0003 11/11/09 16:32:54 INFO mapred.JobClient: Counters: 0 11/11/09 16:32:54 INFO mapred.JobClient: Job Failed: Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=smulcahy, access=EXECUTE, inode=system:hadoop:supergroup:rwx-- . which seems to be due to not being able to create a jobToken file in mapred.system.dir/job id/jobToken I can reset the permissions on mapred.system.dir to something like 777 but when I restart the jobtracker, it resets the permissions back to 700, requiring another permissions reset. This gives rise to a few questions, 1. Should I be able to use a hadoop cluster in this fashion or is this not supported (if not, supported, I guess close this bug as invalid). If it is not supported, it reduces the usability of hadoop for a class of users like myself (but maybe thats a small class). 2. If I should be able to use the cluster like this, should mapred.system.dir/job id/jobToken need to be created if security is disabled? If no, then I guess is the bug that needs to be fixed. If yes, then the jobtracker needs to be modified to allow everyone to create dirs in mapred.system.dir (or the method of creation of the jobToken file needs to be changed). Apologies if this was operator error but I didn't get much feedback on the mailing lists so not sure where/how else to raise this. Changed introduced in MAPREDUCE-2219 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3470) Jobtracker sets permissions on mapred.system.dir to 700 preventing non-superusers from submitting jobs to multi-user cluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu resolved MAPREDUCE-3470. Resolution: Duplicate Fix Version/s: 1.2.0 Jobtracker sets permissions on mapred.system.dir to 700 preventing non-superusers from submitting jobs to multi-user cluster Key: MAPREDUCE-3470 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3470 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 0.20.205.0, 1.0.0 Environment: Debian 6.0 x64_64 java version 1.6.0_26 Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode) Reporter: stephen mulcahy Fix For: 1.2.0 (See thread discussing here - https://mail-archives.apache.org/mod_mbox/hadoop-common-user/20.mbox/%3c4ebac2b3.6090...@deri.org%3E) I have installed a hadoop 0.20.205.0 cluster for use by multiple users, each of which will submit their jobs from remote client systems. I have disabled security property namehadoop.security.authorization/name valuefalse/value /property property namehadoop.security.authentication/name valuesimple/value /property When a user other than super-user attempts to submit a job, they get the following error 11/11/09 16:32:53 INFO mapred.FileInputFormat: Total input paths to process : 2 11/11/09 16:32:53 INFO mapred.JobClient: Running job: job_20091731_0003 11/11/09 16:32:54 INFO mapred.JobClient: map 0% reduce 0% 11/11/09 16:32:54 INFO mapred.JobClient: Job complete: job_20091731_0003 11/11/09 16:32:54 INFO mapred.JobClient: Counters: 0 11/11/09 16:32:54 INFO mapred.JobClient: Job Failed: Job initialization failed: org.apache.hadoop.security.AccessControlException: org.apache.hadoop.security.AccessControlException: Permission denied: user=smulcahy, access=EXECUTE, inode=system:hadoop:supergroup:rwx-- . which seems to be due to not being able to create a jobToken file in mapred.system.dir/job id/jobToken I can reset the permissions on mapred.system.dir to something like 777 but when I restart the jobtracker, it resets the permissions back to 700, requiring another permissions reset. This gives rise to a few questions, 1. Should I be able to use a hadoop cluster in this fashion or is this not supported (if not, supported, I guess close this bug as invalid). If it is not supported, it reduces the usability of hadoop for a class of users like myself (but maybe thats a small class). 2. If I should be able to use the cluster like this, should mapred.system.dir/job id/jobToken need to be created if security is disabled? If no, then I guess is the bug that needs to be fixed. If yes, then the jobtracker needs to be modified to allow everyone to create dirs in mapred.system.dir (or the method of creation of the jobToken file needs to be changed). Apologies if this was operator error but I didn't get much feedback on the mailing lists so not sure where/how else to raise this. Changed introduced in MAPREDUCE-2219 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480692#comment-13480692 ] Hudson commented on MAPREDUCE-4733: --- Integrated in Hadoop-Yarn-trunk #9 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/9/]) MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many reducers complete consecutively. Contributed by Jason Lowe via. (Revision 1400264) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400264 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4733.patch, MAPREDUCE-4733.patch, MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4479) Fix parameter order in assertEquals() in TestCombineInputFileFormat.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480693#comment-13480693 ] Hudson commented on MAPREDUCE-4479: --- Integrated in Hadoop-Yarn-trunk #9 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/9/]) MAPREDUCE-4479. Fix parameter order in assertEquals() in TestCombineInputFileFormat.java (Mariappan Asokan via bobby) (Revision 1400192) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400192 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java Fix parameter order in assertEquals() in TestCombineInputFileFormat.java Key: MAPREDUCE-4479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4479 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: mapreduce-4479.patch, mapreduce-4479.patch, mapreduce-4479.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4479) Fix parameter order in assertEquals() in TestCombineInputFileFormat.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480721#comment-13480721 ] Hudson commented on MAPREDUCE-4479: --- Integrated in Hadoop-Hdfs-0.23-Build #410 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/410/]) svn merge -c 1400192 FIXES: MAPREDUCE-4479. Fix parameter order in assertEquals() in TestCombineInputFileFormat.java (Mariappan Asokan via bobby) (Revision 1400197) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400197 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java Fix parameter order in assertEquals() in TestCombineInputFileFormat.java Key: MAPREDUCE-4479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4479 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: mapreduce-4479.patch, mapreduce-4479.patch, mapreduce-4479.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480720#comment-13480720 ] Hudson commented on MAPREDUCE-4733: --- Integrated in Hadoop-Hdfs-0.23-Build #410 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/410/]) MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many reducers complete consecutively. Contributed by Jason Lowe. svn merge --ignore-ancestry -c 1400264 ../../trunk/ (Revision 1400267) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400267 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4733.patch, MAPREDUCE-4733.patch, MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480726#comment-13480726 ] Hudson commented on MAPREDUCE-4736: --- Integrated in Hadoop-Hdfs-trunk #1201 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1201/]) Moved MAPREDUCE-4736 jira from trunk to Release 2.0.3 section (Revision 1400314) MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO. Contributed by Brandon Li. (Revision 1400313) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400314 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400313 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. Remove it to avoid confusion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480727#comment-13480727 ] Hudson commented on MAPREDUCE-4733: --- Integrated in Hadoop-Hdfs-trunk #1201 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1201/]) MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many reducers complete consecutively. Contributed by Jason Lowe via. (Revision 1400264) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400264 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4733.patch, MAPREDUCE-4733.patch, MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4479) Fix parameter order in assertEquals() in TestCombineInputFileFormat.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480728#comment-13480728 ] Hudson commented on MAPREDUCE-4479: --- Integrated in Hadoop-Hdfs-trunk #1201 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1201/]) MAPREDUCE-4479. Fix parameter order in assertEquals() in TestCombineInputFileFormat.java (Mariappan Asokan via bobby) (Revision 1400192) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400192 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java Fix parameter order in assertEquals() in TestCombineInputFileFormat.java Key: MAPREDUCE-4479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4479 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: mapreduce-4479.patch, mapreduce-4479.patch, mapreduce-4479.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4256) Improve resource scheduling
[ https://issues.apache.org/jira/browse/MAPREDUCE-4256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480731#comment-13480731 ] Radim Kolar commented on MAPREDUCE-4256: About optional resources like GPU for computation. Lets task to specify weight in application resource requests. for example request like gpu:1:.8 means requesting 1 of gpu resources with weight 0.8 - give node 80% bonus if gpu resource is available it could handle data locality as well, because for some compute tasks data locality its not needed because you do not move much data over net. In that case request will be data:1:0.0 Improve resource scheduling --- Key: MAPREDUCE-4256 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4256 Project: Hadoop Map/Reduce Issue Type: Improvement Components: resourcemanager Reporter: Radim Kolar Currently resource manager supports only Memory resource during container allocation. I propose following improvements: 1. add support for CPU utilization. Node CPU used information can be obtained by ResourceCalculatorPlugin. 2. add support for custom resources. In node configuration will be something like: name=node.resource.GPU, value=1 (node has 1 GPU). If job will need to use GPU for computation, it will add GPU=1 requirement to its job config and Resource Manager will allocate container on node with GPU available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4736) Remove obsolete option [-rootDir] from TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-4736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480737#comment-13480737 ] Hudson commented on MAPREDUCE-4736: --- Integrated in Hadoop-Mapreduce-trunk #1231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1231/]) Moved MAPREDUCE-4736 jira from trunk to Release 2.0.3 section (Revision 1400314) MAPREDUCE-4736. Remove obsolete option [-rootDir] from TestDFSIO. Contributed by Brandon Li. (Revision 1400313) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400314 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400313 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java Remove obsolete option [-rootDir] from TestDFSIO Key: MAPREDUCE-4736 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4736 Project: Hadoop Map/Reduce Issue Type: Improvement Components: test Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Priority: Trivial Fix For: 3.0.0, 2.0.3-alpha Attachments: MAPREDUCE-4736.patch Looks like this option is obsolete. Remove it to avoid confusion. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4479) Fix parameter order in assertEquals() in TestCombineInputFileFormat.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480739#comment-13480739 ] Hudson commented on MAPREDUCE-4479: --- Integrated in Hadoop-Mapreduce-trunk #1231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1231/]) MAPREDUCE-4479. Fix parameter order in assertEquals() in TestCombineInputFileFormat.java (Mariappan Asokan via bobby) (Revision 1400192) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400192 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestCombineFileInputFormat.java Fix parameter order in assertEquals() in TestCombineInputFileFormat.java Key: MAPREDUCE-4479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4479 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: mapreduce-4479.patch, mapreduce-4479.patch, mapreduce-4479.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4733) Reducer can fail to make progress during shuffle if too many reducers complete consecutively
[ https://issues.apache.org/jira/browse/MAPREDUCE-4733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480738#comment-13480738 ] Hudson commented on MAPREDUCE-4733: --- Integrated in Hadoop-Mapreduce-trunk #1231 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1231/]) MAPREDUCE-4733. Reducer can fail to make progress during shuffle if too many reducers complete consecutively. Contributed by Jason Lowe via. (Revision 1400264) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1400264 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/Job.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MockJobs.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFetchFailure.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/PartialJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java Reducer can fail to make progress during shuffle if too many reducers complete consecutively Key: MAPREDUCE-4733 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4733 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.3 Reporter: Jason Lowe Assignee: Jason Lowe Fix For: 2.0.3-alpha, 0.23.5 Attachments: MAPREDUCE-4733.patch, MAPREDUCE-4733.patch, MAPREDUCE-4733.patch TaskAttemptListenerImpl implements getMapCompletionEvents by calling Job.getTaskAttemptCompletionEvents with the same fromEvent and maxEvents passed in from the reducer and then filtering the result for just map events. We can't filter the task completion event list and expect the caller's window into the list to match up. As soon as a reducer event appears in the list it means we are redundantly sending map completion events that were already seen by the reducer. Worst case the reducer will hang if all of the events in the requested window are reducer events. In that case zero events will be reported back to the caller and it won't bump up fromEvent on the next call. Reducer then never sees the final map completion events needed to complete the shuffle. This could happen in a case where all maps complete, more than MAX_EVENTS reducers complete consecutively, but some straggling reducers get fetch failures and cause a map to be restarted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4479) Fix parameter order in assertEquals() in TestCombineInputFileFormat.java
[ https://issues.apache.org/jira/browse/MAPREDUCE-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480792#comment-13480792 ] Mariappan Asokan commented on MAPREDUCE-4479: - Hi Robert, Thanks for committing this. -- Asokan Fix parameter order in assertEquals() in TestCombineInputFileFormat.java Key: MAPREDUCE-4479 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4479 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha Reporter: Mariappan Asokan Assignee: Mariappan Asokan Fix For: 3.0.0, 2.0.3-alpha, 0.23.5 Attachments: mapreduce-4479.patch, mapreduce-4479.patch, mapreduce-4479.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4229) Intern counter names in the JT
[ https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miomir Boljanovic updated MAPREDUCE-4229: - Attachment: MAPREDUCE-4229-branch-0.23.patch Intern counter names in the JT -- Key: MAPREDUCE-4229 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobtracker Affects Versions: 1.0.2, 3.0.0, 2.0.2-alpha Reporter: Todd Lipcon Attachments: MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229.patch, MR-4229.txt In our experience, most of the memory in production JTs goes to storing counter names (String objects and character arrays). Since most counter names are reused again and again, it would be a big memory savings to keep a hash set of already-used counter names within a job, and refer to the same object from all tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4229) Intern counter names in the JT
[
https://issues.apache.org/jira/browse/MAPREDUCE-4229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13480854#comment-13480854
]
Hadoop QA commented on MAPREDUCE-4229:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12550156/MAPREDUCE-4229-branch-0.23.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2951//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2951//console
This message is automatically generated.
Intern counter names in the JT
--
Key: MAPREDUCE-4229
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4229
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Affects Versions: 1.0.2, 3.0.0, 2.0.2-alpha
Reporter: Todd Lipcon
Attachments: MAPREDUCE-4229-branch-0.23.patch,
MAPREDUCE-4229-branch-0.23.patch, MAPREDUCE-4229.patch, MR-4229.txt
In our experience, most of the memory in production JTs goes to storing
counter names (String objects and character arrays). Since most counter names
are reused again and again, it would be a big memory savings to keep a hash
set of already-used counter names within a job, and refer to the same object
from all tasks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
