[jira] [Updated] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-3801: -- Assignee: Tsuyoshi OZAWA > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Tsuyoshi OZAWA > Attachments: > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3801) org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator fails intermittently
[ https://issues.apache.org/jira/browse/MAPREDUCE-3801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634881#comment-13634881 ] Tsuyoshi OZAWA commented on MAPREDUCE-3801: --- I got similar results. This is the result of testing: $ mvn test -Dtest=TestRuntimeEstimators --- T E S T S --- Running org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 38.515 sec <<< FAILURE! testLegacyEstimator(org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators) Time elapsed: 20276 sec <<< FAILURE! java.lang.AssertionError: We got the wrong number of successful speculations. expected:<3> but was:<5> at org.junit.Assert.fail(Assert.java:91) at org.junit.Assert.failNotEquals(Assert.java:645) at org.junit.Assert.assertEquals(Assert.java:126) at org.junit.Assert.assertEquals(Assert.java:470) at org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.coreTestEstimator(TestRuntimeEstimators.java:222) at org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.testLegacyEstimator(TestRuntimeEstimators.java:229) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:252) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:141) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:112) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165) at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75) Results : Failed tests: testLegacyEstimator(org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators): We got the wrong number of successful speculations. expected:<3> but was:<5> > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently > -- > > Key: MAPREDUCE-3801 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3801 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.24.0 >Reporter: Robert Joseph Evans > Attachments: > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators-output.txt, > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.txt, > TEST-org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators.xml > > > org.apache.hadoop.mapreduce.v2.app.TestRuntimeEstimators,testExponentialEstimator > fails intermittently -- This message is automatically generated by JIRA. If you think it was sent incorrectly, pl
[jira] [Updated] (MAPREDUCE-5004) Somebody working on Genetic Algorithm library on Map Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Singh Rajput updated MAPREDUCE-5004: Assignee: Vaibhav Singh Rajput > Somebody working on Genetic Algorithm library on Map Reduce > --- > > Key: MAPREDUCE-5004 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5004 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Abhishek Bajpai >Assignee: Vaibhav Singh Rajput > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5163) Update MR App after YARN-441
Vinod Kumar Vavilapalli created MAPREDUCE-5163: -- Summary: Update MR App after YARN-441 Key: MAPREDUCE-5163 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5163 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Xuan Gong YARN-441 is removing some collection APIs that are utility methods on top the base APIs. MR App needs to be updated to not use those anymore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5155) Race condition in test case TestFetchFailure cause it to fail
[
https://issues.apache.org/jira/browse/MAPREDUCE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634784#comment-13634784
]
nemon lou commented on MAPREDUCE-5155:
--
Another test case failed with the same reason:
Failed tests: testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestMRApp):
Expecting 2 more completion events for killed expected:<4> but was:<2>
These lines of code has race condition issue:
{quote}
app.waitForState(task1Attempt, TaskAttemptState.KILLED);
app.waitForState(task2Attempt, TaskAttemptState.KILLED);
events = job.getTaskAttemptCompletionEvents(0, 100);
Assert.assertEquals("Expecting 2 more completion events for killed", 4,
events.length);
{quote}
> Race condition in test case TestFetchFailure cause it to fail
> -
>
> Key: MAPREDUCE-5155
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5155
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.3-alpha
> Environment: Suse x86_64 GNU/Linux
> Java(TM) SE Runtime Environment (build 1.6.0_32-b05
>Reporter: nemon lou
>Priority: Minor
> Attachments:
> org.apache.hadoop.mapreduce.v2.app.TestFetchFailure-output.txt,
> org.apache.hadoop.mapreduce.v2.app.TestFetchFailure.txt
>
>
> I run into this once:
> testFetchFailureWithRecovery(org.apache.hadoop.mapreduce.v2.app.TestFetchFailure):
> Num completion events not correct expected:<1> but was:<0>
> There is a race condition between job.getTaskAttemptCompletionEvents and
> dealing with JOB_TASK_ATTEMPT_COMPLETED event.
> If job.getTaskAttemptCompletionEvents invoked because of task in SUCCEEDED
> state ,but before JOB_TASK_ATTEMPT_COMPLETED event scheduled,the test case
> will fail.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5110) Long task launch delays can lead to multiple parallel attempts of the task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5110: Attachment: mr-5110-half-tt-expiry.patch [~vinodkv], here is a new patch that uses half the tt-expriry-interval as the timeout for task launch. Do you think this is a resonable way to go about it, or do you think it is better to add a job-specific parameter? I ll validate the patch we finalize on a cluster. > Long task launch delays can lead to multiple parallel attempts of the task > -- > > Key: MAPREDUCE-5110 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5110 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.1.2 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: expose-mr-5110.patch, mr-5110-half-tt-expiry.patch, > mr-5110.patch, mr-5110.patch, mr-5110-tt-only.patch > > > If a task takes too long to launch, the JT expires the task and schedules > another attempt. The earlier attempt can start after the later attempt > leading to two parallel attempts running at the same time. This is > particularly an issue if the user turns off speculation and expects a single > attempt of a task to run at any point in time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5110) Kill task early in case of long task launch delays
[ https://issues.apache.org/jira/browse/MAPREDUCE-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5110: Summary: Kill task early in case of long task launch delays (was: Long task launch delays can lead to multiple parallel attempts of the task) > Kill task early in case of long task launch delays > -- > > Key: MAPREDUCE-5110 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5110 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.1.2 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: expose-mr-5110.patch, mr-5110-half-tt-expiry.patch, > mr-5110.patch, mr-5110.patch, mr-5110-tt-only.patch > > > If a task takes too long to launch, the JT expires the task and schedules > another attempt. The earlier attempt can start after the later attempt > leading to two parallel attempts running at the same time. This is > particularly an issue if the user turns off speculation and expects a single > attempt of a task to run at any point in time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5162) Add ability to change the queue a job has been assigned to with JobClient
Jeff Lord created MAPREDUCE-5162: Summary: Add ability to change the queue a job has been assigned to with JobClient Key: MAPREDUCE-5162 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5162 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 0.20.2 Reporter: Jeff Lord Currently the job client does not provide a facility to change the particular queue a job is assigned to. It would be nice to be able to do something like this: mapred job -setqueue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634598#comment-13634598 ] Hudson commented on MAPREDUCE-5128: --- Integrated in Hadoop-trunk-Commit #3625 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3625/]) MAPREDUCE-5128. mapred-default.xml is missing a bunch of history server configs. (sandyr via tucu) (Revision 1469095) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469095 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml > mapred-default.xml is missing a bunch of history server configs > --- > > Key: MAPREDUCE-5128 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, jobhistoryserver >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5128.patch > > > mapred-default.xml is missing many configs that work for the job history > server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, > and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5133) TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race
[ https://issues.apache.org/jira/browse/MAPREDUCE-5133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-5133: -- Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Sandy. Committed to branch-1. > TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race > -- > > Key: MAPREDUCE-5133 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5133 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 1.3.0 > > Attachments: MAPREDUCE-5133.patch > > > At the end of TestSubmitJob.testSecureJobExecution, the test waits for the > job to be done and then asserts that the job submission directory has been > deleted. The directory is deleted by an asynchronous cleanup thread, so the > test can hit the assert before the deletion is run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-5128: -- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Sandy. Committed to trunk and branch-2. > mapred-default.xml is missing a bunch of history server configs > --- > > Key: MAPREDUCE-5128 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, jobhistoryserver >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5128.patch > > > mapred-default.xml is missing many configs that work for the job history > server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, > and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5152) MR App is not using Container from RM
[
https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634577#comment-13634577
]
Hadoop QA commented on MAPREDUCE-5152:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579218/MAPREDUCE-5152-20130417.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3535//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3535//console
This message is automatically generated.
> MR App is not using Container from RM
> -
>
> Key: MAPREDUCE-5152
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.5-beta
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
> Attachments: MAPREDUCE-5152-20130415.1.txt,
> MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt
>
>
> The goal of YARN-486 was to make AMs just pass information encapsulated in
> Container along to NM instead of doing it themselves by duplicating
> information. We still do not do this pass-through as intended as YARN-486
> avoided the individual field duplication but failed to avoid the duplication
> of container itself.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4638) MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634569#comment-13634569 ] Sandy Ryza commented on MAPREDUCE-4638: --- To save future viewers a little work: This wasn't put into branch-2, but it's contents are there now as a result of Tom's merge, commit 660ea022ac6e8a1209951c0990dcee4b5169608d. > MR AppMaster shouldn't rely on YARN_APPLICATION_CLASSPATH providing MR jars > --- > > Key: MAPREDUCE-4638 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4638 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Arun C Murthy >Assignee: Arun C Murthy > Fix For: 2.0.2-alpha > > Attachments: MAPREDUCE-4638.patch > > > Currently YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH provides > $YARN_HOME/share/hadoop/mapreduce/* & > $YARN_HOME/share/hadoop/mapreduce/lib/*. It should not depend on this post > YARN-86. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5154) staging directory deletion fails because delegation tokens have been cancelled
[ https://issues.apache.org/jira/browse/MAPREDUCE-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634563#comment-13634563 ] Alejandro Abdelnur commented on MAPREDUCE-5154: --- LGTM, any chance to have a testcase? Else ensure it works in a secure setup as expected? > staging directory deletion fails because delegation tokens have been cancelled > -- > > Key: MAPREDUCE-5154 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5154 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 1.2.0 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5154.patch > > > In a secure setup, the jobtracker needs the job's delegation tokens to delete > the staging directory. MAPREDUCE-4850 made it so that job cleanup staging > directory deletion occurs asynchronously, so that it could order it with > system directory deletion. This introduced the issue that a job's delegation > tokens could be cancelled before the cleanup thread got around to deleting > it, causing the deletion to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5133) TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race
[ https://issues.apache.org/jira/browse/MAPREDUCE-5133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634550#comment-13634550 ] Alejandro Abdelnur commented on MAPREDUCE-5133: --- +1 > TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race > -- > > Key: MAPREDUCE-5133 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5133 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5133.patch > > > At the end of TestSubmitJob.testSecureJobExecution, the test waits for the > job to be done and then asserts that the job submission directory has been > deleted. The directory is deleted by an asynchronous cleanup thread, so the > test can hit the assert before the deletion is run. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5152) MR App is not using Container from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5152: --- Status: Patch Available (was: Open) > MR App is not using Container from RM > - > > Key: MAPREDUCE-5152 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.5-beta >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-5152-20130415.1.txt, > MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt > > > The goal of YARN-486 was to make AMs just pass information encapsulated in > Container along to NM instead of doing it themselves by duplicating > information. We still do not do this pass-through as intended as YARN-486 > avoided the individual field duplication but failed to avoid the duplication > of container itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5128) mapred-default.xml is missing a bunch of history server configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634545#comment-13634545 ] Alejandro Abdelnur commented on MAPREDUCE-5128: --- +1 > mapred-default.xml is missing a bunch of history server configs > --- > > Key: MAPREDUCE-5128 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5128 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, jobhistoryserver >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5128.patch > > > mapred-default.xml is missing many configs that work for the job history > server. mapreduce.jobhistory.cleaner.enable, mapreduce.jobhistory.done-dir, > and mapreduce.jobhistory.datestring.cache.size are a few examples. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5152) MR App is not using Container from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5152: --- Attachment: MAPREDUCE-5152-20130417.txt Updated patch addressing the review comments. The new tests passes with the code changes and fails without. > MR App is not using Container from RM > - > > Key: MAPREDUCE-5152 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.5-beta >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-5152-20130415.1.txt, > MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt > > > The goal of YARN-486 was to make AMs just pass information encapsulated in > Container along to NM instead of doing it themselves by duplicating > information. We still do not do this pass-through as intended as YARN-486 > avoided the individual field duplication but failed to avoid the duplication > of container itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5152) MR App is not using Container from RM
[ https://issues.apache.org/jira/browse/MAPREDUCE-5152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5152: --- Status: Open (was: Patch Available) > MR App is not using Container from RM > - > > Key: MAPREDUCE-5152 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5152 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.5-beta >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-5152-20130415.1.txt, > MAPREDUCE-5152-20130415.txt, MAPREDUCE-5152-20130417.txt > > > The goal of YARN-486 was to make AMs just pass information encapsulated in > Container along to NM instead of doing it themselves by duplicating > information. We still do not do this pass-through as intended as YARN-486 > avoided the individual field duplication but failed to avoid the duplication > of container itself. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5161) CombineFileInputFormat fix for paths not on default FS merge from branch-1 to branch-1-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated MAPREDUCE-5161:
-
Attachment: MAPREDUCE-5161-branch-1-win.1.patch
I'm attaching a patch for branch-1-win that puts the fix into agreement with
the version from MAPREDUCE-1806 that had been committed to branch-1. I
verified this on Mac and Windows.
> CombineFileInputFormat fix for paths not on default FS merge from branch-1 to
> branch-1-win
> --
>
> Key: MAPREDUCE-5161
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1
>Affects Versions: 1-win
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: MAPREDUCE-5161-branch-1-win.1.patch
>
>
> MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
> paths that are not on the default file system. This same bug was fixed
> independently on branch-1-win. The code was slightly different, but
> equivalent to the branch-1 fix. This jira will apply the branch-1 fix to
> branch-1-win to keep the 2 code lines in agreement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5156: - Priority: Blocker (was: Major) > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-5156: - Target Version/s: 2.0.5-beta > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen >Priority: Blocker > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5161) CombineFileInputFormat fix for paths not on default FS merge from branch-1 to branch-1-win
Chris Nauroth created MAPREDUCE-5161:
Summary: CombineFileInputFormat fix for paths not on default FS
merge from branch-1 to branch-1-win
Key: MAPREDUCE-5161
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5161
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
MAPREDUCE-1806 fixed a bug related to use of {{CombineFileInputFormat}} with
paths that are not on the default file system. This same bug was fixed
independently on branch-1-win. The code was slightly different, but equivalent
to the branch-1 fix. This jira will apply the branch-1 fix to branch-1-win to
keep the 2 code lines in agreement.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha resolved MAPREDUCE-5140. --- Resolution: Fixed Committed to trunk and branch-2 > MR part of YARN-514 > --- > > Key: MAPREDUCE-5140 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5140.1.patch > > > In YARN-514, application store needs to be delayed to unblock application > submission, such that a new state of MRApp needs to be created. On mapreduce > side, there's some function to map yarn states to mapreduce ones. This > mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634391#comment-13634391 ] Hudson commented on MAPREDUCE-5140: --- Integrated in Hadoop-trunk-Commit #3623 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3623/]) MAPREDUCE-5140. MR part of YARN-514 (Zhijie Shen via bikas) (Revision 1469060) Result = SUCCESS bikas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469060 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/ClientServiceDelegate.java > MR part of YARN-514 > --- > > Key: MAPREDUCE-5140 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5140.1.patch > > > In YARN-514, application store needs to be delayed to unblock application > submission, such that a new state of MRApp needs to be created. On mapreduce > side, there's some function to map yarn states to mapreduce ones. This > mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5140) MR part of YARN-514
[ https://issues.apache.org/jira/browse/MAPREDUCE-5140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-5140: -- Summary: MR part of YARN-514 (was: The mapping between yarn and mapreduce states needs to be updated when application store is delayed) > MR part of YARN-514 > --- > > Key: MAPREDUCE-5140 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5140 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Attachments: MAPREDUCE-5140.1.patch > > > In YARN-514, application store needs to be delayed to unblock application > submission, such that a new state of MRApp needs to be created. On mapreduce > side, there's some function to map yarn states to mapreduce ones. This > mapping needs to be updated due to the newly added state. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634375#comment-13634375
]
Hadoop QA commented on MAPREDUCE-4443:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579168/MAPREDUCE-4443-trunk-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3534//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3534//console
This message is automatically generated.
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-3.patch, MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Status: Patch Available (was: Open)
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-3.patch, MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..2012-07-12
>
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Status: Open (was: Patch Available)
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-3.patch, MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..2012-07-12
>
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Status: Patch Available (was: Open)
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-3.patch, MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..2012-07-12
>
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Attachment: MAPREDUCE-4443-trunk-3.patch
Fixing findbugs warnings
Thanks,
Mayank
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-3.patch, MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoo
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Status: Open (was: Patch Available)
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..2012-07-12
> 17:31:43,503 INFO [Thread-1]
[jira] [Commented] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634285#comment-13634285
]
Zhijie Shen commented on MAPREDUCE-5160:
The exception observed when running aggregatewordhist:
{code}
2013-04-17 11:04:13,471 FATAL [IPC Server handler 7 on 61283]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1366220242098_0004_m_07_0 - exited : java.lang.RuntimeException:
Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:425)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1489)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 17 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.AggregateWordHistogram$AggregateWordHistogramPlugin
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance(UserDefinedValueAggregatorDescriptor.java:63)
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.createAggregator(UserDefinedValueAggregatorDescriptor.java:70)
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.(UserDefinedValueAggregatorDescriptor.java:84)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.(UserDefinedValueAggregatorDescriptor.java:55)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggregatorDescriptor(ValueAggregatorJobBase.java:58)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregatorDescriptors(ValueAggregatorJobBase.java:69)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMySpec(ValueAggregatorJobBase.java:78)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(ValueAggregatorJobBase.java:46)
... 22 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.AggregateWordHistogram$AggregateWordHistogramPlugin
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:249)
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance(UserDefinedValueAggregatorDescriptor.java:58)
... 29 more
{code}
> Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find
> their inner classes when running on Yarn
> -
[jira] [Commented] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634282#comment-13634282
]
Zhijie Shen commented on MAPREDUCE-5160:
The exception observed when running aggregatewordcount:
{code}
2013-04-17 11:21:02,068 FATAL [IPC Server handler 7 on 62303]
org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
attempt_1366220242098_0005_m_01_0 - exited : java.lang.RuntimeException:
Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:425)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1489)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:72)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:130)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:103)
... 17 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.AggregateWordCount$WordCountPlugInClass
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance(UserDefinedValueAggregatorDescriptor.java:63)
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.createAggregator(UserDefinedValueAggregatorDescriptor.java:70)
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.(UserDefinedValueAggregatorDescriptor.java:84)
at
org.apache.hadoop.mapred.lib.aggregate.UserDefinedValueAggregatorDescriptor.(UserDefinedValueAggregatorDescriptor.java:55)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getValueAggregatorDescriptor(ValueAggregatorJobBase.java:58)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.getAggregatorDescriptors(ValueAggregatorJobBase.java:69)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.initializeMySpec(ValueAggregatorJobBase.java:78)
at
org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJobBase.configure(ValueAggregatorJobBase.java:46)
... 22 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.examples.AggregateWordCount$WordCountPlugInClass
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:249)
at
org.apache.hadoop.mapreduce.lib.aggregate.UserDefinedValueAggregatorDescriptor.createInstance(UserDefinedValueAggregatorDescriptor.java:58)
... 29 more
{code}
> Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find
> their inner classes when running on Yarn
> ---
[jira] [Updated] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5160: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 > Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find > their inner classes when running on Yarn > --- > > Key: MAPREDUCE-5160 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 > due to > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller) is not available on hadoop-2 (see MAPREDUCE-5159). > After I changed Aggregatewordcount and Aggregatewordhist to use > createValueAggregatorJob(String args[], Class ValueAggregatorDescriptor>[] descriptors), which is available on hadoop-2, > the two examples could be accepted and run on Yarn. > However, the two examples still failed, because their inner classes, > WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in > runtime, respectively. Both the plugin classes extend > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5160) Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn
Zhijie Shen created MAPREDUCE-5160: -- Summary: Aggregatewordcount and aggregatewordhist in hadoop-1 examples can not find their inner classes when running on Yarn Key: MAPREDUCE-5160 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5160 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Aggregatewordcount and Aggregatewordhist of hadoop-1 cannot run on hadoop-2 due to org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String args[], Class caller) is not available on hadoop-2 (see MAPREDUCE-5159). After I changed Aggregatewordcount and Aggregatewordhist to use createValueAggregatorJob(String args[], Class[] descriptors), which is available on hadoop-2, the two examples could be accepted and run on Yarn. However, the two examples still failed, because their inner classes, WordCountPlugInClass and AggregateWordHistogramPlugin, cannot be found in runtime, respectively. Both the plugin classes extend org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorBaseDescriptor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5158) Cleanup required when mapreduce.job.restart.recover is set to false
[ https://issues.apache.org/jira/browse/MAPREDUCE-5158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634221#comment-13634221 ] Arun C Murthy commented on MAPREDUCE-5158: -- Good catch Yesha, thanks for filing this! > Cleanup required when mapreduce.job.restart.recover is set to false > --- > > Key: MAPREDUCE-5158 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5158 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 1.2.0 >Reporter: yeshavora > > When mapred.jobtracker.restart.recover is set as true and > mapreduce.job.restart.recover is set to false for a MR job, Job clean up > never happens for that job if JT restarts while job is running. > .staging and job-info file for that job remains on HDFS forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
[ https://issues.apache.org/jira/browse/MAPREDUCE-5159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5159: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 > Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary > compatible with hadoop-2 mapred.lib.aggregate > -- > > Key: MAPREDUCE-5159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Both examples in hadoop-1 use > org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String > args[], Class caller), which no longer exists after upgrading to > hadoop-2. Therefore, they cannot not find the accordant function with their > function signature in the binary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5159) Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate
Zhijie Shen created MAPREDUCE-5159: -- Summary: Aggregatewordcount and aggregatewordhist in hadoop-1 examples are not binary compatible with hadoop-2 mapred.lib.aggregate Key: MAPREDUCE-5159 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5159 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen Both examples in hadoop-1 use org.apache.hadoop.mapred.lib.aggregate.ValueAggregatorJob.createValueAggregatorJob(String args[], Class caller), which no longer exists after upgrading to hadoop-2. Therefore, they cannot not find the accordant function with their function signature in the binary. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5158) Cleanup required when mapreduce.job.restart.recover is set to false
yeshavora created MAPREDUCE-5158: Summary: Cleanup required when mapreduce.job.restart.recover is set to false Key: MAPREDUCE-5158 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5158 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobtracker Affects Versions: 1.2.0 Reporter: yeshavora When mapred.jobtracker.restart.recover is set as true and mapreduce.job.restart.recover is set to false for a MR job, Job clean up never happens for that job if JT restarts while job is running. .staging and job-info file for that job remains on HDFS forever. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[
https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634193#comment-13634193
]
Zhijie Shen commented on MAPREDUCE-5157:
Bellow is the log I observed when executing hadoop-2.0.4-alpha/bin/hadoop jar
hadoop-1.1.1/hadoop-examples-1.1.1.jar
{code}
java.lang.NoClassDefFoundError:
org/apache/hadoop/mapred/lib/InputSampler$Sampler
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2436)
at java.lang.Class.getMethod0(Class.java:2679)
at java.lang.Class.getMethod(Class.java:1605)
at
org.apache.hadoop.util.ProgramDriver$ProgramDescription.(ProgramDriver.java:60)
at org.apache.hadoop.util.ProgramDriver.addClass(ProgramDriver.java:103)
at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.mapred.lib.InputSampler$Sampler
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 12 more
{code}
> Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
> ---
>
> Key: MAPREDUCE-5157
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Zhijie Shen
>
> In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in
> hadoop-1. However, after upgrading to hadoop-2,
> org.apache.hadoop.mapred.lib.InputSampler is modified to extend
> org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class,
> Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not
> find Sampler when it runs with hadoop-2 framework.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
[ https://issues.apache.org/jira/browse/MAPREDUCE-5157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5157: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 > Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib > --- > > Key: MAPREDUCE-5157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in > hadoop-1. However, after upgrading to hadoop-2, > org.apache.hadoop.mapred.lib.InputSampler is modified to extend > org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, > Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not > find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5157) Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib
Zhijie Shen created MAPREDUCE-5157: -- Summary: Sort in hadoop-1 examples is not binary compatible with hadoop-2 mapred.lib Key: MAPREDUCE-5157 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5157 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen In SORT example, org.apache.hadoop.mapred.lib.InputSampler.Sampler is used in hadoop-1. However, after upgrading to hadoop-2, org.apache.hadoop.mapred.lib.InputSampler is modified to extend org.apache.hadoop.mapreduce.lib.partition.InputSampler, and the inner class, Sampler, has been moved to the superclass. Therefore, hadoop-1 SORT can not find Sampler when it runs with hadoop-2 framework. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4502) Node-level aggregation with combining the result of maps
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-4502: -- Summary: Node-level aggregation with combining the result of maps (was: Multi-level aggregation with combining the result of maps per node/rack) > Node-level aggregation with combining the result of maps > > > Key: MAPREDUCE-4502 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Affects Versions: 3.0.0 >Reporter: Tsuyoshi OZAWA >Assignee: Tsuyoshi OZAWA > Attachments: design_v2.pdf, design_v3.pdf, MAPREDUCE-4502.1.patch, > MAPREDUCE-4502.2.patch, MAPREDUCE-4502.3.patch, MAPREDUCE-4502.4.patch, > MAPREDUCE-4502.5.patch, MAPREDUCE-4502.6.patch, MAPREDUCE-4525-pof.diff, > speculative_draft.pdf > > > The shuffle costs is expensive in Hadoop in spite of the existence of > combiner, because the scope of combining is limited within only one MapTask. > To solve this problem, it's a good way to aggregate the result of maps per > node/rack by launch combiner. > This JIRA is to implement the multi-level aggregation infrastructure, > including combining per container(MAPREDUCE-3902 is related), coordinating > containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5147) Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly
[ https://issues.apache.org/jira/browse/MAPREDUCE-5147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634104#comment-13634104 ] Thomas Graves commented on MAPREDUCE-5147: -- Thanks Rob, mostly looks good. Can you provide branch-2 and branch-0.23 patches as well. > Maven build should create hadoop-mapreduce-client-app-VERSION.jar directly > -- > > Key: MAPREDUCE-5147 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5147 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: trunk >Reporter: Robert Parker >Assignee: Robert Parker > Attachments: MAPREDUCE-5147.patch > > > Currently the build creates mr-app.jar and links it to the proper name. All > hard links to mr-app.jar appear to have been removed. The maven build should > be simplified to directly build the jar. > Related -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5015) Coverage fix for org.apache.hadoop.mapreduce.tools.CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634056#comment-13634056 ] Hudson commented on MAPREDUCE-5015: --- Integrated in Hadoop-Mapreduce-trunk #1402 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1402/]) MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI (Aleksey Gorshkov via tgraves) (Revision 1468483) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468483 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/job_1329348432655_0001-10.jhist > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > -- > > Key: MAPREDUCE-5015 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5015 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5015-branch-0.23-a.patch, > MAPREDUCE-5015-branch-0.23-b.patch, MAPREDUCE-5015-branch-0.23.patch, > MAPREDUCE-5015-branch-2-a.patch, MAPREDUCE-5015-branch-2-b.patch, > MAPREDUCE-5015-branch-2.patch, MAPREDUCE-5015-trunk-a.patch, > MAPREDUCE-5015-trunk-b.patch, MAPREDUCE-5015-trunk.patch > > > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > MAPREDUCE-5015-trunk.patch patch for trunk > MAPREDUCE-5015-branch-2.patch for branch-2 > MAPREDUCE-5015-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5065) DistCp should skip checksum comparisons if block-sizes are different on source/target.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634055#comment-13634055 ] Hudson commented on MAPREDUCE-5065: --- Integrated in Hadoop-Mapreduce-trunk #1402 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1402/]) MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes are different on source/target. Contributed by Mithun Radhakrishnan. (Revision 1468629) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468629 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java > DistCp should skip checksum comparisons if block-sizes are different on > source/target. > -- > > Key: MAPREDUCE-5065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5065 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5065.branch-0.23.patch, > MAPREDUCE-5065.branch-2.patch > > > When copying files between 2 clusters with different default block-sizes, one > sees that the copy fails with a checksum-mismatch, even though the files have > identical contents. > The reason is that on HDFS, a file's checksum is unfortunately a function of > the block-size of the file. So you could have 2 different files with > identical contents (but different block-sizes) have different checksums. > (Thus, it's also possible for DistCp to fail to copy files on the same > file-system, if the source-file's block-size differs from HDFS default, and > -pb isn't used.) > I propose that we skip checksum comparisons under the following conditions: > 1. -skipCrc is specified. > 2. File-size is 0 (in which case the call to the checksum-servlet is moot). > 3. source.getBlockSize() != target.getBlockSize(), since the checksums are > guaranteed to differ in this case. > I have a patch for #3. > Edit: I've modified the fix to warn the user (instead of skipping the > checksum-check). Skipping parity-checks is unsafe. The code now fails the > copy, and suggests that the user either use -pb to preserve block-size, or > consider -skipCrc (and forgo copy validation entirely). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5015) Coverage fix for org.apache.hadoop.mapreduce.tools.CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634000#comment-13634000 ] Hudson commented on MAPREDUCE-5015: --- Integrated in Hadoop-Hdfs-trunk #1375 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1375/]) MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI (Aleksey Gorshkov via tgraves) (Revision 1468483) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468483 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/job_1329348432655_0001-10.jhist > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > -- > > Key: MAPREDUCE-5015 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5015 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5015-branch-0.23-a.patch, > MAPREDUCE-5015-branch-0.23-b.patch, MAPREDUCE-5015-branch-0.23.patch, > MAPREDUCE-5015-branch-2-a.patch, MAPREDUCE-5015-branch-2-b.patch, > MAPREDUCE-5015-branch-2.patch, MAPREDUCE-5015-trunk-a.patch, > MAPREDUCE-5015-trunk-b.patch, MAPREDUCE-5015-trunk.patch > > > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > MAPREDUCE-5015-trunk.patch patch for trunk > MAPREDUCE-5015-branch-2.patch for branch-2 > MAPREDUCE-5015-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5065) DistCp should skip checksum comparisons if block-sizes are different on source/target.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633999#comment-13633999 ] Hudson commented on MAPREDUCE-5065: --- Integrated in Hadoop-Hdfs-trunk #1375 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1375/]) MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes are different on source/target. Contributed by Mithun Radhakrishnan. (Revision 1468629) Result = FAILURE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468629 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java > DistCp should skip checksum comparisons if block-sizes are different on > source/target. > -- > > Key: MAPREDUCE-5065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5065 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5065.branch-0.23.patch, > MAPREDUCE-5065.branch-2.patch > > > When copying files between 2 clusters with different default block-sizes, one > sees that the copy fails with a checksum-mismatch, even though the files have > identical contents. > The reason is that on HDFS, a file's checksum is unfortunately a function of > the block-size of the file. So you could have 2 different files with > identical contents (but different block-sizes) have different checksums. > (Thus, it's also possible for DistCp to fail to copy files on the same > file-system, if the source-file's block-size differs from HDFS default, and > -pb isn't used.) > I propose that we skip checksum comparisons under the following conditions: > 1. -skipCrc is specified. > 2. File-size is 0 (in which case the call to the checksum-servlet is moot). > 3. source.getBlockSize() != target.getBlockSize(), since the checksums are > guaranteed to differ in this case. > I have a patch for #3. > Edit: I've modified the fix to warn the user (instead of skipping the > checksum-check). Skipping parity-checks is unsafe. The code now fails the > copy, and suggests that the user either use -pb to preserve block-size, or > consider -skipCrc (and forgo copy validation entirely). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4974) Optimising the LineRecordReader initialize() method
[ https://issues.apache.org/jira/browse/MAPREDUCE-4974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sachin Jose updated MAPREDUCE-4974: --- Affects Version/s: (was: 0.23.5) > Optimising the LineRecordReader initialize() method > --- > > Key: MAPREDUCE-4974 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4974 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2, performance >Affects Versions: 2.0.2-alpha > Environment: Hadoop Linux >Reporter: Arun A K >Assignee: Gelesh > Labels: patch, performance > Fix For: trunk, 2.0.5-beta > > Attachments: MAPREDUCE-4974.2.patch, MAPREDUCE-4974.3.patch, > MAPREDUCE-4974.4.patch, MAPREDUCE-4974.5.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > I found there is a a scope of optimizing the code, over initialize() if we > have compressionCodecs & codec instantiated only if its a compressed input. > Mean while Gelesh George Omathil, added if we could avoid the null check of > key & value. This would time save, since for every next key value generation, > null check is done. The intention being to instantiate only once and avoid > NPE as well. Hope both could be met if initialize key & value over > initialize() method. We both have worked on it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5015) Coverage fix for org.apache.hadoop.mapreduce.tools.CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633988#comment-13633988 ] Hudson commented on MAPREDUCE-5015: --- Integrated in Hadoop-Hdfs-0.23-Build #584 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/584/]) MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI (Aleksey Gorshkov via tgraves) (Revision 1468485) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468485 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/job_1329348432655_0001-10.jhist > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > -- > > Key: MAPREDUCE-5015 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5015 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5015-branch-0.23-a.patch, > MAPREDUCE-5015-branch-0.23-b.patch, MAPREDUCE-5015-branch-0.23.patch, > MAPREDUCE-5015-branch-2-a.patch, MAPREDUCE-5015-branch-2-b.patch, > MAPREDUCE-5015-branch-2.patch, MAPREDUCE-5015-trunk-a.patch, > MAPREDUCE-5015-trunk-b.patch, MAPREDUCE-5015-trunk.patch > > > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > MAPREDUCE-5015-trunk.patch patch for trunk > MAPREDUCE-5015-branch-2.patch for branch-2 > MAPREDUCE-5015-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5065) DistCp should skip checksum comparisons if block-sizes are different on source/target.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633987#comment-13633987 ] Hudson commented on MAPREDUCE-5065: --- Integrated in Hadoop-Hdfs-0.23-Build #584 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/584/]) MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes are different on source/target. Contributed by Mithun Radhakrishnan. (Revision 1468636) Result = UNSTABLE kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468636 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java * /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java > DistCp should skip checksum comparisons if block-sizes are different on > source/target. > -- > > Key: MAPREDUCE-5065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5065 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5065.branch-0.23.patch, > MAPREDUCE-5065.branch-2.patch > > > When copying files between 2 clusters with different default block-sizes, one > sees that the copy fails with a checksum-mismatch, even though the files have > identical contents. > The reason is that on HDFS, a file's checksum is unfortunately a function of > the block-size of the file. So you could have 2 different files with > identical contents (but different block-sizes) have different checksums. > (Thus, it's also possible for DistCp to fail to copy files on the same > file-system, if the source-file's block-size differs from HDFS default, and > -pb isn't used.) > I propose that we skip checksum comparisons under the following conditions: > 1. -skipCrc is specified. > 2. File-size is 0 (in which case the call to the checksum-servlet is moot). > 3. source.getBlockSize() != target.getBlockSize(), since the checksums are > guaranteed to differ in this case. > I have a patch for #3. > Edit: I've modified the fix to warn the user (instead of skipping the > checksum-check). Skipping parity-checks is unsafe. The code now fails the > copy, and suggests that the user either use -pb to preserve block-size, or > consider -skipCrc (and forgo copy validation entirely). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[
https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633962#comment-13633962
]
Hadoop QA commented on MAPREDUCE-4502:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579117/design_v3.pdf
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3533//console
This message is automatically generated.
> Multi-level aggregation with combining the result of maps per node/rack
> ---
>
> Key: MAPREDUCE-4502
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster, mrv2
>Affects Versions: 3.0.0
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: design_v2.pdf, design_v3.pdf, MAPREDUCE-4502.1.patch,
> MAPREDUCE-4502.2.patch, MAPREDUCE-4502.3.patch, MAPREDUCE-4502.4.patch,
> MAPREDUCE-4502.5.patch, MAPREDUCE-4502.6.patch, MAPREDUCE-4525-pof.diff,
> speculative_draft.pdf
>
>
> The shuffle costs is expensive in Hadoop in spite of the existence of
> combiner, because the scope of combining is limited within only one MapTask.
> To solve this problem, it's a good way to aggregate the result of maps per
> node/rack by launch combiner.
> This JIRA is to implement the multi-level aggregation infrastructure,
> including combining per container(MAPREDUCE-3902 is related), coordinating
> containers by application master without breaking fault tolerance of jobs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated MAPREDUCE-4502: -- Attachment: design_v3.pdf Updated design note including benchmark of node-level aggregation. This revision dedicates only to explain node-level aggregation design and improvement of performance. The rack-level aggregation still needs big change, so I'd like to discuss in the other ticket. > Multi-level aggregation with combining the result of maps per node/rack > --- > > Key: MAPREDUCE-4502 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, mrv2 >Affects Versions: 3.0.0 >Reporter: Tsuyoshi OZAWA >Assignee: Tsuyoshi OZAWA > Attachments: design_v2.pdf, design_v3.pdf, MAPREDUCE-4502.1.patch, > MAPREDUCE-4502.2.patch, MAPREDUCE-4502.3.patch, MAPREDUCE-4502.4.patch, > MAPREDUCE-4502.5.patch, MAPREDUCE-4502.6.patch, MAPREDUCE-4525-pof.diff, > speculative_draft.pdf > > > The shuffle costs is expensive in Hadoop in spite of the existence of > combiner, because the scope of combining is limited within only one MapTask. > To solve this problem, it's a good way to aggregate the result of maps per > node/rack by launch combiner. > This JIRA is to implement the multi-level aggregation infrastructure, > including combining per container(MAPREDUCE-3902 is related), coordinating > containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5015) Coverage fix for org.apache.hadoop.mapreduce.tools.CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633947#comment-13633947 ] Hudson commented on MAPREDUCE-5015: --- Integrated in Hadoop-Yarn-trunk #186 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/186/]) MAPREDUCE-5015. Coverage fix for org.apache.hadoop.mapreduce.tools.CLI (Aleksey Gorshkov via tgraves) (Revision 1468483) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468483 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/resources/job_1329348432655_0001-10.jhist > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > -- > > Key: MAPREDUCE-5015 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5015 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.5 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5015-branch-0.23-a.patch, > MAPREDUCE-5015-branch-0.23-b.patch, MAPREDUCE-5015-branch-0.23.patch, > MAPREDUCE-5015-branch-2-a.patch, MAPREDUCE-5015-branch-2-b.patch, > MAPREDUCE-5015-branch-2.patch, MAPREDUCE-5015-trunk-a.patch, > MAPREDUCE-5015-trunk-b.patch, MAPREDUCE-5015-trunk.patch > > > Coverage fix for org.apache.hadoop.mapreduce.tools.CLI > MAPREDUCE-5015-trunk.patch patch for trunk > MAPREDUCE-5015-branch-2.patch for branch-2 > MAPREDUCE-5015-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5065) DistCp should skip checksum comparisons if block-sizes are different on source/target.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633946#comment-13633946 ] Hudson commented on MAPREDUCE-5065: --- Integrated in Hadoop-Yarn-trunk #186 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/186/]) MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes are different on source/target. Contributed by Mithun Radhakrishnan. (Revision 1468629) Result = SUCCESS kihwal : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1468629 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java * /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java > DistCp should skip checksum comparisons if block-sizes are different on > source/target. > -- > > Key: MAPREDUCE-5065 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5065 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 2.0.3-alpha, 0.23.5 >Reporter: Mithun Radhakrishnan >Assignee: Mithun Radhakrishnan > Fix For: 3.0.0, 2.0.5-beta, 0.23.8 > > Attachments: MAPREDUCE-5065.branch-0.23.patch, > MAPREDUCE-5065.branch-2.patch > > > When copying files between 2 clusters with different default block-sizes, one > sees that the copy fails with a checksum-mismatch, even though the files have > identical contents. > The reason is that on HDFS, a file's checksum is unfortunately a function of > the block-size of the file. So you could have 2 different files with > identical contents (but different block-sizes) have different checksums. > (Thus, it's also possible for DistCp to fail to copy files on the same > file-system, if the source-file's block-size differs from HDFS default, and > -pb isn't used.) > I propose that we skip checksum comparisons under the following conditions: > 1. -skipCrc is specified. > 2. File-size is 0 (in which case the call to the checksum-servlet is moot). > 3. source.getBlockSize() != target.getBlockSize(), since the checksums are > guaranteed to differ in this case. > I have a patch for #3. > Edit: I've modified the fix to warn the user (instead of skipping the > checksum-check). Skipping parity-checks is unsafe. The code now fails the > copy, and suggests that the user either use -pb to preserve block-size, or > consider -skipCrc (and forgo copy validation entirely). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5108) Review support for Binary Compatibility for MR applications via YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-5108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633897#comment-13633897 ] Steve Loughran commented on MAPREDUCE-5108: --- I got a stack trace trying to submit a pig job; the submission API has changed from the pig 0.10 library > Review support for Binary Compatibility for MR applications via YARN > > > Key: MAPREDUCE-5108 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5108 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.3-alpha >Reporter: Arun C Murthy >Assignee: Zhijie Shen >Priority: Blocker > Fix For: 2.0.5-beta > > Attachments: Binary Backward Compatibility.pdf, > mr1_mr2_api_diff.tar.gz > > > As we get ready to ship out a beta/stable version of hadoop-2, it makes sense > to spend time reviewing support for existing MR applications (hadoop-1) to > migrate seamlessly. > We've done various pieces of work over time, let's track progress and > document things clearly. [~zjshen] has done a bunch of testing and results > look very promising so far. > The aim is to support applications using org.apache.hadoop.mapred.* api in a > binary compatible manner in hadoop-2 - thus, users can just take existing MR > applications jars, point them at YARN clusters and things just work. > Clearly, we might have some corner cases (haven't seen many so far), > including semantics (not just apis); however the intent is to, at least, > document them throughly if not actually fix them as feasible. > Also, it's clear that we will *not* be able to support > org.apache.hadoop.mapreduce api in a *binary compatible* manner due to the > interface changes we made in hadoop-0.21 (sigh), and hence, users using the > _new_ apis will have to re-compile (i.e. source compatible only). > Net, given that vast majority of users use the org.apache.hadoop.mapred api, > it's a very reasonable way to ease migration to hadoop-2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633886#comment-13633886
]
Hadoop QA commented on MAPREDUCE-4443:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579104/MAPREDUCE-4443-trunk-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3532//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3532//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3532//console
This message is automatically generated.
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.
[jira] [Updated] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5156: --- Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-5108 > Hadoop-examples-1.x.x.jar cannot run on Yarn > > > Key: MAPREDUCE-5156 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > M/R examples are run through ProgramDriver.driver. ProgramDriver.driver > returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the > function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5156) Hadoop-examples-1.x.x.jar cannot run on Yarn
Zhijie Shen created MAPREDUCE-5156: -- Summary: Hadoop-examples-1.x.x.jar cannot run on Yarn Key: MAPREDUCE-5156 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5156 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Zhijie Shen Assignee: Zhijie Shen M/R examples are run through ProgramDriver.driver. ProgramDriver.driver returns void in hadoop-1, while it returns int in hadoop-2. Therefore, the function signatures in the example jar and yarn are incompatible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
[ https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633849#comment-13633849 ] Andrey Klochkov commented on MAPREDUCE-4980: The failure is expected due to dependency on HDFS-4491 > Parallel test execution of hadoop-mapreduce-client-core > --- > > Key: MAPREDUCE-4980 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0 >Reporter: Tsuyoshi OZAWA >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch, > MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch > > > The maven surefire plugin supports parallel testing feature. By using it, the > tests can be run more faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Status: Patch Available (was: Open)
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..2012-07-12
> 17:31:43,503 INFO [Thread-1]
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Attachment: MAPREDUCE-4443-trunk-2.patch
Attaching patch
Thanks,
Mayank
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting,
[jira] [Updated] (MAPREDUCE-4443) MR AM and job history server should be resilient to jobs that exceed counter limits
[
https://issues.apache.org/jira/browse/MAPREDUCE-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mayank Bansal updated MAPREDUCE-4443:
-
Status: Open (was: Patch Available)
> MR AM and job history server should be resilient to jobs that exceed counter
> limits
>
>
> Key: MAPREDUCE-4443
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4443
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Rahul Jain
>Assignee: Mayank Bansal
> Labels: usability
> Attachments: am_failed_counter_limits.txt,
> MAPREDUCE-4443-trunk-1.patch, MAPREDUCE-4443-trunk-2.patch,
> MAPREDUCE-4443-trunk-draft.patch
>
>
> We saw this problem migrating applications to MapReduceV2:
> Our applications use hadoop counters extensively (1000+ counters for certain
> jobs). While this may not be one of recommended best practices in hadoop, the
> real issue here is reliability of the framework when applications exceed
> counter limits.
> The hadoop servers (yarn, history server) were originally brought up with
> mapreduce.job.counters.max=1000 under core-site.xml
> We then ran map-reduce job under an application using its own job specific
> overrides, with mapreduce.job.counters.max=1
> All the tasks for the job finished successfully; however the overall job
> still failed due to AM encountering exceptions as:
> {code}
> 2012-07-12 17:31:43,485 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks
> : 712012-07-12 17:31:43,502 FATAL [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher threa
> dorg.apache.hadoop.mapreduce.counters.LimitExceededException: Too many
> counters: 1001 max=1000
> at
> org.apache.hadoop.mapreduce.counters.Limits.checkCounters(Limits.java:58)
>at org.apache.hadoop.mapreduce.counters.Limits.incrCounters(Limits.java:65)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounter(AbstractCounterGroup.java:77)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.addCounterImpl(AbstractCounterGroup.java:94)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.findCounter(AbstractCounterGroup.java:105)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounterGroup.incrAllCounters(AbstractCounterGroup.java:202)
> at
> org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:337)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.constructFinalFullcounters(JobImpl.java:1212)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.mayBeConstructFinalFullCounters(JobImpl.java:1198)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.createJobFinishedEvent(JobImpl.java:1179)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.logJobHistoryFinishedEvent(JobImpl.java:711)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.checkJobCompleteSuccess(JobImpl.java:737)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.checkJobForCompletion(JobImpl.java:1360)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1340)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl$TaskCompletedTransition.transition(JobImpl.java:1323)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:380)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:666)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl.handle(JobImpl.java:113)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:890)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher.handle(MRAppMaster.java:886)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:125)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:74)
> at java.lang.Thread.run(Thread.java:662)
> 2012-07-12 17:31:43,502 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..2012-07-12
> 17:31:43,503 INFO [Thread-1]
[jira] [Commented] (MAPREDUCE-4980) Parallel test execution of hadoop-mapreduce-client-core
[
https://issues.apache.org/jira/browse/MAPREDUCE-4980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13633846#comment-13633846
]
Hadoop QA commented on MAPREDUCE-4980:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579103/MAPREDUCE-4980--n4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 119
new or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3531//console
This message is automatically generated.
> Parallel test execution of hadoop-mapreduce-client-core
> ---
>
> Key: MAPREDUCE-4980
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4980
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0
>Reporter: Tsuyoshi OZAWA
>Assignee: Tsuyoshi OZAWA
> Attachments: MAPREDUCE-4980.1.patch, MAPREDUCE-4980--n3.patch,
> MAPREDUCE-4980--n4.patch, MAPREDUCE-4980.patch
>
>
> The maven surefire plugin supports parallel testing feature. By using it, the
> tests can be run more faster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
