[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719194#comment-13719194 ] Avner BenHanoch commented on MAPREDUCE-4049: Thanks Alejandro, You are faster than a rocket! Avner > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: performance, task, tasktracker >Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0 >Reporter: Avner BenHanoch >Assignee: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Fix For: 2.0.3-alpha, 1.3.0 > > Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, > MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, > MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) > # I am providing link for downloading UDA - Mellanox's open source plugin > that implements generic shuffle service using RDMA and levitated merge. > Note: At this phase, the code is in C++ through JNI and you should consider > it as beta only. Still, it can serve anyone that wants to implement or > contribute to levitated merge. (Please be advised that levitated merge is > mostly suit in very fast networks) - > [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719166#comment-13719166 ] Jason Lowe commented on MAPREDUCE-5386: --- Patch looks pretty good overall. Couple of comments: * Was there an intent for a more comprehensive test? I see a lot of refactoring for methods that seem unrelated to this patch otherwise (e.g.: scanDirectoryForHistoryFilesWrapper, deleteDir, etc.) * scheduleHistoryCleaner should either always set futureHistoryCleaner (even if that means setting it to null for no cleaner) or we need to set it to null after cancelling it. * Nits: some lines are over 80 columns, arg lists without whitespace between args, and some methods were added without whitespace between it and the subsequent method. > Refresh job retention time,job cleaner interval, enable/disable cleaner > --- > > Key: MAPREDUCE-5386 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver >Affects Versions: 2.1.0-beta >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Labels: features > Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt, > JOB_RETENTION-3.txt, JOB_RETENTION-4.txt > > > We want to be able to refresh following job retention parameters > without having to bounce the history server : > 1. Job retention time - mapreduce.jobhistory.max-age-ms > 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms > 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5411: -- Status: Open (was: Patch Available) The patch has gone stale, could you refresh it? Also a few comments: * I think the refresh protocol should have a way of conveying success/failure back to clients. A boolean result would be a minimum, or potentially throwing an exception if the refresh fails with details on the failure. That would allow the CLI to report a proper exit code if the refresh did not succeed. * On a related note, the refresh should note it failed to the caller if a CachedHistoryStorage is not being used. Currently it doesn't even log if that's the case and just silently proceeds. > Refresh size of loaded job cache on history server > -- > > Key: MAPREDUCE-5411 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver >Affects Versions: 2.1.0-beta >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Labels: features > Attachments: LOADED_JOB_CACHE_MR5411-1.txt > > > We want to be able to refresh size of the loaded job > cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server > through history server's admin interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719145#comment-13719145
]
Hadoop QA commented on MAPREDUCE-5414:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594080/MAPREDUCE-5414.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3898//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3898//console
This message is automatically generated.
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch, MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43
[jira] [Commented] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719142#comment-13719142 ] Jason Lowe commented on MAPREDUCE-5251: --- THanks Ashwin. I think the patch is almost there, but I noticed that when we log an error we don't say anything at all about the error itself -- we probably should at least log the .getMessage() of the error if we're going to bother logging there was an error. Also the handling of the unknown host error text is somewhat misleading -- one could interpret the "unknown" referring to the local error that occurred rather than the fact it couldn't lookup the node name. > Reducer should not implicate map attempt if it has insufficient space to > fetch map output > - > > Key: MAPREDUCE-5251 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Jason Lowe >Assignee: Ashwin Shankar > Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt, > MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt, MAPREDUCE-5251-6.txt > > > A job can fail if a reducer happens to run on a node with insufficient space > to hold a map attempt's output. The reducer keeps reporting the map attempt > as bad, and if the map attempt ends up being re-launched too many times > before the reducer decides maybe it is the real problem the job can fail. > In that scenario it would be better to re-launch the reduce attempt and > hopefully it will run on another node that has sufficient space to complete > the shuffle. Reporting the map attempt is bad and relaunching the map task > doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nemon lou updated MAPREDUCE-5414:
-
Status: Patch Available (was: Open)
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch, MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4P
[jira] [Updated] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nemon lou updated MAPREDUCE-5414:
-
Attachment: MAPREDUCE-5414.patch
Updating patch as Ravi Prakash suggested.Thanks for your advise.
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch, MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
[jira] [Updated] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nemon lou updated MAPREDUCE-5414:
-
Status: Open (was: Patch Available)
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>
[jira] [Commented] (MAPREDUCE-5417) Propagate task settings to AM when job runs in uber mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719124#comment-13719124
]
Jason Lowe commented on MAPREDUCE-5417:
---
bq. When the settings for map/reduce tasks conflict, it should be okay to set
to the maximum value.
For heap/container settings I agree this will work for most cases, but what if
the environment variable settings are not identical? e.g.: passing environment
variables to maps but not reduces or worse, passing different values for the
same variable to maps vs. reduces.
I suspect when there are conflicts in map vs. reduce task settings we have to
not run the job as uber or have some hybrid where we run the maps within the AM
and the reduces as separate tasks.
> Propagate task settings to AM when job runs in uber mode
>
>
> Key: MAPREDUCE-5417
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5417
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Jason Lowe
>
> When a job runs in Uber-AM mode the JVM settings (e.g.: heap sizing),
> container sizing, and environment variables are not propagated to the AM.
> This means running in uber-AM mode is not as simple as enabling
> {{mapreduce.job.ubertask.enable}} since lack of proper heap/container
> settings can doom the AM to crash or lack of proper environment variables can
> prevent the task code from properly running.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5385) JobContext cache files api are broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5385: - Affects Version/s: 2.1.0-beta > JobContext cache files api are broken > - > > Key: MAPREDUCE-5385 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5385 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: YARN-916-20130712.1.patch > > > I just checked there are issues with latest distributed cache api. > * JobContext.getCacheFiles is broken returns null. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5385) JobContext cache files api are broken
[ https://issues.apache.org/jira/browse/MAPREDUCE-5385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5385: - Priority: Blocker (was: Major) > JobContext cache files api are broken > - > > Key: MAPREDUCE-5385 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5385 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi >Priority: Blocker > Attachments: YARN-916-20130712.1.patch > > > I just checked there are issues with latest distributed cache api. > * JobContext.getCacheFiles is broken returns null. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5418) JobHistoryServer has no information about applications if the MR-AM crashes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719044#comment-13719044 ] Alejandro Abdelnur commented on MAPREDUCE-5418: --- dont think you'll get counters in case of MRAM crash. I think what we could craft for the jobhistory entry like: * user (from job.xml?) * job.xml * start time (aprox by using job.xml file timestamp from the localized file) * end time (time the container gone event arrives to the auxiliary service) * queue (from job.xml) * all counters to zero * no known tasks * AM logs Not sure there is a better way of doing this. > JobHistoryServer has no information about applications if the MR-AM crashes > --- > > Key: MAPREDUCE-5418 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5418 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mr-am >Affects Versions: 2.0.5-alpha >Reporter: Karthik Kambatla >Priority: Critical > Attachments: JHS-web-ui.png, RM-web-ui.png > > > Currently, the AM writes the job-specific information to HDFS only after it > finishes; the JHS needs this info to display anything. If the AM fails, this > info is not written and the JHS fails to display anything for that job. > While JHS on top of AHS might address this issue, it would be nice to have a > solution in the interim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5418) JobHistoryServer has no information about applications if the MR-AM crashes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719018#comment-13719018 ] Karthik Kambatla commented on MAPREDUCE-5418: - Possible approach: - An auxiliary service on the NM can write diagnostic information, container logs and other info (like Counters) to a pre-defined directory in HDFS. The JHS can subsequently read this information, in the same way it reads information from completed applications. > JobHistoryServer has no information about applications if the MR-AM crashes > --- > > Key: MAPREDUCE-5418 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5418 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mr-am >Affects Versions: 2.0.5-alpha >Reporter: Karthik Kambatla >Priority: Critical > Attachments: JHS-web-ui.png, RM-web-ui.png > > > Currently, the AM writes the job-specific information to HDFS only after it > finishes; the JHS needs this info to display anything. If the AM fails, this > info is not written and the JHS fails to display anything for that job. > While JHS on top of AHS might address this issue, it would be nice to have a > solution in the interim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5418) JobHistoryServer has no information about applications if the MR-AM crashes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5418: Attachment: RM-web-ui.png JHS-web-ui.png To reproduce this, ran sleep job thrice. Killed the AM for the second job. As in the attached screen shots, the second job is missing in the JHS UI. > JobHistoryServer has no information about applications if the MR-AM crashes > --- > > Key: MAPREDUCE-5418 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5418 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mr-am >Affects Versions: 2.0.5-alpha >Reporter: Karthik Kambatla >Priority: Critical > Attachments: JHS-web-ui.png, RM-web-ui.png > > > Currently, the AM writes the job-specific information to HDFS only after it > finishes; the JHS needs this info to display anything. If the AM fails, this > info is not written and the JHS fails to display anything for that job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5418) JobHistoryServer has no information about applications if the MR-AM crashes
[ https://issues.apache.org/jira/browse/MAPREDUCE-5418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5418: Description: Currently, the AM writes the job-specific information to HDFS only after it finishes; the JHS needs this info to display anything. If the AM fails, this info is not written and the JHS fails to display anything for that job. While JHS on top of AHS might address this issue, it would be nice to have a solution in the interim. was: Currently, the AM writes the job-specific information to HDFS only after it finishes; the JHS needs this info to display anything. If the AM fails, this info is not written and the JHS fails to display anything for that job. > JobHistoryServer has no information about applications if the MR-AM crashes > --- > > Key: MAPREDUCE-5418 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5418 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mr-am >Affects Versions: 2.0.5-alpha >Reporter: Karthik Kambatla >Priority: Critical > Attachments: JHS-web-ui.png, RM-web-ui.png > > > Currently, the AM writes the job-specific information to HDFS only after it > finishes; the JHS needs this info to display anything. If the AM fails, this > info is not written and the JHS fails to display anything for that job. > While JHS on top of AHS might address this issue, it would be nice to have a > solution in the interim. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5418) JobHistoryServer has no information about applications if the MR-AM crashes
Karthik Kambatla created MAPREDUCE-5418: --- Summary: JobHistoryServer has no information about applications if the MR-AM crashes Key: MAPREDUCE-5418 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5418 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mr-am Affects Versions: 2.0.5-alpha Reporter: Karthik Kambatla Priority: Critical Currently, the AM writes the job-specific information to HDFS only after it finishes; the JHS needs this info to display anything. If the AM fails, this info is not written and the JHS fails to display anything for that job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5417) Propagate task settings to AM when job runs in uber mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719008#comment-13719008
]
Karthik Kambatla commented on MAPREDUCE-5417:
-
When the settings for map/reduce tasks conflict, it should be okay to set to
the maximum value.
> Propagate task settings to AM when job runs in uber mode
>
>
> Key: MAPREDUCE-5417
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5417
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Jason Lowe
>
> When a job runs in Uber-AM mode the JVM settings (e.g.: heap sizing),
> container sizing, and environment variables are not propagated to the AM.
> This means running in uber-AM mode is not as simple as enabling
> {{mapreduce.job.ubertask.enable}} since lack of proper heap/container
> settings can doom the AM to crash or lack of proper environment variables can
> prevent the task code from properly running.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13719000#comment-13719000 ] Alejandro Abdelnur commented on MAPREDUCE-4049: --- Thanks Avner. Committed to branch-1. > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: performance, task, tasktracker >Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0 >Reporter: Avner BenHanoch >Assignee: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Fix For: 2.0.3-alpha > > Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, > MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, > MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) > # I am providing link for downloading UDA - Mellanox's open source plugin > that implements generic shuffle service using RDMA and levitated merge. > Note: At this phase, the code is in C++ through JNI and you should consider > it as beta only. Still, it can serve anyone that wants to implement or > contribute to levitated merge. (Please be advised that levitated merge is > mostly suit in very fast networks) - > [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-4049: -- Fix Version/s: 1.3.0 > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: performance, task, tasktracker >Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0 >Reporter: Avner BenHanoch >Assignee: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Fix For: 2.0.3-alpha, 1.3.0 > > Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, > MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, > MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) > # I am providing link for downloading UDA - Mellanox's open source plugin > that implements generic shuffle service using RDMA and levitated merge. > Note: At this phase, the code is in C++ through JNI and you should consider > it as beta only. Still, it can serve anyone that wants to implement or > contribute to levitated merge. (Please be advised that levitated merge is > mostly suit in very fast networks) - > [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718998#comment-13718998 ] Alejandro Abdelnur commented on MAPREDUCE-4049: --- [~avnerb], never mind, seems some whitespace issue, just run all testcases successfully (no test-patch for branch-1). Committing momentarily. > plugin for generic shuffle service > -- > > Key: MAPREDUCE-4049 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: performance, task, tasktracker >Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0 >Reporter: Avner BenHanoch >Assignee: Avner BenHanoch > Labels: merge, plugin, rdma, shuffle > Fix For: 2.0.3-alpha > > Attachments: HADOOP-1.x.y.patch, Hadoop Shuffle Plugin Design.rtf, > MAPREDUCE-4049--branch-1.patch, MAPREDUCE-4049--branch-1.patch, > MAPREDUCE-4049--branch-1.patch, mapreduce-4049.patch > > > Support generic shuffle service as set of two plugins: ShuffleProvider & > ShuffleConsumer. > This will satisfy the following needs: > # Better shuffle and merge performance. For example: we are working on > shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, > or Infiniband) instead of using the current HTTP shuffle. Based on the fast > RDMA shuffle, the plugin can also utilize a suitable merge approach during > the intermediate merges. Hence, getting much better performance. > # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden > dependency of NodeManager with a specific version of mapreduce shuffle > (currently targeted to 0.24.0). > References: > # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu > from Auburn University with others, > [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] > # I am attaching 2 documents with suggested Top Level Design for both plugins > (currently, based on 1.0 branch) > # I am providing link for downloading UDA - Mellanox's open source plugin > that implements generic shuffle service using RDMA and levitated merge. > Note: At this phase, the code is in C++ through JNI and you should consider > it as beta only. Still, it can serve anyone that wants to implement or > contribute to levitated merge. (Please be advised that levitated merge is > mostly suit in very fast networks) - > [http://www.mellanox.com/content/pages.php?pg=products_dyn&product_family=144&menu_section=69] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5213) Re-assess TokenCache methods marked @Private
[ https://issues.apache.org/jira/browse/MAPREDUCE-5213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-5213: -- Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Karthik. Committed to trunk, branch-2 and branch-2.1. > Re-assess TokenCache methods marked @Private > > > Key: MAPREDUCE-5213 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5213 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla >Priority: Minor > Fix For: 2.1.0-beta > > Attachments: mr-5213-1.patch, mr-5213-2.patch > > > While looking at the source, noticed that TokenCache#loadTokens methods are > marked @Private but not used anywhere. > We should either remove those methods or mark them Public or LimitedPrivate. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5288) ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues
[
https://issues.apache.org/jira/browse/MAPREDUCE-5288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alejandro Abdelnur updated MAPREDUCE-5288:
--
Resolution: Fixed
Fix Version/s: 1.3.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks Karthik. Thanks Harsh for reviewing it. Committed to branch-1.
> ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero
> issues
> ---
>
> Key: MAPREDUCE-5288
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5288
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1
>Affects Versions: 1.2.0
>Reporter: Harsh J
>Assignee: Karthik Kambatla
> Fix For: 1.3.0
>
> Attachments: mr-5288-1.patch
>
>
> The computation in the above mentioned class-method is below:
> {code}
> long estimate = Math.round(((double)inputSize *
> completedMapsOutputSize * 2.0)/completedMapsInputSize);
> {code}
> Given
> http://docs.oracle.com/javase/6/docs/api/java/lang/Math.html#round(double),
> its possible that the returned estimate could be Long.MAX_VALUE if
> completedMapsInputSize is determined to be zero.
> This can be proven with a simple code snippet:
> {code}
> class Foo {
> public static void main(String... args) {
> long inputSize = 600L + 2;
> long estimate = Math.round(((double)inputSize *
> 1L * 2.0)/0L);
> System.out.println(estimate);
> }
> }
> {code}
> The above conveniently prints out: {{9223372036854775807}}, which is
> Long.MAX_VALUE (or 8 Exbibytes per MapReduce).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5417) Propagate task settings to AM when job runs in uber mode
[
https://issues.apache.org/jira/browse/MAPREDUCE-5417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718883#comment-13718883
]
Jason Lowe commented on MAPREDUCE-5417:
---
There's some issues with what to do if properties conflict (e.g.: different
environment settings for map vs. reduce tasks), but it would be nice if in the
common case the client code could make some smart decisions re: AM settings so
only enabling uber mode would "just work".
> Propagate task settings to AM when job runs in uber mode
>
>
> Key: MAPREDUCE-5417
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5417
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 2.0.4-alpha
>Reporter: Jason Lowe
>
> When a job runs in Uber-AM mode the JVM settings (e.g.: heap sizing),
> container sizing, and environment variables are not propagated to the AM.
> This means running in uber-AM mode is not as simple as enabling
> {{mapreduce.job.ubertask.enable}} since lack of proper heap/container
> settings can doom the AM to crash or lack of proper environment variables can
> prevent the task code from properly running.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5417) Propagate task settings to AM when job runs in uber mode
Jason Lowe created MAPREDUCE-5417:
-
Summary: Propagate task settings to AM when job runs in uber mode
Key: MAPREDUCE-5417
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5417
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 2.0.4-alpha
Reporter: Jason Lowe
When a job runs in Uber-AM mode the JVM settings (e.g.: heap sizing), container
sizing, and environment variables are not propagated to the AM. This means
running in uber-AM mode is not as simple as enabling
{{mapreduce.job.ubertask.enable}} since lack of proper heap/container settings
can doom the AM to crash or lack of proper environment variables can prevent
the task code from properly running.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4366) mapred metrics shows negative count of waiting maps and reduces
[ https://issues.apache.org/jira/browse/MAPREDUCE-4366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718870#comment-13718870 ] Luke Lu commented on MAPREDUCE-4366: The patch looks reasonable to me as well. It'd be nice to get this into 1.3. > mapred metrics shows negative count of waiting maps and reduces > --- > > Key: MAPREDUCE-4366 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4366 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobtracker >Affects Versions: 1.0.2 >Reporter: Thomas Graves >Assignee: Sandy Ryza > Attachments: MAPREDUCE-4366-branch-1-1.patch, > MAPREDUCE-4366-branch-1.patch > > > Negative waiting_maps and waiting_reduces count is observed in the mapred > metrics. MAPREDUCE-1238 partially fixed this but it appears there is still > issues as we are seeing it, but not as bad. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5416) hadoop-mapreduce-client-common depends on hadoop-yarn-server-common
Hitesh Shah created MAPREDUCE-5416: -- Summary: hadoop-mapreduce-client-common depends on hadoop-yarn-server-common Key: MAPREDUCE-5416 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5416 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Hitesh Shah mapreduce-client-app and mapreduce-client-jobclient modules also depend on yarn-server-common but only in test scope. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[ https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718833#comment-13718833 ] Sandy Ryza commented on MAPREDUCE-5403: --- bq. why are we using a literal string instead of a programming friendly constant? Good point. Will add a constant in. bq. Why have we changed YARN_APPLICATION_CLASSPATH to NM_APPLICATION_CLASSPATH? The convention for configs that are used (only) by the NodeManager is NM (e.g. NM_HEALTH_CHECK_INTERVAL_MS, NM_LINUX_CONTAINER_EXECUTOR_PATH, etc.). Nearly all configs in this category will be the same across all nodes. bq. Regardless, is this a configuration for the yarn application or the NM? It is a configuration for the NM, specifying where the framework jars are located on the node. bq. Mixing NM_APPLICATION_CLASSPATH with YARN_APPLICATION_CLASSPATH_DEFAULT??? Ops my bad. > yarn.application.classpath requires client to know service internals > > > Key: MAPREDUCE-5403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5403-1.patch, MAPREDUCE-5403.patch > > > yarn.application.classpath is a confusing property because it is used by > MapReduce and not YARN, and MapReduce already has > mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[
https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718807#comment-13718807
]
Hadoop QA commented on MAPREDUCE-5403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12594004/MAPREDUCE-5403-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site:
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
org.apache.hadoop.mapreduce.security.TestMRCredentials
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3897//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3897//console
This message is automatically generated.
> yarn.application.classpath requires client to know service internals
>
>
> Key: MAPREDUCE-5403
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.0.5-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5403-1.patch, MAPREDUCE-5403.patch
>
>
> yarn.application.classpath is a confusing property because it is used by
> MapReduce and not YARN, and MapReduce already has
> mapreduce.application.classpath, which provides the same functionality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[ https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718806#comment-13718806 ] Bikas Saha commented on MAPREDUCE-5403: --- Jira also needs to be moved to YARN project and MR needs its own jira for MR changes. > yarn.application.classpath requires client to know service internals > > > Key: MAPREDUCE-5403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5403-1.patch, MAPREDUCE-5403.patch > > > yarn.application.classpath is a confusing property because it is used by > MapReduce and not YARN, and MapReduce already has > mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[
https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718797#comment-13718797
]
Bikas Saha commented on MAPREDUCE-5403:
---
why are we using a literal string instead of a programming friendly constant?
Literal string is error-prone and hard to maintain.
Why have we changed YARN_APPLICATION_CLASSPATH to NM_APPLICATION_CLASSPATH? The
config will be identical across NM's in the common case. Regardless, is this a
configuration for the yarn application or the NM?
Mixing NM_APPLICATION_CLASSPATH with YARN_APPLICATION_CLASSPATH_DEFAULT???
{code}
+classPathEnv.append(File.pathSeparatorChar)
+ .append("$YARN_APPLICATION_CLASSPATH");
{code}
{code}
- Map environment = launchContext.getEnvironment();
+ LinkedHashMap environment = new LinkedHashMap();
+ for (String c : conf.getStrings(
+ YarnConfiguration.NM_APPLICATION_CLASSPATH,
+ YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)) {
+Apps.addToEnvironment(environment, "YARN_APPLICATION_CLASSPATH", c
+.trim());
+ }
+ environment.putAll(launchContext.getEnvironment());
{code}
> yarn.application.classpath requires client to know service internals
>
>
> Key: MAPREDUCE-5403
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Affects Versions: 2.0.5-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5403-1.patch, MAPREDUCE-5403.patch
>
>
> yarn.application.classpath is a confusing property because it is used by
> MapReduce and not YARN, and MapReduce already has
> mapreduce.application.classpath, which provides the same functionality.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[ https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5403: -- Attachment: MAPREDUCE-5403-1.patch > yarn.application.classpath requires client to know service internals > > > Key: MAPREDUCE-5403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5403-1.patch, MAPREDUCE-5403.patch > > > yarn.application.classpath is a confusing property because it is used by > MapReduce and not YARN, and MapReduce already has > mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[ https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718744#comment-13718744 ] Sandy Ryza commented on MAPREDUCE-5403: --- Thanks for taking a look Jason. Uploading a patch that updates the documentation both in WritingYarnApplications and yarn-default.xml. I also changed the name of the config to yarn.nodemanager.application.classpath, because it's a NodeManager property. Will we need to deprecate the old one to the new one? > yarn.application.classpath requires client to know service internals > > > Key: MAPREDUCE-5403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5403.patch > > > yarn.application.classpath is a confusing property because it is used by > MapReduce and not YARN, and MapReduce already has > mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[
https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718704#comment-13718704
]
Hadoop QA commented on MAPREDUCE-5386:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593994/JOB_RETENTION-4.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3896//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3896//console
This message is automatically generated.
> Refresh job retention time,job cleaner interval, enable/disable cleaner
> ---
>
> Key: MAPREDUCE-5386
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobhistoryserver
>Affects Versions: 2.1.0-beta
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Labels: features
> Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt,
> JOB_RETENTION-3.txt, JOB_RETENTION-4.txt
>
>
> We want to be able to refresh following job retention parameters
> without having to bounce the history server :
> 1. Job retention time - mapreduce.jobhistory.max-age-ms
> 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms
> 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5386) Refresh job retention time,job cleaner interval, enable/disable cleaner
[ https://issues.apache.org/jira/browse/MAPREDUCE-5386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5386: -- Attachment: JOB_RETENTION-4.txt > Refresh job retention time,job cleaner interval, enable/disable cleaner > --- > > Key: MAPREDUCE-5386 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5386 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver >Affects Versions: 2.1.0-beta >Reporter: Ashwin Shankar >Assignee: Ashwin Shankar > Labels: features > Attachments: JOB_RETENTION-1.txt, JOB_RETENTION-2.txt, > JOB_RETENTION-3.txt, JOB_RETENTION-4.txt > > > We want to be able to refresh following job retention parameters > without having to bounce the history server : > 1. Job retention time - mapreduce.jobhistory.max-age-ms > 2. Cleaner interval - mapreduce.jobhistory.cleaner.interval-ms > 3. Enable/disable cleaner -mapreduce.jobhistory.cleaner.enable -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718592#comment-13718592 ] Alejandro Abdelnur commented on MAPREDUCE-5379: --- [~daryn], happy to have a call if you want to quickly this discuss this, then I'll summarize the offline discussions here. > Include FS delegation token ID in job conf > -- > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379.patch > > > Making a job's FS delegation token ID accessible will allow external services > to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718591#comment-13718591 ] Alejandro Abdelnur commented on MAPREDUCE-5379: --- +1 from my side, IMO [~daryn] concerns have been addressed, [~daryn]? > Include FS delegation token ID in job conf > -- > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379.patch > > > Making a job's FS delegation token ID accessible will allow external services > to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718565#comment-13718565 ] Sandy Ryza commented on MAPREDUCE-5379: --- I wasn't able to write a test case because security must be on for tokens to be fetched. I couldn't find any minicluster test cases that use security. (Am I missing them?). I verified the patch manually on a Kerberos cluster. > Include FS delegation token ID in job conf > -- > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379.patch > > > Making a job's FS delegation token ID accessible will allow external services > to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718408#comment-13718408
]
Ravi Prakash commented on MAPREDUCE-5414:
-
and you can probably just pass in new Token(), instead of naming a variable and
then passing it
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(J
[jira] [Commented] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718398#comment-13718398
]
Ravi Prakash commented on MAPREDUCE-5414:
-
I was seeing the same failures. Applying the patch fixes it for me. You have
just replaced the mocked objects with a real instance of Token. I am not
familiar with tokens much, but that should be fine. One nit is you have used
tabs for indenting the "TaskAttemptImpl taImpl =" line. Otherwise lgtm
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.ju
[jira] [Commented] (MAPREDUCE-5403) yarn.application.classpath requires client to know service internals
[ https://issues.apache.org/jira/browse/MAPREDUCE-5403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718390#comment-13718390 ] Jason Lowe commented on MAPREDUCE-5403: --- After looking at the patch, I tend to agree that this approach is probably better long-term for managing the classpath. It's easier to access in general and enables heterogeneous YARN deploys since the classpath is bound by the node handling the container instead of the client. Specifically I'm thinking of the rolling upgrades case where one node has the old YARN classpath layout and the new YARN version has a different one. This hides that difference nicely. I think the patch looks good overall, but we should document this somewhere like hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/WritingYarnApplications.apt.vm. > yarn.application.classpath requires client to know service internals > > > Key: MAPREDUCE-5403 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5403 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Affects Versions: 2.0.5-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5403.patch > > > yarn.application.classpath is a confusing property because it is used by > MapReduce and not YARN, and MapReduce already has > mapreduce.application.classpath, which provides the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5409) MRAppMaster throws InvalidStateTransitonException: Invalid event: TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl
[
https://issues.apache.org/jira/browse/MAPREDUCE-5409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718206#comment-13718206
]
Devaraj K commented on MAPREDUCE-5409:
--
Initially TaskAttemptImpl is SUCCEEDED. TaskAttemptImpl state moved from
SUCCEEDED to KILLED state with the reason as "Diagnostics report from
attempt_1374560536158_0003_m_07_0: Container released on a *lost* node".
After some time JobImpl is getting the JOB_TASK_ATTEMPT_FETCH_FAILURE for the
same task attempt and during transition it is triggering the
TA_TOO_MANY_FETCH_FAILURE for the TaskAttemptImpl which is causing for this
invalid transition.
I think here we should not raise the event TA_TOO_MANY_FETCH_FAILURE for task
attempt when the state is KILLED, or we can ignore this event at KILLED for
TaskAttemptImpl.
> MRAppMaster throws InvalidStateTransitonException: Invalid event:
> TA_TOO_MANY_FETCH_FAILURE at KILLED for TaskAttemptImpl
> -
>
> Key: MAPREDUCE-5409
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5409
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 2.0.5-alpha
>Reporter: Devaraj K
>Assignee: Devaraj K
>
> {code:xml}
> 2013-07-23 12:28:05,217 INFO [IPC Server handler 29 on 50796]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1374560536158_0003_m_40_0 is : 0.0
> 2013-07-23 12:28:05,221 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Too many fetch-failures
> for output of task attempt: attempt_1374560536158_0003_m_07_0 ... raising
> fetch failure to map
> 2013-07-23 12:28:05,222 ERROR [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Can't handle
> this event at current state for attempt_1374560536158_0003_m_07_0
> org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event:
> TA_TOO_MANY_FETCH_FAILURE at KILLED
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:445)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1032)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:143)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1123)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher.handle(MRAppMaster.java:1115)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:130)
> at
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
> at java.lang.Thread.run(Thread.java:662)
> 2013-07-23 12:28:05,249 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
> job_1374560536158_0003Job Transitioned from RUNNING to ERROR
> 2013-07-23 12:28:05,338 INFO [IPC Server handler 16 on 50796]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1374560536158_0003_m_40_0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5411) Refresh size of loaded job cache on history server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718175#comment-13718175
]
Hadoop QA commented on MAPREDUCE-5411:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593729/LOADED_JOB_CACHE_MR5411-1.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3895//console
This message is automatically generated.
> Refresh size of loaded job cache on history server
> --
>
> Key: MAPREDUCE-5411
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5411
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: jobhistoryserver
>Affects Versions: 2.1.0-beta
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Labels: features
> Attachments: LOADED_JOB_CACHE_MR5411-1.txt
>
>
> We want to be able to refresh size of the loaded job
> cache(mapreduce.jobhistory.loadedjobs.cache.size) of history server
> through history server's admin interface.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[
https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718161#comment-13718161
]
Hadoop QA commented on MAPREDUCE-5251:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593790/MAPREDUCE-5251-6.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3891//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3891//console
This message is automatically generated.
> Reducer should not implicate map attempt if it has insufficient space to
> fetch map output
> -
>
> Key: MAPREDUCE-5251
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.7, 2.0.4-alpha
>Reporter: Jason Lowe
>Assignee: Ashwin Shankar
> Attachments: MAPREDUCE-5251-2.txt, MAPREDUCE-5251-3.txt,
> MAPREDUCE-5251-4.txt, MAPREDUCE-5251-5.txt, MAPREDUCE-5251-6.txt
>
>
> A job can fail if a reducer happens to run on a node with insufficient space
> to hold a map attempt's output. The reducer keeps reporting the map attempt
> as bad, and if the map attempt ends up being re-launched too many times
> before the reducer decides maybe it is the real problem the job can fail.
> In that scenario it would be better to re-launch the reduce attempt and
> hopefully it will run on another node that has sufficient space to complete
> the shuffle. Reporting the map attempt is bad and relaunching the map task
> doesn't change the fact that the reducer can't hold the output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1981) Improve getSplits performance by using listFiles, the new FileSystem API
[
https://issues.apache.org/jira/browse/MAPREDUCE-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718160#comment-13718160
]
Hadoop QA commented on MAPREDUCE-1981:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593773/MAPREDUCE-1981.branch-0.23.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3893//console
This message is automatically generated.
> Improve getSplits performance by using listFiles, the new FileSystem API
>
>
> Key: MAPREDUCE-1981
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1981
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission
>Affects Versions: 0.23.0
>Reporter: Hairong Kuang
>Assignee: Hairong Kuang
> Attachments: mapredListFiles1.patch, mapredListFiles2.patch,
> mapredListFiles3.patch, mapredListFiles4.patch, mapredListFiles5.patch,
> mapredListFiles.patch, MAPREDUCE-1981.branch-0.23.patch, MAPREDUCE-1981.patch
>
>
> This jira will make FileInputFormat and CombinedFileInputForm to use the new
> API, thus reducing the number of RPCs to HDFS NameNode.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5367) Local jobs all use same local working directory
[
https://issues.apache.org/jira/browse/MAPREDUCE-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718144#comment-13718144
]
Hadoop QA commented on MAPREDUCE-5367:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593094/MAPREDUCE-5367-b1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3890//console
This message is automatically generated.
> Local jobs all use same local working directory
> ---
>
> Key: MAPREDUCE-5367
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5367
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
>Affects Versions: 1.2.0
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5367-b1.patch
>
>
> This means that local jobs, even in different JVMs, can't run concurrently
> because they might delete each other's files during work directory setup.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718143#comment-13718143
]
Hadoop QA commented on MAPREDUCE-5414:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12593885/MAPREDUCE-5414.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3889//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3889//console
This message is automatically generated.
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.la
[jira] [Updated] (MAPREDUCE-5415) LeaseRenewer throw java.util.ConcurrentModificationException when timeout
[ https://issues.apache.org/jira/browse/MAPREDUCE-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhaoyunjiong updated MAPREDUCE-5415: Attachment: MAPREDUCE-5415.patch This patch avoid to create iterator to prevent throw ConcurrentModificationException. > LeaseRenewer throw java.util.ConcurrentModificationException when timeout > - > > Key: MAPREDUCE-5415 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5415 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: zhaoyunjiong >Assignee: zhaoyunjiong > Attachments: MAPREDUCE-5415.patch > > > In LeaseRenewer, when renew() throw SocketTimeoutException, c.abort() will > remove one dfsclient from dfsclients. Here will throw a > ConcurrentModificationException because dfsclients changed after the iterator > created by "for(DFSClient c : dfsclients)": > Exception in thread "org.apache.hadoop.hdfs.LeaseRenewer$1@75fa1077" > java.util.ConcurrentModificationException > at > java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) > at java.util.AbstractList$Itr.next(AbstractList.java:343) > at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:406) > at > org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69) > at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273) > at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nemon lou updated MAPREDUCE-5414:
-
Status: Patch Available (was: Open)
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>
[jira] [Updated] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nemon lou updated MAPREDUCE-5414:
-
Attachment: MAPREDUCE-5414.patch
> TestTaskAttempt fails jdk7 with NullPointerException
>
>
> Key: MAPREDUCE-5414
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.5-alpha
>Reporter: nemon lou
>Assignee: nemon lou
> Labels: java7
> Attachments: MAPREDUCE-5414.patch
>
>
> Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
> once in a while when i run all of them together.
> {code:xml}
> Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
> FAILURE!
> Results :
> Tests in error:
>
> testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
>
> testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
> {code}
> But if i run a single test case,taking testContainerCleanedWhileRunning for
> example,it will fail without doubt.
> {code:xml}
> classname="org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt"
> name="testContainerCleanedWhileRunning">
> type="java.lang.NullPointerException">java.lang.NullPointerException
> at org.apache.hadoop.security.token.Token.write(Token.java:216)
> at
> org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
> at
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
>
[jira] [Created] (MAPREDUCE-5415) LeaseRenewer throw java.util.ConcurrentModificationException when timeout
zhaoyunjiong created MAPREDUCE-5415: --- Summary: LeaseRenewer throw java.util.ConcurrentModificationException when timeout Key: MAPREDUCE-5415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5415 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.0 Reporter: zhaoyunjiong Assignee: zhaoyunjiong In LeaseRenewer, when renew() throw SocketTimeoutException, c.abort() will remove one dfsclient from dfsclients. Here will throw a ConcurrentModificationException because dfsclients changed after the iterator created by "for(DFSClient c : dfsclients)": Exception in thread "org.apache.hadoop.hdfs.LeaseRenewer$1@75fa1077" java.util.ConcurrentModificationException at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372) at java.util.AbstractList$Itr.next(AbstractList.java:343) at org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:406) at org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69) at org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273) at java.lang.Thread.run(Thread.java:662) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4579) TestTaskAttempt fails jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718101#comment-13718101 ] nemon lou commented on MAPREDUCE-4579: -- I have created MAPREDUCE-5414. > TestTaskAttempt fails jdk7 > -- > > Key: MAPREDUCE-4579 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4579 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha >Reporter: Thomas Graves >Assignee: Thomas Graves > Labels: java7 > Fix For: 3.0.0, 2.0.2-alpha > > Attachments: MAPREDUCE-4579.patch > > > --- > Test set: org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt > --- > Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.205 sec > <<< > FAILURE!testAttemptContainerRequest(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt) > Time elapsed: 0.032 sec <<< ERROR! > java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > at > org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) > at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) > at org.apache.hadoop.io.Text.readFields(Text.java:280) > at org.apache.hadoop.security.token.Token.readFields(Token.java:165) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5414) TestTaskAttempt fails jdk7 with NullPointerException
nemon lou created MAPREDUCE-5414:
Summary: TestTaskAttempt fails jdk7 with NullPointerException
Key: MAPREDUCE-5414
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5414
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 2.0.5-alpha
Reporter: nemon lou
Assignee: nemon lou
Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt fails
once in a while when i run all of them together.
{code:xml}
Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
FAILURE!
Results :
Tests in error:
testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
{code}
But if i run a single test case,taking testContainerCleanedWhileRunning for
example,it will fail without doubt.
{code:xml}
java.lang.NullPointerException
at org.apache.hadoop.security.token.Token.write(Token.java:216)
at
org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(Prov
[jira] [Commented] (MAPREDUCE-4579) TestTaskAttempt fails jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718073#comment-13718073 ] Devaraj K commented on MAPREDUCE-4579: -- [~nemon], can you raise a new JIRA with the details? > TestTaskAttempt fails jdk7 > -- > > Key: MAPREDUCE-4579 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4579 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha >Reporter: Thomas Graves >Assignee: Thomas Graves > Labels: java7 > Fix For: 3.0.0, 2.0.2-alpha > > Attachments: MAPREDUCE-4579.patch > > > --- > Test set: org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt > --- > Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.205 sec > <<< > FAILURE!testAttemptContainerRequest(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt) > Time elapsed: 0.032 sec <<< ERROR! > java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > at > org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) > at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) > at org.apache.hadoop.io.Text.readFields(Text.java:280) > at org.apache.hadoop.security.token.Token.readFields(Token.java:165) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4579) TestTaskAttempt fails jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718066#comment-13718066
]
nemon lou commented on MAPREDUCE-4579:
--
logs for running
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt#testContainerCleanedWhileRunning
{code:xml}
java.lang.NullPointerException
at org.apache.hadoop.security.token.Token.write(Token.java:216)
at
org.apache.hadoop.mapred.ShuffleHandler.serializeServiceData(ShuffleHandler.java:205)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createCommonContainerLaunchContext(TaskAttemptImpl.java:695)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.createContainerLaunchContext(TaskAttemptImpl.java:751)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1309)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl$ContainerAssignedTransition.transition(TaskAttemptImpl.java:1282)
at
org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:357)
at
org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:298)
at
org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:43)
at
org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:443)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl.handle(TaskAttemptImpl.java:1009)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt.testContainerCleanedWhileRunning(TestTaskAttempt.java:410)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at
org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at
org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at
org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
2013-07-24 10:32:27,664 INFO [main] util.RackResolver
(RackResolver.java:coreResolve(100)) - Resolved 127.0.0.1 to /default-rack
2013-07-24 10:32:27,665 INFO [main] impl.TaskAttemptImpl
(TaskAttemptImpl.java:handle(1020)) - attempt_1_0002_m_01_1 TaskAttempt
Transitioned from NEW to UNASSIGNED
2013-07-24 10:32:27,666 INFO [main] util.RackResolver
(RackResolver.java:coreResolve(100)) - Resolved 127.0.0.1 to /default-rack
2013-07-24 10:32:27,668 INFO [main] impl.TaskAttemptImpl
(TaskAttemptImpl.java:createCommonContainerLaunchContext(636)) - Job jar is not
present. Not adding any jar to the list of resources.
2013-07-24 10:32:27
[jira] [Commented] (MAPREDUCE-4579) TestTaskAttempt fails jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718064#comment-13718064
]
nemon lou commented on MAPREDUCE-4579:
--
Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt still can
fail after apply this patch.
It fails once in a while when you run all of them together.
But if you run a single test case,taking testDoubleTooManyFetchFailure for
example,it will fail without doubt.
I will fire a jira if it's an issue .
{code:xml}
Running org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
Tests run: 9, Failures: 0, Errors: 4, Skipped: 0, Time elapsed: 7.893 sec <<<
FAILURE!
Results :
Tests in error:
testLaunchFailedWhileKilling(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
testContainerCleanedWhileRunning(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
testContainerCleanedWhileCommitting(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
testDoubleTooManyFetchFailure(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
Tests run: 9, Failures: 0, Errors: 4, Skipped: 0
{code}
> TestTaskAttempt fails jdk7
> --
>
> Key: MAPREDUCE-4579
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4579
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha
>Reporter: Thomas Graves
>Assignee: Thomas Graves
> Labels: java7
> Fix For: 3.0.0, 2.0.2-alpha
>
> Attachments: MAPREDUCE-4579.patch
>
>
> ---
> Test set: org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt
> ---
> Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.205 sec
> <<<
> FAILURE!testAttemptContainerRequest(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt)
> Time elapsed: 0.032 sec <<< ERROR!
> java.io.EOFException
> at java.io.DataInputStream.readByte(DataInputStream.java:267)
> at
> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329)
> at org.apache.hadoop.io.Text.readFields(Text.java:280)
> at org.apache.hadoop.security.token.Token.readFields(Token.java:165)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4579) TestTaskAttempt fails jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13718054#comment-13718054 ] nemon lou commented on MAPREDUCE-4579: -- Test case org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt still can fail after apply this patch. > TestTaskAttempt fails jdk7 > -- > > Key: MAPREDUCE-4579 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4579 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.23.3, 3.0.0, 2.0.2-alpha >Reporter: Thomas Graves >Assignee: Thomas Graves > Labels: java7 > Fix For: 3.0.0, 2.0.2-alpha > > Attachments: MAPREDUCE-4579.patch > > > --- > Test set: org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt > --- > Tests run: 10, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 7.205 sec > <<< > FAILURE!testAttemptContainerRequest(org.apache.hadoop.mapreduce.v2.app.job.impl.TestTaskAttempt) > Time elapsed: 0.032 sec <<< ERROR! > java.io.EOFException > at java.io.DataInputStream.readByte(DataInputStream.java:267) > at > org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:308) > at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:329) > at org.apache.hadoop.io.Text.readFields(Text.java:280) > at org.apache.hadoop.security.token.Token.readFields(Token.java:165) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
