[jira] [Commented] (MAPREDUCE-2636) Scheduling over disks horizontally
[ https://issues.apache.org/jira/browse/MAPREDUCE-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537755#comment-13537755 ] Qinghe Jin commented on MAPREDUCE-2636: --- Hi Steve, although the number of disks may several times than the number of nodes, but I think there is only several bits different to identify it.Does it really matter that much? It's good idea to consider output and itermediate data, but do we need to think about it for each task? I think the best configuration is to ensure the locality of each task, which means it reads, writes to the same disk. In this way, it makes more sense to the sheduler or user. Conflict detection is necessary.If we rush to assign task to the busy nodes, it's not only more harmful to the running tasks, but also will cause load unblance problem. For conflict detection, there are two ways:1, find out how many task running on the node; 2,monitor the actual usage of different resources(for disk, we can use disk waiting time). I prefer the second method for there maybe more than one hadoop deployment. Scheduling over disks horizontally -- Key: MAPREDUCE-2636 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2636 Project: Hadoop Map/Reduce Issue Type: Improvement Components: job submission Reporter: Evert Lammerts Priority: Minor Based on this message: http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201106.mbox/browser The JT schedules tasks on nodes based on metadata it gets from the NN. The namenode does not know on which disk a block resides. It might happen that on a node running 4 tasks, all read from the same disk. This can affect performance. An optimization might be to schedule horizontally over disks instead of nodes. Any ideas? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4762) repair test org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan A. Veselovsky updated MAPREDUCE-4762: -- Resolution: Invalid Status: Resolved (was: Patch Available) The class DelegationTokenRenewal was completely removed from all the branches as a result of fix MAPREDUCE-4861 . So, closing this request. repair test org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal - Key: MAPREDUCE-4762 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4762 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Ivan A. Veselovsky Attachments: MAPREDUCE-4762--b.patch, MAPREDUCE-4762-branch-0.23--b.patch, MAPREDUCE-4762-trunk.patch The test org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal is @Ignor-ed. Due to that several classes in package org.apache.hadoop.mapreduce.security.token have zero unit-test coverage. The problem is that the test assumed that class org.apache.hadoop.mapreduce.security.token.TestDelegationTokenRenewal.Renewer is used as a custom implementation of the org.apache.hadoop.security.token.TokenRenewer service, but that did not happen, because this custom service implementation was not registered. We solved this problem by using special classloader that is invoked to find the resource META-INF/services/org.apache.hadoop.security.token.TokenRenewer , and supplies some custom content for it. This way the custom service implementation gets instantiated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4502) Multi-level aggregation with combining the result of maps per node/rack
[ https://issues.apache.org/jira/browse/MAPREDUCE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537783#comment-13537783 ] Tsuyoshi OZAWA commented on MAPREDUCE-4502: --- I've fixed the bug! Now I start to create patch sets. Multi-level aggregation with combining the result of maps per node/rack --- Key: MAPREDUCE-4502 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4502 Project: Hadoop Map/Reduce Issue Type: Improvement Components: applicationmaster, mrv2 Reporter: Tsuyoshi OZAWA Assignee: Tsuyoshi OZAWA Attachments: design_v2.pdf, MAPREDUCE-4525-pof.diff, speculative_draft.pdf The shuffle costs is expensive in Hadoop in spite of the existence of combiner, because the scope of combining is limited within only one MapTask. To solve this problem, it's a good way to aggregate the result of maps per node/rack by launch combiner. This JIRA is to implement the multi-level aggregation infrastructure, including combining per container(MAPREDUCE-3902 is related), coordinating containers by application master without breaking fault tolerance of jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4845) ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-4845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537825#comment-13537825 ] Hudson commented on MAPREDUCE-4845: --- Integrated in Hadoop-Hdfs-0.23-Build #470 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/]) MAPREDUCE-4845. ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2 (Sandy Ryza via tgraves) (Revision 1424729) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1424729 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/ClusterStatus.java ClusterStatus.getMaxMemory() and getUsedMemory() exist in MR1 but not MR2 -- Key: MAPREDUCE-4845 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4845 Project: Hadoop Map/Reduce Issue Type: Improvement Components: client Affects Versions: 1.1.1, 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 1.2.0, 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4845-branch-1.patch, MAPREDUCE-4845.patch, MAPREDUCE-4845.patch For backwards compatibility, these methods should exist in both MR1 and MR2. Confusingly, these methods return the max memory and used memory of the jobtracker, not the entire cluster. I'd propose to add them to MR2 and return -1, and deprecate them in both MR1 and MR2. Alternatively, I could add plumbing to get the resource manager memory stats. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4856) TestJobOutputCommitter uses same directory as TestJobCleanup
[ https://issues.apache.org/jira/browse/MAPREDUCE-4856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537827#comment-13537827 ] Hudson commented on MAPREDUCE-4856: --- Integrated in Hadoop-Hdfs-0.23-Build #470 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/]) MAPREDUCE-4856. TestJobOutputCommitter uses same directory as TestJobCleanup (Sandy Ryza via tgraves) (Revision 1424616) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1424616 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestJobOutputCommitter.java TestJobOutputCommitter uses same directory as TestJobCleanup Key: MAPREDUCE-4856 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4856 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4856.patch This can cause problems if one of the tests fails to delete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4861) Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal
[ https://issues.apache.org/jira/browse/MAPREDUCE-4861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537824#comment-13537824 ] Hudson commented on MAPREDUCE-4861: --- Integrated in Hadoop-Hdfs-0.23-Build #470 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/470/]) MAPREDUCE-4861. Cleanup: Remove unused mapreduce.security.token. DelegationTokenRenewal (Karthik Kambatla via tgraves) (Revision 1424595) Result = UNSTABLE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1424595 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/security/token/TestDelegationTokenRenewal.java Cleanup: Remove unused mapreduce.security.token.DelegationTokenRenewal -- Key: MAPREDUCE-4861 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4861 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Karthik Kambatla Assignee: Karthik Kambatla Fix For: 2.0.3-alpha, 0.23.6 Attachments: mr-4861.patch, mr-4861.patch, mr-4861.patch mapreduce.security.token.DelegationTokenRenewal doesn't seem to be used in branch-2 at all. grep on trunk yields no results, not even ReflectionUtils related suff. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4887) Rehashing partitioner for better distribution
[ https://issues.apache.org/jira/browse/MAPREDUCE-4887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13537884#comment-13537884 ] Hudson commented on MAPREDUCE-4887: --- Integrated in Hadoop-Mapreduce-trunk #1291 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1291/]) MAPREDUCE-4887. Add RehashPartitioner, to smooth distributions with poor implementations of Object#hashCode(). Contributed by Radim Kolar. (Revision 1424158) Result = SUCCESS cutting : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1424158 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/partition/RehashPartitioner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/partition * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestRehashPartitioner.java Rehashing partitioner for better distribution - Key: MAPREDUCE-4887 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4887 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Radim Kolar Assignee: Radim Kolar Fix For: 3.0.0 Attachments: rehash1.txt, rehash2.txt, rehash3.txt, rehash4.txt rehash value returned by Object.hashCode() to get better distribution -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
[ https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-1700: - Attachment: MAPREDUCE-1700.patch Kihwal, that's true - thanks for pointing it out. I've modified the patch to take care of that case, by setting the classloader for the MRAppMaster (when the configured of course). I've also created YARN-286 for the YARN part of this patch so it can be committed separately. This patch is a combined patch so that Jenkins can test it as a whole. User supplied dependencies may conflict with MapReduce system JARs -- Key: MAPREDUCE-1700 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700 Project: Hadoop Map/Reduce Issue Type: Bug Components: task Reporter: Tom White Assignee: Tom White Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch If user code has a dependency on a version of a JAR that is different to the one that happens to be used by Hadoop, then it may not work correctly. This happened with user code using a different version of Avro, as reported [here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081]. The problem is analogous to the one that application servers have with WAR loading. Using a specialized classloader in the Child JVM is probably the way to solve this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4842) Shuffle race can hang reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538108#comment-13538108 ] Jason Lowe commented on MAPREDUCE-4842: --- This still isn't quite right. If an exception occurs during the merge, numPending will be set to 0 and then decremented to -1 by the finally block. If we're going to explicitly set the value to 0 for exceptions then we shouldn't be decrementing in the finally block. Instead we can decrement in the try block after the merge completes. Shuffle race can hang reducer - Key: MAPREDUCE-4842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4842 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Mariappan Asokan Priority: Blocker Attachments: MAPREDUCE-4842-2.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch Saw an instance where the shuffle caused multiple reducers in a job to hang. It looked similar to the problem described in MAPREDUCE-3721, where the fetchers were all being told to WAIT by the MergeManager but no merge was taking place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538109#comment-13538109 ] Daryn Sharp commented on MAPREDUCE-4894: I'm not sure if there is a problem canceling JH tokens when the task completes. Tokens are intended to take the place of kerberos auth within a task. The token allows the oozie launcher or pig's controlling task to poll for its sub-job statuses. The external process that launched the job had kerberos auth to submit the job, so it will have kerberos auth to query the JH server. Renewal / cancellation of JobHistory tokens --- Key: MAPREDUCE-4894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MAPREDUCE-4894_wip.txt, MR-4894.txt Equivalent of YARN-50 for JobHistory tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538116#comment-13538116
]
Alejandro Abdelnur commented on MAPREDUCE-4896:
---
* ClientRMService, instead doing a return in the middle of the method I'd
invert the condition of the IF and move the follow statements into the IF block.
* ResourceMgrDelegate, the change is adding a duplicate call
super.getQueueInfo(queueName), we should do:
{code}
org.apache.hadoop.yarn.api.records.QueueInfo queueInfo =
super.getQueueInfo(queueName);
return (queueInfo == null) ? null : TypeConverter.fromYarn(queueInfo, conf);
{code}
mapred queue -info spits out ugly exception when queue does not exist
---
Key: MAPREDUCE-4896
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, scheduler
Affects Versions: 2.0.2-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4896.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1700) User supplied dependencies may conflict with MapReduce system JARs
[
https://issues.apache.org/jira/browse/MAPREDUCE-1700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538117#comment-13538117
]
Hadoop QA commented on MAPREDUCE-1700:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562082/MAPREDUCE-1700.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3157//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3157//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3157//console
This message is automatically generated.
User supplied dependencies may conflict with MapReduce system JARs
--
Key: MAPREDUCE-1700
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1700
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Reporter: Tom White
Assignee: Tom White
Attachments: MAPREDUCE-1700-ccl.patch, MAPREDUCE-1700-ccl.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch, MAPREDUCE-1700.patch, MAPREDUCE-1700.patch,
MAPREDUCE-1700.patch
If user code has a dependency on a version of a JAR that is different to the
one that happens to be used by Hadoop, then it may not work correctly. This
happened with user code using a different version of Avro, as reported
[here|https://issues.apache.org/jira/browse/AVRO-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12852081#action_12852081].
The problem is analogous to the one that application servers have with WAR
loading. Using a specialized classloader in the Child JVM is probably the way
to solve this.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4842) Shuffle race can hang reducer
[
https://issues.apache.org/jira/browse/MAPREDUCE-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538141#comment-13538141
]
Mariappan Asokan commented on MAPREDUCE-4842:
-
Hi Jason,
When the exceptions happen, the thread will terminate(there is a {{return}}
inside the {{catch}} blocks.) It is okay if {{numPending}} ends up being -1.
The method {{waitForMerge()}} will return immediately. From the point of view
of users of the class, there is no problem.
If you have any more questions, please let me know. Otherwise, I think the fix
is good.
-- Asokan
Shuffle race can hang reducer
-
Key: MAPREDUCE-4842
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4842
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.2-alpha, 0.23.5
Reporter: Jason Lowe
Assignee: Mariappan Asokan
Priority: Blocker
Attachments: MAPREDUCE-4842-2.patch, mapreduce-4842.patch,
mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch,
mapreduce-4842.patch, mapreduce-4842.patch, MAPREDUCE-4842.patch,
MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch
Saw an instance where the shuffle caused multiple reducers in a job to hang.
It looked similar to the problem described in MAPREDUCE-3721, where the
fetchers were all being told to WAIT by the MergeManager but no merge was
taking place.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4842) Shuffle race can hang reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538240#comment-13538240 ] Jason Lowe commented on MAPREDUCE-4842: --- Yes, I suppose it's OK in that sense. I think it's oddly written to have it set the value to zero but really -1 because of the finally block. But in the end, that's a nit and not necessary to fix. +1 for the patch, will commit shortly. Shuffle race can hang reducer - Key: MAPREDUCE-4842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4842 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Mariappan Asokan Priority: Blocker Attachments: MAPREDUCE-4842-2.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch Saw an instance where the shuffle caused multiple reducers in a job to hang. It looked similar to the problem described in MAPREDUCE-3721, where the fetchers were all being told to WAIT by the MergeManager but no merge was taking place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4842) Shuffle race can hang reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4842: -- Resolution: Fixed Fix Version/s: 0.23.6 2.0.3-alpha Status: Resolved (was: Patch Available) Thanks, Mariappan! I committed this to trunk, branch-2, and branch-0.23. Shuffle race can hang reducer - Key: MAPREDUCE-4842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4842 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Mariappan Asokan Priority: Blocker Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4842-2.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch Saw an instance where the shuffle caused multiple reducers in a job to hang. It looked similar to the problem described in MAPREDUCE-3721, where the fetchers were all being told to WAIT by the MergeManager but no merge was taking place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4842) Shuffle race can hang reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538288#comment-13538288 ] Hudson commented on MAPREDUCE-4842: --- Integrated in Hadoop-trunk-Commit #3149 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3149/]) MAPREDUCE-4842. Shuffle race can hang reducer. Contributed by Mariappan Asokan (Revision 1425071) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1425071 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/MergeThread.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java Shuffle race can hang reducer - Key: MAPREDUCE-4842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4842 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Mariappan Asokan Priority: Blocker Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4842-2.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch Saw an instance where the shuffle caused multiple reducers in a job to hang. It looked similar to the problem described in MAPREDUCE-3721, where the fetchers were all being told to WAIT by the MergeManager but no merge was taking place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4842) Shuffle race can hang reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538290#comment-13538290 ] Mariappan Asokan commented on MAPREDUCE-4842: - Hi Jason, It was a pleasure working with all of you. I know this race condition is very hard to reproduce let alone debug. You did an excellent job. All your feedback and challenges encouraged me to find the best possible solution. -- Asokan Shuffle race can hang reducer - Key: MAPREDUCE-4842 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4842 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Mariappan Asokan Priority: Blocker Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4842-2.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, mapreduce-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch, MAPREDUCE-4842.patch Saw an instance where the shuffle caused multiple reducers in a job to hang. It looked similar to the problem described in MAPREDUCE-3721, where the fetchers were all being told to WAIT by the MergeManager but no merge was taking place. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-4833: - Attachment: MAPREDUCE4833.patch Added a test case. Test fails without the fix and passes with the fix Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Attachments: MAPREDUCE4833-23.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4894) Renewal / cancellation of JobHistory tokens
[ https://issues.apache.org/jira/browse/MAPREDUCE-4894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538338#comment-13538338 ] Siddharth Seth commented on MAPREDUCE-4894: --- A launcher which submits jobs using delegation tokens (since it doesn't have kerberos credentials) may need to pull job details after the job completes. That would require the history server delegation tokens. I believe the Oozie launcher works like this for certain Oozie action types. Renewal / cancellation of JobHistory tokens --- Key: MAPREDUCE-4894 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4894 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver, mrv2 Affects Versions: 0.23.4 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Blocker Attachments: MAPREDUCE-4894_wip.txt, MR-4894.txt Equivalent of YARN-50 for JobHistory tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[
https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538339#comment-13538339
]
Hadoop QA commented on MAPREDUCE-4833:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562116/MAPREDUCE4833.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 2016 javac
compiler warnings (more than the trunk's current 2015 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3158//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3158//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3158//console
This message is automatically generated.
Task can get stuck in FAIL_CONTAINER_CLEANUP
Key: MAPREDUCE-4833
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.5
Reporter: Robert Joseph Evans
Assignee: Robert Parker
Priority: Critical
Attachments: MAPREDUCE4833-23.patch, MAPREDUCE4833.patch
If an NM goes down and the AM still tries to launch a container on it the
ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the
RM may notice that the NM has gone away and inform the AM of this, this
triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl
before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try
to kill the container, but the ContainerLauncherImpl will not send back a
TA_CONTAINER_CLEANED event causing the attempt to be stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
Derek Dagit created MAPREDUCE-4899: -- Summary: Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-4833: - Attachment: MAPREDUCE4833-1.patch fixed compiler warnings Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-4833: - Attachment: (was: MAPREDUCE4833-23.patch) Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538403#comment-13538403
]
Robert Kanter commented on MAPREDUCE-4898:
--
I just ran all of the tests under {{hadoop-mapreduce-project}} successfully.
I also just checked where those two methods are used and there's no issues
because:
1) Whenever {{checkOutputSpecs}} is used, the code handles an {{IOException}}
instead of either of the two types of {{FileAlreadyExistsException}}, and both
types of {{FileAlreadyExistsException}} inherit from {{IOException}}
2) Whenever {{setOutputPath}} is used, there's always other method calls that
can throw {{IOException}}s anyway
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath
incompatible with MR1
--
Key: MAPREDUCE-4898
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.3-alpha
Attachments: MAPREDUCE-4898.patch
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it
throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making
them incompatible.
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}}
doesn't throw any exceptions but now it throws an {{IOException}}, making
them incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-4896: -- Attachment: MAPREDUCE-4896-1.patch mapred queue -info spits out ugly exception when queue does not exist --- Key: MAPREDUCE-4896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[ https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538404#comment-13538404 ] Sandy Ryza commented on MAPREDUCE-4896: --- Latest patch makes those changes. mapred queue -info spits out ugly exception when queue does not exist --- Key: MAPREDUCE-4896 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896 Project: Hadoop Map/Reduce Issue Type: Bug Components: client, scheduler Affects Versions: 2.0.2-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[
https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538405#comment-13538405
]
Hadoop QA commented on MAPREDUCE-4833:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562133/MAPREDUCE4833-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 2016 javac
compiler warnings (more than the trunk's current 2015 warnings).
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3159//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3159//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3159//console
This message is automatically generated.
Task can get stuck in FAIL_CONTAINER_CLEANUP
Key: MAPREDUCE-4833
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.5
Reporter: Robert Joseph Evans
Assignee: Robert Parker
Priority: Critical
Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833.patch
If an NM goes down and the AM still tries to launch a container on it the
ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the
RM may notice that the NM has gone away and inform the AM of this, this
triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl
before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try
to kill the container, but the ContainerLauncherImpl will not send back a
TA_CONTAINER_CLEANED event causing the attempt to be stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4898) FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath incompatible with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-4898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538414#comment-13538414
]
Karthik Kambatla commented on MAPREDUCE-4898:
-
Thanks for taking a closer look, Robert.
+1
FileOutputFormat.checkOutputSpecs and FileOutputFormat.setOutputPath
incompatible with MR1
--
Key: MAPREDUCE-4898
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4898
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Robert Kanter
Assignee: Robert Kanter
Fix For: 2.0.3-alpha
Attachments: MAPREDUCE-4898.patch
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs}}
throws {{org.apache.hadoop.mapred.FileAlreadyExistsException}} but now it
throws {{org.apache.hadoop.fs.FileAlreadyExistsException}} instead, making
them incompatible.
In MR1,
{{org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath}}
doesn't throw any exceptions but now it throws an {{IOException}}, making
them incompatible.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Parker updated MAPREDUCE-4833: - Attachment: MAPREDUCE4833-2.patch Actually saving the file before creating the patch this time. Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833-2.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[
https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538440#comment-13538440
]
Hadoop QA commented on MAPREDUCE-4833:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562139/MAPREDUCE4833-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3161//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3161//console
This message is automatically generated.
Task can get stuck in FAIL_CONTAINER_CLEANUP
Key: MAPREDUCE-4833
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, mrv2
Affects Versions: 0.23.5
Reporter: Robert Joseph Evans
Assignee: Robert Parker
Priority: Critical
Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833-2.patch,
MAPREDUCE4833.patch
If an NM goes down and the AM still tries to launch a container on it the
ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the
RM may notice that the NM has gone away and inform the AM of this, this
triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl
before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try
to kill the container, but the ContainerLauncherImpl will not send back a
TA_CONTAINER_CLEANED event causing the attempt to be stuck.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4890) Invalid TaskImpl state transitions when task fails while speculating
[ https://issues.apache.org/jira/browse/MAPREDUCE-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538451#comment-13538451 ] Thomas Graves commented on MAPREDUCE-4890: -- +1 looks good. Thanks Jason. Go ahead and commit. Invalid TaskImpl state transitions when task fails while speculating Key: MAPREDUCE-4890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4890 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-4890.patch There are a couple of issues when a task fails while speculating (i.e.: multiple attempts are active): # The other active attempts are not killed. # TaskImpl's FAILED state does not handle the T_ATTEMPT_* set of events which can be sent from the other active attempts. These all need to be handled since they can be sent asynchronously from the other active task attempts. Failure to handle this properly means jobs that are configured to normally tolerate failures via mapreduce.map.failures.maxpercent or mapreduce.reduce.failures.maxpercent and also speculate can easily end up failing due to invalid state transitions rather than complete successfully with a few explicitly allowed task failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899-branch-0.23.patch Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899.patch Patch that provides MapReduceTrackingUriPlugin, which basically returns a URI to the configured history server with the appropriate string substitution on the application id. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4793) Problem with adding resources when using both -files and -file to hadoop streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4793: -- Attachment: MAPREDUCE-4793.patch Straigthforward fix that I manually tested. Problem with adding resources when using both -files and -file to hadoop streaming -- Key: MAPREDUCE-4793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4793 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3 Reporter: Thomas Graves Attachments: MAPREDUCE-4793.patch It seems when both -files and -file are present, it will trigger this IAE, and the error message is just misleading. hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar -files hdfs://host:port/user/foo/access.log#test -input 'input' -output 'output' -mapper egrep '.*' -file tmp.file Below is the error message 12/11/02 07:37:54 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/haiyang/.staging/job_1351804437209_0575 Exception in thread main java.lang.IllegalArgumentException: Resource name must be relative at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:383) at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:324) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4793) Problem with adding resources when using both -files and -file to hadoop streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4793: -- Assignee: Jason Lowe Target Version/s: 2.0.3-alpha, 0.23.6 Status: Patch Available (was: Open) Problem with adding resources when using both -files and -file to hadoop streaming -- Key: MAPREDUCE-4793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4793 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Jason Lowe Attachments: MAPREDUCE-4793.patch It seems when both -files and -file are present, it will trigger this IAE, and the error message is just misleading. hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar -files hdfs://host:port/user/foo/access.log#test -input 'input' -output 'output' -mapper egrep '.*' -file tmp.file Below is the error message 12/11/02 07:37:54 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/haiyang/.staging/job_1351804437209_0575 Exception in thread main java.lang.IllegalArgumentException: Resource name must be relative at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:383) at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:324) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4896) mapred queue -info spits out ugly exception when queue does not exist
[
https://issues.apache.org/jira/browse/MAPREDUCE-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538458#comment-13538458
]
Hadoop QA commented on MAPREDUCE-4896:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562136/MAPREDUCE-4896-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3160//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3160//console
This message is automatically generated.
mapred queue -info spits out ugly exception when queue does not exist
---
Key: MAPREDUCE-4896
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4896
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, scheduler
Affects Versions: 2.0.2-alpha
Reporter: Sandy Ryza
Assignee: Sandy Ryza
Attachments: MAPREDUCE-4896-1.patch, MAPREDUCE-4896.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4793) Problem with adding resources when using both -files and -file to hadoop streaming
[
https://issues.apache.org/jira/browse/MAPREDUCE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538470#comment-13538470
]
Hadoop QA commented on MAPREDUCE-4793:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562151/MAPREDUCE-4793.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-streaming:
org.apache.hadoop.streaming.TestFileArgs
org.apache.hadoop.streaming.TestStreamingTaskLog
org.apache.hadoop.streaming.TestStreamingBadRecords
org.apache.hadoop.streaming.TestMultipleCachefiles
org.apache.hadoop.streaming.TestSymLink
org.apache.hadoop.streaming.TestMultipleArchiveFiles
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3162//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3162//console
This message is automatically generated.
Problem with adding resources when using both -files and -file to hadoop
streaming
--
Key: MAPREDUCE-4793
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4793
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Thomas Graves
Assignee: Jason Lowe
Attachments: MAPREDUCE-4793.patch
It seems when
both -files and -file are present, it will trigger this IAE, and the error
message is just misleading.
hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar
-files
hdfs://host:port/user/foo/access.log#test
-input 'input' -output 'output' -mapper egrep '.*' -file tmp.file
Below is the error message
12/11/02 07:37:54 INFO mapreduce.JobSubmitter: Cleaning up the staging area
/user/haiyang/.staging/job_1351804437209_0575
Exception in thread main java.lang.IllegalArgumentException: Resource name
must be relative
at
org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:383)
at
org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:324)
at
org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419)
at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538472#comment-13538472 ] Jason Lowe commented on MAPREDUCE-4833: --- +1, thanks for writing a test. Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833-2.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4833: -- Resolution: Fixed Fix Version/s: 0.23.6 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks, Robert. I committed this to trunk, branch-2, and branch-0.23. Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833-2.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4833) Task can get stuck in FAIL_CONTAINER_CLEANUP
[ https://issues.apache.org/jira/browse/MAPREDUCE-4833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538489#comment-13538489 ] Hudson commented on MAPREDUCE-4833: --- Integrated in Hadoop-trunk-Commit #3151 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3151/]) MAPREDUCE-4833. Task can get stuck in FAIL_CONTAINER_CLEANUP. Contributed by Robert Parker (Revision 1425167) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1425167 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java Task can get stuck in FAIL_CONTAINER_CLEANUP Key: MAPREDUCE-4833 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4833 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster, mrv2 Affects Versions: 0.23.5 Reporter: Robert Joseph Evans Assignee: Robert Parker Priority: Critical Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE4833-1.patch, MAPREDUCE4833-2.patch, MAPREDUCE4833.patch If an NM goes down and the AM still tries to launch a container on it the ContainerLauncherImpl can get stuck in an RPC timeout. At the same time the RM may notice that the NM has gone away and inform the AM of this, this triggers a TA_FAILMSG. If the TA_FAILMSG arrives at the TaskAttemptImpl before the TA_CONTAINER_LAUNCH_FAILED message then the task attempt will try to kill the container, but the ContainerLauncherImpl will not send back a TA_CONTAINER_CLEANED event causing the attempt to be stuck. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899.patch Fixes previous set of patches Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899-branch-0.23.patch Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Status: Patch Available (was: Open) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4660) Update task placement policy for NetworkTopology with 'NodeGroup' layer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu resolved MAPREDUCE-4660. Resolution: Fixed Fix Version/s: 1.2.0 Committed (after whitespace cleanup) to branch-1. Thanks Junping! Update task placement policy for NetworkTopology with 'NodeGroup' layer --- Key: MAPREDUCE-4660 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4660 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker, mrv1, scheduler Reporter: Junping Du Assignee: Junping Du Fix For: 1.2.0 Attachments: MAPREDUCE-4660.patch, MAPREDUCE-4660-v2.patch, MAPREDUCE-4660-v3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538502#comment-13538502 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4899: Patch looks good, will push in after taking care of YARN-285. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538504#comment-13538504
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562170/MAPREDUCE-4899.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3163//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4793) Problem with adding resources when using both -files and -file to hadoop streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538507#comment-13538507 ] Jason Lowe commented on MAPREDUCE-4793: --- Thanks for the review, Tom. All of the test failures are unrelated and known issues, see MAPREDUCE-4884. When the patch from MAPREDUCE-4884 is also applied, all of the streaming tests pass. Problem with adding resources when using both -files and -file to hadoop streaming -- Key: MAPREDUCE-4793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4793 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Jason Lowe Attachments: MAPREDUCE-4793.patch It seems when both -files and -file are present, it will trigger this IAE, and the error message is just misleading. hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar -files hdfs://host:port/user/foo/access.log#test -input 'input' -output 'output' -mapper egrep '.*' -file tmp.file Below is the error message 12/11/02 07:37:54 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/haiyang/.staging/job_1351804437209_0575 Exception in thread main java.lang.IllegalArgumentException: Resource name must be relative at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:383) at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:324) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4793) Problem with adding resources when using both -files and -file to hadoop streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4793: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I committed this to trunk, branch-2, and branch-0.23. Problem with adding resources when using both -files and -file to hadoop streaming -- Key: MAPREDUCE-4793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4793 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Jason Lowe Attachments: MAPREDUCE-4793.patch It seems when both -files and -file are present, it will trigger this IAE, and the error message is just misleading. hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar -files hdfs://host:port/user/foo/access.log#test -input 'input' -output 'output' -mapper egrep '.*' -file tmp.file Below is the error message 12/11/02 07:37:54 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/haiyang/.staging/job_1351804437209_0575 Exception in thread main java.lang.IllegalArgumentException: Resource name must be relative at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:383) at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:324) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant
[
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538521#comment-13538521
]
Hudson commented on MAPREDUCE-4272:
---
Integrated in Hadoop-trunk-Commit #3152 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3152/])
MAPREDUCE-4272. SortedRanges.Range#compareTo was not spec compliant. (Yu
Gao via llu) (Revision 1425180)
Result = SUCCESS
llu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1425180
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/SortedRanges.java
SortedRanges.Range#compareTo is not spec compliant
--
Key: MAPREDUCE-4272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 1.0.3
Reporter: Luke Lu
Assignee: Yu Gao
Labels: alt-jdk
Attachments: mapreduce-4272-branch-1.0.patch,
mapreduce-4272-branch-2.patch, mapreduce-4272.patch
SortedRanges.Range#compareTo does not satisfy the requirement of
Comparable#compareTo, where the implementor must ensure
{noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and
y.
This is manifested as TestStreamingBadRecords failures in alternative JDKs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4793) Problem with adding resources when using both -files and -file to hadoop streaming
[ https://issues.apache.org/jira/browse/MAPREDUCE-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538522#comment-13538522 ] Hudson commented on MAPREDUCE-4793: --- Integrated in Hadoop-trunk-Commit #3152 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3152/]) MAPREDUCE-4793. Problem with adding resources when using both -files and -file to hadoop streaming. Contributed by Jason Lowe (Revision 1425177) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1425177 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/main/java/org/apache/hadoop/streaming/StreamJob.java Problem with adding resources when using both -files and -file to hadoop streaming -- Key: MAPREDUCE-4793 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4793 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 0.23.3 Reporter: Thomas Graves Assignee: Jason Lowe Attachments: MAPREDUCE-4793.patch It seems when both -files and -file are present, it will trigger this IAE, and the error message is just misleading. hadoop jar $HADOOP_PREFIX/share/hadoop/tools/lib/hadoop-streaming.jar -files hdfs://host:port/user/foo/access.log#test -input 'input' -output 'output' -mapper egrep '.*' -file tmp.file Below is the error message 12/11/02 07:37:54 INFO mapreduce.JobSubmitter: Cleaning up the staging area /user/haiyang/.staging/job_1351804437209_0575 Exception in thread main java.lang.IllegalArgumentException: Resource name must be relative at org.apache.hadoop.mapreduce.v2.util.MRApps.parseDistributedCacheArtifacts(MRApps.java:383) at org.apache.hadoop.mapreduce.v2.util.MRApps.setupDistributedCache(MRApps.java:324) at org.apache.hadoop.mapred.YARNRunner.createApplicationSubmissionContext(YARNRunner.java:419) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:288) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4272) SortedRanges.Range#compareTo is not spec compliant
[
https://issues.apache.org/jira/browse/MAPREDUCE-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Lu updated MAPREDUCE-4272:
---
Resolution: Fixed
Fix Version/s: 1.1.2
1.2.0
Target Version/s: 2.0.0-alpha, 1.2.0 (was: 1.2.0, 2.0.0-alpha)
Status: Resolved (was: Patch Available)
+1. Committed to branch-{1,1.1}. Thanks Yu!
SortedRanges.Range#compareTo is not spec compliant
--
Key: MAPREDUCE-4272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4272
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: task
Affects Versions: 1.0.3
Reporter: Luke Lu
Assignee: Yu Gao
Labels: alt-jdk
Fix For: 1.2.0, 1.1.2
Attachments: mapreduce-4272-branch-1.0.patch,
mapreduce-4272-branch-2.patch, mapreduce-4272.patch
SortedRanges.Range#compareTo does not satisfy the requirement of
Comparable#compareTo, where the implementor must ensure
{noformat}sgn(x.compareTo(y)) == -sgn(y.compareTo(x)){noformat} for all x and
y.
This is manifested as TestStreamingBadRecords failures in alternative JDKs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4397) Introduce HADOOP_SECURITY_CONF_DIR for task-controller
[ https://issues.apache.org/jira/browse/MAPREDUCE-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu resolved MAPREDUCE-4397. Resolution: Fixed Fix Version/s: 1.1.2 1.2.0 Hadoop Flags: Reviewed +1. Committed to branch-1 and 1.1. Thanks Yu! Introduce HADOOP_SECURITY_CONF_DIR for task-controller -- Key: MAPREDUCE-4397 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4397 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task-controller Reporter: Luke Lu Assignee: Yu Gao Fix For: 1.2.0, 1.1.2 Attachments: mapreduce-4397-branch-1.patch, test-patch.result The linux task controller concurrently hard code the directory to look for its config file at compile time (via the HADOOP_CONF_DIR macro). Adding a new environment variable to look for task-controller's conf dir (with strict permission checks) would make installation much more flexible. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4660) Update task placement policy for NetworkTopology with 'NodeGroup' layer
[ https://issues.apache.org/jira/browse/MAPREDUCE-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luke Lu updated MAPREDUCE-4660: --- Hadoop Flags: Reviewed Update task placement policy for NetworkTopology with 'NodeGroup' layer --- Key: MAPREDUCE-4660 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4660 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobtracker, mrv1, scheduler Reporter: Junping Du Assignee: Junping Du Fix For: 1.2.0 Attachments: MAPREDUCE-4660.patch, MAPREDUCE-4660-v2.patch, MAPREDUCE-4660-v3.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899-branch-0.23.patch Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899.patch Plugin overrides setConf to force loading of mapreduce config Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538557#comment-13538557
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562180/MAPREDUCE-4899.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3164//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899-branch-0.23.patch Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899.patch Patch to fix plugin. setConf calls super.setConf, and checks for null Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538580#comment-13538580
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562185/MAPREDUCE-4899.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3165//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899.patch Fixes artifact version in new pom.xml Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Status: Open (was: Patch Available) Looks good now. Can you fix the version in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml for the trunk patch to point to 3.0 snapshot.? Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538592#comment-13538592 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-4899: commenting race, let me quickly look at the latest patch. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Derek Dagit updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899.patch Fixing the other incorrect artifact version. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) Thanks. The latest patch looks fine. Time to wake up Jenkins. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538616#comment-13538616
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562190/MAPREDUCE-4899.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3166//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3166//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs-plugins.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3166//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538624#comment-13538624
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562190/MAPREDUCE-4899.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3167//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3167//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs-plugins.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3167//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Status: Open (was: Patch Available) Good catch findBugs. It is a code issue.. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899-20121221.txt Patch fixing the bug. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-20121221.txt, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Status: Patch Available (was: Open) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 0.23.5, 3.0.0 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-20121221.txt, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538634#comment-13538634
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562201/MAPREDUCE-4899-20121221.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3168//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3168//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-20121221.txt,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Attachment: MAPREDUCE-4899-branch-0.23-20121221.txt Updated patch for 23. Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Attachments: MAPREDUCE-4899-20121221.txt, MAPREDUCE-4899-branch-0.23-20121221.txt, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4890) Invalid TaskImpl state transitions when task fails while speculating
[ https://issues.apache.org/jira/browse/MAPREDUCE-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538636#comment-13538636 ] Hudson commented on MAPREDUCE-4890: --- Integrated in Hadoop-trunk-Commit #3154 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3154/]) MAPREDUCE-4890. Invalid TaskImpl state transitions when task fails while speculating. Contributed by Jason Lowe (Revision 1425223) Result = SUCCESS jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1425223 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java Invalid TaskImpl state transitions when task fails while speculating Key: MAPREDUCE-4890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4890 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Attachments: MAPREDUCE-4890.patch There are a couple of issues when a task fails while speculating (i.e.: multiple attempts are active): # The other active attempts are not killed. # TaskImpl's FAILED state does not handle the T_ATTEMPT_* set of events which can be sent from the other active attempts. These all need to be handled since they can be sent asynchronously from the other active task attempts. Failure to handle this properly means jobs that are configured to normally tolerate failures via mapreduce.map.failures.maxpercent or mapreduce.reduce.failures.maxpercent and also speculate can easily end up failing due to invalid state transitions rather than complete successfully with a few explicitly allowed task failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538638#comment-13538638
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562201/MAPREDUCE-4899-20121221.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3169//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3169//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-20121221.txt,
MAPREDUCE-4899-branch-0.23-20121221.txt, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[
https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538640#comment-13538640
]
Hadoop QA commented on MAPREDUCE-4899:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562202/MAPREDUCE-4899-branch-0.23-20121221.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3170//console
This message is automatically generated.
Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R
appllications given the ID
Key: MAPREDUCE-4899
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 3.0.0, 0.23.5
Reporter: Derek Dagit
Assignee: Derek Dagit
Attachments: MAPREDUCE-4899-20121221.txt,
MAPREDUCE-4899-branch-0.23-20121221.txt, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch,
MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch,
MAPREDUCE-4899.patch
Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce
tracking links to the History server.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4890) Invalid TaskImpl state transitions when task fails while speculating
[ https://issues.apache.org/jira/browse/MAPREDUCE-4890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4890: -- Resolution: Fixed Fix Version/s: 0.23.6 2.0.3-alpha Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review, Tom. I committed this to trunk, branch-2, and branch-0.23. Invalid TaskImpl state transitions when task fails while speculating Key: MAPREDUCE-4890 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4890 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am Affects Versions: 2.0.2-alpha, 0.23.5 Reporter: Jason Lowe Assignee: Jason Lowe Priority: Critical Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4890.patch There are a couple of issues when a task fails while speculating (i.e.: multiple attempts are active): # The other active attempts are not killed. # TaskImpl's FAILED state does not handle the T_ATTEMPT_* set of events which can be sent from the other active attempts. These all need to be handled since they can be sent asynchronously from the other active task attempts. Failure to handle this properly means jobs that are configured to normally tolerate failures via mapreduce.map.failures.maxpercent or mapreduce.reduce.failures.maxpercent and also speculate can easily end up failing due to invalid state transitions rather than complete successfully with a few explicitly allowed task failures. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-4899: --- Resolution: Fixed Fix Version/s: 0.23.6 2.0.3-alpha Status: Resolved (was: Patch Available) I just committed this to trunk, branch-2 and branch-0.23. Thanks Derek! Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4899-20121221.txt, MAPREDUCE-4899-branch-0.23-20121221.txt, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch Create a Map/Reduce specific plugin for use with the Yarn RM Proxy to produce tracking links to the History server. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4899) Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID
[ https://issues.apache.org/jira/browse/MAPREDUCE-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13538643#comment-13538643 ] Hudson commented on MAPREDUCE-4899: --- Integrated in Hadoop-trunk-Commit #3155 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3155/]) MAPREDUCE-4899. Implemented a MR specific plugin for tracking finished applications that YARN's ResourceManager doesn't keep track off anymore. Contributed by Derek Dagit. (Revision 1425225) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1425225 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache/hadoop/mapreduce * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache/hadoop/mapreduce/v2 * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache/hadoop/mapreduce/v2/hs * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/MapReduceTrackingUriPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2 * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs-plugins/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestMapReduceTrackingUriPlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml Provide a plugin to the Yarn Web App Proxy to generate tracking links for M/R appllications given the ID Key: MAPREDUCE-4899 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4899 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 3.0.0, 0.23.5 Reporter: Derek Dagit Assignee: Derek Dagit Fix For: 2.0.3-alpha, 0.23.6 Attachments: MAPREDUCE-4899-20121221.txt, MAPREDUCE-4899-branch-0.23-20121221.txt, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899-branch-0.23.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch, MAPREDUCE-4899.patch
