[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671208#comment-13671208
]
Hadoop QA commented on MAPREDUCE-5275:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585558/MAPREDUCE-5275-trunk-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3707//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3707//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3707//console
This message is automatically generated.
> Mapreduce API: TokenCache incompatibility issues with MR1
> -
>
> Key: MAPREDUCE-5275
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5275-trunk-1.patch,
> MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch
>
>
> There're following incompatibility issues:
> * Token getDelegationToken(Credentials, String) is
> removed
> * Credentials loadTokens(String, Configuration) changes to Credentials
> loadTokens(String, JobConf)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5275: - Attachment: MAPREDUCE-5275-trunk-3.patch Thanks [~vinodkv] for review and test. Updating the latest patch. Thanks, Mayank > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5275: - Status: Patch Available (was: Open) > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch, MAPREDUCE-5275-trunk-3.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671198#comment-13671198 ] Hudson commented on MAPREDUCE-5176: --- Integrated in Hadoop-trunk-Commit #3820 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3820/]) MAPREDUCE-5176. Add annotation for tagging tasks as responsive to preemption. Contributed by Carlo Curino, cdouglas (Revision 1488103) Result = SUCCESS cdouglas : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488103 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Reducer.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialFileOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/PartialOutputCommitter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/annotation/Checkpointable.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/output/TestPreemptableFileOutputCommitter.java > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5176: - Resolution: Fixed Hadoop Flags: Incompatible change,Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-2. Thanks Carlo > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]
[ https://issues.apache.org/jira/browse/MAPREDUCE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671141#comment-13671141 ] Hudson commented on MAPREDUCE-5286: --- Integrated in Hadoop-trunk-Commit #3818 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3818/]) MAPREDUCE-5286. Change MapReduce to use ContainerTokenIdentifier instead of the entire Container in the startContainer call - YARN-684. Contributed by Vinod Kumar Vavilapalli. (Revision 1488087) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488087 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java > startContainer call should use the ContainerToken instead of Container > [YARN-684] > - > > Key: MAPREDUCE-5286 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Vinod Kumar Vavilapalli > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5286.txt > > > MapReduce counterpart of YARN-684. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]
[ https://issues.apache.org/jira/browse/MAPREDUCE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth resolved MAPREDUCE-5286. --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Committed to trunk and branch-2. Thanks Vinod. > startContainer call should use the ContainerToken instead of Container > [YARN-684] > - > > Key: MAPREDUCE-5286 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Vinod Kumar Vavilapalli > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5286.txt > > > MapReduce counterpart of YARN-684. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]
[ https://issues.apache.org/jira/browse/MAPREDUCE-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5286: -- Attachment: MAPREDUCE-5286.txt MR part of Vinod's patch on YARN-684. > startContainer call should use the ContainerToken instead of Container > [YARN-684] > - > > Key: MAPREDUCE-5286 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Vinod Kumar Vavilapalli > Attachments: MAPREDUCE-5286.txt > > > MapReduce counterpart of YARN-684. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5286) startContainer call should use the ContainerToken instead of Container [YARN-684]
Siddharth Seth created MAPREDUCE-5286: - Summary: startContainer call should use the ContainerToken instead of Container [YARN-684] Key: MAPREDUCE-5286 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5286 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Siddharth Seth Assignee: Vinod Kumar Vavilapalli MapReduce counterpart of YARN-684. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671110#comment-13671110 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5176: Looks good. +1. We can do more testing, but will track that in the related JIRAs. Feel free to commit it. If you want others to look at it, leave it around. I'll commit it first thing tomorrow morning if this is still around and no one has objections. > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5275: --- Status: Open (was: Patch Available) I just did a quick experiment, it is a source code compatible change but breaks binary compatibility. For mapreduce, we are only focusing on source compatibility, but it's worth making the change here. > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671096#comment-13671096
]
Hadoop QA commented on MAPREDUCE-5176:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585536/MAPREDUCE-5176.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3706//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3706//console
This message is automatically generated.
> Preemptable annotations (to support preemption in MR)
> -
>
> Key: MAPREDUCE-5176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch,
> MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch
>
>
> Proposing a patch that introduces a new annotation @Preemptable that
> represents to the framework property of user-supplied classes (e.g., Reducer,
> OutputCommiter). The intended semantics is that a tagged class is safe to be
> preempted between invocations.
> (this is in spirit similar to the Output Contracts of [Nephele/PACT |
> https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[
https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671093#comment-13671093
]
Hadoop QA commented on MAPREDUCE-5176:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585535/MAPREDUCE-5176.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 2
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3705//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3705//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3705//console
This message is automatically generated.
> Preemptable annotations (to support preemption in MR)
> -
>
> Key: MAPREDUCE-5176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Reporter: Carlo Curino
>Assignee: Carlo Curino
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch,
> MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch
>
>
> Proposing a patch that introduces a new annotation @Preemptable that
> represents to the framework property of user-supplied classes (e.g., Reducer,
> OutputCommiter). The intended semantics is that a tagged class is safe to be
> preempted between invocations.
> (this is in spirit similar to the Output Contracts of [Nephele/PACT |
> https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf])
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5176: - Attachment: MAPREDUCE-5176.3.patch Missed some license headers... > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5176: - Status: Open (was: Patch Available) > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5176: - Status: Patch Available (was: Open) > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.3.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5176: - Attachment: MAPREDUCE-5176.2.patch Thanks for the review, Vinod. Changed the names, visibility, and audience. > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated MAPREDUCE-5176: - Status: Patch Available (was: Open) > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.2.patch, > MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671047#comment-13671047 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5176: bq. It'd be great for the annotation to be part of 2.0.5-beta. +1, let's push forward with this for 2.0.5 or whatever the immediate 'big' release is. > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5176) Preemptable annotations (to support preemption in MR)
[ https://issues.apache.org/jira/browse/MAPREDUCE-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671046#comment-13671046 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5176: bq. BTW I think we are in a great spot to carry along this conversation, since we have one very specific example of these annotations: @Preemptable for which we have the entire end-to-end usage scenario (all the preemption in mapreduce stuff tracked in MAPREDUCE-5189, MAPREDUCE-5192, MAPREDUCE-5194, MAPREDUCE-5196, MAPREDUCE-5197 and few upcoming ones), and plenty more ideas coming up from people. Yeah, clearly for now, this is a MR only thing. We can promote it to YARN when we see more uses of it. Even so, seems like here there is the usage of preemption and checkpointing interchangeably, but it feels like this is better off called @Checkpointable. If you agree - some javadoc in the patch also needs to be fixed - PreemptableFileOutputCommitter name is fine? I don't know for sure. Still haven't looked at the patch in the context of the whole thing, but had a quick look at it: - Call PartialOutputCommitter and PreemptableFileOutputCommitter to be public but evolving? - mapreduce/lib/output/FileOutputCommitter: getCommittedTaskPath can be protected? - hdfu://user/hadoop/out : Typo? It perhaps doesn't matter, but.. Also some statements, please correct me if I am wrong: - The clean up of partitial outputs from previous attempts is done by AM/task. - When the reducer gets the preempt signal, it uses the regular commit api to commit the output and then for successive commits too - The current abortTask/cleanupTask won't be enough as it will only delete the current TaskAttempts output. cleanUpPartialOutputForTask deletes all the previous tasks' output. > Preemptable annotations (to support preemption in MR) > - > > Key: MAPREDUCE-5176 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5176 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5176.1.patch, MAPREDUCE-5176.patch > > > Proposing a patch that introduces a new annotation @Preemptable that > represents to the framework property of user-supplied classes (e.g., Reducer, > OutputCommiter). The intended semantics is that a tagged class is safe to be > preempted between invocations. > (this is in spirit similar to the Output Contracts of [Nephele/PACT | > https://stratosphere.eu/sites/default/files/papers/ComparingMapReduceAndPACTs_11.pdf]) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5250) Searching for ';' in JobTracker History throws ArrayOutOfBoundException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671042#comment-13671042
]
Hadoop QA commented on MAPREDUCE-5250:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585521/mr-5250.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3704//console
This message is automatically generated.
> Searching for ';' in JobTracker History throws ArrayOutOfBoundException
>
>
> Key: MAPREDUCE-5250
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5250
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Labels: history
> Attachments: mr-5250.patch
>
>
> Searching for ';' in JobTracker History throws ArrayOutOfBoundException
> {noformat}
> Problem accessing /jobhistoryhome.jsp. Reason:
> 0
> Caused by:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapred.jobhistoryhome_jsp._jspService(jobhistoryhome_jsp.java:221)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5250) Searching for ';' in JobTracker History throws ArrayOutOfBoundException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5250:
Status: Patch Available (was: Open)
> Searching for ';' in JobTracker History throws ArrayOutOfBoundException
>
>
> Key: MAPREDUCE-5250
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5250
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Labels: history
> Attachments: mr-5250.patch
>
>
> Searching for ';' in JobTracker History throws ArrayOutOfBoundException
> {noformat}
> Problem accessing /jobhistoryhome.jsp. Reason:
> 0
> Caused by:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapred.jobhistoryhome_jsp._jspService(jobhistoryhome_jsp.java:221)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5250) Searching for ';' in JobTracker History throws ArrayOutOfBoundException
[
https://issues.apache.org/jira/browse/MAPREDUCE-5250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5250:
Attachment: mr-5250.patch
Trivial patch that adds a length check before accessing the array element.
To verify, started a pseudo-dist cluster and ran pi job. Searching for ";"
didn't throw an exception with the patch.
> Searching for ';' in JobTracker History throws ArrayOutOfBoundException
>
>
> Key: MAPREDUCE-5250
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5250
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 1.2.0
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Labels: history
> Attachments: mr-5250.patch
>
>
> Searching for ';' in JobTracker History throws ArrayOutOfBoundException
> {noformat}
> Problem accessing /jobhistoryhome.jsp. Reason:
> 0
> Caused by:
> java.lang.ArrayIndexOutOfBoundsException: 0
> at
> org.apache.hadoop.mapred.jobhistoryhome_jsp._jspService(jobhistoryhome_jsp.java:221)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:914)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance
[
https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13671018#comment-13671018
]
Hadoop QA commented on MAPREDUCE-5268:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585515/mr-5268.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3703//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3703//console
This message is automatically generated.
> Improve history server startup performance
> --
>
> Key: MAPREDUCE-5268
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobhistoryserver
>Affects Versions: 0.23.7, 2.0.4-alpha
>Reporter: Jason Lowe
>Assignee: Karthik Kambatla
> Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch,
> mr-5268-prelim.patch
>
>
> The history server can easily take many minutes to startup when there are a
> significant number of jobs to scan in the done directory. However the
> scanning of files is not the bottleneck, rather it's the heavy use of
> ConcurrentSkipListMap.size in HistoryFileManager.
> ConcurrentSkipListMap.size is a very expensive operation, especially on maps
> with many entries, as it has to scan every entry to compute the size. We
> should avoid calling this method or at least minimize its use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5268) Improve history server startup performance
[ https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5268: Attachment: mr-5268.patch Updated the patch to address Jason's comments: # use AtomicInteger instead of synchronized methods # wrapper not to use templates > Improve history server startup performance > -- > > Key: MAPREDUCE-5268 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Jason Lowe >Assignee: Karthik Kambatla > Attachments: mr-5268.patch, mr-5268.patch, mr-5268.patch, > mr-5268-prelim.patch > > > The history server can easily take many minutes to startup when there are a > significant number of jobs to scan in the done directory. However the > scanning of files is not the bottleneck, rather it's the heavy use of > ConcurrentSkipListMap.size in HistoryFileManager. > ConcurrentSkipListMap.size is a very expensive operation, especially on maps > with many entries, as it has to scan every entry to compute the size. We > should avoid calling this method or at least minimize its use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5285) Update MR App to use immutable ApplicationAttemptID, ContainerID, NodeID after YARN-735
Jian He created MAPREDUCE-5285: -- Summary: Update MR App to use immutable ApplicationAttemptID, ContainerID, NodeID after YARN-735 Key: MAPREDUCE-5285 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5285 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Jian He -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670999#comment-13670999 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5275: I think the changes to loadTokens will breaking binary compatibility. Why not just add a new method which calls the existing method? > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670993#comment-13670993 ] Zhijie Shen commented on MAPREDUCE-5275: +1. Looks good to me now. > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670990#comment-13670990 ] Hudson commented on MAPREDUCE-5228: --- Integrated in Hadoop-trunk-Commit #3816 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3816/]) MAPREDUCE-5228. Bring back FileInputFormat.Counter and FileOuputFormat.Counter for binary compatibility with 1.x mapred APIs. Contributed by Mayank Bansal. (Revision 1488060) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488060 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/Counters.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat_Counter.properties * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileOutputFormat_Counter.properties * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat_Counter.properties * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileOutputFormat_Counter.properties * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/output/FileOutputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestJobCounters.java > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, > MAPREDUCE-5228-trunk-6.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670988#comment-13670988
]
Hadoop QA commented on MAPREDUCE-5275:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585503/MAPREDUCE-5275-trunk-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3702//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3702//console
This message is automatically generated.
> Mapreduce API: TokenCache incompatibility issues with MR1
> -
>
> Key: MAPREDUCE-5275
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5275-trunk-1.patch,
> MAPREDUCE-5275-trunk-2.patch
>
>
> There're following incompatibility issues:
> * Token getDelegationToken(Credentials, String) is
> removed
> * Credentials loadTokens(String, Configuration) changes to Credentials
> loadTokens(String, JobConf)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670987#comment-13670987 ] Niranjan Singh commented on MAPREDUCE-5183: --- Thanks,for the clarification Sandy Ryza. > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5228: --- Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed this to trunk and branch-2. Thanks Mayank! Tx for the reivews Zhijie.. > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, > MAPREDUCE-5228-trunk-6.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670976#comment-13670976 ] Sandy Ryza commented on MAPREDUCE-5183: --- It's not your fault. The QA bot only tries to apply and verify the patch against trunk. Since your patch is for branch-1, it (understandably) does not apply. > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5275: - Attachment: MAPREDUCE-5275-trunk-2.patch Thanks [~zjshen] for the review. Updated the latest patch. Thanks, Mayank > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670972#comment-13670972 ] Niranjan Singh commented on MAPREDUCE-5183: --- @Sandy Ryza- Can you please let me know why Hadoop QA is generating [ -1 patch. The patch command could not apply the patch.]. Is there something I am missing out? > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5275: - Status: Patch Available (was: Open) > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch, > MAPREDUCE-5275-trunk-2.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated MAPREDUCE-5275: --- Status: Open (was: Patch Available) > Mapreduce API: TokenCache incompatibility issues with MR1 > - > > Key: MAPREDUCE-5275 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5275-trunk-1.patch > > > There're following incompatibility issues: > * Token getDelegationToken(Credentials, String) is > removed > * Credentials loadTokens(String, Configuration) changes to Credentials > loadTokens(String, JobConf) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670951#comment-13670951 ] Zhijie Shen commented on MAPREDUCE-5228: +1. Looks good to me. > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, > MAPREDUCE-5228-trunk-6.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5275) Mapreduce API: TokenCache incompatibility issues with MR1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670949#comment-13670949
]
Zhijie Shen commented on MAPREDUCE-5275:
bq. I had an offline discussion with vinod, For this to work we would need
build time dependency. What we should do here?
One way to walk around is as follows:
{code}
public static Token
getDelegationToken(Credentials credentials, String namenode) {
return (Token)
credentials.getToken(new Text(namenode));
}
{code}
No additional dependency is required, but is binary compatible.
> Mapreduce API: TokenCache incompatibility issues with MR1
> -
>
> Key: MAPREDUCE-5275
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5275
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5275-trunk-1.patch
>
>
> There're following incompatibility issues:
> * Token getDelegationToken(Credentials, String) is
> removed
> * Credentials loadTokens(String, Configuration) changes to Credentials
> loadTokens(String, JobConf)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670945#comment-13670945
]
Hadoop QA commented on MAPREDUCE-5228:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585494/MAPREDUCE-5228-trunk-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3700//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3700//console
This message is automatically generated.
> Enum Counter is removed from FileInputFormat and FileOutputFormat of both
> mapred and mapreduce
> --
>
> Key: MAPREDUCE-5228
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5228-trunk-1.patch,
> MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch,
> MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch,
> MAPREDUCE-5228-trunk-6.patch
>
>
> The enum was used by findCounter(Enum key) to find a specific counter object.
> Now it seems to be replaced by FileInputFormatCounter and
> FileOutputFormatCounter. Now the enum seems to be only used internally, but
> not sure whether it will be used externally when users extend FileFormat.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance
[ https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670942#comment-13670942 ] Karthik Kambatla commented on MAPREDUCE-5268: - bq. Actually I'm a bit surprised to see a templated interface for the private class. I initially had a custom/ un-templated implementation, but added the template to limit the changes to JobListCache. Let me get rid of it. bq. I was a bit puzzled to see the synchronized methods instead of the AtomicInteger. What was the reasoning behind that change? Agree using an AtomicInteger would be faster. However, my thinking was that given we are abstracting it out, someone working on JobListCache in the future might be oblivious to the potential race of mapSize not changing along with the collection. But, I suppose we could explicitly comment that behavior. Let me update the patch with the two changes. > Improve history server startup performance > -- > > Key: MAPREDUCE-5268 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Jason Lowe >Assignee: Karthik Kambatla > Attachments: mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch > > > The history server can easily take many minutes to startup when there are a > significant number of jobs to scan in the done directory. However the > scanning of files is not the bottleneck, rather it's the heavy use of > ConcurrentSkipListMap.size in HistoryFileManager. > ConcurrentSkipListMap.size is a very expensive operation, especially on maps > with many entries, as it has to scan every entry to compute the size. We > should avoid calling this method or at least minimize its use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[
https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670936#comment-13670936
]
Hadoop QA commented on MAPREDUCE-5283:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585497/MAPREDUCE-5283.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs:
org.apache.hadoop.mapreduce.v2.hs.webapp.TestHsWebServicesJobsQuery
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3701//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3701//console
This message is automatically generated.
> Over 10 different tests have near identical implementations of AppContext
> -
>
> Key: MAPREDUCE-5283
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster, test
>Affects Versions: 2.0.4-alpha
>Reporter: Sandy Ryza
>Assignee: Sandy Ryza
> Attachments: MAPREDUCE-5283.patch
>
>
> I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go
> into nearly every test file for MR web services to make sure their
> TestAppContext implements it. I propose having a common implementation of
> AppContext that all these tests can use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5268) Improve history server startup performance
[ https://issues.apache.org/jira/browse/MAPREDUCE-5268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670930#comment-13670930 ] Jason Lowe commented on MAPREDUCE-5268: --- bq. Considered implementing a full-blown implementation of ConcurrentSkipListMapWithSize extending ConcurrentSkipList and adding it to hadoop-common, but decided against it as that seemed a little overboard. Agreed. Actually I'm a bit surprised to see a templated interface for the private class. The whole purpose of the class is to be a map of JobId to HistoryFileInfo so templating seems a bit overkill. Not a big deal, though. I was a bit puzzled to see the synchronized methods instead of the AtomicInteger. What was the reasoning behind that change? It seems a bit odd to see a ConcurrentSkipListMap behind synchronized accesses. BTW, I built a local test of 50,000 jhist files with the history server configured to cache 20,000. Without this change the history server takes over 60 seconds to process the existing jobs, and after the patch it processes them in around 6 seconds. > Improve history server startup performance > -- > > Key: MAPREDUCE-5268 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5268 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 0.23.7, 2.0.4-alpha >Reporter: Jason Lowe >Assignee: Karthik Kambatla > Attachments: mr-5268.patch, mr-5268.patch, mr-5268-prelim.patch > > > The history server can easily take many minutes to startup when there are a > significant number of jobs to scan in the done directory. However the > scanning of files is not the bottleneck, rather it's the heavy use of > ConcurrentSkipListMap.size in HistoryFileManager. > ConcurrentSkipListMap.size is a very expensive operation, especially on maps > with many entries, as it has to scan every entry to compute the size. We > should avoid calling this method or at least minimize its use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5283: -- Status: Patch Available (was: Open) > Over 10 different tests have near identical implementations of AppContext > - > > Key: MAPREDUCE-5283 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, test >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5283.patch > > > I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go > into nearly every test file for MR web services to make sure their > TestAppContext implements it. I propose having a common implementation of > AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670917#comment-13670917 ] Sandy Ryza commented on MAPREDUCE-5283: --- Uploaded a straightforward patch with implementations of AppContext and HistoryContext that the tests use instead of their own. > Over 10 different tests have near identical implementations of AppContext > - > > Key: MAPREDUCE-5283 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, test >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5283.patch > > > I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go > into nearly every test file for MR web services to make sure their > TestAppContext implements it. I propose having a common implementation of > AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5283: -- Attachment: MAPREDUCE-5283.patch > Over 10 different tests have near identical implementations of AppContext > - > > Key: MAPREDUCE-5283 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, test >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5283.patch > > > I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go > into nearly every test file for MR web services to make sure their > TestAppContext implements it. I propose having a common implementation of > AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5280) Mapreduce API: ClusterMetrics incompatibility issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670909#comment-13670909 ] Zhijie Shen commented on MAPREDUCE-5280: +1. Looks good to me. > Mapreduce API: ClusterMetrics incompatibility issues with MR1 > - > > Key: MAPREDUCE-5280 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5280 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5280-trunk-1.patch > > > 1. Constructor has one fewer parameters: numGraylistedTrackers > 2. getGrayListedTaskTrackerCount() is removed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670816#comment-13670816 ] Sandy Ryza commented on MAPREDUCE-5183: --- +1, the updated patch looks good to me > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Status: Patch Available (was: Open) > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, > MAPREDUCE-5228-trunk-6.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Status: Open (was: Patch Available) > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, > MAPREDUCE-5228-trunk-6.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Attachment: MAPREDUCE-5228-trunk-6.patch Updating the latest patch. Thanks, Mayank > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch, > MAPREDUCE-5228-trunk-6.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build
[ https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670805#comment-13670805 ] Hudson commented on MAPREDUCE-5261: --- Integrated in Hadoop-trunk-Commit #3814 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3814/]) MAPREDUCE-5261. Reverting the patch as it is no longer needed. (Revision 1488032) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1488032 Files : * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java > TestRMContainerAllocator is exiting and failing the build > - > > Key: MAPREDUCE-5261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.5-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5261.20130521.1.patch, > MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, > MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, > MAPREDUCE-5261.patch > > > Recent builds are failing because TestRMContainerAllocator is exiting rather > than succeeding or failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[
https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670797#comment-13670797
]
Hadoop QA commented on MAPREDUCE-5183:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585492/MAPREDUCE-5183.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3699//console
This message is automatically generated.
> In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by
> percent sign
> --
>
> Key: MAPREDUCE-5183
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv1, tasktracker
>Affects Versions: 1.1.2
>Reporter: Sandy Ryza
>Priority: Minor
> Labels: newbie
> Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch
>
>
> This makes looking at progress in the logs unnecessarily confusing. It would
> probably look prettiest to keep the percentage sign and have the numbers
> between 0 and 100.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated MAPREDUCE-5183: -- Attachment: MAPREDUCE-5183.patch > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670792#comment-13670792
]
Zhijie Shen commented on MAPREDUCE-5228:
Some more comments:
1. FileInputFormat_Counter.properties and FileOutputFormat_Counter.properties
of *mapreduce.lib.input/output* are missing.
2. The following and its analog asserts seem to be unnecessary, because you
have assert that it equals a non-zero number afterwards.
{code}
+assertTrue(counter.findCounter(FileInputFormat.Counter.BYTES_READ)
+ .getValue() != 0);
{code}
> Enum Counter is removed from FileInputFormat and FileOutputFormat of both
> mapred and mapreduce
> --
>
> Key: MAPREDUCE-5228
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5228-trunk-1.patch,
> MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch,
> MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch
>
>
> The enum was used by findCounter(Enum key) to find a specific counter object.
> Now it seems to be replaced by FileInputFormatCounter and
> FileOutputFormatCounter. Now the enum seems to be only used internally, but
> not sure whether it will be used externally when users extend FileFormat.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated MAPREDUCE-5183: -- Attachment: (was: MAPREDUCE-5183.patch) > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build
[ https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5261: --- Resolution: Invalid Hadoop Flags: (was: Reviewed) Status: Resolved (was: Patch Available) Tx Omkar. Looks good. Applied the 'revert-patch' to trunk and branch-2. Closing this as invalid as one of the other tickets put in a core-site.xml in resource-manager module essentially fixing these tests. > TestRMContainerAllocator is exiting and failing the build > - > > Key: MAPREDUCE-5261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.5-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5261.20130521.1.patch, > MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, > MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, > MAPREDUCE-5261.patch > > > Recent builds are failing because TestRMContainerAllocator is exiting rather > than succeeding or failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5183) In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign
[ https://issues.apache.org/jira/browse/MAPREDUCE-5183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated MAPREDUCE-5183: -- Attachment: MAPREDUCE-5183.patch Fixed coding style > In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by > percent sign > -- > > Key: MAPREDUCE-5183 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5183 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, tasktracker >Affects Versions: 1.1.2 >Reporter: Sandy Ryza >Priority: Minor > Labels: newbie > Attachments: MAPREDUCE-5183.patch, MAPREDUCE-5183.patch > > > This makes looking at progress in the logs unnecessarily confusing. It would > probably look prettiest to keep the percentage sign and have the numbers > between 0 and 100. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670770#comment-13670770 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5228: Looks good, +1. Checking this in.. > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670692#comment-13670692
]
Hadoop QA commented on MAPREDUCE-5228:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585456/MAPREDUCE-5228-trunk-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3698//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3698//console
This message is automatically generated.
> Enum Counter is removed from FileInputFormat and FileOutputFormat of both
> mapred and mapreduce
> --
>
> Key: MAPREDUCE-5228
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5228-trunk-1.patch,
> MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch,
> MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch
>
>
> The enum was used by findCounter(Enum key) to find a specific counter object.
> Now it seems to be replaced by FileInputFormatCounter and
> FileOutputFormatCounter. Now the enum seems to be only used internally, but
> not sure whether it will be used externally when users extend FileFormat.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5282) Update MR App to use immutable ApplicationID after YARN-716
[ https://issues.apache.org/jira/browse/MAPREDUCE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670674#comment-13670674 ] Hudson commented on MAPREDUCE-5282: --- Integrated in Hadoop-trunk-Commit #3813 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3813/]) YARN-716. Making ApplicationID immutable. Contributed by Siddharth Seth. MAPREDUCE-5282. Updating MR App to use immutable ApplicationID after YARN-716. Contributed by Siddharth Seth. (Revision 1487994) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487994 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRuntimeEstimators.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TestTaskImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAppController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/TypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/TestTypeConverter.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapreduce/v2/util/TestMRApps.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestShufflePlugin.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/JobHistory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestResourceMgrDelegate.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/ApplicationId.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ApplicationIdPBImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestYarnClient.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/Apps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/BuilderUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/ConverterUtils.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/MockApps.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestContainerLaunchRPC.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/TestRPC.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestEventFlow.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerReboot.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeStatusUpdater.java * /hadoop/common/trunk
[jira] [Resolved] (MAPREDUCE-5282) Update MR App to use immutable ApplicationID after YARN-716
[ https://issues.apache.org/jira/browse/MAPREDUCE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5282. Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Reviewed and committed this to trunk and branch-2 together with YARN-716. Thanks Sid! > Update MR App to use immutable ApplicationID after YARN-716 > --- > > Key: MAPREDUCE-5282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5282 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5282.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Status: Patch Available (was: Open) > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Attachment: MAPREDUCE-5228-trunk-5.patch Thanks [~vinodkv] for review. I updated the patch with actually checking file written bytes in validateoldfilecounter. Thanks, Mayank > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Status: Open (was: Patch Available) > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch, MAPREDUCE-5228-trunk-5.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5281) Mapreduce API: Counter changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670621#comment-13670621 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5281: Clearly we don't want to mess with the class-signature. This is one of the main reasons why MAPREDUCE-5108 targeted only source-compatibility for mapreduce APIs. I think we can do the remaining - adding back equals etc. But no to the signature change. > Mapreduce API: Counter changes from non-abstract class to interface > --- > > Key: MAPREDUCE-5281 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5281 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Therefore, significant changes in Counter: > 1. Two Constructors are removed; > 2. Following methods are removed: > * boolean equals(Object) > * int hashCode() > * void readFields(DataInput) > * void write(DataOutput) > Fix of this issue may break 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5284) Mapreduce API: CounterGroup changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670619#comment-13670619 ] Vinod Kumar Vavilapalli commented on MAPREDUCE-5284: Yes, clearly we don't want to mess with this. This is one of the main reasons why MAPREDUCE-5108 targeted only source-compatibility for mapreduce APIs. We should close this as Won't Fix. > Mapreduce API: CounterGroup changes from non-abstract class to interface > > > Key: MAPREDUCE-5284 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5284 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Therefore, constructors and implemented methods are removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5274) Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils
[ https://issues.apache.org/jira/browse/MAPREDUCE-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670595#comment-13670595 ] Zhijie Shen commented on MAPREDUCE-5274: +1. Straight forward patch. Copy the missing method from branch-1. > Mapreduce API: String toHex(byte[]) is removed from SecureShuffleUtils > -- > > Key: MAPREDUCE-5274 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5274 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5274-trunk-1.patch > > > String toHex(byte[]) is removed from SecureShuffleUtils in mapreduce after > upgrading to M/R 2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5025) Key Distribution and Management for supporting crypto codec in Map Reduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670593#comment-13670593 ] Owen O'Malley commented on MAPREDUCE-5025: -- This is a really problematic change. In particular, making changes to each input format seems like a very error-prone idea. I'm also not comfortable with special casing the crypto codec in the input/output formats. > Key Distribution and Management for supporting crypto codec in Map Reduce > - > > Key: MAPREDUCE-5025 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5025 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: security >Affects Versions: trunk >Reporter: Jerry Chen >Assignee: Jerry Chen > Attachments: MAPREDUCE-5025.patch > > Original Estimate: 504h > Remaining Estimate: 504h > > This task defines the work to enable Map Reduce to utilize the Crypto Codec > framework to support encryption and decryption of data during MapReduce Job. > According to the some real use case and discussions from the community, for > encryption and decryption files in Map Reduce, we have the following > requirements: > 1. Different stages (input, output, intermediate output) should have the > flexibility to choose whether encrypt or not, as well as which crypto codec > to use. > 2. Different stages may have different scheme of providing the keys. > 3. Different Files (for example, different input files) may have or use > different keys. > 4. Support a flexible way of retrieving keys for encryption or decryption. > So this task defines and provides the framework for supporting these > requirements as well as the implementations for common use and key retrieving > scenarios. > The design document of this part is included in the Hadoop Crypto Design > attached in HADOOP-9331. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5273) Protected variables are removed from CombineFileRecordReader in both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670573#comment-13670573 ] Zhijie Shen commented on MAPREDUCE-5273: +1. Straightforward patch. > Protected variables are removed from CombineFileRecordReader in both mapred > and mapreduce > - > > Key: MAPREDUCE-5273 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5273 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5273-trunk-1.patch > > > Two protected variables are removed from CombineFileRecordReader in both > mapred and mapreduce: > * FileSystem fs > * Class> rrClass -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670574#comment-13670574 ] Jason Lowe commented on MAPREDUCE-5211: --- Is this actually a problem in 2.0.4? I believe after MAPREDUCE-2264, which is in 2.0.3, MapOutput objects are not used to generate the pathnames and the problem does not occur. The pathnames can get very long, since they concatenate two absolute paths to form the new path, but I didn't think it was possible for two paths to collide for the same reducer. Also the reduce attempt ID should appear at least once in the pathname, so it should also be impossible for paths to collide between reducers. > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Labels: 2.0.4.2 > Fix For: 0.23.8 > > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5282) Update MR App to use immutable ApplicationID after YARN-716
[ https://issues.apache.org/jira/browse/MAPREDUCE-5282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5282: -- Attachment: MAPREDUCE-5282.txt MR part of the YARN-716 patch. > Update MR App to use immutable ApplicationID after YARN-716 > --- > > Key: MAPREDUCE-5282 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5282 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth > Attachments: MAPREDUCE-5282.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build
[
https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670550#comment-13670550
]
Hadoop QA commented on MAPREDUCE-5261:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585436/MAPREDUCE-5261.20130529.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3697//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3697//console
This message is automatically generated.
> TestRMContainerAllocator is exiting and failing the build
> -
>
> Key: MAPREDUCE-5261
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.5-beta
>Reporter: Jason Lowe
>Assignee: Omkar Vinit Joshi
> Fix For: 2.0.5-beta
>
> Attachments: MAPREDUCE-5261.20130521.1.patch,
> MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch,
> MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch,
> MAPREDUCE-5261.patch
>
>
> Recent builds are failing because TestRMContainerAllocator is exiting rather
> than succeeding or failing.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build
[ https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5261: - Status: Patch Available (was: Open) > TestRMContainerAllocator is exiting and failing the build > - > > Key: MAPREDUCE-5261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.5-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5261.20130521.1.patch, > MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, > MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, > MAPREDUCE-5261.patch > > > Recent builds are failing because TestRMContainerAllocator is exiting rather > than succeeding or failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build
[ https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5261: - Status: Open (was: Patch Available) resubmitting.. > TestRMContainerAllocator is exiting and failing the build > - > > Key: MAPREDUCE-5261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.5-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5261.20130521.1.patch, > MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, > MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, > MAPREDUCE-5261.patch > > > Recent builds are failing because TestRMContainerAllocator is exiting rather > than succeeding or failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5261) TestRMContainerAllocator is exiting and failing the build
[ https://issues.apache.org/jira/browse/MAPREDUCE-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Omkar Vinit Joshi updated MAPREDUCE-5261: - Attachment: MAPREDUCE-5261.20130529.patch > TestRMContainerAllocator is exiting and failing the build > - > > Key: MAPREDUCE-5261 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5261 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.5-beta >Reporter: Jason Lowe >Assignee: Omkar Vinit Joshi > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5261.20130521.1.patch, > MAPREDUCE-5261.20130521.2.patch, MAPREDUCE-5261.20130521.patch, > MAPREDUCE-5261.20130529.patch, MAPREDUCE-5261.20130529.patch, > MAPREDUCE-5261.patch > > > Recent builds are failing because TestRMContainerAllocator is exiting rather > than succeeding or failing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik reopened MAPREDUCE-5211: --- This needs to go into the next stabilization release of 2.0.4.2 as well. Adding the label to that effect and re-opening the ticket. > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Labels: 2.0.4.2 > Fix For: 0.23.8 > > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-5211: -- Labels: 2.0.4.2 (was: ) > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Labels: 2.0.4.2 > Fix For: 0.23.8 > > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5211) Reducer intermediate files can collide during merge
[ https://issues.apache.org/jira/browse/MAPREDUCE-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-5211: -- Target Version/s: 2.0.5-beta, 0.23.8 (was: 0.23.8) > Reducer intermediate files can collide during merge > --- > > Key: MAPREDUCE-5211 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5211 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.7 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Blocker > Labels: 2.0.4.2 > Fix For: 0.23.8 > > Attachments: MAPREDUCE-5211.branch-0.23.patch > > > The OnDiskMerger.merge method constructs an output path that is not unique to > a reduce attempt, and as a result can result in a file collision with other > reducers from the same app that are running on the same node. In addition > the name of the output file is based on MapOutput.toString which may not be > unique in light of multi-pass merges on disk since the mapId will be null and > the basename ends up as "MapOutput(null, DISK)" -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670495#comment-13670495 ] Konstantin Boudnik commented on MAPREDUCE-3685: --- This has been clearly backported to 2.0.4-alpha in r1453367. Marking the ticket as such. > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Critical > Fix For: 0.23.7, 2.0.5-beta, 2.0.4-alpha > > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Boudnik updated MAPREDUCE-3685: -- Fix Version/s: 2.0.4-alpha > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Critical > Fix For: 0.23.7, 2.0.5-beta, 2.0.4-alpha > > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.branch-0.23.patch, > MAPREDUCE-3685.branch-0.23.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, MAPREDUCE-3685.patch, > MAPREDUCE-3685.patch, MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5284) Mapreduce API: CounterGroup changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670489#comment-13670489 ] Zhijie Shen commented on MAPREDUCE-5284: [~jlowe] Yes, it would be. So will MAPREDUCE-5281. More investigation and discussion are required before deciding to fix them or not. > Mapreduce API: CounterGroup changes from non-abstract class to interface > > > Key: MAPREDUCE-5284 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5284 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Therefore, constructors and implemented methods are removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4394) ReduceTask.MapOutputCopier.copyOutput() signature doesn't match the corresponding Javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670461#comment-13670461 ] Niranjan Singh commented on MAPREDUCE-4394: --- Mismatched method signature to the corresponding Javadoc rectified. > ReduceTask.MapOutputCopier.copyOutput() signature doesn't match the > corresponding Javadoc > - > > Key: MAPREDUCE-4394 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4394 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Priority: Minor > Labels: docs, newbie > Attachments: MAPREDUCE-4394.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4394) ReduceTask.MapOutputCopier.copyOutput() signature doesn't match the corresponding Javadoc
[ https://issues.apache.org/jira/browse/MAPREDUCE-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Niranjan Singh updated MAPREDUCE-4394: -- Attachment: MAPREDUCE-4394.patch > ReduceTask.MapOutputCopier.copyOutput() signature doesn't match the > corresponding Javadoc > - > > Key: MAPREDUCE-4394 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4394 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 1.0.3 >Reporter: Karthik Kambatla >Priority: Minor > Labels: docs, newbie > Attachments: MAPREDUCE-4394.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5284) Mapreduce API: CounterGroup changes from non-abstract class to interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-5284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670365#comment-13670365 ] Jason Lowe commented on MAPREDUCE-5284: --- Fixing this would be a significant breakage for binary compatibility between 0.23 and 2.x, correct? > Mapreduce API: CounterGroup changes from non-abstract class to interface > > > Key: MAPREDUCE-5284 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5284 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > > Therefore, constructors and implemented methods are removed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670349#comment-13670349 ] Hudson commented on MAPREDUCE-5263: --- Integrated in Hadoop-Mapreduce-trunk #1441 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1441/]) MAPREDUCE-5263. Bring back old methods and fields in filecache.DistributedCache for binary compatibility with mapred in 1.x. Contributed by Zhijie Shen. (Revision 1487695) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java > filecache.DistributedCache incompatiblity issues with MR1 > - > > Key: MAPREDUCE-5263 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, > MAPREDUCE-5263.3.patch > > > A couple of methods and variables have been removed: > void addLocalArchives(Configuration, String) > void addLocalFiles(Configuration, String) > void createAllSymlink(Configuration, File, File) > FileStatus getFileStatus(Configuration, URI) > long getTimestamp(Configuration, URI) > void setArchiveTimestamps(Configuration, String) > void setFileTimestamps(Configuration, String) > void setLocalArchives(Configuration, String) > void setLocalFiles(Configuration, String) > String CACHE_ARCHIVES > String CACHE_ARCHIVES_SIZES > String CACHE_ARCHIVES_TIMESTAMPS > String CACHE_FILES > String CACHE_FILES_SIZES > String CACHE_FILES_TIMESTAMPS > String CACHE_LOCALARCHIVES > String CACHE_LOCALFILES > String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5283) Over 10 different tests have near identical implementations of AppContext
[ https://issues.apache.org/jira/browse/MAPREDUCE-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670333#comment-13670333 ] Jason Lowe commented on MAPREDUCE-5283: --- +1, I think people have avoided placing items into AppContext because the tests make it so painful to do so. > Over 10 different tests have near identical implementations of AppContext > - > > Key: MAPREDUCE-5283 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5283 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster, test >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > I'm trying to add a method to AppContext for MAPREDUCE-5171, and I have to go > into nearly every test file for MR web services to make sure their > TestAppContext implements it. I propose having a common implementation of > AppContext that all these tests can use. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670317#comment-13670317 ] Hudson commented on MAPREDUCE-5263: --- Integrated in Hadoop-Hdfs-trunk #1415 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1415/]) MAPREDUCE-5263. Bring back old methods and fields in filecache.DistributedCache for binary compatibility with mapred in 1.x. Contributed by Zhijie Shen. (Revision 1487695) Result = FAILURE vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java > filecache.DistributedCache incompatiblity issues with MR1 > - > > Key: MAPREDUCE-5263 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, > MAPREDUCE-5263.3.patch > > > A couple of methods and variables have been removed: > void addLocalArchives(Configuration, String) > void addLocalFiles(Configuration, String) > void createAllSymlink(Configuration, File, File) > FileStatus getFileStatus(Configuration, URI) > long getTimestamp(Configuration, URI) > void setArchiveTimestamps(Configuration, String) > void setFileTimestamps(Configuration, String) > void setLocalArchives(Configuration, String) > void setLocalFiles(Configuration, String) > String CACHE_ARCHIVES > String CACHE_ARCHIVES_SIZES > String CACHE_ARCHIVES_TIMESTAMPS > String CACHE_FILES > String CACHE_FILES_SIZES > String CACHE_FILES_TIMESTAMPS > String CACHE_LOCALARCHIVES > String CACHE_LOCALFILES > String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5279) mapreduce scheduling deadlock
[
https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670280#comment-13670280
]
Hadoop QA commented on MAPREDUCE-5279:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585399/MAPREDUCE-5279-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3696//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3696//console
This message is automatically generated.
> mapreduce scheduling deadlock
> -
>
> Key: MAPREDUCE-5279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: PengZhang
> Fix For: trunk
>
> Attachments: MAPREDUCE-5279.patch, MAPREDUCE-5279-v2.patch
>
>
> YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't
> take into account virtual cores while scheduling reduce tasks.
> This may cause more reduce tasks to be scheduled because memory is enough.
> And on a small cluster, this will end with deadlock, all running containers
> are reduce tasks but map phase is not finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5279) mapreduce scheduling deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengZhang updated MAPREDUCE-5279: - Attachment: MAPREDUCE-5279-v2.patch removed unrelated changed file > mapreduce scheduling deadlock > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: PengZhang > Fix For: trunk > > Attachments: MAPREDUCE-5279.patch, MAPREDUCE-5279-v2.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to mrv2
[ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-2723: --- Fix Version/s: (was: 0.24.0) > MR-279: port MAPREDUCE-2324 to mrv2 > --- > > Key: MAPREDUCE-2723 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > MRV2 currently does not take reduce disk usage into account when trying to > schedule a container. For feature parity with the original map reduce it > should be extended to allow for disk space requests within containers along > with RAM requests. We then also need to port MAPREDUCE-2324 to the scheduler > to allow it to avoid starvation of containers that might never get the > resources that they need. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5263) filecache.DistributedCache incompatiblity issues with MR1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670227#comment-13670227 ] Hudson commented on MAPREDUCE-5263: --- Integrated in Hadoop-Yarn-trunk #225 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/225/]) MAPREDUCE-5263. Bring back old methods and fields in filecache.DistributedCache for binary compatibility with mapred in 1.x. Contributed by Zhijie Shen. (Revision 1487695) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1487695 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestMRWithDistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/filecache/DistributedCache.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ConfigUtil.java > filecache.DistributedCache incompatiblity issues with MR1 > - > > Key: MAPREDUCE-5263 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5263 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Zhijie Shen > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5263.1.patch, MAPREDUCE-5263.2.patch, > MAPREDUCE-5263.3.patch > > > A couple of methods and variables have been removed: > void addLocalArchives(Configuration, String) > void addLocalFiles(Configuration, String) > void createAllSymlink(Configuration, File, File) > FileStatus getFileStatus(Configuration, URI) > long getTimestamp(Configuration, URI) > void setArchiveTimestamps(Configuration, String) > void setFileTimestamps(Configuration, String) > void setLocalArchives(Configuration, String) > void setLocalFiles(Configuration, String) > String CACHE_ARCHIVES > String CACHE_ARCHIVES_SIZES > String CACHE_ARCHIVES_TIMESTAMPS > String CACHE_FILES > String CACHE_FILES_SIZES > String CACHE_FILES_TIMESTAMPS > String CACHE_LOCALARCHIVES > String CACHE_LOCALFILES > String CACHE_SYMLINK -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5279) mapreduce scheduling deadlock
[
https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670190#comment-13670190
]
Hadoop QA commented on MAPREDUCE-5279:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585385/MAPREDUCE-5279.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3695//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3695//console
This message is automatically generated.
> mapreduce scheduling deadlock
> -
>
> Key: MAPREDUCE-5279
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
>Affects Versions: 2.0.3-alpha
>Reporter: PengZhang
> Fix For: trunk
>
> Attachments: MAPREDUCE-5279.patch
>
>
> YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't
> take into account virtual cores while scheduling reduce tasks.
> This may cause more reduce tasks to be scheduled because memory is enough.
> And on a small cluster, this will end with deadlock, all running containers
> are reduce tasks but map phase is not finished.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5279) mapreduce scheduling deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengZhang updated MAPREDUCE-5279: - Fix Version/s: trunk Status: Patch Available (was: Open) > mapreduce scheduling deadlock > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: PengZhang > Fix For: trunk > > Attachments: MAPREDUCE-5279.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5279) mapreduce scheduling deadlock
[ https://issues.apache.org/jira/browse/MAPREDUCE-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] PengZhang updated MAPREDUCE-5279: - Attachment: MAPREDUCE-5279.patch > mapreduce scheduling deadlock > - > > Key: MAPREDUCE-5279 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5279 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.3-alpha >Reporter: PengZhang > Attachments: MAPREDUCE-5279.patch > > > YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't > take into account virtual cores while scheduling reduce tasks. > This may cause more reduce tasks to be scheduled because memory is enough. > And on a small cluster, this will end with deadlock, all running containers > are reduce tasks but map phase is not finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[
https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670170#comment-13670170
]
Hadoop QA commented on MAPREDUCE-5228:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12585379/MAPREDUCE-5228-trunk-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3694//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3694//console
This message is automatically generated.
> Enum Counter is removed from FileInputFormat and FileOutputFormat of both
> mapred and mapreduce
> --
>
> Key: MAPREDUCE-5228
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
>Reporter: Zhijie Shen
>Assignee: Mayank Bansal
> Attachments: MAPREDUCE-5228-trunk-1.patch,
> MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch,
> MAPREDUCE-5228-trunk-4.patch
>
>
> The enum was used by findCounter(Enum key) to find a specific counter object.
> Now it seems to be replaced by FileInputFormatCounter and
> FileOutputFormatCounter. Now the enum seems to be only used internally, but
> not sure whether it will be used externally when users extend FileFormat.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13670163#comment-13670163 ] Tsuyoshi OZAWA commented on MAPREDUCE-5221: --- I've confirmed all tests are pass correctly. It's absolutely odd, because of the difference of o.a.h.mapred.Reducer and o.a.h.mapreduce.Reduce as you pointed out. Therefore, I've analysed the source code, and I've found some problems: 1. There are no codes to test with MiniMRYarnCluster with launching reduce-side combiner. TestMRAppWithCombiner seems to be enough, but reduce-side combiners are actually not launched because data size is too small to launch combiners. 2. ShuffleConsumerPlugin.Contexts only accepts Class when constructing. It's needed to add new ShuffleConsumerPlugin.Contexts which accepts Class. # Or, change huffleConsumerPlugin.Contexts to accepts Class and check the type with Java reflection API when constructing. 3. Shuffle#MergeManager only accepts Class. It's needed to add new MergeManager which accepts Class. I'll create a subtasks to deal with these problems step by step. > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, > MAPREDUCE-5221.3.patch > > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Attachment: MAPREDUCE-5228-trunk-4.patch Thanks [~vinodkv] for the review. Updating the patch. Thanks, Mayank > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5228) Enum Counter is removed from FileInputFormat and FileOutputFormat of both mapred and mapreduce
[ https://issues.apache.org/jira/browse/MAPREDUCE-5228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mayank Bansal updated MAPREDUCE-5228: - Status: Patch Available (was: Open) > Enum Counter is removed from FileInputFormat and FileOutputFormat of both > mapred and mapreduce > -- > > Key: MAPREDUCE-5228 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5228 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Reporter: Zhijie Shen >Assignee: Mayank Bansal > Attachments: MAPREDUCE-5228-trunk-1.patch, > MAPREDUCE-5228-trunk-2.patch, MAPREDUCE-5228-trunk-3.patch, > MAPREDUCE-5228-trunk-4.patch > > > The enum was used by findCounter(Enum key) to find a specific counter object. > Now it seems to be replaced by FileInputFormatCounter and > FileOutputFormatCounter. Now the enum seems to be only used internally, but > not sure whether it will be used externally when users extend FileFormat. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
